WO2015164668A2 - High-performance virtualized applications platform - Google Patents

Abstract

Methods, systems, and computer readable media for a high- performance virtual application platform are disclosed. According to one aspect, a system for providing a high-performance virtual application includes a set of metadata that describes resource requirements and inter- VM resource dependencies for a virtual application comprising a plurality of virtual machines, and a management layer for receiving a request to instantiate an instance of the virtual application, and in response to receiving the request, using the set of metadata for at least the virtual application to make instantiation decisions.

Description

DESCRIPTION

HIGH-PERFORMANCE VIRTUALIZED APPLICATIONS PLATFORM

PRIORITY CLAIM

This application claims the priority benefit of U.S. Provisional Patent Application Serial No. 61/983,427, filed April 23, 2014, which is incorporated by reference herein in its entirety.

TECHNICAL FIELD

The subject matter described herein relates to methods and systems for virtualization of applications. More particularly, the subject matter described herein relates to a high-performance virtualized applications platform.

BACKGROUND

When we speak of virtualization, we are referring to systems comprising a hypervisor, also known as a virtual machine monitor (VMM), executing on physical hardware (i.e., a server); the hypervisor exposes virtual hardware to Virtual Machines (VMs); the VMs, in turn, are where we run our operating systems and software. The hypervisor and hardware comprise the "host". We refer to a VM as a "guest" or "guest OS". The hypervisor exposes virtual hardware upon which the VMs execute. The hypervisor manages the server's physical hardware resources, such as CPU cores, RAM, and I/O devices, and multiplexes these resources among guests. From a guest OS's perspective, it is just running on ordinary hardware. The hypervisor maintains physical isolation between guests - e.g., they cannot share memory or, in principle, communicate with one another (other than by canonical means such as network connections or shared volumes).

VMs, under the covers, are just data: a set of disk images and some configuration information. If this data is stored on NAS (Network-Attached Storage) or a SAN (Storage Area Network), then the respective VM may be executed on any host with access to that storage. Among other things, this enables a feature called "migration", wherein a running VM can be moved from one host to another with essentially no interruption to its operation. Additionally, we can make copies - "clones" - of a VM such that multiple identical (modulo guest-instance-specific configuration) instances may be deployed. A common pattern is to create instances of some "system" by making clones of a master image.

The implementation of hypervisorsA MMs is beyond the scope of the present discussion. It is worth noting, however, that all VMMs introduce some non-zero overhead. All things being equal, the difference in an application's performance between bare metal and a VM is typically less than 5-10%. The exact value is highly-dependent on the specific hypervisor, hardware, and the application in question.

Commonly, some management software ("management layer") is used to control and administer numerous hosts. For example, management software is used to invoke and control migration operations.

A Virtual Infrastructure, then, consists of a set of hosts, one or more networks, storage, a management layer, and possibly other resources.

A Virtual Application, briefly, is a long-lived collection of VMs providing end-user functionality or services; they execute on a virtual infrastructure.

Recall that, physically, a VM, when powered down, is really just a set of files. When we choose to execute the VM, we may use said files directly or, as discussed above, make copies of them and run off of the copies. So, on one hand, "VM" may describe a particular image running on a host or, very reasonably, it could describe the definition (data) used to create a new instance of a particular VM. In all cases, if we are discussing a running VM, then we necessarily are discussing an instance.

Likewise, a Virtual Application is a collection of VMs - perhaps including multiple clones of specific VMs (e.g., multiple front-end HTTP servers). We refer to the definition of such a collection (e.g., "these seven VMs constitute my CRM application") as a Virtual Application. When such a collection is running, we refer to it as an "instance" of the given application. In our parlance, then, a Virtual Application is analogous to a Class and an instance of a Virtual Application is analogous to an Object (of some Class). One function of the management layer is to create, destroy, start, and stop VMs. Consider the case of creating a new instance of some particular VM. Broadly, the management layer must locate a set of resources for this VM. This set includes a) the choice of which host will execute the VM and b) allocation of other resources, such as durable storage, needed by the VM (e.g., space to store the copies of its disk images). By assigning the VM to a hypervisor, resources managed by that hypervisor, e.g., CPU, RAM, and network bandwidth, are (implicitly or explicitly) assigned to the VM.

As used in the literature and herein, (VM) "placement" refers to choosing which host will execute the VM. We make a distinction between placement and the allocation of other resources (such as volumes, services, licenses, etc.) because the set of resources allocated via placement must all be sourced from the same host. For example, we cannot supply a particular VM instance with 4GB of RAM from one host and 2.2GB of RAM from some other host; likewise, a VM instance cannot share the CPUs of multiple hosts.

Placement of an instance of a Virtualized Application, then, comprises the placement of its constituent VMs, assignment of their respective resources, and, in the generalized case, assignment of resources specific to said instance but not specific to a particular VM.

Industry and the literature also use the term SLA (Service Level

Agreement) and the weaker term, SLO (Service Level Objective) to describe sets of requirements (such as performance, availability, or compliance) that must (or should) be met by a given service. In general, SLAs typically refer to contractual terms between a customer and a provider while SLOs refer to the operator's target for service quality. As used herein, SLA should be more broadly-construed such that failing to meet an SLA will result in degraded end-user experience or the infeasibility of deploying an application on a given infrastructure.

For our purposes, an SLA comprises, inter alia, a set of quantitative metrics that are to be met by an application. Metrics are application-specific. For example, an IP telephony application might be concerned with media quality while a computer vision application might be concerned with response times. An application's SLA may comprise multiple metrics; for example, a transaction-processing system may be concerned with both transaction throughput and response times. Broad classes of applications may be concerned with availability (e.g., "five nines"). Without loss of generality, it is assumed that any metric stated in an SLA can either be measured directly (such as "is it running?") or can be obtained by communicating with the application or processing application and/or system outputs, such as log files or notifications.

In traditional "bare-metal" deployments, the topology of and dependencies within an application are largely self-evident. One can go into the machine room and physically identify the constituent hardware, examine network connections, etc. Performance-sensitive components are (or canbe) contained and isolated from other applications. For example, a database in a web service is often given dedicated compute and storage hardware.

As we begin a discussion of structure, consider that such bare-metal deployments have inherent structure. And, absent a physical change, this structure is fixed. Considering the structures of their resource dependencies, these, too, are generally fixed - if I reboot the application cluster, each server will still contain the same software and be connected to the same storage and the same switches. Importantly, in substantial part, the invariance of inherent structure is maintained passively - without human intervention.

In industry jargon, applications become "silos" - providing some levels of determinism at the cost of under-utilized hardware. These silos, however, have the significant virtue of independence (within and among applications) and localization of performance concerns and change management.

In addition to the many benefits of virtualization, however, there are distinct potential pitfalls. For example, consolidation - i.e., transitioning N servers running on bare metal to N VMs running on M hosts (where M significantly smaller than N) - is one of virilization's its major purported benefits - minimally, one saves money on energy and hardware because fewer servers are used to perform the same work. However, the expectation is that the applications will continue to perform as they did on bare metal (modulo inherent virtualization overhead). This is harder to achieve in practice. More specifically, in large virtualized infrastructures, the structures of applications - including applications' constituent virtual machines (VMs or "guests"), network topologies, storage, and other key resource dependencies - become opaque over time.

For conventional implementations of VMs, we might say that an application's structure is not captured or respected post-virtualization. For this reason, answers to simple questions such as, "Which physical servers and networks does my application depend on," are often difficult to answer with precision (if at all). Questions regarding storage dependencies, network bandwidth, and other performance-critical elements are even more difficult to address.

The application undeniably still has structure and resource dependencies - it wouldn't function otherwise! However, in contrast to the inherent structure of the bare-metal system, the now-virtualized system's structure is implicit, substantially opaque, and, most importantly, subject to

(potentially-arbitrary) change.

A second, perhaps more subtle effect, is that previously unshared resources (such as memory busses, network adapters, or local disks) become shared resources. Importantly, who shares what with whom is unspecified - implicit.

The fact that potentially-diverse applications share the same hardware has been shown to adversely impact performance and QoS in d iff icu It-to-pred ict ways (e.g., [3], [7], [10], [1 1]). The details of what is being shared by whom also change over time as VMs are created, destroyed, and migrate and as hardware is added and removed from the infrastructure - i.e., as placement decisions are made.

As a simple example, under conventional implementations of VMs, starting a set of applications (each comprising multiple VMs) at different points in time or in different orders will generally result in different combinations of VMs being co-located on the same hosts. Even if performance characteristics are well-understood at some point in time, restarting an application - resulting in different VM placement - can cause unpredictable changes in performance.

These factors collude to make even routine server maintenance a risky affair ("Which applications are affected by shutting down or moving this server?"). Moreover, achieving strong SLAs, such as tight latency or throughput requirements, becomes difficult or impossible without resorting to excessive engineering safety margins, potentially reducing the efficacy of consolidation and leading to greater hardware and energy costs (thus negating the motivation and benefit of virtualizing the application).

We define the "Management Layer" as comprising the software that manages, monitors, and provides interfaces to the virtualized infrastructure and the VMs and applications contained therein. See, e.g., [30], [34], [37]. Such software is necessary to even begin to consider building and managing a non-trivially-sized deployment. By way of example, it is readily-apparent that VMware has shifted its business model from providing an enterprise- grade hypervisor to providing the management tools necessary to run a virtualized infrastructure. The hypervisor has become a commodity - that problem is substantially solved for the vast majority of enterprise applications. However, many aspects of management remain open problems.

One function performed by the Management Layer is VM placement: given a request to start a VM, the Management Layer must choose where to start it (i.e., which physical server) and possibly make other resource assignments on its behalf. In canonical Infrastructure as a Service (laaS) systems, such as EC2 [33], the application administrator and the infrastructure provider are separate physical (and usually business) entities and the application administrator/owner has no control (or visibility) into placement decisions. Similarly, the infrastructure provider has little or no visibility into application structure or semantics. This is also seen in deployments where the application and infrastructure are owned by the same entity - due to limitations in existing Management Layer implementations. Despite their many rich features, present-day Management Layer software packages provide little help in capturing applications' structure, internal transitive and external dependencies, or making effective VM placement and resource allocation decisions in light of these dependencies. This places fundamental limits on what SLA or QoS assurances can be made.

In all virtualization systems, VMs are well-defined objects, typically represented by a text configuration file, such as [50]. VM definitions necessarily contain virtual hardware specifications such as number of CPUs, network adapters, MAC addresses, volumes, etc. These definitions can contain a considerable amount of detail, allowing one to specify and tweak many low-level aspects of the virtual hardware. Such a specification is, however, in a vacuum and is not tied to other VMs.

To varying degrees, applications, if represented at all in management systems, group sets of VMs to provide, for example, resource pooling (a given set of VMs is reserved or limited to a certain set of resources), elastic scaling (starting or stopping VMs within a group based upon demand), high- availability (restarting or replicating VMs), group instantiation, migration, and power state management, and a grouping function on (system) performance metrics. We do not mean to trivialize the value of these features; the observation, however, is that they fail to capture critical properties of applications, such as resource dependencies.

In contrast to the richness with which individual VMs are represented, applications' representations in current systems are akin to shoving a bunch of VM definitions into a directory; the directory's name becomes the application name and that's about it. It should be apparent, then, that much of the structure of a bare-metal implementation of an application is lost in translation, as it were.

This management problem is bad enough - both for individual guests (VMs) and applications (composed of multiple guests) - that a new class of software tools has emerged to help IT organizations essentially reverse- engineer (by inference) the architecture of their deployments (e.g., [31 ]). Similar challenges and solution patterns are seen when addressing security concerns with such deployments.

If we presume that IT decision-makers vote with their feet then we must conclude that the advantages of virtualization are seen to outweigh these and other difficulties, costs, and perils.

Notwithstanding this observation, the problem of providing application-level predictability, assurances, and SLAs in such environments remains open.

Accordingly, in light of these disadvantages associated with conventional implementations of a VM Management Layer's failure to consider the potential negative impact of a VM on other Virtual Machines that may also use shared resources when making instantiation decisions, there exists a need for a high-performance virtualized applications platform. SUMMARY

According to one aspect, a system for providing a high-performance virtual application includes a set of metadata that describes resource requirements and inter-VM resource dependencies for a virtual application comprising a plurality of virtual machines, and a management layer for receiving a request to instantiate an instance of the virtual application, and in response to receiving the request, using the set of metadata for at least the virtual application to make instantiation decisions.

According to another aspect, the subject matter described herein includes a method for providing high-performance virtualized applications. The method includes receiving a request to instantiate an instance of a virtual application comprising a plurality of virtual machines, and, in response to receiving the request, making instantiation decisions related to the instance of the virtual application based on metadata that describes resource requirements and inter-VM resource dependencies of the virtual application.

As used herein, a "derivative" of a first virtualized application is another virtualized application or virtualized application instance whose metadata definitions are based upon, inherit from, and/or are otherwise derivatives of those of the first virtual application. The subject matter described herein can be implemented in software in combination with hardware and/or firmware. For example, the subject matter described herein can be implemented in software executed by a processor. In one exemplary implementation, the subject matter described herein can be implemented using a non-transitory computer readable medium having stored thereon computer executable instructions that when executed by the processor of a computer control the computer to perform steps. Exemplary computer readable media suitable for implementing the subject matter described herein include non-transitory computer-readable media, such as disk memory devices, chip memory devices, programmable logic devices, and application specific integrated circuits. In addition, a computer readable medium that implements the subject matter described herein may be located on a single device or computing platform or may be distributed across multiple devices or computing platforms.

BRIEF DESCRIPTION OF THE DRAWINGS

Preferred embodiments of the subject matter described herein will now be explained with reference to the accompanying drawings, wherein like reference numerals represent like parts, of which:

Figure 1 depicts an exemplary three-tier web application based upon a real contact management system, used for the purpose of illustrating the characterization of the application using metadata and the use of that metadata when making instantiation decisions according to an embodiment of the subject matter described herein;

Figure 2 is a block diagram of a simplified web application that may be characterized using metadata according to an embodiment of the subject matter described herein;

Figure 3 is a simplified (non-normative syntax) metadata example according to an embodiment of the subject matter described herein;

Figure 4 is a flowchart of an exemplary process for using metadata that describes inter-VM resource dependencies to make virtual machine instantiation decisions and to update virtual machine instances after instantiation; and Figure 5 is a block diagram illustrating a virtual application management layer using metadata that describes inter-VM dependencies according to an embodiment of the subject matter described herein. DETAILED DESCRIPTION

In accordance with the subject matter disclosed herein, systems, methods, and computer readable media for high-performance virtualized applications platform are provided. Reference will now be made in detail to exemplary embodiments of the present invention, examples of which are illustrated in the accompanying drawings. Wherever possible, the same reference numbers will be used throughout the drawings to refer to the same or like parts.

Explicit specification of structure, semantics, SLAs, and resource requirements of Virtualized Applications can be used to provide better, more predictable performance than existing management/placement solutions and enables the deployment of more performance-sensitive applications in virtualized infrastructures. Additionally, the study of such applications' behavior, informed by explicit specifications, enables the characterization and quantification of second-order resources and interference sensitivities thereby enabling further enhancement of placement and resource allocation.

We propose a solution to the problem of VM placement and resource allocation in the context of existing and envisioned futuristic Virtualized Applications (long-lived collections of VMs providing end-user functionality or services). The goal is to provide high-assurance, predictable application SLAs that include performance metrics not realizable with today's placement and resource allocation mechanisms. Our solution includes three main parts: placement and resource allocation based upon explicit structural and semantic characterization and specification of applications and their resource dependencies, interpretation of application telemetry informed by applications' specifications, and iterative refinement of performance and resource specifications.

Dr. Fred Brooks of the University of North Carolina has been quoted as having said, "It is better to be specific and wrong than to be vague." In the present context, we claim that, in order to achieve high-performance SLAs, Virtualized Applications must be specific - explicit - in terms of their structure and resource requirements.

Our first contribution is to determine and define which of these elements are needed to inform accurate placement and resource assignment. The second and third contributions use the application's explicit structure and specific resource requirements to analyze its actual performance and inform refinements to its specifications.

Should there be a mismatch between the application's specifications and runtime behavior (i.e., we have been "specific and wrong"), we at least know what was requested from and provided by the infrastructure (and why) and we can make a better request next time (both for the present application and future instances thereof). In the alternative (i.e., if we are "vague", as present systems are), we must begin by reverse-engineering our application and its dependencies, and any insights garnered in this process are, at best, memorialized outside of the infrastructure and may rapidly become obsolete.

Our higher-level ethos is that, given the choice between an architecture reliant or based upon inference and one based upon explicit specifications, the latter is superior. As an analogy, consider a piece of software where various bits of code interact via an ad-hoc set of function calls, shared data structures, etc. If studied carefully, we may well find that interfaces between functional components exist but are just not well-defined or enforced and lack consistent semantics; we may also find that the nature of certain behavior is not immediately obvious. In contrast, consider a piece of software with clear interfaces and boundaries between its functional parts; assuming these boundaries are not violated, then reasoning about the program and the interaction between its modules is greatly simplified.

Our assertion is that by creating, in essence, an interface between applications and the management layer such that resource requirements and SLAs can be richly-specified, performing correct placement and later reasoning about application performance is likewise greatly simplified.

A Virtualized Application exposes a set of end-user functions and/or services running as a set of VMs on a virtualized infrastructure. For our purposes, we assume that the infrastructure and the applications are within the same autonomous system. We contrast this with canonical Infrastructure-as-a-Service (laaS) platforms, such as Amazon EC2 [33], where application owners do not have direct control or visibility into the placement of their virtual machines or the underlying hypervisors and infrastructure.

Of interest are long-lived applications - applications whose lifetime is not fixed or known at instantiation. Such applications may service users for indefinite periods of time. Examples of such present-day applications abound - email, CRM, Wiki, logistics, accounting, medical records, etc. Virtualized Applications of the future, with stricter performance requirements, may include: consumer applications with a low-latency/high-update computer vision/navigation component, or medical/home health care applications with a cybernetic device control component. Present-day applications will benefit from our contributions. Future virtualized applications cannot be realized without much stricter performance assurances; our research is an essential enabler for the future of virtualized computing.

Such applications are fundamentally different in many respects from applications seen in batch-processing, middleware- and application container-based systems, job-oriented, scheduled, most "grid"-based applications, or other similar environments (see, e.g., [9], [14], [19], [22], [24]). These differences include, among other things: Virtualized Applications consist of collections of VMs; multiple applications share hardware at runtime and thus their VMs are subject to interference from one another; application resources are committed and SLAs must be assured for indefinite periods of time; there is generally an aspect of real-time interaction between users and the application; and user experience is dependent on short-term performance characteristics of applications. We are also interested, in particular, in infrastructures hosting a heterogeneous set of applications on (possibly) heterogeneous hardware.

In the context of the present research, we define a Virtualized Application as set of Virtual Machines (VMs) providing end-user functionality and/or services ("application"); a Virtual Application, under our definition, also has the following properties:

^■ The end-users of the application can or do include humans. We specifically include user-interactive applications in our definition.

^■ All the VMs comprising the application are located within the same physical infrastructure and, by implication, are within the same autonomous domain. This is a statement about control over the infrastructure, not necessarily the physical location of its constituent parts.

^■ All physical resources used by the VMs comprising the application are within this same autonomous domain and under the (logical) exclusive control of the same management layer that controls the VMs.

^■ The application proper is long-lived, i.e., once instantiated, it exists and provides services to its users for an indeterminate time. In particular, we assume that its lifetime is not specified (or known precisely) a priori. This does not necessarily preclude applications with known finite durations within our framework, but such lifetime is not included in the calculus of the placement of the application. In particular, our results may be applicable to a broader class of "applications", such as the task-, job-, and batch-oriented processing seen in, e.g., Grid computing. However, we make no claims as to our efficacy with respect to such applications and we do not intend to compare ourselves against, e.g., Grid-oriented placement, scheduling, and/or resource allocation algorithms.

^■ The application has a quantitative SLA that is known a priori. To the extent that it is desirable or necessary to characterize qualitative measures, such as "user experience", such characterization must be mapped to quantitative metrics for inclusion in the SLA. A common mapping uses a Likert Scale [51] wherein a range of answers, e.g., Strongly Agree... Strongly Disagree, is mapped to a range of ordinal values, e.g., 1...10. This presumes, of course, that feedback is from humans. An alternative is to map quantitative performance metrics, such as transaction latencies, to a model of user experience; e.g., latencies of below 1000ms are satisfactory. More sophisticated algorithms exist to generate synthetic "scores" from performance metrics; for example, the Mean Opinion Score (MOS) is used to measure the expected human perception of the quality of a call in a voice telephony system based upon quantitative inputs such as delay, jitter, packet loss, and the audio CODEC in use. The contents of a particular SLA will depend upon the respective application and its semantics. For completeness, we permit applications to be specified with a vacuous SLA (or an SLA function that always returns true), though this is not of particular interest. Without loss of generality, we assume that all SLA metrics can be determined directly from the state of the infrastructure (e.g., is this application running?) or by communicating with the application proper (e.g., an SNMP MIB).

Without loss of generality, we assume that the number of VMs comprising the application is fixed at instantiation. The architecture of the application may include so-called elastic scaling, wherein additional VMs are brought online to provide more capacity, e.g., during peak busy hours. Assuming we decide to support such applications (which is a desirable but non-essential objective), our model will be such that resources are reserved as if all VMs would be running continuously. Said another way, VMs designated to provide "reserve capacity" are placed and allocated resources whether or not they are initially powered on. This scheme ensures that the "spare- capacity" VMs are guaranteed resources (and thus, according to our assertions, performance) at all times. This in no way prevents such VMs from being placed on hosts which are usually powered down (for energy savings). Without loss of generality, it is assumed that the underlying elastic scaling mechanism takes into account the start-up time of spare VMs.

Note that this does necessarily exclude applications requiring arbitrary (unbounded) elastic scaling (e.g., where an additional web server is launched for every K additional units of load). On the other hand, no finite infrastructure can provide tight SLA assurances to such an application; any assurances would be best-effort. Such applications are not within the present scope of work.

In our architecture, applications also specify the set of resources they require (discussed in much more detail below) and may specify how SLA parameters are dependent upon resources.

The "contract" between our part of the Management Layer and the application is: if the declared resource requirements are met, then the application will meet its SLA with high assurance. We do not claim that any resource allocation scheme for a human-interactive application can guarantee SLAs 100% of the time. Even telephone carriers can't do this. We have explicitly not defined "high assurance" at this point. Among other things, achievable performance will depend, in part, on the specific infrastructure and implementation details therein and, more specifically, of the Management Layer. Additionally, the notion of high assurance will differ depending on the application, its users, and the business arrangements underlying the SLA. Formulated another way, the level of assurance could be viewed as a property of the SLA proper; this is standard practice in industry, for example, when specifying SLAs for service availability (e.g., five-nines availability).

Note that our definition of a Virtualized Application applies to a large proportion of presently-deployed enterprise applications. One potential difference is that, in our case, resource requirements SLAs are explicit - declared a priori - whereas, in present systems, (actual) resource requirements may not be well-known and SLAs, or components thereof, are implicit - e.g., the (implicit) SLA is being met as long as no one is complaining.

Our approach begins with defining an application in a human- and machine-readable grammar, which we refer to as application metadata. (In the preferred embodiment, "human-readable" implies that a person can easily comprehend, create, and modify metadata without the need for specialized software tools. In other embodiments, human-readability may be achieved by means of specialized software tools, though this is less desirable.) Metadata is expressed in a document format - such as XML or JSON or a domain-specific language; the specific format and encoding is an implementation detail and will not be discussed further herein; metadata examples herein are non-normative. The salient point is that the metadata provides a closed, clean, well-defined interface between application architects, administrators, etc. and the underlying Management Layer.

This definition specifies, among other things, the constituent VMs, hardware requirements (such as CPU and RAM as well as second-order resources, point-to-point and per-protocol network dependencies, storage requirements, key performance indicators, and SLA specifications. The metadata language will permit resource specifications to be functions of SLA parameters and/or other resource specifications. Among other things, this permits the analysis of transitive dependencies between VMs. Thusly- described, applications' structures and performance-related semantics are made explicit, or equivalently, "specific" or "declarative" (as opposed to "implicit" or "inferred") and are tied to application instances throughout their lifecycles.

Figure 1 depicts an exemplary three-tier web application based upon a real contact management system, used for the purpose of illustrating the characterization of the application using metadata and the use of that metadata when making instantiation decisions according to an embodiment of the subject matter described herein. The application consists of two front- end HTTP servers ("http{1 ,2}), two application servers ("was{1 ,2}), labeled 100 and 102, a database cluster 104, an NFS server 106, and ancillary services. In Figure 1 , the ancillary services include a load balancer 108 that load balances traffic among web servers 100 and 102, gateways 110, 112, and 114 that separate internal and external networks, web application servers 114 and 116 that provide access to database cluster 104, a de- network address translator (DNAT) 118 that translates between internal and external addresses, a logger 120 that logs events, and a monitor 122 that monitors operations in the network. Database cluster 104 is further divided into load balancer 124, Databases 126 and 128 and backing node 130. Each block 100 through 130 illustrated in Figure 1 represents a virtual machine. Virtual machines 100-130 together implement the three-tier web application. In this example, the web application is a contact management system. For clarity, some network links, such as those to monitor node 122 and logger 120 are not shown. The arrows on the left-hand side of each VM 100-130 indicate inbound network connections and the arrows on the right- hand side of each VM represent outbound network connections. The dark boxes at the bottom of each VM 100-122 indicate that each VM is a clone of a reference image. The data driven definition enables and encourages component reuse-a side benefit of this type of architecture. SLAs specified for the architecture illustrated in Figure 1 may include response time for web services and portals, capacity in terms of numbers of users, and throughput in transactions per second.

The diagram in Figure 1 is typical of what is found in architectural design and operational documents in industry. The near incomprehensibility of this diagram is a reflection of the inherent complexity of real applications. And this diagram does not address details at the level of OS and software content and configuration. Furthermore, it captures the application, at best, at a specific moment in time.

Notwithstanding this complexity, imagine this application is deployed into a virtualized environment according to the above design. This may involve manual and/or automated provisioning and configuration (see, e.g., [29], [17], [2]). The new application instance is validated and approved for production. Consider this application instance as time moves forward. Configuration changes, performance tuning, software upgrades, VM and storage migrations - the actions of many people (and possibly management software) - will induce non-trivial changes in the structure and behavior of the application. These changes may alter its resource requirements. And, regardless of what form of (present-day) change management may or may not be in place, structure and resource requirements will diverge from the (nominal) design and become increasingly difficult to capture in a coherent and application-wide format. Indeed, we assert that after some time, the design is no longer known. Because the application is virtualized, many of the underlying details, such as which VMs are co-resident on the same hypervisors, which storage systems host the application's volumes, etc., are substantially opaque to the application's administrators. Likewise, due to the initial complexity and divergent evolution of the application, the semantics and requirements of the application's components become increasingly opaque to the infrastructure and its administrators. Note that the administrators of the application and the infrastructure may or may not be the same individuals or even from the same organization.

Consider now the following situation (which is closely-based upon a real-world experience): one day, response times for inbound HTTP transactions spike well-above acceptable values. Typical KPIs (Key Performance Indicators) do not indicate any CPU or network bottlenecks. Hours later, systems engineers investigating the problem discover that NFS operations are experiencing extreme latency and/or timing out, causing user transaction failures. Yet, the application's demand towards the NFS server seems normal. Upon further investigation, it is discovered that the NFS device is completely saturated (the NFS controller's CPU is at 100%). The root cause is determined to be that, while the HTTP servers' demand upon the NFS resource had not changed, the backing store for the logging subsystem happened to be on the same NFS device. And, while debugging an issue the previous night, Support had left debug logging levels set higher than normal on some systems. The resulting logging output saturated the NFS device, causing a near-outage situation and violating the application's SLAs for availability and performance. This sort of I/O comprises many small write transactions - one per line of each log file, in the worst case. The actual volume, in terms of bytes of I/O was relatively small, but the transaction rate swamped the NFS device. This is an excellent example of the non-triviality of specifying and provisioning storage, of inter-VM interference via a shared resource, and, depending on how it is characterized, an example of one or more second-order resources.

Restating the root cause in the resource domain, we could say one or more of the following: the logging server's demand upon the NFS resource exceeded its design specification; the NFS resource's capacity was insufficient to meet the demands of the logger in certain operational modes; or the combination of the demand from the HTTP servers and logging exceeded the capacity of the NFS resource.

Examining the architectural specification, it is not immediately obvious that both the logging and NFS data volumes might be co-located on the same storage device (or that this is permitted). Moreover, since volume allocation during instantiation was not sensitive to this (potential) resource dependency, the outcome of the allocation decision is effectively opaque to monitoring tools and personnel. And, because hardware details are encapsulated away from the application, developers and technicians often may not be able to tell where the storage resides (which is hopefully known by someone managing the infrastructure).

Diagnosing and resolving this issue - which we claim could have been avoided had a placement and management system such as the one we propose been used - was very costly (in time and personnel) to the respective service provider. The economic and business impacts of the incident SLA violations are not known to us; suffice it to say that many end- users were dissatisfied. The resolution process was inherently inferential - one couldn't just look somewhere and see the structure of the application. The existence of a shared resource was now known but not memorialized in the Management Layer - and so the application of lessons learned in this incident to future troubleshooting or subsequent deployments depends on human actors to persist and convey the information. A more robust mechanism, such as ours, would be desirable.

To give a better feeling as to the nature of the metadata itself, we will consider a much simpler application (Figure 2) and a simplified version of its metadata (Figure 3).

Figure 2 is a block diagram of a simplified web application that may be characterized using metadata according to an embodiment of the subject matter described herein. In Figure 2, W1 and W2 are web servers labeled with reference numbers 200 and 202, receiving HTTP requests on their IN interfaces and performing file I/O towards an NFS device 204. NFS device 204, in turn, communicates with some SAN (Storage Area Network) 206 via iSCSI, to address a volume 208 allocated to the application.

Figure 3 is a simplified metadata example according to an embodiment of the subject matter described herein. (Non-normative syntax.) Note that some resource specifications are expressed as functions of other resources and of performance metrics (SLA parameters). In the illustrated example, the SLA parameters txn_rate and rsp_time respectively specify a transmission rate and response time SLA parameters for the application. Metadata 302, 304, and 306 specifies each virtual machine that is part of the application. Metadata 302, 304, and 306 are subordinate to the SLA parameters resource specification. Metadata 308 characterizes network connections for the virtual machines. Metadata 310 characterizes a NAS volume assignment for the virtual machines.

Within an exemplary virtual machine metadata specification, such as metadata 302, each virtual machine is characterized by input and output interfaces the protocol used on the interfaces, and whether the network is public or backbone. Metadata 302 also specifies resources, such as CPU, RAM, inbound transmission rate, inbound receive rate, outbound transmission rate, and outbound receive rate. Metadata 302 also specifies performance metrics, including transmission rate and response time. Metadata 302 also characterizes performance dependencies, including transmission rate, CPU utilization as a function of transmission rate, RAM requirements as a function of transmission rate and response time, input receive bandwidth as a function of transmission bandwidth, NFS transmission bandwidth as a function of input receive bandwidth, input receive bandwidth as a function of transmission bandwidth, and input transmission bandwidth as a function of output interface receive bandwidth. It is these interdependencies between performance parameters of a single VM and between VMs that allow more intelligent application placement and other VM related resource management decisions.

The metadata of Figure 3 has been simplified somewhat for brevity - in particular, each VM ("subordinate") would have additional properties, such as the location for disk images and more detailed virtual hardware parameters. The omitted information is analogous to what would be found in an OVF (Open Virtualization Format) [38] or VMware VMX file [50]. This metadata is a non-normative example.

Note that the entire application is defined in a single (logical and possibly physical) document - capturing the complete structure of the application and its resource dependencies. In this instance, we have parameterized the application with a variable SLA (its capacity setting) which then feeds, in a data-driven manner, into resource specifications.

Network relationships are defined by pairs of input and output interfaces (inbound interfaces are to the left of each VM and outbound interfaces are to the right). (See also Figure 1 for an example of a much more complex application.) All network connections are explicit, protocol- specific, and defined (and modeled, but not necessarily implemented) on a point-to-point basis.

In the example illustrated in Figure 3, all network connectivity is declared explicitly. For example, the metadata for each VM specifies the network to which each interface illustrated in Figure 2 is connected in the protocol used on the interface. The two networks specified in Figure 3 are public and backbone. In addition, point to point network connections between the VMs are specified by metadata 308. There is no ambiguity as to who communicates with whom and we are able to carefully-specify bandwidth and other requirements for each connection. We are also able to traverse these dependencies to reason about application behavior and performance. This obviates the need for inferential, ongoing reverse- engineering of applications and/or using "network/application discovery" software after-the-fact (e.g., [31], [32]).

Durable storage is likewise explicitly-defined and associated with specific VMs; subsequently-allocated volumes are unambiguously linked to the specific application instance. For example, each VM is provided and interfaced to the NAS data volume NAS_data. The transmit and receive rate on the interface is also specified. The application's metadata contains all of this information (and more) and, indeed, is sufficient to reconstruct the above diagram. The metadata illustrated in Figure 3 contains examples of inter-VM resource dependencies and also functional (in the mathematical sense) relationships between SLA parameters and resource requirements. Examining metadata 302 for server W1 , we see that, for example, its inbound receive bandwidth (in.rx) is a function of transaction rate. In turn, its outbound bandwidth towards/demand upon the NFS server (NAS), given by nfs.tx, is a function of in.rx and in.tx is given as a function of nfs.rx - i.e., the bandwidth back to the user is a function of the volume of data received from the NFS server. The specification of the inbound transmission bandwidth (in.tx) of server W1 as a function of the demand received from the nfs server (nfs.rx) is an example of an inter-VM dependency. We have not concretized the (disk) I/O requirements (e.g., data.iops), but they would be specified in like manner. We also allude to the concept of second-order capacity requirements (defined and discussed below) such as data.io_profile which, might, for example, specify the expected read vs. write ratio demanded of the respective volume.

In practice, we envision the metadata document proper being created or developed in one or a combination of ways. First, note that the document follows a well-defined, closed (in the mathematical sense) grammar - one that is specifically intended to be machine- and human-generable and readable. An IDE (Integrated Development Environment), such as Eclipse [47] would be well-suited for the development of tools to assist in writing and maintaining metadata files. Metadata might also be written or modified by hand or even be the output of an application's build process. Graphical editing and development tools - such as those present in AppLogic [37] and tools for creating and editing UML - demonstrate the feasibility and virtues of abstract point-and-click visual editors for domain-specific languages. We also anticipate that new applications' metadata will be based upon (branched off of) those of existing applications; in particular, the creation of metadata documents from scratch will be uncommon and, with the help of various tool chains, non-onerous. Note that we do not plan to spend a significant amount of time building tools to create and edit metadata documents. The salient point is that, from a practical standpoint, they will be easy to create and manage.

Identifying exact contents, structure, and semantics of an application's metadata definition is one of the initial steps in our research and contributions thereof. For example, how detailed must resource definitions be to make placement decisions that provide SLA assurances? The determination of what information is required to make SLA-preserving placement and resource allocation decisions - including a deeper and more expansive study of what constitutes a "resource" - is one of the key contributions of the present research. The details of the grammar we use - i.e., our specific syntax and encoding - is an implementation detail.

The idea of such specification grammars for virtualized applications is not itself new. Similar, but far less-expressive mechanisms for specifying applications as groups of VMs exist, such as those seen in OVF, AppLogic, and as explored in, e.g., [46] and [2]. The foci of these mechanisms, however, are on application deployment and high-level management functions. The grammars and semantics thereof lack the expressiveness to capture inter-VM functional and resource dependencies.

For example, AppLogic [37] permits specification of per-VM network bandwidth as a single scalar quantity specifying a bound on the sum of all ingress and egress traffic to and from that VM. In contrast, we observe that the ability to specify per-interface receive and transmit bandwidth requirements is necessary to achieve even basic assurances with respect to network bandwidth. (Recall that, in our formulation (and in AppLogic'sl), each interface represents a protocol-specific point-to-point connection between VMs.)

Our primary focus is upon VM placement and allocation of system, network, and storage resources for applications. Note, however, that our design and philosophy broadly-construes the concepts of "resource" and QoS/SLA. For example, next generation infrastructures might include farms of graphics processing units (GPUs), digital signal processing (DSP) resources or ASIC-based services in general, cryptographic accelerators, security services and policies or compliance requirements, or even wireless spectra as other potential resource types. Certain resource-SLA pairings also follow naturally, e.g., a backing store with specific access restrictions, encryption, and backup policies may be a necessary condition for regulatory compliance. In a mixed-classification environment, the fact that a particular application's SLA depends on a specific type of storage would be captured in its metadata and becomes a necessary condition for it to be instantiated or relocated. The power of explicit structural and semantic specification becomes particularly evident in this situation. Ensuring such compliance in a mixed-classification environment by human-driven administrative policies entails greater expense and is more error-prone.

One exemplary implementation of the subject matter described herein is built upon an existing Virtual Infrastructure management software package, such as VMware vCenter [29]. In other implementations, the functionality of the subject matter described herein may be designed into or tightly-coupled to the balance of the Management Layer; this may be desirable, for example, if developing a net-new Virtual Infrastructure management software package. In any case, the subject matter described herein may be implemented, in substantial part, as an element of or in the Management Layer.

Our software uses application metadata to instantiate, place, allocate resources to, monitor, and manage Virtualized Applications. One implementation of the software does not necessarily replace the existing monitoring and management functions of off-the-shelf management software. It instead augments existing functionality vis-a-vis our metadata and semantics. The explicit structure and semantics captured in the metadata define how the application is deployed: to change the application's architecture, one changes the metadata (as well as, perhaps, the underlying application software and configuration) and the Management Layer re- configures and/or re-deploys the application pursuant to the new specifications. In this manner, the runtime structure and architecture (and, to a significant extent, semantics) of the application are always well-known to both the infrastructure and the humans charged with administering it. The Application is, throughout its lifecycle, as a first-class citizen/object; placement of and resource allocation to an application's constituent VMs must respect their individual, several, and transitive resource requirements. Applications' structure and dependencies remain transparent (indeed, explicit) throughout its lifecycle. To the best of our knowledge, no one to date has modelled Virtualized Applications in this manner and no system exists having the capabilities described above.

The definition of the metadata grammar and semantics, alone, is believed to improve the functionality a virtual machine management layer, the virtual machines themselves, and the underlying hardware that implements the virtual machines and management layers.

One distinguishing characteristic of our placement algorithm, in contrast to those in present use, is that we compute the placement of the entire application atomically such that application-wide constraints are satisfied. In contrast, in much of the literature and in commercial products such as VMware and AppLogic, placement is computed based solely on fitting each VM onto a host. To be fair, AppLogic does consider whether all the VMs in an application "fit" in the infrastructure before initiating startup. However, the definition of "fit" is on a per-VM basis, rather than on a Virtual Application basis.

The placement problem is an instance of a family of problems commonly-known as "packing problems" [26], [27], [52]. This family includes one- and multi-dimensional formulations of the knapsack problem, bin- packing, and so-called vector bin-packing. See [idem] for many examples of formal definitions of instances of these problems. All of the packing problems reviewed are NP-Hard in the strong sense and, depending on the formulation, may or may not admit bounded approximation algorithms with polynomial complexity [idem]. We will discuss a few of these problems at a high level to provide insight into the relation of the placement problem to these other problems.

In One Dimension:

Consider the one-dimensional bin-packing problem. We are given a set S of N items, each with some integral scalar size (or weight). The problem is to partition S into the smallest number of subsets such that the sum of the sizes of the items in each set is less than some integer B (the bin capacity), where a partition of a set S is a set P of disjoint subsets of S such that the union of all elements in P is exactly S. For example, a partition of the set { a, b, c, d } is { { b}, {a, d}, {c} }. Note that, in this traditional formulation, the problem is stated as an optimization problem. An example from real life is the problem of moving the contents of one's garage to a storage unit using a vehicle of finite capacity (in mass) such that the number of round-trips is minimized; in this case, the bin is the vehicle and the items are the contents of the garage.

In Multiple Dimensions:

Extending bin-packing to multiple dimensions (resulting in a problem in the class of multidimensional bin-packing or vector bin-packing problems) is straightforward. In this case, each item has a vector of integral scalar sizes (or weights) and each bin has a vector (of the same dimensionality) of capacities. The statement of the optimization problem follows naturally from above. Returning to our example, this problem is analogous to the problem of fitting (in three spatial dimensions) all of our items into the minimum number of storage units. The wrinkle here is that as each item is assigned to a bin, it consumes resources (space, in this case) in multiple dimensions, whereas, in the previous example, it consumed a single resource (mass).

Equivalence to Decision Problem:

Again, the problems above are stated in terms of optimization (minimizing the number of bins required). However, altering the problem such that we ask whether the set of items can be packed into some specific number of bins (e.g., can I move all my stuff in three round trips?), i.e., converting the optimization problem into a decision problem, is no less algorithmically difficult.

Mapping to the Placement Problem:

Returning to Virtualized Applications, let us consider a simplified instance of the placement problem. Let V be the set of VMs in some application A. Each element of V requires three types of resources: CPU, RAM, and network bandwidth (all of which, without loss of generality, are integer-valued). Let H be a set of hosts (onto which we will place V), having capacities of CPU, RAM, and network bandwidth. The placement problem can be stated, then, as follows: does there exist a mapping of the VMs in V onto the hosts H such that, for each host, the capacity consumed by the VMs placed on the host does not exceed the capacity of the host? If such a mapping exists, then we have a feasible placement - i.e., it is possible to place A such that its resource requirements are respected. The same construction also permits optimization problems, e.g., to minimize the number of hosts used in the placement or to maximize the remaining free RAM on each host after placement.

As discussed above, to a first approximation, the placement problem is a multi-dimensional (or vector) bin-packing problem, and even a highly- simplified form, is NP-complete in the strong sense [25-27], [52]. However, relaxing the problem from finding an optimal solution (for some definition of optimality) to finding a feasible solution and the imposition of certain lightweight congruency constraints on resource quantities enables the use of well-understood (pseudo-)polynomial approximation algorithms. See, e.g., [39], [52]. Common implementations use variants of Integer Linear Programming (ILP) solvers (the problem itself can be reduced to ILP) [idem]; maintaining linearity is important (see below).

The subject matter described herein strikes a balance between tractability and optimality (with respect to one or more criteria).

Returning to algorithmic complexity and tractability - placement decisions are made at application instantiation (and later, start-up) time and need not be instantaneous. A placement algorithm whose execution time is on the order of a few minutes would be quite acceptable and represent a small fraction of the total instantiation time (which is typically dominated by large I/O operations). The broader class of online and offline bin-packing techniques for (re)allocating resources has been explored in the literature;

[39] is a good exemplar and discusses a number of contemporary approaches in §6.

Acceptable real-time computational overhead is thusly achieved. Other embodiments may also comprise longer-running off-line optimizations, for example, to reduce power consumption or improve hardware utilization. The characterization, quantification, and metadata-grammatical mappings of second-order resources is another facet of the present disclosure.

For example, disk I/O capacities, including IOPS, throughput, read/write ratio, and access patterns (e.g., sequential vs. random accesses) are inter-related. These relations may be device-dependent - for example, the trade-off between IOPS and throughput is dramatically different when comparing SSDs (Solid-State Disks) with Winchester disks or when comparing storage arrays using different technologies or comprised of unequal numbers of spindles.

Importantly, for a given system, the consumption of one capacity may cause a reduction in another capacity. For example, the cost of reads and writes varies among different RAID topologies (because of the number of spindles touched per operation) [53]. This is actually a result of a transitive dependency: the number of IOPS required of logical volume by a VM may not correspond one-to-one with the IOPS required of the backing store to service the demand. For example, according to [idem] a single write operation to a RAID-5 volume requires four times as many writes to the backing store. This in turn may reduce overall IOPS capacity and, in turn, possibly reduce throughput capacity. Minimally, the cost of placement of a volume may be different depending on which storage device is chosen. Said another way, the "weight" of an item changes depending on which bin it is placed in. This is non-independence.

There do exist packing algorithms that address non-linearity and/or dependence, though they are more complicated and more computationally- expensive. However, we expect that each instance of the placement problem (a request to place an application instance upon a virtual infrastructure) having such non-linearity or dependence can be reduced or transformed into an equivalent linear, independent problem and computed using off-the-shelf solvers.

Viewing runtime telemetry (system and application performance metrics, for example) through the lens of explicit structure and semantics enables more precise, less onerous, and more efficient and intuitive analytic and diagnostic techniques than are presently feasible. Importantly, the semantic construction of application telemetry is data-driven, with respect to the explicit metadata. In contrast, classical and present approaches rely on intuition, anecdotal and implementation-specific knowledge, and inference; these factors collude to obstruct efforts to document and/or automate monitoring and analytics and to adapt the same to (opaque) changes in application structure or behavior.

An infrastructure thusly-constructed and managed enables multiple further avenues of investigation, such monitoring and analyzing runtime telemetry informed by the explicit structure, resource requirements, and dependencies specified in the metadata.

We use the word "telemetry" to encapsulate the large equivalence class of quantitative measurement and metrics emitted and collected by the system. Rich sources of such data at the guest OS level are well- established, for example the sysstat package [54], SNMP MIBs, and the performance monitoring tools in Microsoft Windows [55]. Existing Management Layer products provide performance data pertaining to hypervisors and guests (from the hypervisors' perspective) and may well already integrate the collection of guest-specific data.

Evaluating and monitoring SLA compliance, however, requires exposing application-specific metrics. In our model, applications are responsible for exposing SLA-relevant metrics via one or more standardized but yet-to-be-defined interfaces (the definition is an implementation detail). Thinking about this presently, it would make sense to include metadata constructs used to define how these data are communicated to the Management Layer (e.g., a URI for a canonical web service or an SNMP OID). In the example of Figure 1 , the "mon" VM 122 serves as a sink for monitoring data and is responsible for either relaying such data to the Management Layer (not shown in Figure 1 ) or exposing a canonical web service interface for such data to be retrieved.

In practice, we expect that telemetry would flow automatically (through some mechanism) to the appropriate bits of the Management Later. Many systems already exist to facilitate such monitoring. Our part of the story is the interpretation of these data through the lens of our structural and semantic metadata.

Some telemetric analysis and monitoring can be performed automatically. This is standard practice today, albeit with less structure than we envision.

An example of an automated metadata-driven analysis would be an off-line audit of resource reservations versus actual utilization. Such analysis would identify VMs which may be allocated more capacity than they ever use - or guests whose demand is approaching or exceeds their allocated capacity (modulo some safety margin).

Another, more challenging operation, would be to consider the placement of all active applications in light of their reservations. This would be an opportunity to use an optimization algorithm to, for example, determine if rearranging guests would be beneficial; the application of optimization algorithms to this problem is seen in the literature, e.g., [6], [16]. Two new directions for such optimization we see are rearranging VMs to "defragment" resources and analytics leading to metadata refinement (see below).

Analysis may also be performed by humans. (We certainly are not claiming that our contributions can replace skilled systems engineers when things go wrong.) This is a place where our methods will shine: explicit dependencies will be used to organize, present, and analyze telemetry; we expect that this will lead to more obvious and intuitive determination of cause-and-effect, for example.

Consider an application analogous to those of Figure 1 and Figure 2.

Suppose the measured transaction rate is not meeting the specified SLA. We (or our tools) know the graph of VMs and resources that are declared as affecting transaction rate. For example, the front-end web servers' performance depends on the performance of the NFS volume. "Hot spots" along this dependency chain can be identified mechanically based upon telemetry and the known resource requirements and semantics. If, for example, a particular network interface is overloaded, we know exactly what traffic depends upon it and, assuming an accurate set of resource specifications, the source(s) of excess traffic.

Working in the other direction, if we come to know that a particular device is overloaded or perhaps in need of maintenance, we can easily determine which application instances and what specific aspects of their SLAs would be affected. This ties low-level aspects of the infrastructure directly, and with rich semantics, to end-user experience.

Going back to our top-level philosophy, by being specific (explicit) as to the structure and deployment of our Virtualized Applications, we are always in a position of knowing rich end-to-end detail about our applications and their interaction with our infrastructure. We see this as a very close approximation of analyzing a bare-metal system (which was our initial basis for our discussion of structure). Indeed, we would argue that our system offers more (independent of virtualization), in that SLA characteristics are directly tied to detailed resource requirements.

We see our work opening the door to research into visualization techniques for application telemetry. Providing a graphical view, including structural context, of system performance and resource consumption (read: high-dimensional data sets) over time is, at the very least, an interesting challenge. We believe that metadata will allow the derivation of semantically- meaningful spatial mapping of telemetry which, in turn, enables the use of visualization tools that, to our knowledge, are not commonly-applied to these sorts of Systems problems. Such visualization work is outside the scope of the present proposal (and our skill-set), but is an opportunity for collaboration with other groups at UNC.

Our system insists that one be specific (explicit) in specifying an application's resource requirements. But where do these requirements come from? And what if they are incorrect or prove to be insufficient as time moves on? Having been explicit at the outset, and armed with our metadata and telemetric capabilities, we are in a powerful position to make adjustments.

Specifically, at the heart of our third contribution, we address the very likely circumstance where an application's performance specifications (i.e., given resources X, I guarantee performance Y) are incorrect or fail to hold due to changes in, for example, user behavior. We observe that comparison of runtime telemetry (empirical data) to specifications, with (asymptotically- )full knowledge of application structure and semantics, should lead to refinement of the specifications.

For example, we may find that a resource specification is too large

(as is often the case), resulting in underutilization of the respective hardware (and wasted energy); in this instance, we may be able to refine the metadata and alter the application's resource allocations. The same argument holds in the case where the resource specification was insufficient (the application's demand exceeds its allocation) and/or SLAs are missed. Other approaches to this problem have been considered in, e.g., [8], [9], [24]. Finally, we may find that certain important resources are unspecified, again leading to refinement of the metadata.

We claim that the overall process - specify, run, refine - will converge to the correct specification. The resource tuning undertaken is explicit, transparent, and memorialized and can readily be applied to new or existing instances of the same or similar applications. Our work refining a given application instance thus scales as it is reused for subsequent instances of the same or similar applications (or components thereof).

Said another way, as an application instance is tuned, the definition of that application becomes more and more specific (explicit). Nuances are (hopefully) captured in a standard way via metadata and remain memorialized. In contrast, as traditional systems are tuned and tweaked, their definitions become increasingly vague as they diverge from their design and deployment specifications.

Explicit structural and semantic specification leads to a feedback loop tending towards specificity, determinism, and transparency. Implicit and inferential methods lead to a feedback loop tending towards vagueness, unpredictability, and opacity.

Returning to the concept of creating a new instance of an application based upon (refined) metadata: suppose an application is developed by one research group to support protein visualization. In traditional bare-metal and present-day virtualized systems, the (very likely) single instance of this application will be tightly-coupled to the environment and the process of tuning it to its present level of performance will not be well-documented (or well-understood). In fact, we expect that researchers would shy away from virtualization for such applications for precisely the concerns regarding performance and predictability (SLAs) addressed by our work.

Transferring this application to a new lab or sharing it with another research group would be quite an undertaking - potentially discouraging collaboration. Anecdotally, we have learned that this is indeed a problem. Our infrastructure, however, would have necessarily captured, at least at a systems level, all of the application's performance-critical dependencies. We would expect that a "clone" of the application in the same or another infrastructure (perhaps at another university) would perform equivalently to the original (indeed, this is one of the criteria we will use to evaluate our implementation),

The knowledge gained scales - and does so in a transparent, data- driven, explicit manner. This gives us the power to clone or relocate applications with high confidence that, assuming their resource requirements are met, they will perform predictably in other environments. In this regard, we actualize one of the central (purported) value propositions of hardware virtualization: freeing applications from dependence on specific physical infrastructure.

Interestingly, a likely finding in the iterative refinement process is that resources are under-specified - more specifically, that some resource is unspecified or a resource is, in effect, a hidden variable. As previously discussed, the co-location of multiple guests on the same host can result in unanticipated performance consequences; indeed, interference between guests has been studied and characterized (in part) in the literature (see e.g., [3], [7], [10]). For example, the co-location of two guests with particular CPU, RAM, or storage access patterns may result in disproportionate degradation in both guests' performance. I.e., two VMs that appear to "fit" onto a particular hypervisor fail to do so. Viewed another way, in addition to traditional "first-order" resource capacities (e.g., RAM, IOPS MIPS), in certain contexts, "second-order" resource dependencies become relevant.

Second-order resources are characteristics of the way in which first- order resources are used - usage patterns. They exist and are consumed by applications even when resources are not shared; they become visible, however, when resources are shared. Simply, the effective capacity of a resource, such as a disk, may decrease in the presence of contention or interference. Whether the capacity decreases or by how much depends on the way in which the contenders use it.

We do not claim to have discovered this fundamental property of systems. We do claim that we can characterize it more precisely in the context of Virtualized Applications and use such characterization to make placement decisions that minimize such interference - interference-aware, mechanical placement and resource allocation.

Clearly, as such requirements are discovered, the respective second- order resources will become candidates for specification in application metadata and resource allocation, thus leading to higher long-term predictability in terms of performance and satisfaction of SLAs. Figuring out how to characterize second-order resources in a manner compatible with our larger resource allocation problem will be another contribution of our work.

Second-Order Resources - Example

Referring to storage specifically, one should observe that merely stating a size/capacity requirement is woefully insufficient from a performance standpoint. Our characterization of the resource requirements of a volume minimally includes IOPS, read and write throughput (MiB/second), and storage capacity. A further refinement is to provide at least coarse-grained modeling information about the expected I/O towards the device (read-to-write ratio, random versus sequential, small versus large, etc.). The "capacities" in the second group are arguably second-order. They are, however, reasonably-well-understood and easily-measurable. If the Management Layer has knowledge of the respective capacities of the storage devices it manages, it can make intelligent decisions about where to place a particular volume. Consider also the importance of these factors if the storage subsystem itself is virtualized (i.e., virtualized storage). Because the Management Layer knows the per-application requirements of the other volumes on its devices and the requirements of a proposed new volume, it can place that volume such that interference with and from other volumes is minimized. Alternatively, it may determine that there is insufficient capacity to place the application without adverse effects. What may appear to be an infrastructure with abundant resources may in fact have exhausted critical second-order resources.

Thus, we propose an application framework and run-time management system wherein processing, storage, and communication are initially mapped to local, wearable resources (e.g., glasses), local portable resources (e.g., smartphone, laptop), and remote resources (e.g., cloud) to realize QoS requirements broadly construed. The approach is to develop a sufficiently-expressive mechanism (a "grammar") for specifying the architecture and resource requirements of applications in sufficient detail to enable provisioning of virtualized or physical local and remote resources and to monitor, orchestrate, and adapt application-component to resource mappings to meet performance requirements. Existing VM management and placement systems lack the ability to make placement decisions that respect intra-application resource dependencies. Indeed, existing technology lacks the ability to describe such dependencies and requirements with specificity sufficient to address the problem in the first instance.

In conjunction with the specification means, we will propose the analysis of application runtime telemetry (e.g., time series of OS- and application-specific performance metrics) to enable developers to adapt an application's grammar (performance and architecture specification) on-the- fly and enable the management layer to adapt allocations by leveraging the application's explicitly specified structure and transitive resource dependencies (as specified in its grammar). We posit that runtime analysis guided and enabled by the application's explicitly-defined relationships and properties, i.e., in the metadata used to specify the application, is strictly more powerful than current techniques that, at best, rely on inferential knowledge (from other external analytic software and/or ad-hoc human inputs).

The subject matter described herein constitutes an ideal mixture of performance sensitivities (with respect to resources) and, by the nature of the research projects in question, a source of objective, concrete evaluation of approaches and solutions to the locus of relevant Virtualized Application resource allocation, placement, and analysis problems.

Figure 4 illustrates a process for providing a high performance virtual application. Referring to Figure 4, in step 400, a request to instantiate or start up an instance of a virtual application comprising a plurality of virtual machines is received. In one example, the request may be received via a user interface of a virtual application management layer that makes instantiation or start up decisions for virtual applications and that iteratively refines metadata associated with the applications as new performance data is received.

In step 402, in response to receiving to the request, an instantiation or start up decision related to the instance of the virtual application is made based on metadata that describes resource requirements and inter-VM dependencies of the virtual application. For example, the virtual application management layer may utilize metadata, such as that illustrated in Figure 3 and data about underlying physical or virtual hardware to determine whether to instantiate or start up a virtual application.

In step 404, it is determined whether available resources are sufficient to support instantiation of the virtual application. If resources are not sufficient, control proceeds to step 406 where the instantiation is rejected and then control returns to step 400 where a new request is received.

If, in step 404, it is determined that the available resources are sufficient to support the instantiation, control proceeds to the step 408 where the virtual application is instantiated on the available resources. For example, multiple virtual machines may be instantiated or started up on hardware resources to implement a desired service, such as the web application service illustrated in Figure 1 or Figure 2.

In step 410, performance data associated with the instance of the virtual application is received. The performance data may include resource utilization by the virtual application, remaining resources, or a combination thereof. The performance data may be received from a monitor application, such as monitor application 122 illustrated in Figure 1.

In step 412, the performance data is analyzed to determine SLA compliance or resource usage of the instance. As stated above, SLA parameters may be specified in the resource metadata. The monitor performance may be compared with the SLA parameters set forth in the resource metadata to determine whether the SLA specifications are being complied with. Absent SLA specifications, the performance data may be analyzed to determine resource utilization of the virtual application.

In step 414, the metadata may be modified to adjust the performance or resource usage of the virtual application. As stated above, the metadata associated with a virtual application may be iteratively modified based on performance data to further include the performance of the application.

Figure 5 is a block diagram illustrating a virtual application management layer that utilizes virtual application metadata in performance data to instantiate and refine performance of virtual applications. In the illustrated example, a virtual application management layer 500 resides in memory 502 of a virtual application management layer platform 504. Virtual application management layer platform 504 may be a computing platform including one or more processors 506 as well as memory 502. I/O interfaces and other hardware associated with virtual application management layer platform 504 are omitted for simplicity. It is understood however that virtual application management layer platform 504 may include one or more network interfaces for communicating with other devices in the network. Virtual application management layer 500 receives virtual application metadata 508 that describes resource requirements and inter-VM dependencies of one or more virtual applications. The metadata may be any of the metadata described above. The metadata may be received from a user via a graphical user interface, read from a file, etc. Virtual application management layer 500 may make virtual application instantiation decisions to instantiate virtual applications 510, 512, and 514 on physical resources 516 and 518. It is understood that physical resources 516 and 518 may be located on the same platform as virtual application management layer 500 or on different platforms from virtual application management layer 500. In addition, multiple virtual applications may be instantiated on the same platform and/or a single virtual application may be distributed across multiple platforms. A monitor 520 monitors the resource utilization of virtual applications 510, 512, and 514 and the available resources 516 and 518. Monitor 520 reports this data to virtual application management layer 500. Virtual application management layer 500 may iteratively change virtual applications 510, 512, and 514 by making changes to the corresponding metadata 508.

In one embodiment, a method of characterizing resource requirements for virtualized applications or components thereof includes characterizing resource requirements based on the effects of interactions among a plurality of components. The method may include characterizing resource requirements based on the effects of interactions among a plurality of components that use a shared resource. In one embodiment, at least one of the plurality of components may be a virtual machine. In one embodiment, the shared resource would not be shared in the absence of virtualization. In one embodiment, the effects of interactions may be due, in whole or in part, to the behavior of the Virtual Machine Monitors (VMMs) or hypervisors on which the virtual machine (VM) components of the application are executed. In one embodiment, the resource requirements may be quantified and/or specified using scalar numerical quantities, vectors of numerical values, Boolean values, bitmasks, zero or more elements of a set of nominal values, an abstract, heuristic, or concrete model pertaining to how the resource is used, strings of characters, a distinguished null or undefined value, functions and/or algebraic expressions, functions and/or algebraic expressions parameterized by one or more resource requirements, binary data, or other appropriate means. In one embodiment, characterizing resource requirements may include characterization, quantification, and/or specification of the resource capacities of hardware, hypervisors, devices, software systems, network services, and/or other extra-application dependencies. In one embodiment, the characterization, quantification, and/or specification may include consideration of capacities arising from and/or due to the resource being shared. In one embodiment, the resource whose capacity is being characterized, quantified, and/or specified may be shared among individual virtual machines (VMs), virtual machines that are components of virtualized applications, or devices, resources, services, or other entities placing demand upon the resource in question. In one embodiment, quantification or specification of capacity may use scalar numerical quantities, vectors of numerical values, Boolean values, bitmasks, zero or more elements of a set of nominal values, an abstract, heuristic, or concrete model pertaining to how the resource is used, strings of characters, a distinguished null or undefined value, functions and/or algebraic expressions, functions and/or algebraic expressions parameterized by one or more resource requirements, binary data, or other appropriate means. In one embodiment, the generation and/or computation such characterization, quantification, and or specification may be automated. In one embodiment, the method may include implementing the storage and retrieval of the concrete characterizations, quantifications, and/or specifications of such capacities. The storage and retrieval may be implemented using, for example, a relational and/or non-relational database, a shared file system, shared storage, remote database access, web services, RESTful web services, SOAP web services, CORBA interfaces, RPC mechanisms, network services, messaging busses and/or enterprise service busses (ESBs), and/or mechanisms following publish-subscribe and/or notify- subscribe semantics. In one embodiment, the method may include using the values of the capacities for determining and/or optimizing placement, provisioning, and/or configuration of virtual machines and/or components of virtualized applications.

In one embodiment, a system for characterizing resource requirements for virtualized applications or components thereof may include multiple virtualized applications, each having at least one component, and a characterization module comprising hardware and for characterizing resource requirements based on the effects of interactions among the plurality of components. In one embodiment, the characterization module characterizes resource requirements based on the effects of interactions among a plurality of components that use a shared resource. In one embodiment, at least one of the components may be a virtual machine. In one embodiment, the shared resource would not be shared in the absence of virtualization. In one embodiment, the effects of interactions may be due, in whole or in part, to the behavior of the Virtual Machine Monitors (VMMs) or hypervisors on which the virtual machine (VM) components of the application are executed. In one embodiment, the resource requirements may be quantified and/or specified using scalar numerical quantities, vectors of numerical values, Boolean values, bitmasks, zero or more elements of a set of nominal values, an abstract, heuristic, or concrete model pertaining to how the resource is used, strings of characters, a distinguished null or undefined value, functions and/or algebraic expressions, functions and/or algebraic expressions parameterized by one or more resource requirements, binary data, or other appropriate means. In one embodiment, characterizing resource requirements may include characterization, quantification, and/or specification of the resource capacities of hardware, hypervisors, devices, software systems, network services, and/or other extra-application dependencies. In one embodiment, the characterization, quantification, and/or specification may include consideration of capacities arising from and/or due to the resource being shared. In one embodiment, the resource whose capacity is being characterized, quantified, and/or specified may be shared among individual virtual machines (VMs), virtual machines that are components of virtualized applications, and/or devices, resources, services, or other entities placing demand upon the resource in question. In one embodiment, quantification or specification of capacity may use scalar numerical quantities, vectors of numerical values, Boolean values, bitmasks, zero or more elements of a set of nominal values, an abstract, heuristic, or concrete model pertaining to how the resource is used, strings of characters, a distinguished null or undefined value, functions and/or algebraic expressions, functions and/or algebraic expressions parameterized by one or more resource requirements, binary data, or other appropriate means. In one embodiment, the generation and/or computation such characterization, quantification, and or specification may be automated. In one embodiment, the characterization module may implement the storage and retrieval of the concrete characterizations, quantifications, and or specifications of such capacities. In one embodiment, the storage and retrieval may be implemented using a relational and/or non-relational database, a shared file system, shared storage, remote database access, web services, RESTful web services, SOAP web services, CORBA interfaces, RPC mechanisms, network services, messaging busses and/or enterprise service busses (ESBs), mechanisms following publish-subscribe and/or notify-subscribe semantics, or other means. In one embodiment, the characterization module may use the values of the capacities for determining and/or optimizing placement, provisioning, and/or configuration of virtual machines and/or components of virtualized applications.

Thus, a virtual application manager that uses metadata that describes interdependencies between virtual machines of a virtual application improves the performance of virtual applications by making better instantiation decisions. In addition, the performance of virtual applications is iteratively improved by monitoring the performance of the virtual applications and making changes to the virtual machines that make up the applications by modifying the corresponding metadata. Such a management scheme is an improvement over existing management schemes that do not consider inter-VM dependencies when making instantiation decisions and/or don't iteratively modify the virtual applications by modifying the corresponding metadata.

The disclosure of each of the following references is incorporated herein by reference in its entirety:

[1] [D. Jayasinghe, C. Pu, T Eilam, M. Steinder, I. Whalley, and E. Snible] "Improving Performance and Availability of Services Hosted on laaS Clouds with Structural Constraint-aware [sic] Virtual Machine Placement". 201 1 . IEEE International Conference on Services Computing, pp. 72-79.

[2] [G. Breiter and M Behrendt] "Life cycle and characteristics of services in the world of cloud computing". 2009. IBM Journal of Research and Development, Volume 53, Number 4, Paper 3. pp. 3:1 -3:8.

[3] [A. Do, J. Chen, C. Wang, Y. Lee, A. Zomaya, and B. Zhou] "Profiling Applications for Virtual Machine Placement in Clouds". 201 1 . IEEE 4th International Conference on Cloud Computing, pp. 660-665.

[4] [L. Badger, T. Grance, R. Patt-Corner, and J. Voas] "Cloud Computing Synopsis and Recommendations". NIST Special Publication 800-146. 2012. (full document -- about 80 pages).

[5] [J. Wan, P. Shan, and X. Xu] "A Utility-based Adaptive Resource Allocation Policy in Virtualization Environments". 2009 Eighth IEEE International Conference on Dependable, Autonomic, and Secure Computing, pp. 63-68.

[6] [O. Niehorster and A. Brinkmann] "Autonomic Resource Management Handling Delayed Configuration Effects". 201 1 Third IEEE Conference on Cloud Computing Technology and Science, pp. 138-145.

[7] [X. Pu, L. Liu, Y. Mei, S. Sivathanu, Y. Koh, and C. Pu] "Understanding Performance Interference of I/O Workload in Virtualized Cloud Environments". 2010 IEEE 3rd International Conference on Cloud Computing, pp. 51 -58.

[8] [W. Iqbal, M. Dailey, D. Carrera] "Black-Box Approach to Capacity Identification for Multi-Tier Applications Hosted on Virtualized Platforms". 201 1 IEEE International Conference on Cloud and Service Computing, pp. 1 1 1 -1 17.

[9] [F. Vraalsen] "Performance Contracts: Predicting and Monitoring Grid Application Behavior" (M. Sc. Thesis, University of Illinois at Urbana- Champaign) 2002. [10] [S, Kraft, et al.] "Performance Models of Storage Contention in Cloud Environments". Springer J. Software and Systems Modeling, 2012.

[1 1 ] [G. Casale, et al] "Modeling Exogenous Variability in Cloud Deployments". ACM Pert. Eval. Review, March 2013. [12] [W. Otte, A. Gokhale, D. Schmidt] "Efficient and Deterministic Application Deployment in Component-based Enterprise Distributed Realtime and Embedded Systems". Information and Software Technology. 2012.

[13] "Deployment and Configuration of Component-based Distributed Applications", v.4.0, OMG, Document formal/2006-04-02 edition. 2006. [14] [G. Deng D. Schmidt] "DAnCE: A QoS-enabled Component

Deployment and Configuration Engine". (Publication status unknown) -2006.

[15] [R. Sambasivan, et al.] "Diagnosing performance changes by comparing request flows". 8th USENIX Symposium on Networked Systems Design and Implementation (NSDI) (201 1). [16] [A. Totok] "Exploiting Service Usage Information for Optimizing Server Resource Management". ACM Transactions on Internet Technology, Vol. 1 1 , No. 1 , Article 1 , Publication date: July 201 1

[17] [A. Konstantinou et al.] "An Architecture for Virtual Solution Composition and Deployment in Infrastructure Clouds". ACM VTDC 9, June 15, 2009.

[18] [R. Kumar, et al.] "Enabling Semantic Communications for Virtual Machines via iConnect," Virtualization Technology in Distributed Computing (VTDC), 2007 Second International Workshop on , vol., no., pp.1 ,8, 12-12 Nov. 2007. doi: 10.1 145/1408654.1408655 [19] [N. Furmento, et al] "An Integrated Grid Environment for Component Applications". : GRID 2001 , LNCS 2242, pp. 26-37, 2001 . Springer-Verlag. [20] [P. Padala, et al.] "Automated Control of Multiple Virtualized Resources". EuroSys'09, April 1-3, 2009, ACM.

[21] [B. Sotomayor, et al] Overhead Matters: A Model for Virtual Resource Management". VTDC 2006. [22] [F. Berman, et al.] "Adaptive Computing on the Grid Using AppLeS". IEEE Transactions on Parallel and Distributed Systems. 2003.

[23] [H. Dail, et al.] "A decoupled scheduling approach for Grid application development environments". Journal of Parallel and Distributed Computing. 2003. [24] [P. Shivam, ... J. Chase] "Learning Application Models for Utility Resource Planning". ICAC 2006.

[25] [C. Papadimitriou, et al.] "Combinatorial optimization : algorithms and complexity." Prentice-Hall (1981 )

[26] [M. Garey, et al.] "Resource constrained scheduling as generalized bin packing." J. Comb. Theory, Ser. A 21 (3) (1976) 257-298:

[27] [C. Chekuri and S. Khanna] "On multidimensional packing problems." SIAM Journal on Computing 33(4) (2004) 837-851 :

[28] "VMware Customers", http://www.vmware.com/company/customers/. Accessed January 2014. [29] "VMware vCenter Operations Manager Suite". Datasheet. VMware Corporation. 2013.

[30] [Davis, D.] "Stop the Finger-Pointing: Managing Tier 1 Applications with VMware vCenter™ Operations Management Suite™". Whitepaper. VMware Corporation. 2012. [31] "vCenter Application Discovery Manager". Datasheet. VMware Corporation. 2010. [32] "Datacenter Operational Excellence Through Automated Application Discovery & Dependency Mapping". Technical Whitepaper. VMware Corporation. 2010.

[33] Amazon EC2. http://aws.amazon.com/ec2 [34] Amazon AWS OpsWorks. http://aws.amazon.com/opsworks/

[35] Amazon AWS CloudFormation. http://aws.amazon.com/cloudformation

[36] [D. Chappell] "Introducing Microsoft Azure". David Chappell and Associates. 2010.

[37] CA Technologies AppLogic, http://www.ca.com/us/cloud-platform.aspx [38] Open Virtualization Format, http://www.dmtf.org/standards/ovf

[39] [W. Song, Z. Xiao, Q. Chen, and H. Luo], "Adaptive Resource Provisioning for the Cloud Using Online Bin Packing," IEEE Transactions on Computers, vol. 99, no. PrePrints, p. 1 , , 2013

[40] [T. Duong, X. Li, R.S.M Goh, X. Tang, and W. Cai] "QoS-Aware Revenue-Cost Optimization for Latency-Sensitive Services in laaS Clouds", Distributed Simulation and Real Time Applications (DS-RT), 2012 IEEE/ACM 16th International Symposium on, On page(s): 1 1 - 18.

[41] [P. Leitner, W. Hummer, B. Satzger, C. Inzinger, and S. Dustdar] "Cost- Efficient and Application SLA-Aware Client Side Request Scheduling in an Infrastructure-as-a-Service Cloud", Cloud Computing (CLOUD), 2012 IEEE 5th International Conference on, On page(s): 2 3 - 220.

[42] [T. Duong, X. Li, and R.S.M. Goh] "A Framework for Dynamic Resource Provisioning and Adaptation in laaS Clouds", Cloud Computing Technology and Science (CloudCom), 201 1 IEEE Third International Conference on, On page(s): 312 - 319.

[43] [A.L. Freitas, N. Parlavantzas, and J.-L. Pazat] "An Integrated Approach for Specifying and Enforcing SLAs for Cloud Services", Cloud Computing (CLOUD), 2012 IEEE 5th International Conference on, On page(s): 376 - 383.

[44] [Y. Song, H. Wang, Y. Li, B. Feng, and Y. Sun] "Multi-Tiered On- Demand Resource Scheduling for VM-Based Data Center," Cluster Computing and the Grid, 2009. CCGRID '09. 9th IEEE/ACM International Symposium on , vol., no., pp.148, 155, 18-21 May 2009.

[45] [H. Madhyastha et al.] "sec: cluster storage provisioning informed by application characteristics and SLAs." FAST'12 (2012).

[46] [T.C. Chieu, A. Mohindra, A. Karve, and A. Segal] "Solution-based deployment of complex application services on a Cloud," Service Operations and Logistics and Informatics (SOLI), 2010 IEEE International Conference on , vol., no., pp.282,287, 15-17 July 2010

[47] Eclipse IDE. http://www.eclipse.org

[48] Xen Hypervisor. http://www.xenproject.org/ [49] VMware. http://www.vmware.com

[50] "Configuration Examples and Troubleshooting for VMDirectPath". Technical Note. VMware. 2010. Pages 2-3.

[51] [D. Bertram] "Likert scales." (2006): 1-10. http://poincare.matf.bg.ac.rs/~kristina/topic-dane-likert.pdf. Accessed January 2014.

[52] [J. Puchinger, G. Raidl, and U. Pferschy] "The multidimensional knapsack problem: Structure and algorithms." INFORMS Journal on Computing 22, no. 2 (2010): 250-265.

[53] [S. Lowe] "Calculate IOPS in a storage array". http://www.techrepublic.com/blog/the-enterprise-cloud/calculate-iops-in-a- storage-array/#. (2010). Accessed January 2014. [54] Sysstat. http://sebastien.godard.pagesperso-orange.fr/. Accessed January 2014.

[55] Windows Performance and Reliability Monitoring. http://technet.microsoft.com/en-us/library/cc771692.aspx. (2007) Accessed January 2014.

[56] [D. Nguyen, F. Lelli, M. Papazoglou, and W-J. Van Den Heuvel.] "Blueprinting approach in support of cloud computing." Future Internet 4, no. 1 (2012): 322-346.

[57] [D. Nguyen] Blueprint model and language for engineering cloud applications. No. urn: nbn: nl: ui: 12-5928387. Tilburg University, 2013.

[58] [A. Reuter] "An Extensible Application Topology Definition and Annotation Framework." PhD Dissertation. (2013)

[59] [L. Sun, H. Dong, and J. Ashraf] 2012. "Survey of Service Description Languages and Their Issues in Cloud Computing", in J. Guerrero (ed), 8th International Conference on Semantics, Knowledge and Grids (SKG), Oct 22-24 2012, pp. 128-135. Beijing, China: IEEE.

[60] [G. Hughes, D. Al-Jumeily, and A. Hussain] "A Declarative Language Framework for Cloud Computing Management," Developments in eSystems Engineering (DESE), 2009 Second International Conference on , vol., no., pp. 279,284, 14-16 Dec. 2009 doi: 10.1109/DeSE.2009.16

[61 ] [E. Albert, F. de Boer, R. Hahnle, E. Johnsen, and C. Laneve] "Engineering virtualized services." In Proceedings of the Second Nordic Symposium on Cloud Computing & Internet Technologies, pp. 59-63. ACM, 2013. [62] [B. Eckart, F. Aderholdt, J. Yoo, X. He, and S. Scott.] "A Top-Down Approach to Dynamically Tune I/O for HPC Virtualization." (2010).

[63] [Z. Gong and X. Gu] "Pac: Pattern-driven application consolidation for efficient cloud computing." In Modeling, Analysis & Simulation of Computer and Telecommunication Systems (MASCOTS), 2010 IEEE International Symposium on, pp. 24-33. IEEE, 2010.

[64] Google, http://www.google.com

[65] Apache Hadoop. http://hadoop.apache.org [66] SalesForce.com. http://www.salesforce.com

[67] Facebook. http://www.facebook.com

It will be understood that various details of the subject matter described herein may be changed without departing from the scope of the subject matter described herein. Furthermore, the foregoing description is for the purpose of illustration only, and not for the purpose of limitation.

Claims

What is claimed is:

1 . A method for providing a high-performance virtual application, the method comprising:

at a virtual application management layer executed by at least one processer:

receiving a request to instantiate or start up an instance of a virtual application comprising a plurality of virtual machines;

in response to receiving the request, making instantiation and/or startup decisions related to the instance of the virtual application based on metadata that describes resource requirements and inter-VM resource dependencies of the virtual application.

2. The method of claim 1 wherein the metadata describes at least one of:

physical hardware and/or virtual hardware requirements;

static performance specifications;

derived and/or parameterized performance specifications;

resource requirements and/or specifications derived and/or parameterized by performance specifications;

resource requirements and/or specifications derived and/or parameterized by other resource requirements and/or specifications; point-to-point and/or per-protocol network dependencies;

network bandwidth and/or latency requirements; network services requirements;

storage requirements;

application component-specific performance requirements; high-availability requirements and/or constraints; compliance requirements and/or constraints;

virtual machine requirements;

virtual machine dependencies;

virtual machine resource interference properties; the physical, logical, and/or semantic structure of the virtual application;

transitive performance and/or resource dependencies among virtual machines;

key performance indicators; and

service level agreement specifications.

The method of claim 1 where the metadata is encoded such that it is human-readable, machine-readable, or both.

The method of claim 1 wherein resource requirements and/or specifications are described or denoted as at least one of:

scalar numerical values;

vectors of numerical values;

Boolean values;

bitmasks;

zero or more elements of a set of nominal values;

an abstract, heuristic, or concrete model pertaining to how the resource is used;

strings of characters;

a distinguished null or undefined value;

functions and/or algebraic expressions;

functions and/or algebraic expressions parameterized by one or more resource requirements; and

binary data.

The method of claim 1 where instantiation and/or start-up decisions related to the instance of the virtual application are made based also on metadata that describes requirements and dependencies for other virtual applications or virtual machines that will share resources with the instance of the virtual application.

The method of claim 1 wherein making instantiation and/or start-up decisions based on metadata includes making decisions based on at least one of: which virtual machines are co-resident on the same hypervisors;

which storage systems host the application's volumes;

system-wide availability and/or redundancy requirements;

a virtual machine's sensitivities to interference from other virtual machines via directly- and/or indirectly-shared resources;

characterization or knowledge of a virtual machine's interference upon other virtual machines via directly- and/or indirectly- shared resources;

capacity of shared network services, storage, and/or database resources; and

capacity of shared graphics processing unit (GPU), cryptographic, field-programmable gate array (FPGA) units, or other hardware resources.

The method of claim 1 comprising:

receiving and/or retrieving performance data associated with the instance of the virtual application;

interpreting the performance data based on metadata associated with at least the instance to determine performance, service level agreement (SLA) compliance, and/or resource usage of the instance; and

modifying the metadata to adjust performance and/or resource usage of the virtual application.

The method of claim 7 wherein interpreting the performance data based on metadata associated with at least the instance comprises interpreting the performance data based also on metadata associated with other instances of virtual applications.

The method of claim 7 wherein receiving and/or retrieving performance data associated with the instance of the virtual application comprises receiving and/or retrieving telemetry data about the instance of the virtual application where such data comprises at least one of:

telemetry from the hypervisor or virtual machine monitor (VMM);

telemetry from at least one of the individual virtual machines comprising the virtual application;

telemetry from at least one of the application components comprising the virtual application;

telemetry from external monitoring components, devices, and/or systems;

telemetry from or derived from feedback from end-users of the virtual application;

telemetry from at least one of the resources, devices, and/or systems upon which at least one of the components comprising the virtual application has dependencies;

telemetry from other applications with which the virtual application interacts; and

online or offline archives of past telemetry data and/or aggregations, summaries, or analyses thereof.

The method of claim 7 wherein receiving and/or retrieving performance data associated with the instance of the virtual application comprises receiving and/or retrieving performance data from a resource that is used by the instance of the virtual application and/or shared among a plurality of virtual applications.

The method of claim 7 wherein modifying the metadata to adjust performance, SLA compliance, and/or resource usage of the virtual application comprises performing such modification automatically.

The method of claim 7 wherein modifying the metadata to adjust performance, SLA compliance, and/or resource usage of the virtual application is performed and/or assisted by human interaction with the system. The method of claim 7 wherein modifying the metadata to adjust performance, SLA compliance, and/or resource usage of the virtual application comprises modifying the instance of the virtual application such that it is consistent with its modified metadata.

The method of claim 7 wherein modifying the metadata to adjust performance, SLA compliance, and/or resource usage of the virtual application comprises at least one of:

modifying the metadata to adjust performance, SLA compliance, and/or resource usage of the instance of the virtual application;

modifying the metadata to adjust performance, SLA compliance, and/or resource usage of subsequent instantiations of the virtual application;

modifying the instance of the virtual application such that it is consistent with its modified metadata; and

modifying the metadata of other related virtualized applications and/or virtualized application instances whose metadata definitions are based upon, inherit from, and/or are otherwise derivatives of those of the virtual application to adjust performance, SLA compliance, and/or resource usage of instances or subsequent instantiations of such related virtualized applications.

A system for providing high-performance virtualized applications, the system comprising:

at least one processor and at least one memory;

a set of metadata stored in the at least one memory that describes resource requirements inter-virtual machine dependencies for a virtual application comprising a plurality of machines; and

a virtual application management layer executed by the at least one processor for receiving a request to instantiate and/or start-up an instance of the virtual application, and in response to receiving the request, using the set of metadata for at least the virtual application to make instantiation and/or start-up decisions. The system of claim 15 wherein the metadata describes at least one of:

physical hardware and/or virtual hardware requirements;

static performance specifications;

derived and/or parameterized performance specifications;

resource requirements and/or specifications derived and/or parameterized by performance specifications;

resource requirements and/or specifications derived and/or parameterized by other resource requirements and/or specifications; point-to-point and/or per-protocol network dependencies;

network bandwidth and/or latency requirements;

network services requirements;

storage requirements;

application component-specific performance requirements; high-availability requirements and/or constraints;

compliance requirements and/or constraints;

virtual machine requirements;

virtual machine dependencies;

virtual machine resource interference properties;

the physical, logical, and/or semantic structure of the virtual application;

transitive performance and/or resource dependencies among virtual machines;

key performance indicators; and

service level agreement specifications.

The system of claim 15 where the metadata is encoded such that it is human-readable, machine-readable, or both.

The system of claim 15 wherein resource requirements and/or specifications are described or denoted as at least one of:

scalar numerical values;

vectors of numerical values;

boolean values; bitmasks;

zero or more elements of a set of nominal values;

an abstract, heuristic, or concrete model pertaining to how the resource is used;

strings of characters;

a distinguished null or undefined value;

functions and/or algebraic expressions;

functions and/or algebraic expressions parameterized by one or more resource requirements; and

binary data.

The system of claim 15 wherein the management layer makes instantiation and/or start-up decisions related to the instance of the virtual application based also on metadata that describes requirements and/or dependencies for other virtual applications or virtual machines that will share resources with the instance of the virtual application.

The system of claim 15 wherein the management layer makes instantiation and/or start-up decisions related to the instance of the virtual application based on at least one of:

which virtual machines are co-resident on the same hypervisors;

which storage systems host the application's volumes;

system-wide availability and/or redundancy requirements; a virtual machine's sensitivities to interference from other virtual machines via directly- and/or indirectly-shared resources;

characterization or knowledge of a virtual machine's interference upon other virtual machines via directly- and/or indirectly- shared resources;

capacity of shared network services, storage, and/or database resources; and

capacity of shared GPU, cryptographic, field-programmable gate array (FPGA) units, or other hardware resources. The system of claim 15 wherein the management layer receives and/or retrieves performance data associated with at least the instance of the virtual application, interprets the performance data based on metadata associated with the instance to determine performance, service level agreement (SLA) compliance, and/or resource usage of the instance, and modifies the metadata to adjust performance, SLA compliance, and/or resource usage of the virtual application.

The system of claim 15 wherein the management layer interprets the performance data based also on metadata associated with other instances of virtual applications.

The system of claim 21 wherein receiving or retrieving performance data comprises receiving or retrieving telemetry data about at least the instance of the virtual application where such data comprises at least one of:

telemetry from the hypervisor or virtual machine monitor (VMM);

telemetry from at least one of the individual virtual machines comprising the virtual application;

telemetry from at least one of the application components comprising the virtual application;

telemetry from external monitoring components, devices, and/or systems;

telemetry from or derived from feedback from end-users of the virtual application;

telemetry from at least one of the resources, devices, and/or systems upon which at least one of the components comprising the virtual application has dependencies;

telemetry from other applications with which the virtual application interacts; and

online or offline archives of past telemetry data and/or aggregations, summarizations, or analyses thereof. The system of claim 21 wherein receiving and/or retrieving performance data comprises receiving and/or retrieving performance data from a resource that is used by the instance of the virtual application and/or that is shared among a plurality of virtual applications.

The system of claim 21 wherein modifying the metadata to adjust performance, SLA compliance, and/or resource usage of the virtual application comprises performing such modification automatically.

The system of claim 21 wherein modifying the metadata to adjust performance, SLA compliance, and/or resource usage of the virtual application is performed and/or assisted by human interaction with the system.

The system of claim 21 wherein modifying the metadata to adjust performance, SLA compliance, and/or resource usage of the virtual application comprises at least one of:

modifying the metadata to adjust performance, SLA compliance, and/or resource usage of the instance of the virtual application;

modifying the metadata to adjust performance, SLA compliance, and/or resource usage of subsequent instantiations of the virtual application;

modifying the instance of the virtual application such that it is consistent with its modified metadata; and

modifying the metadata of other related virtualized applications and/or virtualized application instances whose metadata definitions are based upon, inherit from, and/or are otherwise derivatives of that of the virtual application "A" to adjust performance, SLA compliance, and/or resource usage of instances or subsequent instantiations of such related virtualized applications.

28. A non-transitory computer readable medium having stored thereon executable instructions that when executed by the processor of a computer control the computer to perform steps comprising:

receiving a request to instantiate and/or start-up an instance of a virtual application comprising at least one virtual machine;

in response to receiving the request, making instantiation and/or start-up decisions related to the instance of the virtual application based on metadata that describes requirements and/or dependencies of the virtual application. 29. A method of characterizing resource requirements for virtualized applications or components thereof, the method comprising:

characterizing resource requirements based on the effects of interactions among a plurality of components.

30. The method of claim 29 comprising characterizing resource requirements based on the effects of interactions among a plurality of components that use a shared resource.

31. The method of claim 30 wherein at least one of the plurality of components comprises a virtual machine.

32. The method of claim 30 wherein the shared resource would not be shared in the absence of virtualization.

33. The method of claim 29 wherein the effects of interactions are due, in whole or in part, to the behavior of the Virtual Machine Monitors (VMMs) or hypervisors on which the virtual machine (VM) components of the application are executed. 34. The method of claim 29 wherein the resource requirements are quantified and/or specified using at least one of:

scalar numerical quantities;

vectors of numerical values;

Boolean values;

bitmasks; zero or more elements of a set of nominal values;

an abstract, heuristic, or concrete model pertaining to how the resource is used;

strings of characters;

a distinguished null or undefined value;

functions and/or algebraic expressions;

functions and/or algebraic expressions parameterized by one or more resource requirements; and

binary data.

The method of claim 29 wherein characterizing resource requirements comprises characterization, quantification, and/or specification of the resource capacities of at least one of: hardware, hypervisors, devices, software systems, network services, and/or other extra-application dependencies.

The method of claim 35 wherein the characterization, quantification, and/or specification comprises consideration of capacities arising from and/or due to the resource being shared.

The method of claim 35 wherein the resource whose capacity is being characterized, quantified, and/or specified is shared among at least one of:

individual virtual machines (VMs);

virtual machines that are components of virtualized applications; and

devices, resources, services, or other entities placing demand upon the resource in question.

The method of claim 35 wherein such quantification or specification of capacity uses at least one of:

scalar numerical quantities;

vectors of numerical values;

Boolean values;

bitmasks; zero or more elements of a set of nominal values;

an abstract, heuristic, or concrete model pertaining to how the resource is used;

strings of characters;

a distinguished null or undefined value;

functions and/or algebraic expressions;

functions and/or algebraic expressions parameterized by one or more resource requirements; and

binary data.

The method of claim 29 wherein the generation and/or computation such characterization, quantification, and or specification is automated.

The method of claim 29 further comprising implementing the storage and retrieval of the concrete characterizations, quantifications, and or specifications of such capacities.

The method of claim 40 wherein the storage and retrieval is implemented using at least one of:

a relational and/or non-relational database;

a shared file system;

shared storage;

remote database access;

web services;

RESTful web services;

SOAP web services;

CORBA interfaces;

RPC mechanisms;

network services;

messaging busses and/or enterprise service busses (ESBs); and

mechanisms following publish-subscribe and/or notify- subscribe semantics.

42. The method or claim 35 comprising using the values of the capacities for determining and/or optimizing placement, provisioning, and/or configuration of virtual machines and/or components of virtualized applications. 43. A system for characterizing resource requirements for virtualized applications or components thereof, the system comprising:

a plurality of virtualized applications, each having at least one component; and

a characterization module comprising hardware and for characterizing resource requirements based on the effects of interactions among the plurality of components.

44. The system of claim 43 wherein the characterization module characterizes resource requirements based on the effects of interactions among a plurality of components that use a shared resource.

45. The system of claim 44 wherein at least one of the plurality of components comprises a virtual machine.

46. The system of claim 44 wherein the shared resource would not be shared in the absence of virtualization. 47. The system of claim 43 wherein the effects of interactions are due, in whole or in part, to the behavior of the Virtual Machine Monitors (VMMs) or hypervisors on which the virtual machine (VM) components of the application are executed.

48. The system of claim 43 wherein the resource requirements are quantified and/or specified using at least one of:

scalar numerical quantities;

vectors of numerical values;

Boolean values;

bitmasks;

zero or more elements of a set of nominal values; an abstract, heuristic, or concrete model pertaining to how the resource is used;

strings of characters;

a distinguished null or undefined value;

functions and/or algebraic expressions;

functions and/or algebraic expressions parameterized by one or more resource requirements; and

binary data.

The system of claim 43 wherein characterizing resource requirements comprises characterization, quantification, and/or specification of the resource capacities of at least one of: hardware, hypervisors, devices, software systems, network services, and/or other extra-application dependencies.

The system of claim 49 wherein the characterization, quantification, and/or specification comprises consideration of capacities arising from and/or due to the resource being shared.

The system of claim 49 wherein the resource whose capacity is being characterized, quantified, and/or specified is shared among at least one of:

individual virtual machines (VMs);

virtual machines that are components of virtualized applications; and

devices, resources, services, or other entities placing demand upon the resource in question.

The system of claim 49 wherein such quantification or specification of capacity uses at least one of:

scalar numerical quantities;

vectors of numerical values;

Boolean values;

bitmasks;

zero or more elements of a set of nominal values; an abstract, heuristic, or concrete model pertaining to how the resource is used;

strings of characters;

a distinguished null or undefined value;

functions and/or algebraic expressions;

functions and/or algebraic expressions parameterized by one or more resource requirements; and

binary data.

The system of claim 43 wherein the generation and/or computation such characterization, quantification, and or specification is automated.

The system of claim 43 wherein the characterization module implements the storage and retrieval of the concrete characterizations, quantifications, and or specifications of such capacities.

The system of claim 54 wherein the storage and retrieval is implemented using at least one of:

a relational and/or non-relational database;

a shared file system;

shared storage;

remote database access;

web services;

RESTful web services;

SOAP web services;

CORBA interfaces;

RPC mechanisms;

network services;

messaging busses and/or enterprise service busses (ESBs); and

mechanisms following publish-subscribe and/or notify- subscribe semantics. The system of claim 49 wherein the characterization module uses the values of the capacities for determining and/or optimizing placement, provisioning, and/or configuration of virtual machines and/or components of virtualized applications.