WO2017025781A1

WO2017025781A1 - Global data center energy management

Info

Publication number: WO2017025781A1
Application number: PCT/IB2015/056183
Authority: WO
Inventors: Claude Gauthier; Yves Lemieux
Original assignee: Telefonaktiebolaget Lm Ericsson (Publ)
Priority date: 2015-08-13
Filing date: 2015-08-13
Publication date: 2017-02-16

Abstract

An energy management method and entity for optimizing energy consumption by applications in a data center. The method includes obtaining information, where the information includes calculated energy utilization for at least one application within a data center, the calculated energy consumption based on at least one trigger factor. The method further includes identifying an energy optimization opportunity for at least one of the applications based on at least the obtained information, and validating the energy optimization opportunity for at least one of the applications based at least in part on energy optimization for the data center.

Description

GLOBAL DATA CENTER ENERGY MANAGEMENT

TECHNICAL FIELD

The present disclosure relates to a method and entity for energy management, and in particular a method and entity for managing energy consumption by applications in a data center.

BACKGROUND

Cloud computing consists of transitioning computer services to offsite locations available on the Internet. The computers making up the cloud system can be virtualized in order to maximize the resources of the available physical computers. In today's cloud computing industry, cloud-based models are being used as high economic opportunities to host public and private services by shifting applications to the cloud. However, managing and optimizing energy consumption and providing a green sustainable infrastructure is one of the major challenges facing data centers. Monitoring the energy consumption of each server in the data center in real time is a challenge. Typically, using power models, the master server periodically computes energy, related consumption data and quantifies energy saving actions where sustainable energy sources can be introduced into the power input of the data center. This is typically done as a macro view on the overall data center.

Based on the different energy and environmental metrics, energy issues can be remediated by generating service requests to take manual action for resolving issues such as shutting down assets, or making changes to the cooling system of the data center. However, changes to the data center such as these may not result in overall optimal energy utilization globally or at the data center on a per application basis. SUMMARY

The present disclosure advantageously provides a method and entity for optimizing the energy consumed in a data center based upon identified energy optimization opportunities at the application level for applications running on servers at the data center. An energy optimization opportunity might be, for example, to relocate an application to a different server at the data center or even to a different data center that is located in a more geographically favorable location. The decision to relocate the application can be based on climate, emergency situations or other factors. The target data center could be found in in other geographic locations such as other cities or countries having more optimal data sustainability. The relocation of the applications is transparent to the end-users requiring the services.

According to one broad aspect of the disclosure, a data center energy management method is provided. The method includes obtaining information, the information including calculated energy utilization for at least one application within a data center, the calculated energy consumption based on at least one trigger factor, identifying an energy optimization opportunity for at least one of the at least one application based on at least the obtained information, and validating the energy optimization opportunity for at least one of the at least one application based at least in part on energy optimization for the data center.

According to another embodiment of this aspect, the method further includes reporting the validated energy optimization opportunity for the at least one of the at least one application to a data center management entity. According to another embodiment of this aspect, information is received from the data center management entity.

According to another embodiment of this aspect, the at least one trigger factor includes at least one of a power usage effectiveness factor, a carbon usage effectiveness factor, a water usage effectiveness factor, weather information, and disaster recovery information.

According to another embodiment of this aspect, the identified energy optimization opportunity includes moving at least one of the at least one application to an alternate data center located at a different geographic region from the data center. According to another embodiment of this aspect, moving the at least one of the at least one application to the alternate data center is transparent to a user of the at least one application.

According to another embodiment of this aspect, the method further includes obtaining subsequent information, the subsequent information including energy utilization updates for the at least one application, the energy utilization updates based at least upon the reported validated energy optimization opportunity. According to another embodiment of this aspect, the subsequent information is obtained after a predetermined optimization period. According to another embodiment of this aspect, the information is obtained from a data repository shared with a data center management entity.

According to another broad aspect of the disclosure, an energy management entity is provided. The energy management entity includes a communication interface configured to obtain information. The information includes calculated energy utilization for at least one application within a data center where the calculated energy consumption is based on at least one trigger factor. The energy management entity also includes a processor, and a memory storing instructions that, when executed, configure the processor to identify an energy optimization opportunity for at least one of the at least one application based on at least the obtained information, and validate the energy optimization opportunity for at least one of the at least one application based at least in part on energy optimization for the data center.

According to another embodiment of this aspect, the communication interface is further configured to report the validated energy optimization opportunity for the at least one of the at least one application to a data center management entity.

According to another embodiment of this aspect, the communication interface receives the information from the data center management entity.

According to another embodiment of this aspect, the at least one trigger factor includes a power usage effectiveness factor, a carbon usage effectiveness factor, a water usage effectiveness factor, weather information, and disaster recovery information.

According to another embodiment of this aspect, the identified energy optimization opportunity includes moving at least one of the at least one application to an alternate data center located at a different geographic region than the data center. According to another embodiment of this aspect, moving the at least one of the at least one application to the alternate data center is transparent to a user of the at least one application.

According to another embodiment of this aspect, the communication interface further configured to obtain subsequent information, the subsequent information including energy utilization updates for the at least one application, the energy utilization updates based at least upon the reported validated energy optimization opportunity. According to another embodiment of this aspect, the subsequent information is obtained after a predetermined optimization period. According to another embodiment of this aspect, the information is obtained from a data repository shared with a data center management entity.

According to another broad aspect of the disclosure, an energy management system is provided. The energy management system includes a database configured to store information, where the information includes calculated energy utilization for at least one application within a data center, the calculated energy consumption based on at least one trigger factor. The energy management system further includes an application level energy manager configured to identify an energy optimization opportunity for at least one of the at least one application based on at least the stored information. The energy management system also includes an optimization controller configured to, upon an optimization request from the application level energy manager, run an optimization model and provide operational data based on the optimization model to the application level energy manager. The application level energy manager is configured to validate the energy optimization opportunity for at least one of the at least one application based at least in part on the operational data.

According to another embodiment of this aspect, the application level energy manager is further configured to report the validated energy optimization opportunity for the at least one of the at least one application based at least in part on energy optimization for a data center.

According to another embodiment of this aspect, the identified energy optimization opportunity includes moving at least one of the at least one application to an alternate data center located at a different geographic region than the data center. According to another embodiment of this aspect, moving the at least one of the at least one application to the alternate data center is transparent to a user of the at least one application. According to another embodiment of this aspect, the database is further configured to store subsequent information, the subsequent information including energy utilization updates for the at least one application, the energy utilization updates based at least upon the validated energy optimization opportunity. According to another embodiment of this aspect, the subsequent information is stored after a predetermined optimization period.

According to another broad aspect of the disclosure, an energy management entity is provided. The energy management entity includes a communication module configured to obtain information, where the information includes calculated energy utilization for at least one application within a data center, the calculated energy consumption based on at least one trigger factor. The energy management entity further includes an energy optimization module configured to identify an energy optimization opportunity for at least one of the at least one application based on at least the obtained information. The energy management entity also includes an energy validation module configured to validate the energy optimization opportunity for at least one of the at least one application based at least in part on energy optimization for the data center.

BRIEF DESCRIPTION OF THE DRAWINGS

A more complete understanding of the present disclosure, and the attendant advantages and features thereof, will be more readily understood by reference to the following detailed description when considered in conjunction with the accompanying drawings wherein:

FIG. 1 is a block diagram of an exemplary system for managing and optimizing energy utilization in a data center in accordance with principles of the present disclosure;

FIG. 2 is a block diagram of an alternate exemplary system for managing and optimizing energy utilization in a data center in accordance with principles of the present disclosure;

FIG. 3 is a block diagram of an exemplary data center energy management entity in accordance with principles of the present disclosure; FIG. 4 is a block diagram of an alternate data center energy management entity in accordance with principles of the present disclosure;

FIG. 5 is a flow diagram illustrating the interaction between the data center energy management entity and the data center energy management entity in accordance with principles of the present disclosure;

FIG. 6 is a flow diagram illustrating the interaction between components of the data center energy management entity in accordance with principles of the present disclosure;

FIG. 7 is a flow diagram illustrating application of the present disclosure to a particular use case scenario in accordance with principles of the present disclosure;

FIG. 8 is a flow diagram illustrating application of the present disclosure to an alternate use case scenario in accordance with principles of the present disclosure;

FIG. 9 is a flow diagram illustrating application of the present disclosure to yet another use case scenario in accordance with principles of the present disclosure;

FIG. 10 is a flow diagram illustrating an exemplary method for managing and optimization energy utilization in a data center in accordance with principles of the present disclosure; and

FIG. 11 is a block diagram of yet another exemplary data center energy management entity in accordance with principles of the present disclosure.

DETAILED DESCRIPTION

The method and system described herein advantageously provide energy utilization and management of applications within a data center. By considering triggers parameters that may indicate underutilization usage of power, elevated carbon dioxide emissions, and/or excessive water usage, often due to high temperature conditions, streamlined energy opportunities for the data center can be identified. For example, upon indication of an application having one or more sub-optimal conditions, a data center management entity can recommend that certain actions be taken. The recommendations can include suggested load balancing, hard switching to duplicate server blades, migration of certain applications to other data centers located in more favorable weather regions, server shutdowns, and/or spatial volume resiliency suggestions. Further, disaster recovery notifications indicating the occurrence or imminent occurrence of a natural disaster, i.e., a hurricane, can be received by the data center management entity, forwarded to the energy management entity and used by the energy management entity to provide actions to be taken with respect to applications residing on servers at the data center. These actions can be transparent to the application user and result in minimal operational cost impact. Advantageously, the present disclosure identifies energy optimization opportunities at the application level, and then validates the opportunities with the data center to assure that any modifications or actions taken with respect to individual applications result in an overall energy optimization for the data center.

A benefit of the methods and system described herein is that current data center implementations can be provided with real time energy optimization opportunities with regard to applications running on servers at the data center as well as a real time assessment of the overall impact the identified opportunities will have on the data center. Data centers are constantly being scaled and it becomes necessary to be aware of conditions such as application consumption, application CO2 emissions, application patterns, and application movements in order to provide for an overall optimization strategy for the data center. The present disclosure

advantageously provides an energy management entity that works together with a data center management entity and considers trigger factors such as power usage, carbon usage, water usage, as well as real time indicators of temperature fluctuations and notifications of disasters or imminent disasters. Based on some or all of these trigger factors, services at the data center could automatically be shut down or moved, in a transparent and validated manner without end users being aware or being affected by the migration of services.

Before describing in detail exemplary embodiments that are in accordance with the disclosure, it is noted that the embodiments reside primarily in combinations of apparatus components and processing steps related to providing higher order modulation determination for wireless devices. Accordingly, components have been represented where appropriate by conventional symbols in drawings, showing only those specific details that are pertinent to understanding the embodiments of the disclosure so as not to obscure the disclosure with details that will be readily apparent to those of ordinary skill in the art having the benefit of the description herein. As used herein, relational terms, such as "first," "second," "top" and "bottom," and the like, may be used solely to distinguish one entity or element from another entity or element without necessarily requiring or implying any physical or logical relationship or order between such entities or elements. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the concepts described herein. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises," "comprising," "includes" and/or "including" when used herein, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. In addition, the terms "class" and "category" are used interchangeably herein as well as the terms

"classifying" and "categorizing."

Unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It will be further understood that terms used herein should be interpreted as having a meaning that is consistent with their meaning in the context of this specification and the relevant art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

In embodiments described herein, the joining term, "in communication with" and the like, may be used to indicate electrical or data communication, which may be accomplished by physical contact, induction, electromagnetic radiation, radio signaling, infrared signaling or optical signaling, for example. One having ordinary skill in the art will appreciate that multiple components may interoperate and modifications and variations are possible of achieving the electrical and data communication.

Referring now to drawing figures in which like reference designators refer to like elements there is shown in FIG. 1 an exemplary system for optimization of energy utilization in a data center designated generally as "10". System 10 includes a data center management entity ("DCME") 12 that monitors overall energy consumption and utilization in data center 14. Data center 14 may include multiple servers 15a, 15b, and 15c (referred to collectively herein as "servers 15") running various applications and services. Although three servers 15a, 15b and 15c are shown, the disclosure is not limited to three servers. System 10 also includes a database (DB) 16, and an energy management entity ("EME") 18. DCME 12 monitors data center 14 with regard to its physical, asset, power and downtime capabilities, in addition to other factors like carbon and water usage. It is within the scope of the present disclosure that DB 16 be considered any type of database including but not limited to any combination of one or more of a relational database, an operational database, or a distributed database.

Data centers 14 strive for optimal energy utilization. Ideally, a hypothetically efficient data center is one where energy is used exclusively to power information technology (IT). An index known as Power Usage Effectiveness ("PUE") 20 can be used to measure energy efficiency. A PUE index 20 of 1.0 represents an ideal data center where no energy is lost to the surrounding elements. For example, if a data center is located in a climate zone where the temperature is exceedingly hot for extended periods of time, the data center's PUE index 20 will be affected due to the cooling required to maintain servers and other hardware components at an optimal temperature. This will increase the PUE index 20 above 1.0 (e.g., an increase to 1.20 for 20% more than the ideal temperature), and in some cases significantly above 1.0.

An index used to measure carbon utilization at data center 14 is Carbon Usage

Effectiveness ("CUE") 22. CUE index 22 measures the green footprint of data center 14 by calculating the total carbon dioxide emission equivalent (CCheq) of the facility's energy consumption. A perfect CUE index 22 of 0.0 means that no carbon use is associated with the data center's operations. This index may be measured in kilograms of carbon dioxide per kilowatt-hour.

Another index used in the data center is Water Usage

Effectiveness ("WUE") 24 where the annual site water usage (for example, in liters) is dictated by the Information Technology (IT) equipment energy usage (for example, in Kwh). Water usage includes water used for cooling, regulating humidity and producing electricity on-site. The PUE index 20, CUE index 22, and WUE index 24 are affected by the environment surrounding the data center. For example, as the temperature outside the data center increases, the PUE index 20, the CUE index 22, and the WUE index 24 are affected accordingly. Thus, the PUE 20, CUE 22, and WUE 24 indices and their values with respect to a baseline or expected value for a particular geographic region represent "triggers," each or some of which may be used by EME 18 to identify energy optimization opportunities for at least one of the applications at data center 14. Although only these three indices are discussed herein, data center 14 may utilize other indices for monitoring environmental conditions and energy consumption and to trigger energy optimization actions.

DCME 12 also receives weather information from, for example, weather satellite 26. In one embodiment, weather satellite 26 can provide real time global weather information to DCME 12 such as information identifying weather trends, for example, heat extremes. Because high-temperature regions may adversely affect the performance of servers 15 and applications and services running on servers 15, up-to- date weather information containing prediction and trends represents another "trigger" which, as explained in greater detail below, can be used by EME 18 to identify energy optimization opportunities for applications running on servers 15 at data center 14.

DCME 12 can also receive disaster recovery notifications 28. In one embodiment, these notifications could be instances of events such as, for example, floods, hurricanes, terrorist attacks, or other events that might cause servers to be shut down or be adversely affected. These notifications 28 also represent "triggers" that, along with the PUE 20, CUE 22, and WUE 24 indices, and information from weather satellite 26 can be used by EME 18, in any combination or weighted fashion, to trigger, i.e., identify, energy optimization opportunities for one or many application running on servers 15 at data center 14. Thus, the trigger factor may include at least one of PUE index 20, CUE index 22, WUE index 24, weather information and disaster recovery information 28. The present disclosure is not limited to the specific triggers identified above. Thus, other factors, notifications, conditions, and/or indices may be used as triggers, and these triggers used in any combination or weighted manner to provide energy optimization opportunities.

EME 18 provides a "micro" view of the energy utilized by applications running on servers 15 at data center 14. The optimization opportunities identified by EME 18 can be used in conjunction with the overall "macro" view of data center 14 provided by DCME 12 to provide an overall optimization strategy that may result in, for example, automatically shutting down or moving some services to other servers or data centers situated in different geographic locations, transparently to the users, without affecting the operation of the services. Thus, advantageously, EME 18 can both identify energy optimization opportunities for individual applications at data center 14 and validate the energy optimization opportunities for the applications by assuring that the application-level energy optimization recommendations provide an overall energy optimization benefit for data center 14.

Thus, referring again to FIG. 1 , EME 18 calculates information, or obtains the information, either by obtaining the information from DB 16 or by receiving the information from DCME 12, where the information includes calculated energy utilization for at least one application within data center 14, the calculated energy consumption being based on at least one trigger factor. In one embodiment, the information is obtained from a database, e.g., DB 16 that is shared with DCME 12. It should be noted that although FIG. 1 shows EME 18 and DCME 12 as two separate entities, it is within the scope of the present disclosure to provide a single entity that performs the functions of both EME 18 and DCME 12. Further, the single entity may also include DB 16. In another embodiment, DCME 12 and EME 18 may themselves be distributed among other entities and/or locations. In another embodiment, EME 18 calculates the utilization for at least one application. An application can comprise a plurality of components, or sub-components, including virtual machines hosted by underlying physical hardware (e.g. servers 15). EME 18 can identify servers 15 hosting the components of an application and measure (or receive measurements of) the energy utilization of the identified servers 15. These measurements can be used to calculate the energy utilization on a per application basis.

Based on this received or obtained, or calculated information, EME 18 identifies an energy optimization opportunity for at least one of the applications in data center 14 based on at least the obtained information. An energy optimization opportunity could be, for example, a determination, based at least on the trigger factors described above, that certain applications or services, e.g., cloud services, or virtual machines, running at data center 14 may need to be moved to different servers 15, and/or different types of servers 15 within data center 14 or to an alternate data center located in a different geographic region than data center 14, i.e., in more weather-friendly climates. In one embodiment, moving at least one application to an alternate data center is transparent to the user of the at least one application. Another energy optimization opportunity might be to shut down one or more overheated or over-used servers to conserve energy. Another opportunity might be to shut down servers that are being underutilized and migrate the applications on that server, e.g., server 15a, to other servers, e.g., server 15b and/or server 15c, or even one or more servers 15 at another physical location.

EME 18 then validates the energy optimization opportunity for at least one of the applications based at least in part on energy optimization for the data center. Validation includes confirming that the actions with respect to individual applications provide an overall energy optimization benefit to the entire data center 14. In other words, a suggestion to move a particular application to a server in another geographic region may have an adverse effect on the overall data center performance. Thus, such a recommended action would not be validated since EME 18 would balance the efficiency of the recommended action on the application level with the impact the recommended action would have to the overall data center 14 and determine that such an action should not be validated. In one embodiment, for energy optimization opportunities that are validated, EME 18 reports the opportunity validated energy optimization opportunity for at least one of the applications to, for example, DCME 12.

Thus, EME 18 works in collaboration with DCME 12 to provide sustainable actions that benefit the overall energy optimization at data center 14. Based upon information either calculated by EME 18, or obtained from either from DB 16 or DCME 12, EME 18 measures the utilization and power consumed per application running on for example, a server, a cloud system or virtual machine and determines which, if any applications need to be relocated. This data can be extrapolated to hundreds or thousands of servers 15 and leveraged further by recommending to DCME 12 that these applications automatically be moved to other servers or other data centers.

In one embodiment, EME 18 obtains the information from DB 16 or DCME

12, the information including data relating to the various triggers discussed above, and compares this data to a policy engine. This results in a "micro assessment" of each of the various server blades, virtual machines, and applications, etc., at data center 14 and enables EME 18 to determine a real time energy footprint for each application. EME 18 can then identify energy utilization opportunities. In one embodiment, the identified energy optimization opportunity includes moving at least one of the applications to an alternate data center located at a different geographic region than data center 14. In other embodiments, EME 18 may provide an energy optimization recommendation to DCME 12 or another managerial entity associated with data center 14 that includes a migration of workload, i.e., moving applications and/or virtual machines towards a more optimal server within data center 14. Other recommended actions might include, for example, shutting down dormant or unused components, lowering central processing unit (CPU) frequency, decreasing CPU voltage, or prioritizing newer or lower-power consumption blades. In one embodiment, EME 18, within a resolution time established by the frequency of the incoming notifications to DCME 12, requests from DCME 12 a full status of each running application at data center 14 at a per-virtual machine basis. The virtual machines and/or applications affected by the various triggers are selected and prioritized for possible workload migration and/or resource enabling and/or disabling.

FIG. 2 illustrates an alternate embodiment of system 10. In this embodiment, energy management system 13 combines the functions of DCME 12, EME 18, and DB 16 into one entity. Thus, DCME 12 and DME 18 still work cooperatively, as discussed above, to provide and implement energy optimization opportunities within data center 14. However, a single energy management system 13 can perform these functions. Energy management system 13 may include a database or have access to a database which stores the trigger information obtained by DCME 12.

FIG. 3 illustrates an exemplary EME 18 in accordance with the present disclosure. EME 18 can include a communication interface 30 configured to obtain information, the information including calculated energy utilization for at least one application within data center 14, the calculated energy consumption based on at least one trigger factor. In one embodiment, the information is obtained from a data repository, such as, for example, DB 16, shared with DCME 12.

EME 18 also includes circuitry having memory 32 and a processor 34 where memory 32 stores instructions that, when executed, configure the processor to identify an energy optimization opportunity for at least one of the at least one application based on at least the obtained information, and validate the energy optimization opportunity for at least one of the at least one application based at least in part on energy optimization for data center 14. Memory 32 may be any volatile or non- volatile storage device capable of storing data including, for example, solid-state memory, optical storage and magnetic storage.

In another example, processor 34 may include one or more hardware components such as application specific integrated circuits (ASICs) that provide some or all of the functionality described above. In another embodiment, processor 34 may include one or more hardware components, e.g., Central Processing Units (CPUs), and some or all of the functionality described above is implemented in software stored in, e.g., the memory 32 and executed by the processor 34. In yet another

embodiment, the processor 34 and memory 32 form processing means (not shown) configured to perform the functionality described herein.

In one embodiment, communication interface 30 is further configured to report the validated energy optimization opportunity for the at least one of the at least one application to DCME 12. In another embodiment, communication interface 30 receives the information from DCME 12. In another embodiment, the at least one trigger factor includes PUE index 20, CUE index 22, WUE index 24, weather information obtained, for example, from weather satellite 26, and disaster recovery information 28. In one embodiment, the identified energy optimization opportunity includes moving at least one of the at least one application to an alternate data center located at a different geographic region than data center 14. In one embodiment, moving the at least one of the at least one application to the alternate data center is transparent to a user of the at least one application. In one embodiment,

communication interface 30 is further configured to obtain subsequent information, the subsequent information including energy utilization updates for the at least one application, the energy utilization updates based at least upon the reported validated energy optimization opportunity. In one embodiment, the subsequent information is obtained after a predetermined optimization period.

FIG. 4 illustrates an alternate embodiment of EME 18. As described in greater detail below and shown in FIG. 4, EME 18 includes DB 16 which is configured to store information the information including calculated energy utilization for at least one application within data center 14, the calculated energy consumption based at least on one trigger factor, as discussed above. Application level energy manager 36 includes memory 32 and processor 34 where memory 32 stores instructions that, when executed, configure the processor to identify an energy optimization opportunity for at least one of the applications run on servers 15 at data center 14 based on at least the stored information in DB 16. In other words, application level energy manager accesses this information and compares it to an associated power table (P-table) containing baseline value to determine if certain actions need to be taken with regard to particular applications or hardware at data center 14. Based upon the result of the comparison of the information in DB 16 with the associated power table, application energy manager 36 can request that an optimization controller 38 run an optimization model based on various factors in order to calculate operational data based on the optimization model. Thus, optimization controller 38 is configured to, upon an optimization request from application level energy manager 36, run an optimization model and provide operational data based on the optimization model to application level energy manager 36. This operational data is sent back to application level energy manager 36. The factors used to calculate the operational data could be, for example, the total energy consumption, CO2 footprint, and life cycle assessment (LCA) source mix at data center 14. Based upon the received operational data, application level energy manager 36 is configured to validate the energy optimization opportunity for at least one of the applications at data center 14 based at least in part on the operational data.

In one embodiment, application level energy manager 36 is further configured to report the validated energy optimization opportunity for the at least one of the at least one application based at least in part on energy optimization for data center 14. In one embodiment, the at least one trigger factor includes PUE index 20, CUE index 22, WUE index 24, weather information obtained, for example, from weather satellite 26, and disaster recovery information 28. In one embodiment, the identified energy optimization opportunity includes moving at least one of the at least one application to an alternate data center located at a different geographic region than data center 14. In one embodiment, moving the at least one of the at least one application to the alternate data center is transparent to a user of the at least one application. In one embodiment, DB 16 is further configured to store subsequent information, the subsequent information including energy utilization updates for the at least one application, the energy utilization updates based at least upon the validated energy optimization opportunity. In one embodiment, the subsequent information is stored after a predetermined optimization period.

FIG. 5 illustrates a flowchart showing an exemplary messaging sequence between DCME 12 and EME 18 in an embodiment of the present disclosure. DCME 12 interacts with and receives data from temperature probes indicative of temperature fluctuations (S100). DCME 12 also receives notifications of disaster recovery events (SI 10) and real time weather forecasts from weather satellites (S120). The sequence in FIG. 5 is merely exemplary and DCME 12 can interact with more or fewer outside sources in order to obtain triggers that can be used by DCME 12 and EME 18 to identify energy optimization opportunities for data center 14. Further, the order shown in FIG. 5 regarding interaction with outside sources may vary and is not limited solely to the order shown in FIG. 5. For example, information from weather satellites may be obtained prior to information from the temperature probes and disaster recovery notifications. The triggers provide indication of real time temperature, weather, and other events. The trigger information can be used to formulate indices, as discussed above, which provide a current view of such factors as power, water, and carbon utilization at data center 14.

Based on the received trigger information, DCME 12 may create different trigger handshakes for different uses (S130) and the trigger information is provided to EME 18, or to DB 16 and accessed by EME 18, via the appropriate trigger information (S140). As discussed above, the trigger information may include various indices, such as PUE index 20, CUE index 22, WUE 24, as well as disaster recovery indicators 28 and information received from weather satellites 26. Included in the trigger information is real-time performance and energy consumption data based on virtual machine consumption. This data is used by EME 18 to determine utilization and consumption patterns and to identify energy optimization opportunities for components at data center 14. Note that in FIG. 5, DB 16 is combined with EME 18. However, it is within the scope of this disclosure to provide a database that is separate from EME 18, where EME 18 can access the information sent by DCME 12 to DB 16 (see, for example, FIG. 1).

EME 18 uses the data sent by DCME 12 to run one or more optimization algorithms (SI 50) that identify energy optimization opportunities for applications and services at data center 14. This can be done by comparing the calculated utilization patterns with other available hardware profiles. The hardware profiles are dynamic and may be stored in a logical database such as DB 16. Based at least upon the optimization algorithms run by EME 18, an energy optimization opportunity for the applications are identified. EME 18 validates the identified optimization

opportunities for the individual applications and services at data center 14 by considering the overall impact its recommended action will have to other resources at data center 14. This may include considering global data for data center 14 such as PUE index 20, CUE index 22, WUE index 24, energy source CO2 coefficients, cooling parameters as well as other factors. In other words, EME 18 considers whether the recommended energy optimization opportunity for a particular application improves the overall current energy consumption optimization at data center 14. EME 18 reports the validated energy optimization opportunity for at least one of the applications for data center 14 to DCME 12 (S160). These opportunities may include suggested migration of applications or virtual machines, load balancing, hard switching to a duplicate server blade, shutdown of servers etc. DCME 12 may use historical data, and other information such as whether there are servers in data center 14 that are being underutilized, and generate behavioral predictions regarding future energy demand at data center 14 (SI 70). Based on the energy optimization opportunities identified by EME 18 and actions taken by DCME 12, new input behaviors are sent to EME 18 (SI 80). Thus, based at least on the reported validated energy optimization opportunities provided by EME 18, DCME 12 sends subsequent periodic reports to EME 18. These reports contain subsequent information, where the subsequent information includes energy utilization updates for the applications at data center 14, where the energy utilization updates are based at least upon the reported validated energy optimization opportunities identified by EME 18. These updates may include, for example, updated status of server shutdowns, increased usage, or migration of services. Thus, EME 18 obtains or receives real time energy updates and utilizes this information to identify future optimization opportunities. The entire process of FIG. 5, including the sending of subsequent information containing updated behaviors to EME 18 can be repeated after an optimization period. Thus, EME 18 can obtain or receive subsequent information after a predetermined optimization period. The optimization period can depend on the time needed for data center 14 to incorporate the recommended optimization option, i.e., migration of services, off-loading of workloads, lowering of CPU speed, or shut-down of servers and for the system to stabilize after the recommended optimization actions have occurred.

FIG. 6 illustrates a flowchart showing an exemplary message exchange by components in EME 18 to identify energy optimization opportunities based on the trigger information that EME 18 either obtained from DCME 12 directly or accesses from DB 16. An application level energy manager 36 in EME 18 reads trigger information forwarded by DCME 12 from DB 16 (S190). Application level energy manager 36 reads the trigger data and associated P-table entries. Depending on the comparison of the trigger information with information in the associated P-table, application level energy manager 36 sends a request to an optimization controller 38 in order to obtain the required operational data that application level energy manager 36 needs in order to identify energy optimization opportunities and to validate the energy optimization opportunity to assure that any actions taken by DCME 12 does not compromise the overall optimal energy consumption at data center 14 (S200). The operational data may include, for example, energy consumption, CPU frequency operation points, carbon dioxide equivalent (CC e) footprint, LCA power source mix, and resulting energy profit. Optimization controller 38 runs the optimization model (S210) and forwards the required operation data to application level energy manager 36 (S220). Application level energy manager 36 validates the decision to take actions based on the identified energy optimization opportunity and forwards a validation confirmation to DCME 12 (S230).

FIGS. 7-9 illustrate various templates or handshakes for different use embodiments of the methodology of the present disclosure. For example, (in FIG. 7), DCME 12 may want to determine energy optimization for a particular application when a "hot spot" or overheated server has been identified. In another scenario (in FIG. 8), the energy optimization of a particular hardware component may be desired. In another scenario (in FIG. 9), the real time energy generation and consumption capability of a particular application, i.e., a video conference application, may be desired. In yet another scenario, DCME 12 may want to determine if a particular server 15 at data center 14 is being underutilized.

FIG. 7 illustrates a flow chart showing a use-based application of the present disclosure. In the scenario depicted in FIG. 7, a "hot spot" alarm is identified, where the hot spot may be, for example, an overheated hardware component, such as a server blade within a server cabinet. In one embodiment, in order to optimize applications running on high-temperature servers, DCME 12 identifies the hot spot trigger and automatically sends a request to EME 18 (S240) to verify if EME 18 wants to act upon the alarm. As discussed above, EME 18 runs a verification algorithm by, in certain embodiments, consulting a table of baseline application measurements and, comparing the trigger information to the baseline measurements, determines whether or not actions should be taken. These actions could include, for example, migrating certain applications at data center 14 to another location, i.e., another server 15 at data center 14 or migrating the application to a different data center 14, or, if necessary, to shut down servers at data center 14.

EME 18 may send a request to DCME 12 asking for space -power-cooling

(SPC) information on specific servers at data center 14 (S250) such that the applications currently located on the overheated server 15 can be moved to a safer location. DCME 12 determines if Power- Applications (P-APPS) and SPC for the requested server are adequate to handle additional applications and validates that the SPC for the server requested by EME 18 are adequate for migration (S260). P-Apps may be used to indicate the power consumed at the application level in order to determine if changes are needed for the currently used servers. For example, the P- Apps can be used to decide if some or all of the applications should be moved or not moved to other severs that are better suited to handle the applications. In one embodiment, P-Apps can determine those severs that are more energy efficient, and/or have the latest firmware versions, etc. Thus, if the decision is to migrate the applications, the applications are moved from the "hot spot" server to another server, either within data center 14 or to a data center at a different geographic region than data center 14.

FIG. 8 illustrates a flow chart showing an alternate use-based application of the present disclosure. In the scenario depicted in FIG. 8, energy optimization and asset performance of a hardware component at data center 14 is determined. Figure 8 illustrates the general interaction between DCME 12 and EME 18 and how a specific trigger handshake, in this instance, "hardware asset performance optimization" interacts with system 10. For example, EME 18 requests SPC data for a specific server 15, and sends a request for the SPC data to DCME 12 (S270). DCME 12 validates the SPC and sends an SPC validation acknowledgment to EME 18 (S280). EME 18 then recommends migration of one or more applications running on server 15 based on an optimization sub-algorithm and informs DCME 12 that server 15 should not run any further applications (S290). In one embodiment, DCME 12 actually performs the migration of applications. However, in other embodiments, DCME 12 can instruct another entity to perform the migration.

FIG. 9 illustrates a flow chart showing an alternate use-based application of the present disclosure. In the scenario depicted in FIG. 9, a limited privilege tenant (or virtual operator, or infrastructure provider, etc.) at data center 14 sets real time power generation and consumption constraints on a particular high-bandwidth application and DCME 12 sends a request to EME 18 (S300) requesting that EME 18 determine the capability of performing that application on a given resource. For example, the tenant may request the capability of holding a video conference and inquire if the video conference can take place via an IP multimedia subsystem (IMS) application, based on the real time power generation and consumption constraints set by the tenant. EME 18 sends a request to DB 16 (S310), requesting open-stack monitoring agents read from real time power-generation and real time power- consumption counters, and based on this information, informs DCME 12 that the video conference power measurement capability is activated, or not activated as the case may be (S320). DCME 12 validates that the P-APPS and SPC for the server 15 running the video conference has the capability to do so. EME 18 obtains continuous reports from DB 16 and sends these reports containing real time power generation and consumption information for the video conference application to DCME 12. FIG. 10 illustrates a flowchart showing exemplary steps taken in an embodiment of the present disclosure to identify and validate energy optimization opportunities at data center 14. Communication interface 30 of EME 18 obtains information, the information including calculated energy utilization for at least one application within data center 14, where the calculated energy consumption is based on at least one trigger factor (Block S340). As discussed above, the information can be obtained directly from DCME 12 or retrieved from database DB 16. EME 18 includes circuitry including memory 32 and processor 34, where memory 32 stores instructions that, when executed, configure the processor to identify an energy optimization opportunity for at least one application based on at least the obtained information (Block S350), and validate the energy optimization opportunity for at least one of the applications based at least in part on energy optimization for data center 14 (Block S360).

FIG. 11 illustrates an alternate embodiment of EME 18. In this embodiment, EME 18 includes a communication module 40 configured to obtain information. The information includes calculated energy utilization for at least one application within data center 14, the calculated energy consumption based on at least one trigger factor. EME 18 can receive the information directly from DCME 12 or obtain the information from database DB 16. In another embodiment, EME 18 calculates the information. EME 18 also includes an energy optimization module 42 configured to identify an energy optimization opportunity for at least one of the applications based on at least the obtained, received, or calculated information. EME 18 also includes an energy validation module 44 configured to validate the energy optimization opportunity for at least one of the applications based at least in part on energy optimization for data center 14.

As will be appreciated by one of skill in the art, the concepts described herein may be embodied as a method, data processing system, and/or computer program product. Accordingly, as noted above, the concepts described herein may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects all generally referred to herein as a "circuit" or "module." Furthermore, the disclosure may take the form of a computer program product on a tangible computer usable storage medium having computer program code embodied in the medium that may be executed by a computer. Any suitable tangible computer readable medium may be utilized including hard disks, CD-ROMs, electronic storage devices, optical storage devices, or magnetic storage devices.

Some embodiments are described herein with reference to flowchart illustrations and/or block diagrams of methods, systems and computer program products. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer readable memory or storage medium that may direct a computer or other

programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks.

The computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks. It is to be understood that the functions/acts noted in the blocks may occur out of the order noted in the operational illustrations. For example, two blocks shown in succession may in fact be executed substantially concurrently or the blocks may sometimes be executed in the reverse order, depending upon the functionality/acts involved. Although some of the diagrams include arrows on communication paths to show a primary direction of communication, it is to be understood that

communication may occur in the opposite direction to the depicted arrows. Computer program code for carrying out operations of the concepts described herein may be written in an object oriented programming language such as Java® or C++. However, the computer program code for carrying out operations of the disclosure may also be written in conventional procedural programming languages, such as the "C" programming language. The program code may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer. In the latter scenario, the remote computer may be connected to the user's computer through a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

Many different embodiments have been disclosed herein, in connection with the above description and the drawings. It will be understood that it would be unduly repetitious and obfuscating to literally describe and illustrate every combination and subcombination of these embodiments. Accordingly, all embodiments may be combined in any way and/or combination, and the present specification, including the drawings, shall be construed to constitute a complete written description of all combinations and subcombinations of the embodiments described herein, and of the manner and process of making and using them, and shall support claims to any such combination or subcombination.

It will be appreciated by persons skilled in the art that the present disclosure is not limited to what has been particularly shown and described herein above. In addition, unless mention was made above to the contrary, it should be noted that all of the accompanying drawings are not to scale. A variety of modifications and variations are possible in light of the above teachings without departing from the scope of the disclosure, which is limited only by the following claims.

Claims

What is claimed is:

1. A data center energy management method, the method comprising: obtaining information, the information including calculated energy utilization for at least one application within a data center (14), the calculated energy consumption based on at least one trigger factor (S340);

identifying an energy optimization opportunity for at least one of the at least one application based on at least the obtained information (S350); and

validating the energy optimization opportunity for at least one of the at least one application based at least in part on energy optimization for the data center (14) (S360).

2. The data center energy management method of Claim 1 , wherein obtaining the information includes calculating the energy utilization for the at least one application within the data center (14).

3. The data center energy management method of Claim 1, further comprising:

reporting the validated energy optimization opportunity for the at least one of the at least one application to a data center management entity (12) (SI 60).

4. The data center energy management method of Claim 3, wherein the information is received from the data center management entity (12).

5. The data center energy management method of Claim 1, wherein the at least one trigger factor includes at least one of a power usage effectiveness index (20), a carbon usage effectiveness index (22), a water usage effectiveness index (24), weather information, and disaster recovery information (28).

6. The data center energy management method of Claim 1, wherein the identified energy optimization opportunity includes moving at least one of the at least one application to an alternate data center located at a different geographic region from the data center (14).

7. The data center energy management method of Claim 6, wherein moving the at least one of the at least one application to the alternate data center is transparent to a user of the at least one application.

8. The data center energy management method of Claim 3, further comprising:

obtaining subsequent information, the subsequent information including energy utilization updates for the at least one application, the energy utilization updates based at least upon the reported validated energy optimization opportunity.

9. The data center energy management method of Claim 8, wherein the subsequent information is obtained after a predetermined optimization period.

10. The data center energy management method of Claim 1, wherein the information is obtained from a data repository (16) shared with a data center management entity (12).

11. An energy management entity (18) comprising:

a communication interface (30) configured to obtain information, the information including calculated energy utilization for at least one application within a data center (14), the calculated energy consumption based on at least one trigger factor;

a processor (34); and

a memory (32) storing instructions that, when executed, configure the processor (34) to:

identify an energy optimization opportunity for at least one of the at least one application based on at least the obtained information; and

validate the energy optimization opportunity for at least one of the at least one application based at least in part on energy optimization for the data center (14).

12. The energy management entity (18) of Claim 11, wherein the processor is further configured to calculate the energy utilization for the at least one application within the data center (14).

13. The energy management entity (18) of Claim 11, the communication interface (30) further configured to:

report the validated energy optimization opportunity for the at least one of the at least one application to a data center management entity (12).

14. The energy management entity (18) of Claim 13, wherein the communication interface (30) receives the information from the data center management entity (12).

15. The energy management entity (18) of Claim 11, wherein the at least one trigger factor includes a power usage effectiveness index (20), a carbon usage effectiveness index (22), a water usage effectiveness index (24), weather information, and disaster recovery information (28).

16. The energy management entity (18) of Claim 11, wherein the identified energy optimization opportunity includes moving at least one of the at least one application to an alternate data center located at a different geographic region than the data center (14).

17. The energy management entity (18) of Claim 16, wherein moving the at least one of the at least one application to the alternate data center is transparent to a user of the at least one application.

18. The energy management entity (18) of Claim 11, the communication interface (30) further configured to:

obtain subsequent information, the subsequent information including energy utilization updates for the at least one application, the energy utilization updates based at least upon the reported validated energy optimization opportunity.

19. The energy management entity (18) of Claim 18, wherein the subsequent information is obtained after a predetermined optimization period.

20. The energy management entity (18) of Claim 11, wherein the information is obtained from a data repository (16) shared with a data center management entity (12).

21. An energy management entity (18) comprising:

a database (16) configured to store information, the information including calculated energy utilization for at least one application within a data center (14), the calculated energy consumption based on at least one trigger factor;

an application level energy manager (36) configured to identify an energy optimization opportunity for at least one of the at least one application based on at least the stored information; and

an optimization controller (38) configured to, upon an optimization request from the application level energy manager, run an optimization model and provide operational data based on the optimization model to the application level energy manager (36),

the application level energy manager (36) configured to validate the energy optimization opportunity for at least one of the at least one application based at least in part on the operational data.

22. The energy management entity (18) of Claim 21, the application level energy manager (36) further configured to report the validated energy optimization opportunity for the at least one of the at least one application based at least in part on energy optimization for a data center (14).

23. The energy management entity (18) of Claim 21, wherein the at least one trigger factor includes a power usage effectiveness index (20), a carbon usage effectiveness index (22), a water usage effectiveness index (24), weather information, and disaster recovery information (28).

24. The energy management entity (18) of Claim 21, wherein the identified energy optimization opportunity includes moving at least one of the at least one application to an alternate data center located at a different geographic region than the data center (14).

25. The energy management entity (18) of Claim 24, wherein moving the at least one of the at least one application to the alternate data center is transparent to a user of the at least one application.

26. The energy management entity (18) of Claim 21, the database (16) further configured to store subsequent information, the subsequent information including energy utilization updates for the at least one application, the energy utilization updates based at least upon the validated energy optimization opportunity.

27. The energy management entity (18) of Claim 26, wherein the subsequent information is stored after a predetermined optimization period.

28. An energy management entity (18) comprising:

a communication module (40) configured to obtain information, the information including calculated energy utilization for at least one application within a data center (14), the calculated energy consumption based on at least one trigger factor;

an energy optimization module (42) configured to identify an energy optimization opportunity for at least one of the at least one application based on at least the obtained information; and

an energy validation module (44) configured to validate the energy optimization opportunity for at least one of the at least one application based at least in part on energy optimization for the data center (14).