WO2014205585A1 - Method and system for optimizing the location of data centers or points of presence and software components in cloud computing networks using a tabu search algorithm - Google Patents

Abstract

The ubiquity of cloud applications requires the meticulous design of cloud networks with high quality of service, low costs, and low CO₂ emissions. The present invention relates to a method and system for optimizing the locations of cloud data centers or points of presence and software components while simultaneously finding the information routing and network link capacities by using an extremely efficient tabu search heuristic. The objectives are to optimize the network performance, the CO₂ emissions, the capital expenditures (CAPEX), and the operational expenditures (OPEX). The problem is modeled using a mixed-integer programming model and solved with both an optimization solver and a tabu search heuristic.

Description

METHOD AND SYSTEM FOR OPTIMIZING THE LOCATION OF DATA CENTERS OR POINTS OF PRESENCE AND SOFTWARE COMPONENTS IN CLOUD COMPUTING NETWORKS USING A TABU SEARCH ALGORITHM

FIELD OF THE INVENTION

[0001] The present invention relates to a method and system for optimizing the location of data centers or points of presence (PoPs) and software components in cloud computing networks, particularly data center locations supplied by green energy sources. It also finds the optimal dimension of the telecommunication backbone linking the data centers, the number of required servers in each data center and the routing.

BACKGROUND OF THE INVENTION

[0002] With the evolution of the Internet, we are witnessing the birth of an increasing number of applications that rely on the network; what was previously executed on the user's computers as stand-alone programs has been redesigned to be executed on servers with permanent connections to the Internet, making the information available from any device that has network access. Instead of buying a copy of a program, users can now pay to obtain access to it through the network, which is one of the models of cloud computing [1], Software as a Service (SaaS). The continuous growth of Internet bandwidth has also given rise to new multimedia applications, such as social networks and video over the Internet; and to complete this new paradigm, mobile platforms provide the ubiquity of information that allows people to stay connected.

[0003] Service providers may own servers and data centers or, alternatively, may contract infrastructure providers that use economies of scale to offer access to servers as a service in the cloud computing model, i.e., Infrastructure as a Service (laaS). As users become more dependent on cloud services and mobile platforms increase the ubiquity of the cloud, the quality of service becomes increasingly more important. A fundamental metric that defines the quality of service is the delay of the information as it travels between the user computers and the servers, and between the servers themselves.

[0004] Along with the quality of service and the costs, the energy consumption and the C0₂ emissions are fundamental considerations in regard to planning cloud computing networks. Data centers consumed 1.5% of the energy in the U.S. in 2006, which is equivalent to the amount of energy consumed by 5.8 million households [2]. Each type of energy, such as wind, hydroelectric, nuclear, diesel, and coil, introduces a different amount of C0₂ into the environment, and each potential data center location will be provided by one or more of those energy sources. Thus, choosing data center locations supplied by green energy sources can greatly reduce environmental pollution.

[0005] The present description refers to a number of documents that are listed at the end of this application, the content of which is herein incorporated by reference in their entirety. SUMMARY OF THE INVENTION

[0006] More specifically, in accordance with the present invention, there is provided a method and system for optimizing the location of data centers or points of presence and software components in cloud computing networks using a tabu search algorithm.

[0007] Other objects, advantages and features of the present invention will become more apparent upon reading of the following non-restrictive description of specific embodiments thereof, given by way of example only with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In the appended drawings:

[0009] Figure 1 is a block diagram of a network and application layers, according to a preferred embodiment of the present invention;

[0010] Figure 2 is a block diagram of a network layer of a greedy solution for an example of a web engine;

[0011] Figure 3 is a network topology diagram with 10 cities and 10 potential data center locations;

[0012] Figure 4 is a network topology diagram corresponding to solution A : Delay minimization;

[0013] Figure 5 is a network topology diagram corresponding to solution B : pollution minimization;

[0014] Figure 6 is a network topology diagram corresponding to solution C : cost minimization;

[0015] Figure 7 is graph of a tabu search vs greedy heuristic for large cases and cost priorities; and

[0016] Figure 8 is a block diagram illustrating a method for locating data centers or points of presence and software components in a cloud computing network, according to a preferred embodiment of the present invention.

DESCRIPTION OF ILLUSTRATIVE EMBODIMENTS

[0017] The present invention is illustrated in further details by the following non-limiting examples.

[0018] In this application, we define "data center" as a facility used to house computer systems and associated components, such as telecommunications and storage systems. It may include redundant or backup power supplies, redundant data communications connections, environmental controls (e.g., air conditioning, fire suppression) and various security devices.

[0019] In this application, we define "point of presence (PoP)" as an access point to the Internet. It is a physical location that may house servers, routers, ATM switches and digital/analog call aggregators. Given that servers and software components can be hosted in either points of presence or data centers, we use the term data center to denote both concepts. That is, Point of Presence locations are included in the set of potential data center locations.

[0020] In this application, we propose to solve the problem of designing a cloud computing network by answering the following questions:

What potential data centers should be used in the network?

Which data center should host each of the software components of the cloud applications?

How many servers will be hosted at each data center?

How will the information be routed through the network?

What are the link capacities required to carry that information?

[0021] All these questions are interrelated and have an impact on the quality of service, energy consumption, cost, and pollution. However, in practice, these questions are tackled separately and very often, in different time frames. In this application, we propose the simultaneous decision of data center or point of presence location, software component placement and other planning elements in order to provide a comprehensive optimization framework for future reference. Once the network is designed and in operation, complementary models can be used to manage the system in a dynamic way, and interesting research avenues are open in that direction [3], [4].

[0022] The criteria to choose the optimal solutions in our framework are embedded in a multi-objective function that allows planners to weight each attribute according to their priorities. The objective function is composed of the following metrics: traffic delay, energy consumption, C0₂ emissions, traffic cost, server cost, data center capital expenditures (CAPEX), and data center operational expenditures (OPEX).

[0023] The proposed problem is formalized as a mixed linear-integer programming model and solved, first with AMPL-Cplex, and then with a very efficient tabu search heuristic. Because there are many integer decision variables, the AMPL-Cplex model ran for 58 minutes for a small network with 24 potential data center locations. Furthermore, the cases used for this study were networks with up to 500 access nodes and 1 ,000 potential data center locations; in fact, one of the largest cloud networks has in the order of 1 ,000 small data centers [5]. Thus, a specific heuristic is required to design this type of network. We show that the tabu search algorithm presented in this application achieved very small optimality gaps, and the execution time ranged from some milliseconds to less than 10 minutes.

[0024] The framework presented in this application helps to understand each aspect of a cloud network in a formal way, and multiple actors may be interested in having optimal solutions to this problem. One actor may be an infrastructure provider that needs to deploy or extend a data center network used by service providers. Another actor may be a service provider that needs to choose the cloud data centers to deploy global distributed applications and to decide the number of servers to host in each one of them. Yet another actor may be organizations with their private clouds who want to solve the whole network design in an integrated and optimal way.

[0025] The remainder of this application is structured as follows. Section 2 presents a literature review of related problems and models. Section 3 describes the problem and proposes a Mixed Integer Linear Programming (MILP) model. Section 4 specifies the tabu search heuristic that was developed to solve large problems. Section 5 presents a case study of a search engine application that requires data centers and software components to be placed around the world. Section 6 shows optimal solutions for the case study and analyzes how the multiple objectives interact. This section also shows the optimality gap of the tabu search heuristic and the execution time depending on the instance sizes. Finally, the conclusions are presented in Section 7.

Section 2: Related work

[0026] The problems of facility location, such as capacitated facility location [14], [15], [16], originate in operations research, and extensive literature exists on them [17]. As surveyed in [4], some articles on data center location were recently presented [6], [7], [8], [10]. Table 1 summarizes the problems of data center location and provisioning closest to our work. The authors in [6] considered access nodes that aggregate user traffic, and each one of them must be assigned to a primary data center and to a backup data center. The objective of the proposed linear programming model was to minimize the delay between users and data centers. In [7], a model that minimizes the total cost of traffic, servers, and data centers was presented. Potential locations in the U.S. were studied, considering energy and land costs. That approach is the closest to the model presented in this application because cost, energy, delay, and C0₂ were also considered. The main difference is that our model takes into account distributed applications with traffic between user devices and servers and between the servers themselves, which has an impact on the routing and capacity assignment of the backbone network. Furthermore, our approach includes a multi criteria objective function allowing planners to minimize delay, C0₂ emissions, as well as costs. Regarding the resolution method, [7] proposed a simulated annealing heuristic combined with a linear programming model. [8] focused on the minimization of optical and IP router power consumption as functions of the data center location through linear programming models. The proposed strategy was to place data centers close to energy sources to reduce power losses in the grid. [10] proposed a method to select potential data center locations by comparing multiple criteria, such as energy cost, security, availability of telecommunications networks and labor. The delay between users and data centers was not taken into account in that model.

TABLE 1

Data Center Location and Provisioning Approaches

Article Objective Decisions Resolution

[6] - Net. delay - DC location MILP model

- AN to DC assig. + solver

- AN to backup DC

[7] - Cost - DC location Simulated

- AN to DC assig. anneal. + LP

[8] - Energy - DC location MILP model

- AN to DC assig. + solver

- Routing

[9] - Net. delay - DC location MILP model

- Cost - AN to DC assig. + solver

- Pollution - App. location

- Link capacities

[10] - Risk - DC location ELECTRE

- Social TRI

- Cost

- Pollution

[1 1] - N. delay - App. placement Max. flow

[12] - Energy cost - Workload assig. Minimum

cost flow

[3] - Cost - App. placement CloudSim

- Energy - Resource alloc.

- Resp. time - Workload migration

[13] - Cost - Virtual network LP + multicom.

- Load balance embedding flow

[0027] With respect to the application location, [1 1] was the first to model the software component placement considering the information traffic between the components. The proposed resolution method based on a maximum flow algorithm was restricted to the case of two processors. [12] served web requests from multiple data centers to minimize the total energy cost by changing the workload assignment, depending on the energy price, and guaranteeing the quality of service. The problem was linearized and then reduced to a minimum cost flow problem. [3] studied dynamic provisioning of applications from multiple cloud vendors and did a performance study with the CloudSim toolkit. That paper shows that the cloud federation model yields important benefits in quality of service and costs. [0028] Because the application layer and the network layer are two separate graphs and because the problem includes mapping the application layer on top of the network layer, a related problem is virtual network embedding, which maps a virtual network with virtual routers and virtual links on top of a physical network. Each virtual node must be located on a physical node, and each virtual link must be routed through a physical path. [13] proposed an integer programming model and solved it using linear relaxation and rounding techniques. With respect to defining the routing and the link capacities in multilevel networks, [18] presented a survey on the models and resolution approaches for the network synthesis problem.

[0029] Regarding our previous work, [19] presented a model that simultaneously solved the location of data centers and software components with the routing. This model was an initial model that minimized the network delay but did not consider the energy and the pollution aspects. [9] solved a model that considered energy and pollution aspects through an AMPL- Cplex program by adding the search for optimal link capacities. The new aspect caused us to discard the assumption of Poisson traffic, and the delay of each link was taken from real measurements.

[0030] The present application includes the following original contributions:

1 ) A highly efficient tabu search heuristic to minimize delay, cost, energy, and C0₂ emissions in the jointly determined location of data centers and software components, routing and link dimensions.

2) Calculation of the energy consumed by switches and routers, in addition to the servers.

3) Tradeoff analysis between the multiple objectives.

Section 3: Problem description

[0031] The problem is to define the location of data centers and software components, the number of servers in each data center, the information routing, and the capacity of each link in the network. The design objective is to minimize the network delay, the cost, the energy consumption, and the C0₂ emissions. This problem is formalized as an MILP model that minimizes a multi-criteria objective function.

[0032] The problem setting is an organization that offers cloud services to its users through a data network. The organization must optimize the network design to provide the best possible quality of service while reducing costs, energy consumption, and C0₂ emissions. The applications that will be executed in the network are represented as a graph

G^A(V^A,A^A) where each node is a software component and each arc is a traffic demand from one software component to another. Each software component may be executed on a user computer or on a server. A single software component can even span multiple servers to achieve its purpose. The applications will be deployed on a computer network

G^N(V^N , A^N) composed of access nodes Λ, backbone routers Ή, existing data centers Ί), and possible locations to open new data centers L. The set T> denotes all the potential data centers, i.e., the union of £ and T>. Each access node represents an aggregation of users, which may be an Internet Service Provider (ISP), a mobile company, a city, a region, or a country, depending on the case. The users will access the cloud services hosted in the data centers. The nodes in G^N are connected through the backbone links, typically by optical fiber. The link capacities, measured in number of channels, are decision variables. Because not every node is suitable to host every software component, P is the set of possible assignments of software components in V^A to network nodes in V^N.

[0033] Figure 1 shows an example of a web search engine that must be deployed on a data center network. The application is composed of user interfaces, web servers, and World Wide Web indexes. Each access node aggregates the users of a whole city. There is one user interface for each access node, representing the client component that is executed on the user computers of that access node. There is also a web service that answers requests to that access node. The same web service may require multiple servers to handle all the requests in an efficient way. The web service makes index queries to the web index where the associations between keywords and pages are stored. Each web index may also require multiple servers with multiple hard disks to store the entire index. In the example, W is the master index and all other indexes are replicas of it. In this case, the problem is to choose a subset of the potential data centers and locate each web service and each web index in one of the data centers. The following sections provide the parameters and decision variables of the problem.

A. Network traffic

[0034] The software components send and receive information between them, and this information is divided into packets that will be sent through network paths. Each step in a path adds processing, waiting, transmission and propagation delay. While the traffic in a link is under a maximal allowed utilization, the link delay is bound by a constant. For example, a specific link could present packet delays of less than 1 millisecond while its traffic is less than 70% of its capacity. In the same way, the delay incurred by the packets in a path is bound by the sum of the link delay bounds. The strategy in this problem is to locate software components that exchange traffic in close nodes in terms of delay such that the delay is minimized and the quality of service is increased. That implies, for instance, that web service components will be placed near the users to reduce the round trip time. The following is the network traffic notation: b_j Throughput of demand d e A^A in peak time (in bps).

b² Capacity per link channel (in bps), e.g., 10 Gbps.

_Ue Maximal utilization ratio per channel in link e e A^N (e.g., 0.70).

t_e Delay bound for link e e A^N (in seconds per packet) while its traffic is less than the allowed utilization.

B. Servers

[0035] Cloud computing applications have software components that are executed either on the user devices or on servers. In fact, a single software component may be executed on multiple servers. For instance, in a web search engine, multiple web servers answer HTTP requests. Each server has capacity allocated for each one of its resources, including network bandwidth, CPU, RAM, and hard disks. In this model, we consider that the number of replicas of each software component, i.e., the number of required servers, is a parameter that is estimated in advance and strictly depends on the nature of the application. The server-related notation is as follows:

K Set of resources that each server has, which include 1 = CPU, 2 = network card, 3 = RAM, and 4 = storage.

r)_p Average consumption of resource p e K required by a copy of component i e V^A , in the correspondent units, e.g., GHz, Gbps, or GB.

ri_p Peak consumption of resource p G K required by a copy of component

i e V^A .

rt Number of replicas of software component i .

r⁴ _p Capacity of resource p G K on a server.

Capacity of data center k e T> in number of servers.

r⁶ Number of servers per Local Area Network (LAN) switch.

[0036] The servers and switches in a data center are organized in racks, and each data center has space for a maximal number of racks. Each switch can connect a maximal number of servers, which will determine the number of required switches. In summary, each software component requires a specific number of servers. The model solution defines what software components each data center hosts, hence the number of servers required in each data center. From that, the number of LAN switches needed for each data center in a particular solution can be calculated. The sum of servers and switches must be less than the data center capacity.

C. Energy

[0037] An important objective of cloud computing is to minimize the power consumption of the entire network. The power consumption is primarily from the servers, the network switches, the routers, the data center cooling, and the power provisioning equipment. The server power consumption depends on its utilization, having a minimal idle power and reaching a maximal power consumption when the resources are fully utilized [20]. The following is the energy-related notation:

w The power consumption of a server as a function of the utilization of each resource (in watts).

w Network switch power consumption (in watts).

Router port power consumption (in watts).

Maximum power capacity of data center k e T> (in watts) for IT equipment.

Power Usage Effectiveness (PUE) of data center k e T) (ratio of total power

/ IT equipment power).

C0₂ emissions in data center k e T> (in g / KWh).

Average power consumption of a server with a replica of software component i e V^A .

Peak power consumption of a server with a replica of software component i e V^A .

[0038] C0₂ emissions depend on how the energy is produced because each type of energy generates a certain amount of C0₂ per KWh. For example, hydroelectric energy emits 10 g of C0₂/KWh, and diesel emits 778 g/KWh [21]. Because the power consumption, wi ,

7 8 depends on each piece of hardware of the server used, the values and should be determined experimentally.

D. Cost

[0039] The following are the parameters that define the cost of each cloud element, the cost of the energy and the penalties. Each element has a different amortization period; data centers are amortized between 12 and 15 years, and servers are amortized between 3 and 4 years [22]. Each cost below is the result of the total element cost divided by its amortization years.

Penalty for delay in demand d e A^A (in $ / (ms/packet) / year)

c Cost of a LAN switch (in $/year).

Cost of a router port (in $/year).

Cost of a channel in link e e A^N (in $/year).

Cost of a server (in $/year).

CAPEX for using data center k e T> (in $/year).

OPEX for using data center k e T> (in $/year).

Electricity cost in data center k e V (in $/MWh).

cl Penalty for emitting C0₂ in data center k & V (in $/ton).

[0040] A penalty is applied to the delay for exchanging information between the software components. Some traffic demands may suffer from the delay more than others. For example, a traffic demand could be used for copying files from a data center to calculate statistics once every day. That traffic demand might not be affected by an additional delay; therefore, its delay penalty is low. Traffic demands from the users doing search requests require a very short delay, and in that case, the penalty is high; this penalty represents a loss in revenue because users may change to the competitors.

[0041] The pollution penalty term comes from two sources. One is the country regulations that apply a carbon tax to the energy consumption. In that case, the electricity cost ^ck is set as the base cost without that tax in data center k , and ^cl includes the carbon tax. There is a second source of C0₂ penalty that may be even higher: the losses in corporate image. Nowadays customers reward companies that make efforts to reduce pollution and its quantification should be included in the C0₂ penalty.

E. Decision variables xit 1 if software component i e V^A is located in a network node k e V^N ; 0 otherwise.

y_k 1 if node k e V^N hosts one or more software components; 0 otherwise.

zt Number of servers hosted in data center k e D.

st Number of LAN switches in data center k e Z).

f _de Amount of demand d G A^A traffic carried by link e < A^N (in bps).

b_e Number of channels, e.g., wavelengths or fibers, in link e≡ A^N .

wk Average power consumption of IT equipment in node k e V^N .

[0042] The problem is modeled as a multi-commodity flow problem, in which each flow variable contains the amount of traffic per demand, per link. These variables are subsequently related to the location variables of the software components because moving a software component moves its traffic demands.

F. Objective function

[0043] Each criterion to optimize is included in a term of the objective function. The following are the formulas to calculate each cost. Note that some of these formulas contain constants to change units, e.g., seconds to milliseconds, KWh to MWh, or Gbps to bps.

[0044] The flow variables that define how demands are routed are used to calculate each demand delay, that is the end-to-end delay incurred by packets between software components. For instance, the delay of the demand between a user interface and a web service will be included in this computation. Then, each demand delay is multiplied by its delay penalty, and the total delay penalty is:

[0045] The network traffic cost is composed of the costs of the LAN switches, the router ports and the links. Each pair of link channels, uplink and downlink, of a link is connected to two router ports, one in each end of the link. Therefore, the number of router ports required for a link is equal to the number of channels.

[0046] The following is the cost of all servers in the network:

V

[0047] The capital expenditures (CAPEX) of each data center includes the land cost, the cost of building the facility, the cooling artifacts, and the power provisioning instruments. The total CAPEX of a solution is calculated as follows:

C "¹ 6

k r^N

[0048] The data center operational expenditures (OPEX) includes the human labor, i.e., technicians and security guards, as well as the administrative costs. We exclude the electricity because we consider it in an explicit term.

[0049] The energy cost for a whole year is calculated as a function of the data center energy consumption, its PUE, and the price of energy as follows:

^ =∑^ · 365 · 241ι · 10-⁶Κ⁵

k≡V^N

[0050] In a similar way, the environmental cost is proportional to the energy consumed in each data center, the ratio of C0₂ emissions, and the C0₂ emission penalty: c^C°² =∑^ (^ · 10-⁶)(>ν, · 365 · 241ι · 10-³)^

k^V^N

[0051] Finally, each cost in the objective function is weighted by a parameter, allowing planners to modify the priorities. The objective function is defined as follows: z = ac^D + βα^τ + jc^s + &^c + sc° + ηα^Ε + ψα^€°² (1 ) G. Constraints

Location of software components and data centers

∑ ^χ _Λ = \ V i <= V^A (2)

(i,k)≡P

∑ x_ik = 0 V i e V^A (3) k≡V^N /

i,k)tP

x_ik≤y_k V (i,k) e P (4)

[0052] Equations (2) and (3) state that each software component must be placed in one node, either access node, or data center. They also state that each assignment of a software component to a node must belong to the set of possible assignments P . Constraint (4) specifies that nodes containing software components are open.

• Flow conservation

∑ f_de + x*b_d = ∑ fde + x_]kb_d V d = (i, j) e A^A , k _S V^N (5)

[0053] This constraint relates the software component location to the routing. It associates the application and the network layers. For each demand from software component i e V^A to software component j e V^A located in nodes k_t ,k_j ^ V^N , respectively, the traffic ¾ enters the network in _t and exits in kj . In each intermediate node k from _t to ^y , flow is conserved.

• Link capacity

∑ f_de≤u_e b² b_e V e A^N (6) d≡A^A b = b_fi V (i,j) A^N (7)

[0054] Constraint (6) defines the link capacity variable b_e (in number of channels), requiring that the traffic is less than the maximal utilization threshold, u_e b² b_e . Because full-duplex links are assumed, equation (7) states that the capacity in both directions of the link is the same.

• Number of servers

[0055] The number of servers required in each data center is the sum of the servers required by the software components that are located there. • Number of switches r⁶s_k>z_k V fee© (9)

[0056] Because each LAN switch can connect a maximum number of servers r⁶ _t Constraint (9) defines the minimal number of switches, ¾, required to connect the ¾ servers in data center k .

• Data center capacity z_k+s_k<r_k ⁵ V keO (10)

[0057] The number of servers and LAN switches must be less than the maximal capacity of the data center.

• IT equipment average power consumption

[0058] Equation (11) sums the average amount of power consumed by the server switches, the router ports, and the servers in a node. This value is used to calculate the cost of electricity used in the solution.

• Data center maximal peak power w² _Sk +∑w³b_e +∑w*rx_ik≤w_k ⁴ VfeGD (12)

[0059] The peak power consumption of data center k is calculated using the same methodology as for the previous constraint. This value has a fixed limit in each data center because of the electricity available and the power provisioning equipment in the data center.

• Domain of variables

½e{0,l} V ieV^A,keV^N (13)

¼<≡{o,i} V k≡V^N (14)

¾ ^{e Z}>0 v k e Ώ (15) ^sk≡ ^Z>0 v leB (16) ^be ^eZ>0 V e≡A^N (17) fde ^{G R}>0 V deA^A,eeA^N (18) w_k e R_≥0 V k≡V^N (19) [0060] Each variable domain is defined. All the variables are non-negative. The location variables may take the values zero or one. The number of servers, LAN switches, and network channels are integers. The flow and the data center power consumption variables are real numbers.

Section 4: Resolution approach

[0061] This section describes the tabu search heuristic [23], [24] proposed to solve the cloud network design problem. The algorithm was programmed in C++ and compiled using gcc 4.5.1 with the optimization options activated. The quality of solutions obtained and the execution time are compared with the model in AMPL with Cplex 12.4. Algorithm 1 presents the main structure of the proposed tabu search heuristic.

Algorithm 1 Tabu search for the cloud network design problem

function TABUSEARCH

M <- INITIALGREEDYSOLUTION

Best <- M

L <- { }

repeat

Choose (c,d) e P-L that minimizes z when moving software component c to node d in M .

Move c to d in M

Add (c, d) to tabu list L for ^^|N ₂ ^(M)I iterations.

i <- i + 1

if z( ) < z(Best)then

Best <- M

end if

until i = MAXJT

return Best

end function

function I NITIALGREEDYSOLUTION

M <- ø

for each component c in V^A do

d <- x I (c, x) e P and x is the node that least increases z when placing c in x in M.

M «- u {(c, d)}

end for

return M

end function

A. Solution space

[0062] A solution of the proposed tabu search heuristic is a mapping of the software components to the network nodes, i.e., M : V^A→V^N . In that solution M, each software component is placed at one of the network nodes. P specifies the set of possible assignments; hence, for each software component i , the pair (i, M(i)) must belong to . Thus, the solution space of the proposed heuristic is the set of mappings, as follows:

S = [M : V^A→ V^N/(i, M(i)) e P, Vi E V^A}

[0063] Each solution M determines the location variables ^xv and satisfies the constraints

(2) and (3) in the model. From the software component placement defined by variables ^xv , all the other variables of data center location (>7 ), data center capacities (s_k and ¾ ), routing

(fde ), link capacities (b_e ), and power consumption (w_k) can be calculated in a deterministic way as follows. The variables >7 of the data centers hosting the software components have a value of 1 , as stated by constraint (4). Because the software component location is defined in , the origins and destinations of the traffic demands are fixed. We use the shortest path between each pair of nodes of V^N for routing the demands and defining variables fde that satisfy the flow conservation constraint (5). The distance considered in the shortest path algorithm is the delay bound t_e of each link. The routing defines the total amount of traffic carried by each link; therefore, we can calculate the minimal number of channels required for that link using the channel capacity, c_{e t} and the maximal allowed utilization ratio per channel, u_e , satisfying constraints (6) and (7). The numbers of servers, z_{k i} and switches,

Sk , are also determined by the software components hosted in data center k through the constraints (8) and (9). Variables w¾ that correspond to the amount of power consumed by the IT equipment in data center k are calculated with the formula defined in equation (1 1 ).

Finally, the terms of the objective function that correspond to a delay of traffic demands, c^D ,

T S C O E

network traffic c , servers c , CAPEX c , OPEX c , electricity c , and C0₂ emissions

CO

c ¹ , are determined, and the value of the objective function is defined in equation (1 ). Thus, we see that the location of the software components determines all the variables in the model, and subsequently all the terms of the objective function.

[0064] The solutions in S satisfy all the constraints except the data center capacity constraints (10) and (12). Instead of enforcing those constraints, one penalty term for each constraint is added to the objective function. The number of servers that exceeds the capacity in an overloaded data center is multiplied by a large constant, which is higher than all the other terms of the objective function. The same penalty is applied for power consumption that is higher than the data center limit. Thus, the tabu search will avoid those solutions because the objective function is much higher than in solutions that do not have overloaded data centers. The algorithm will thus accommodate corner cases, which have capacities that make feasible solutions difficult to find. In that case, the tabu search will search to reduce the overloaded data centers until a feasible solution is found.

B. Initial solution

[0065] The greedy heuristic shown in Algorithm 1 locates one software component of V^A at a time. Each step chooses the node in V^N where the objective function has the lowest value when the software component is placed in that node. Because the objective function includes delay, cost, energy, and C0₂ the node chosen will be the best in those aspects, according to the priorities defined. If the first priority is the delay, a web service will be hosted in the closest data center to its users. If it is the C0₂, then the data center with the cleanest energy will be chosen. As in every greedy heuristic, the placement is done in an arbitrary order, and that may be non optimal. The greedy algorithm provides a good starting solution that the tabu search improves to a near optimal solution.

[0066] Figure 2 shows a greedy solution for the example of a web search engine. In the first five steps, each one of the user interfaces Ul, is located in its correspondent node A,, defined in the set P . In those steps, there is no choice because each Ul, has a unique possible assignment in P . Until that moment, the objective function remains zero because those components do not require servers and do not have direct traffic demands among them. Then, the location of the web service WSi that serves the requests from U is the node that increments the objective function the least. If all the data centers have the same costs and capacities and all the links have the same delay bound, the best possible data center for WSi is Li , as shown in Figure 2. That is because U is located in Ai and there is a traffic demand between U and WSi that contributes a delay that is part of the objective function. The shortest path between Ai and Li has three links, and the shortest paths between Ai and the other data center locations L₂ and L₃ have more links; hence more delay, given that all the links have the same delay. In the same way, WS₂, WS₃, WS₄ and WS₅ are located in the data center locations that are closest to the respective user interface. Given that the web indexes Wl, exchange a high amount of traffic with the corresponding web services WS„ the locations that have a smaller increment in the objective function are those where the web index is in the same data center as the web service.

C. Neighborhood relation

[0067] Each iteration of the tabu search moves from the current solution to a neighbor solution. One solution M is neighbor of a solution M' if and only if they differ in the location of only one software component i . The neighborhood of solution M' is the following set: N(M) = {Μ' e S I 3 i e V^A, M'(i)≠ M(i)}

[0068] Thus, one software component is moved from one node to another in each iteration of the tabu search. The chosen movement is that with the lowest objective function. The calculation of the objective function for each neighbor can be time consuming for large instances because the number of possible movements is high. However, the key of the tabu search heuristic is that each iteration must be performed as rapidly as possible. Calculating the objective function in an incremental way achieves that efficiency. Instead of performing the entire calculation for each neighbor, the difference between the current solution and the neighbor solution is calculated. That is, the amount that the objective function would change if a component is moved is what is calculated. In other words, the difference of each objective function term is calculated. Coding this approach with data structures with constant access time is more complicated than the straightforward calculation of the objective function, but the execution time of each iteration is several orders of magnitude shorter when performed incrementally. That means that the tabu search heuristic moves very rapidly among the solutions.

[0069] Limiting the number of neighbors to analyze can also reduce the execution time of each iteration. Instead of looking at the entire neighborhood, the tabu search analyzes a random subset of them [25]. The number of neighbors to analyze is 4,/| N(M) | , a number that grows as the instance size increases but is less than a linear function. The multiplier 4 was adjusted through experiments, making a trade-off between the execution time and the solution quality obtained in each iteration.

D. Tabu list

[0070] The risk of this type of heuristic is to reach a local optimum and keep returning to it iteration after iteration. A mechanism to avoid repeated solutions prevents that situation.

When a software component i is moved to a node /^', then the pair is added to a tabu list, L _t so that movement will not be performed again for a number of iterations. This mechanism effectively avoids local optimums and the searching process continues. The number of iterations to keep the movement in L is related to the neighborhood size and is equal to ^^M)l . This value was carefully adjusted to grow when the neighborhood is larger, but in a less degree than a linear function.

E. Stop criterion

[0071] If the best solution found does not improve after MAX_IT iterations, then the algorithm stops. MAX_IT is defined as the sum of the number of nodes in the network and the number of software components, i.e., \V^N \ + \V^A \ .

Section 5: Case study

[0072] In this section, we study the deployment of a hypothetical web search engine on a network of data centers.

A. Network and application topology

[0073] The network nodes V^N include a set of access nodes Λ, backbone routers Ή, and locations for new data centers L. The nodes are distributed among different locations around the globe. To do that, we considered the 500 largest cities in the world [26], the number of Internet users in each of them [27], and their geographic location [28]. For each city, we created an access node in A e V^N that aggregates all the traffic of its users. The access nodes are connected through intermediate backbone routers. Finally, for each one of those nodes -routers and access nodes-, a potential data center location was created in L. This method allows us to produce scenarios of different sizes, with up to 500 access nodes and 1000 potential data center locations.

[0074] Regarding the application layer G^A _t we used the web search engine topology shown in Figure 1 for n access nodes. There is a user interface Ul, for each access node that aggregates every web client in that city. There is also a web service WS, that answer the users' HTTP requests and make queries to the web indexes Wl,. All the web indexes keep synchronized with a master web index. The decisions to make are which data centers will host each web service and web index, as well as all the network planning variables of the proposed model, to minimize the delay, the cost, energy consumption, and the C0₂ emissions.

B. Network traffic

[0075] The total inter-domain traffic of the Internet was 154 Tbps in March 2013, considering the estimation of 39 Tbps in July 2009 [29] and a 45.5% annual growth. The hypothetical web search engine analyzed manages 1 % of the Internet traffic, that is 1 .54 Tbps. That total amount of traffic was distributed among the access nodes in proportion to the number of Internet users in each city [27]. Those are the traffic demands between the user interfaces

Ul, and the web services WS, (¾ ). The same value was used between each WS, and its Wl,.

The average packet size is 770 bytes ( ), the capacity of each link channel is 10 Gbps (b² ), the maximal channel utilization is 70% and the maximal link delay is 1 ms plus the propagation delay (t_e ). The propagation delay of each link is calculated with the distance between the nodes divided by the speed of light over optical fiber cable (200,000 km/s). C. Servers

[0076] The number of servers required is 200,000, also distributed in proportion to the number of users in each city. From the servers required by each city, 60% are for web

3

services and 40% for the web index (^r; ). Each data center can accommodate either 40, 80, 160, 1 ,000, 32,000, 64,000, or 128,000 IT elements. Each LAN switch can connect up to 40 r⁶

servers ( ip ).

D. Power

7

[0077] The average power consumption of each server (^wi ) was defined as 150 W, and the

8 9 peak power consumption (^wi ), as 300 W. Each LAN switch consumes 400 W (w ), and each 10 Gbps router port consumes 20 W (w³ ). The data center power capacity for IT equipment (^wl ) is either 12 KW, 24 KW, 49 KW, 303 KW, 9.7 MW, 19.4 MW, or 38.7 MW, depending on the data center size. The PUE of the data centers is 1 .08 (^wl ), that is the power provisioning equipment, cooling artifacts and any other power consumption are 8% of the power consumed by the IT equipment. The type of energy that supplies a data center generates a specific amount of C0₂ emissions (^wt ): offshore wind produces 9 g / KWh, hydroelectric 10 g / KWh, geothermal 38 g / KWh, nuclear 66 g / KWh, natural gas 443 g / KWh, diesel 778 g / KWh, and coal 960 g / KWh [21 ]. The amount of C0₂ produced by each data center in this case was randomly chosen from one of those values.

E. Cost

[0078] The delay penalty was defined as $10,000 / (ms / packet) / year for the traffic demands between the Ul„ WS„ and Wl„ and $10 / (ms / packet) / year for the delay between the Wl, themselves (^cd ). That means that the delay between Ul„ WS„ and Wl, is much more important than the delay between the Wl, themselves. The cost of each LAN switch is $1 ,000

/ year (c² ). Each 1 0 Gbps router port costs $100 / year (c³ ) and each directed 10 Gbps link channel, $60,000 / year (c⁴ _e ) [30]. The cost of each server is $4,000 [31 ] and it is amortized in 4 years, resulting in $1 ,000 / year (c⁵ ). The data center CAPEX is $12 / W amortized in

12 years giving a total of $1 / W / year (^ct ), and the OPEX is $0.24 / W / year (^cl ) excluding electricity that was considered as an explicit term of the cost function [22]. Electricity prices were uniformly distributed between $30 / MWh and $70 / MWh (^cl ) [32]. Finally, the penalty

9

for emitting C0₂ was defined as $1 ,000 / ton (^ck ).

Section 6: Results A. Tradeoff between objectives

[0079] As a first step, we executed the AMPL-Cplex model on a case of 10 cities to analyze how the different aspects of the model interact. This case will also demonstrate how planners can use the framework with multiple objectives. The network has 10 access nodes and 10 potential data center locations. In Figure 3, the access nodes are shown as black circles and the potential data center locations are shown as white boxes. The application graph is a web search engine with a user interface, a web service, and a web index for each access node, and the topology of traffic demands as discussed in Section 5.

[0080] The case was solved in three different contexts, depending on the optimization priorities: A) delay oriented, B) pollution oriented, and C) cost oriented. Table 2 presents the results of the three solutions, and Figures 4, 5, and 6 show the selected data centers with black boxes. All the tests were executed on an Intel \1 with 4 cores at 2.8 GHz.

• In Solution A, 10 data centers are open, i.e., all of them are used to host servers and software components. That optimization initially favors the delay because the delay penalty is the highest term in the objective function at $ 856.2 M. In this case, all the access nodes have a close data center that serves their requests, and the average delay is 4.4 ms. The total power consumed by all the data centers is 34.6 MW. The data centers are powered with all types of energy, in particular, 4 of them have high pollution emission ratios, between 443 and 960 grams of C0₂ per KWh. The total C0₂ emission is 74,695 ton per year, and the total cost is $ 678.1 M per year.

TABLE 2

Solutions for 10 Cities with up to 10 Data Centers

Solution: A B C

Delay multiplier 1 1 1

C0₂ multiplier 1 1000 1

Cost multiplier 1 1 1000

Delay 4.4 ms 12.2 ms 21.2 ms

Total power 34.6 MW 34.6 MW 34.6 MW

C0₂ emissions 74,695 ton 3,561 ton 41 ,181 ton

Data centers 10 4 3

Delay penalty 856.2 M 2,563.5 M 5,336.2 M

C0₂ penalty 74.7 M 3.6 M 41.2 M Traffic cost 52.4 M 78.0 M 77.0 M

Server cost 200.0 M 200.0 M 200.0 M

CAPEX 329.1 M 135.5 M 77.4 M

OPEX 79.0 M 32.5 M 18.6 M

Energy cost 17.7 M 16.8 M 17.5 M

Total cost 678.1 M 462.8 M 390.5 M

Execution time 0.7 s 0.4 s 178 s

• Solution B is obtained by augmenting the pollution multiplier in the objective function (ip) to 1000. That makes the C0₂ penalty term greater than the delay penalty term. In this case, the optimal solution is to open 4 data centers. Three of the data centers are powered with hydro-electric energy producing 10 grams of C0₂ per KWh, and one of the centers is powered with geothermal energy introducing 38 grams of C0₂ per KWh. The total C0₂ emissions are 3,561 ton per year, 21 -times smaller than Solution A. The C0₂ penalty decreases with the same ratio. The energy consumption remains the same because most of it depends on the number of servers used and the of the data centers, and in these cases, both remain constant. The total cost is 68% of Solution A because fewer data centers are used; therefore, the CAPEX and the OPEX are reduced. The average delay is 12.2 ms, which is 2.8-times greater than Solution A.

• Solution C is obtained by setting the cost multipliers to 1000, making the total cost the first priority to optimize. The number of open data centers is 3, and they have the minimal capacity to accommodate all the servers. The total cost is reduced to $ 390.5 M, the total C0₂ emissions are 41 ,181 tons per year, and the average delay is 21.2 ms.

[0081] From the three solutions, Solution B appears to be a good compromise between the network performance, pollution, and cost because the four chosen data centers imply a short delay with an acceptable cost and very low C0₂ emissions. Using this framework, planners can evaluate multiple alternative solutions and analyze the tradeoffs between them. Furthermore, once the optimal values for each aspect are known, additional constraints can be added to evaluate solutions around the optimal values. The execution time to find these solutions was between 0.7 s and 178 s. We will see how the Cplex execution time increases very quickly as the network growths and how the tabu search performs very well for small and large cases.

B. Comparison between tabu search and Cplex

[0082] We executed the tabu search heuristic and the AMPL-Cplex model in eleven small cases to evaluate the execution time and the solution gaps in the tabu search with respect to the optimal values. The small cases had from 2 to 12 access nodes and from 4 to 24 potential data center locations. In these cases, the data center capacities were set at either 64,000 or 128,000 to be able to host 200,000 servers at a small number of data centers. Table 3 shows the results when the optimization priority is the delay, Table 4 shows the results with pollution minimization, and Table 5 presents the results when the priority was the network cost. All the costs are expressed in millions of dollars. Each case was executed 10 times using the tabu search heuristic to calculate the averages of each value, as well as the minimal, the average, and the maximal gaps. The minimal gaps are between 0 and 1.95%; most of them are below 0.1 %, showing that the heuristic is very effective in finding near- optimal solutions. The last column of the tables exhibits the gap between the value found with the greedy heuristic and the optimal value. In two cases, greedy found the optimal values but most of the cases were above 10%. That reveals that the tabu search is what effectively improves the solution to near optimality.

[0083] The execution time of the tabu search ranged from less than 1 millisecond to 868 milliseconds, whereas Cplex used more than 58 minutes for the most difficult case. For that complex case, with 12 access nodes, 24 data centers and cost minimization, the tabu search heuristic had an average gap of 1.95% and an average execution time of 868 milliseconds. The results also show that the cost-oriented optimization requires more execution time in all the cases. This extra execution time is needed because there are many near-optimal solutions that cannot be discarded until the optimum is found. In the delay-oriented case, the feasible solutions with small delay are those with servers near the users. Once one of those solutions is found, the components will tend to be fixed near the users, and other solutions are discarded, reducing the search space and the execution time. In the case of C0₂ optimization, feasible solutions with low emissions will discard data centers with high emissions. It is not possible to discard solutions in the cost oriented case because many data centers have the same CAPEX and OPEX, resulting in many combinations with similar costs.

C. Large cases

[0084] The tabu search heuristic is very efficient for small and medium problems. We will analyze larger cases, i.e., 50 to 500 access nodes and 100 to 1000 potential data center locations, to gauge the performance of the tabu search heuristic for large problems. Considering a large number of cities makes it possible to guarantee a good quality of service for most of the Internet users around the world. Large networks, such as Akamai, have small data centers in more than a thousand locations around the world to guarantee a high quality of service [5], which is why it is important to analyze the large cases using an efficient heuristic. In these cases, the data center sizes were 40, 80, 160, 1 ,000, 32,000, 64,000, or 128,000 IT elements; solutions may contain multiple small data centers or a combination of small, medium and large data centers. Tables 6, 7 and 8 show the network delay, costs, optimal values, and execution times for these cases. Each reported value is the average of 10 executions.

[0085] The results show that the execution time was less than 1 minute for the cases with up to 200 access nodes and 400 potential data centers, and it was less than 10 minutes in every instance. The delay was held to approximately 5 ms in the delay-oriented optimization by adding more data centers. The smallest case opened 34 data centers, and the largest case opened 396, i.e., 12-times more, but the cost was only multiplied by 4.5 because smaller data centers can be used. However, the C0₂ emissions were high compared to the pollution minimization cases. The average C0₂ in the first table is 99,557 tons, and it is 7,295 tons in the second table. The average delay in the C0₂ optimization only increased to 5.8 ms, which is less than 1 ms more than the delay-oriented optimization. The average cost decreased from $ 3,543.0 M to $ 3,000.5 M, which are both very large compared to the average optimal cost of $ 531.8 M in the third case. In that cost oriented strategy, the average cost was 18% of the cost of the C0₂ strategy, though the average delay increased to 22 ms, more than four-times that in the delay-oriented strategy.

[0086] In these cases, that are too large for Cplex to solve to optimality, we show the performance of the tabu search compared to the initial greedy solution. Here again the results exhibit a similar behavior to the near optimal results achieved in the small instances because the tabu effectively improves the initial solution: the gap between the greedy solution and the best tabu solution is up to 12.77% in Tables 6 and 7, and up to 61 .72% in Table 8. To illustrate the gap, the cost oriented case AN=200, DC=400 had a total cost of $ 494.2 M in tabu search and $ 630.2 M in greedy. The difference can also be seen in Figure 7 where the values obtained by the tabu and greedy algorithms in the cost oriented cases are shown.

[0087] To our knowledge, these results exhibit the most flexible and efficient algorithm for data center location published in the literature. The flexibility is given by the possibility of defining an application graph that matches the forecasted traffic of the applications, and not only the traffic between users and servers. The model is also flexible in the sense of allowing planners to define the priorities in the multi criteria objective function. The efficiency between different approaches is difficult to compare because each one solves a variant of the problem and for different cases. Nevertheless, we can point that a simulated annealing heuristic combined with linear programming was tested in [7] with up to 500 data center locations and execution times of 30 minutes in a similar environment. Our case with 1000 data centers was solved by tabu search in 10 minutes. The solution quality cannot be compared because optimal solutions cannot be guaranteed in neither of the two approaches. We understand that our approach achieves such results because each tabu search iteration is extremely quick compared to expensive linear programming resolutions. That was accomplished by using specialized data structures to calculate the objective function in an incremental way.

Section 7: Conclusion

[0088] Cloud computing is expanding to increasingly more users, applications and devices; therefore, the network needs to grow in an efficient and sustainable way. In this application, we presented a cloud network planning problem through a mixed integer-linear programming model. The location of data centers, servers, and software components impacts the network efficiency, the pollution, and the total cost. The definition of the network link capacities and the information routing is also very important. The proposed model allows planners to evaluate different solutions and to make variations in the optimization priorities.

[0089] The AMPL-Cplex implementation is useful for analyzing small cases, but its algorithmic complexity makes the execution time too high for real instances. The deployment of global networks needs the evaluation of the traffic from many cities around the world, and hundreds of potential data center locations. We addressed that issue by developing an efficient tabu search heuristic. The tabu search algorithm evaluates potential neighbor solutions by calculating the difference from the current solution using the right data structures. The results showed that the tabu search heuristic could find near optimal solutions in a very short execution time. Instances of up to 500 access nodes and 1000 potential data center locations were solved in less than 10 minutes, showing that every real instance can be modeled and solved in this fashion.

[0090] The mathematical model and the algorithm can be extended in multiple directions. One direction is the server virtualization, a key aspect of cloud computing. With that feature, each server can host multiple independent virtual machines. Then, fewer servers are used, reducing costs and energy consumption. Another possible extension is to consider dynamic demands. In that case, the variations of the resource requirements in each hour of the day and on each day of the week are considered; therefore, the solutions make better use of the data centers, servers and link capacities.

[0091] Referring to Figure 8, there is shown a method 10 for locating data centers or points of presence and software components in a cloud computing network, according to a preferred embodiment of the present invention. The method begins by modeling 12 a network using a mixed integer linear programming (MILP) model and an objective function. The next step is minimizing 14 the objective function by means of an optimization solver and a tabu search heuristic. Then next steps is simultaneously determining 16 an optimal location of data centers or points of presence and data software components in the network and simultaneously determining 18 optimal capacities for backbone links that connect data centers or points of presence, the routing and the number of servers in each data centers or point of presence of the network.

[0092] According to the invention, there is also provided a system for locating data centers or points of presence and software components in a cloud computing network, the system comprising: a mixed integer linear programming (MILP) model for execution in a computer including an objective function for modeling said network; an optimization solver for minimizing said objective function using a tabu search heuristic, wherein the solver is configured to: simultaneously determine an optimal location of said data centers or points of presence and said data software components in said network; simultaneously determine optimal capacities for the backbone links that connect said data centers or points of presence, the routing and the number of servers in each said data centers or point of presence of said network.

[0093] According to the invention, there is also provided a system for locating data centers or points of presence and software components in a cloud computing network, the system comprising: a memory configured to store program instructions and data; a processor configured to read the program instructions from the memory, wherein, in response to execution of the program instructions, the processor is operable to: model said network using a mixed integer linear programming (MILP) model and an objective function; minimize said objective function by means of an optimization solver and a tabu search heuristic; simultaneously determine an optimal location of said data centers or points of presence and said data software components in said network; simultaneously determine optimal capacities for backbone links that connect said data centers or points of presence, the routing and the number of servers in each said data centers or point of presence of said network.

[0094] According to the invention, there is also provided a computer-readable memory medium configured to store program instructions for locating data centers or points of presence and software components in a cloud computing network, wherein the program instructions are configured to direct one or more computers to perform operations comprising: modeling said network using a mixed integer linear programming (MILP) model and an objective function; minimizing said objective function by means of an optimization solver and a tabu search heuristic; simultaneously determining an optimal location of said data centers or points of presence and said data software components in said network; simultaneously determining optimal capacities for backbone links that connect said data centers or points of presence, the routing and the number of servers in each said data centers or point of presence of said network. [0095] The scope of the claims should not be limited by the preferred embodiments set forth in the examples, but should be given the broadest interpretation consistent with the description as a whole.

TABLE 3

Comparison Between Tabu Search and Cplex. Delay priority

#AN #DC Tabu(s) Cplex(s) Delay(ms) C0₂(ton) Cost z Min.gap Avg.gap Max. gap GRgap

2 4 0.001 0.032 2.9 6,520 360.0 593.8 0.00% 0.00% 0.00%, 0.00%,

3 6 0.002 0.090 2.6 49,473 385.7 748.6 0.00% 0.00% 0.00% 24.44%,

4 8 0.013 0.287 2.8 101 ,757 402.9 926.3 0.00% 0.00% 0.00%, 22.44%,

5 10 0.023 0.137 2.4 90,729 462.9 1.060.4 0.00% 0.00% 0.00%, 0.38%,

6 12 0.061 0.272 2.2 43,688 484.0 1.059.9 0.02% 0.02% 0.02% 0.02%

7 14 0.071 0.381 2.3 131 ,135 509.9 1 ,279.1 0.02% 0.02% 0.02% 1.64%

8 16 0.165 0.532 2.5 1 0,208 583.0 1 ,477.1 0.02% 0.08%· 0.62% 0.62%

9 18 0.158 0.564 2.2 96,281 629.1 1 ,495.3 0.02% 0.02%. 0.02%, 0.02%,

10 20 0.276 0.719 2.3 1 12,688 606.8 1 ,614.1 0.01% 0.01 %· 0.01 % 0.01 %

11 0.313 0.822 2.5 143,455 626.0 1 ,836.2 0.01% 0.01 %. 0.01 %, 0.01 %,

12 24 0.373 0.996 2.5 36,043 675.5 1 ,820.1 0.01% 0.01 %. 0.01 % 0.01 %

TABLE 4

Comparison Between Tabu Search and Cplex. C0₂ Priority

#AN #DC Tabufs) Cplex(s) Delay(ras) CO , (ton) Cost z Min.gap Avg.gap Max.gap GR.gap

2 4 0.002 0.031 2.9 6.520 360.0 7,138.7 0.00% 0.00%, 0.00% 4.60%

3 6 0.005 0.130 13.5 5.645 408.8 8,085.0 0.00% 0.00% 0.00%, 9.41 %,

4 8 0.022 0.361 8.6 18.178 393.8 19,983.1 0.00% 1.07%, 10.22%, 13.14%,

5 10 0.033 0.336 6.6 6, 108 435.4 7,998.7 0.00% 0.00%, 0.00%, 22.28%,

6 12 0.047 0.354 10.3 4,650 393.4 8,042.6 0.00% 0.96%, 1.59%, 1.59%,

7 14 0.137 1.084 22.9 6,932 416.6 13,509.8 0.00% 1.67% 4.40% 29.38%

8 16 0.247 0.846 7.4 3,427 429.7 6,437.7 0.00%, 0.00%, 0.00%, 9.47%,

9 18 0.230 0.580 16.7 5,151 501 .5 10,339.3 0.00% 0.59% 4.86% 26.63%

10 20 0.458 4.146 11.9 4,040 432.5 8,714.3 0.00%, 0.23%, 0.95%, 15.39%,

11 22 0.303 3.297 7.9 4,149 435.8 8,480.7 0.03%, 0.03%, 0.03%, 0.03%,

12 24 0.559 0.763 4.6 3,817 512.2 6,456.5 0.00%, 0.00%, 0.00%, 0.00%,

TABLE 5

Comparison Between Tabu Search and Cplex. Cost Priority

SAN #DC Tabu(si Cplex(s) Delay(ms) C0₂(1on) Cost z Min.gap Avg.gap Max.gap GR.gap

2 4 0.001 0.045 2.1 170,026 357.6 357,907.8 0.00% 0.00% 0.00% 16.09%

3 6 0.005 0.283 9.9 87,891 362.9 364,469.4 0.00% 0.45% 4.47% 20.02%,

4 8 0.010 1.163 6.8 210,694 364.9 366,263.7 0.00% 0.28% 0.94% 24.85%

5 10 0.033 2.422 10.8 143,705 374.9 377,336.6 0.00% 1 .90% 3.56% 22.88%

6 12 0.076 5.361 9.6 60,234 365.3 368.031.9 0.10% 0.29% 1 .75% 24.28%

7 14 0.124 58.147 9.9 184.332 365.3 368.744.8 0.10% 0.75% 3.19% 1 .42%

8 16 0.205 83.514 10.0 138,276 373.8 377,319.7 0.06%. 1.99% 6.38% 36.51%

9 18 0.288 40.673 8.7 50,595 367.0 370,745.7 0.03%. 2.45% 4.45%, 37.35%

10 20 0.460 1057.181 11.8 75,585 369.8 374,71 1.4 0.03% 2.00% 5.71%, 28.61%

11 i 0.867 1496.597 12.1 141 ,955 375.2 381 ,295.8 1.05% 3 27% 7.02% 29.95%

12 24 0.868 3482.393 13.5 59,401 380.5 387,864.8 1.95% 4.53% 5.74% 30.69%, TABLE 6

Tabu Search Results for Large Cases. Delay Priority

#AN #DC Exec.(s) #DC open Iterations Delay(ms) CO (ton) Total cost z GR.gap

50 100 0.4 34 405 3.9 91 ,330 1 ,188.3 8,762.1 0.00%

100 200 2.5 73 908 3.6 106,711 1 ,833.2 14,594.4 7.06%

150 300 5.7 117 1,212 3.9 108,790 2,611.4 21 ,098.7 0.85%

200 400 12.1 137 1,669 4.4 99,655 2,976.8 27,455.9 3.81%

250 500 20.8 199 2,149 4.7 79,202 3,735.3 32,103.8 3.35%

300 600 32.3 218 2,631 5.0 97,388 3,597.0 38,335.7 5.03%

350 700 45.6 261 3,061 5.3 103,037 4,201.3 43,899.6 4.67%

400 800 66.1 306 3,681 5.7 101 ,242 4,859.2 49,963.2 10.34%

450 900 85.4 356 4,047 6.2 92,121 5,078.5 54,795.5 10.05%

500 1000 117.5 396 4,718 6.6 116,096 5,349.3 60,327.2 12.41%

TABLE 7

Search Results for large Cases. Pollution Priority

#AN #DC Exec.(s) #DC open Iterations Delay(ms) CO (ton) Total cost z GR.gap

50 100 0.7 20 623 5.9 5,714 874.8 18,156.8 8.44%

100 200 2.2 52 806 4.8 5,344 1 ,259.0 23,013.4 0.74%

150 300 6.2 69 1302 5.0 6,982 1 ,848.2 33,031.9 6.37%

200 400 12.7 93 1744 5.1 7,500 2,330.5 40,687.0 4.05%

250 500 18.9 138 2027 5.3 5,746 2,765.8 42,423.4 0.67%

300 600 31.1 164 2658 5.5 7,171 3,183.9 50,408.5 3.40%

350 700 46.5 189 3094 6.0 7,620 3,705.2 58,609.4 3.41%

400 800 67.0 243 3712 6.4 8,609 4,267.8 65,433.0 7.98%

450 900 86.2 281 4182 6.8 8,325 4,737.6 69,195.0 12.77%

500 1000 112.4 314 4582 7.6 9,944 5,032.5 77,003.9 7.59%

TABLE 8

Tabu Search Results for Large Cases. Cost Priority

#AN #DC Exec.(s) #DC open Iterations Delay(ms) C0₂(ton) Total cost z GR.gap

50 100 1.4 7 1,295 17.6 101,729 426.2 463,528.1 33.84%

100 200 9.6 29 3,342 24.8 101,188 466.2 544,064.0 46.72%

150 300 19.7 44 3,905 22.3 82,974 470.9 590,651.9 46.45%

200 400 52.1 52 6,762 23.4 96,500 494.2 662,006.5 61.72%

250 500 126.8 85 12,418 18.5 149,779 505.1 637,902.5 56.34%

300 600 1 0.3 122 15,330 21.1 123,173 528.7 695,767.9 58.15%

350 700 258.3 176 17,098 22.6 116,601 554.6 795,113.5 50.70%

400 800 459.0 208 24,552 21.6 59,005 584.5 781,621.2 29.18%

450 900 467.2 296 21,193 23.6 76,514 617.3 863,840.6 13.77%

500 1000 553.8 359 21,329 26.1 134,860 670.2 951,827.6 16.10% REFERENCES

1. M. Armbrust, A. Fox, A. Joseph, R. Katz, A. Konwinski, G. Lee, D. Patterson, A.

Rabkin, I. Stoica, and M. zaharia, "Above the clouds: A Berkeley view on cloud computing," University of California at Berkeley, Tech. Rep. UCB/EECS-2009-28, 2009.

2. R. Brown, E. Masanet, B. Nordman, B. Tschudi, A. Shehabi, J. Stanley, J. Koomey, D.

Sartor, P. Chan, J. Loper, S. Capana, B. Hedman, R. Duff, E. Haines, D. Sass, and A. Fanara, "Report to congress on server and data center energy efficiency," Public law, vol. 109, p. 431 , 2007.

3. R. Buyya, R. Ranjan, and R. Calheiros, "InterCloud: utility-oriented federation of cloud computing environments for scaling of application services," in Proc. 10th International Conference on Algorithms and architectures for parallel processing (ICA3PP). Springer, May 2010, pp. 328-336.

4. F. Larumbe and B. Sanso, "Location and dynamic provisioning problems in cloud computing networks," in Communication infrastructures for cloud computing: design and applications, M. Hussein and B. Kantarci, Eds. IGI Global, to be published.

5. E. Nygren, R. Sitaraman, and J. Sun, "The Akamai network: a platform for high- performance internet applications," ACM SIGOPS Operating Systems Review, vol. 44, no. 3, pp. 2-19, 2010.

6. S. Chang, S. Patel, and J. Withers, "An optimization model to determine data center locations for the army enterprise," in World's Premier Military Communications Conference (MILCOM), Oct. 2007, pp. 1-8.

7. ί. Goiri, K. Le, J. Guitart, J. Torres, and R. Bianchini, "Intelligent placement of datacenters for internet services," in 31st International Conference on Distributed Computing Systems (ICDCS 2011), Minneapolis, Minnesota, USA, Jun. 201 1.

8. X. Dong, T. El-Gorashi, and J. Elmirghani, "Green IP over WDM networks with data centers," Journal of Lightwave Technology, vol. 29, no. 12, pp. 1861-1880, 201 1.

9. F. Larumbe and B. Sanso, "Cloptimus: a multi-objective cloud data center and software component location framework," in 1st IEEE International Conference on Cloud Networking (CLOUDNET), Nov. 2012, pp. 23-28.

10. M. Covas, C. Silva, and L. Dias, "Multicriteria decision analysis for sustainable data centers location," International Transactions in Operational Research, vol. 20, no. 3, pp. 269-299, May 2013. H. Stone, "Multiprocessor scheduling with the aid of network flow algorithms," IEEE Transactions on Software Engineering, vol. SE-3, no. 1 , pp. 85-93, 1977.

L. Rao, X. Liu, L. Xie, and W. Liu, "Minimizing electricity cost: Optimization of distributed internet data centers in a multi-electricity-market environment," in Proc. IEEE INFOCOM, 2010, pp. 1-9.

N. Chowdhury, M. Rahman, and R. Boutaba, "Virtual network embedding with coordinated node and link mapping," in Proc. IEEE INFOCOM, 2009, pp. 783-791. A. Geoffrion and R. Bride, "Lagrangean relaxation applied to capacitated facility location problems," AIIE Transactions, vol. 10, no. 1 , pp. 40-47, 1978.

A. Klose and S. Gortz, "A branch-and-price algorithm for the capacitated facility location problem," European Journal of Operational Research, vol. 179, no. 3, pp. 1 109-1 125, 2007.

H. Pirkul, "Efficient algorithms for the capacitated concentrator location problem," Computers & operations research, vol. 14, no. 3, pp. 197-208, 1987.

Z. Drezner and H. Hamacher, Facility location: applications and theory. Germany: Springer, 2004.

M. Plante and B. Sanso, "A typology for multi-technology, multi-service broadband network synthesis," Telecommunication Systems, vol. 19, no. 1 , pp. 39-73, 2002.

F. Larumbe and B. Sanso, "Optimal location of data centers and software components in cloud computing network design," in IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing, Workshop on Cloud Computing Optimization, CCOPT 2012, May 2012, pp. 841-844.

A. Beloglazov, R. Buyya, Y. C. Lee, A. Zomaya et ai, "A taxonomy and survey of energy-efficient data centers and cloud computing systems," /Advances in Computers, vol. 82, no. 2, pp. 47-1 1 1 , 201 1.

B. Sovacool, "Valuing the greenhouse gas emissions from nuclear power: a critical survey," Energy Policy, vol. 36, no. 8, pp. 2950-2963, 2008.

L. Barroso and U. Holzle, "The datacenter as a computer: An introduction to the design of warehouse-scale machines," Synthesis Lectures on Computer Architecture, vol. 4, no. 1 , pp. 1-108, 2009.

F. Glover, "Tabu search— part I," ORSA Journal on Computing, vol. 1 , no. 3, pp. 190- 206, 1989.

, "Tabu search— part II," ORSA Journal on Computing, vol. 2, no. 1 , pp. 4-32, 1990. A. Hertz and D. Werra, "Using tabu search techniques for graph coloring," Computing, vol. 39, no. 4, pp. 345-351 , 1987.

T. Brinkhoff, "City population," Jan. 2013. [Online]. Available: http://www.citypopulation.de/

Internet World Stats, "Usage and population statistics," Jan. 2013. [Online]. Available: http://www.internetworldstats.com

Google, "Google Map," Jan. 2013. [Online]. Available: http://map.google.com

C. Labovitz, S. lekel-Johnson, D. McPherson, J. Oberheide, and F. Jahanian, "Internet inter-domain traffic," in ACM SIGCOMM Computer Communication Review, vol. 40, no. 4, 2010, pp. 75-86.

Telegeography, "Executive summary," Jan. 2013. [Online]. Available: http://www.telegeography.com

Claims

CLAIMS:

1 . An optimization method for locating data centers or points of presence and software components in a cloud computing network, the method comprising:

modeling said network using a mixed integer linear programming (MILP) model and an objective function;

minimizing said objective function by means of an optimization solver and a tabu search heuristic;

simultaneously with said minimizing, determining an optimal location of said data centers or points of presence and said software components in said network;

simultaneously with said minimizing, determining optimal capacities for backbone links that connect said data centers or points of presence, the routing and the number of servers in each said data centers or point of presence of said network.

2. The method of claim 1 , wherein the objective function comprises metrics selected from: traffic delay, energy consumption, C02 emissions, traffic cost, server cost, data center capital expenditures (CAPEX), and data center operational expenditures (OPEX).

3. The method of claim 1 , wherein the objective function is defined by the following formula:

z = ac^D + fic^T + yc^s + 5c^c + sc° + ηο^Ε + yc^c°²

f

where c^D = c_d' 1000 t_e ^f- is the delay penalty cost, c^T = ^ c²s_k + ^ (c³ + c_e ⁴)b_e is the network traffic cost, c^s = ^ c⁵z_t is the server in the network cost,

^■≡v^N

c_h ⁶y_h is the data center capital expenditure (CAPEX) cost,

:≡r^N

c° data center operational expenditures (OPEX) cost,

- 24h - l0^~6)w_k ⁵ is the data center energy cost, ∑ c_k ⁹(w_k ⁶ - I0^~6)(w_k · 365 · 24η · 10^~3)νν£ is the C0₂ emission penalty cost, where ^α' β δ ε η ψ _are parameters for each cost,

and where

b Throughput of demand d e A^A in peak time (in bps) b² Capacity per link channel (in bps)

_Ue Maximal utilization ratio per channel in link e e A^N

t_e Delay bound for link e e A^N (in seconds per packet) while its traffic is less than the allowed utilization

K Set of resources that each server has, which include 1 = CPU , 2 = network card, 3 = RAM, and 4 = storage

r)_p Average consumption of resource p e K required by a copy of component i e V^A , in the correspondent units

rj_p Peak consumption of resource p <≡K required by a copy of component Number of replicas of software component i

4

r_P Capacity of resource p < K on a server

5

k Capacity of data center k e T> in number of servers

6

r Number of servers per Local Area Network (LAN) switch

1

w The power consumption of a server as a function of the utilization of each resource (in watts)

2

w Network switch power consumption (in watts)

3

W Router port power consumption (in watts)

Maximum power capacity of data center k e T> (in watts) for IT equipment. Power Usage Effectiveness (PUE) of data center k e T⁾ (ratio of total power

/ IT equipment power)

C0₂ emissions in data center k e T> (in g / KWh)

w] Average power consumption of a server with a replica of software component i e V^A

w* Peak power consumption of a server with a replica of software component

Cd Penalty for delay in demand d e A^A (in $ / (ms/packet) / year)

c² Cost of a LAN switch (in $/year)

c³ Cost of a router port (in $/year)

Cost of a channel in link e e A^N (in $/year)

5 Cost of a server (in $/year)

6

Ck CAPEX for using data center k e T> (in $/year)

7

Ck OPEX for using data center k e T> (in $/year)

C Electricity cost in data center k e T> (in $/MWh)

Penalty for emitting C0₂ in data center k & V (in $/ton)

xik 1 if software component i e V^A is located in a network node k e V^N otherwise

y_k 1 if node k e V^N hosts one or more software components; 0 otherwise zk Number of servers hosted in data center k e T) sk Number of LAN switches in data center k e T>

f _de Amount of demand d e A^A traffic carried by link e e A^N (in bps) be Number of channels, e.g., wavelengths or fibers, in link e e A^N wk Average power consumption of IT equipment in node k e V^N

4. A system for locating data centers or points of presence and software components in a cloud computing network, the system comprising:

a mixed integer linear programming (MILP) model for execution in a computer including an objective function for modeling said network;

an optimization solver for minimizing said objective function using a tabu search heuristic, wherein the solver is configured to:

simultaneously determine an optimal location of said data centers or points of presence and said data software components in said network;

simultaneously determine optimal capacities for the backbone links that connect said data centers or points of presence, the routing and the number of servers in each said data centers or point of presence of said network.

5. A system for locating data centers or points of presence and software components in a cloud computing network, the system comprising:

a memory configured to store program instructions and data; a processor configured to read the program instructions from the memory, wherein, in response to execution of the program instructions, the processor is operable to:

model said network using a mixed integer linear programming (MILP) model and an objective function;

minimize said objective function by means of an optimization solver and a tabu search heuristic;

simultaneously determine an optimal location of said data centers or points of presence and said data software components in said network;

simultaneously determine optimal capacities for backbone links that connect said data centers or points of presence, the routing and the number of servers in each said data centers or point of presence of said network.

6. A computer-readable memory medium configured to store program instructions for locating data centers or points of presence and software components in a cloud computing network, wherein the program instructions are configured to direct one or more computers to perform operations comprising:

modeling said network using a mixed integer linear programming (MILP) model and an objective function;

minimizing said objective function by means of an optimization solver and a tabu search heuristic;

simultaneously determining an optimal location of said data centers or points of presence and said data software components in said network;

simultaneously determining optimal capacities for backbone links that connect said data centers or points of presence, the routing and the number of servers in each said data centers or point of presence of said network.