WO2017141071A1

WO2017141071A1 - Route optimization for intra- or inter-data centers communication

Info

Publication number: WO2017141071A1
Application number: PCT/IB2016/050858
Authority: WO
Inventors: Farnaz MORADI; Ahmad ROSTAMI
Original assignee: Telefonaktiebolaget Lm Ericsson (Publ)
Priority date: 2016-02-17
Filing date: 2016-02-17
Publication date: 2017-08-24

Abstract

A method for re-routing communication, between two software entities, intra- or inter-data center networks (DCNs). The method includes receiving (900) at a route optimizer entity (230) a list (750) of software entity pairs that experience performance degradation, wherein the list (750) is sent by a data center controller (202/220) of the DCNs, receiving (902) at the route optimizer entity (230) a topology of the DCNs and routing information associated with the software entity pairs; calculating (904), in a path computation element (232), a new route for a first software entity pair of the software entity pairs from the list (750), based on the topology and routing; information of the software entity pairs, to alleviate the performance degradation; and sending (906) the new route to the data center controller (202/220) for modifying an existing route of the first software entity pair.

Description

Route Optimization for Intra- or Inter-Data Centers Communication

TECHNICAL FtE D

I0001J The present invention generally relates to a method and apparatus that re-routes software entities intra- or inter-data centers communication and, more particularly, to mechanisms and techniques for optimizing routes in the data center network.

BACKGROUND

00021 The number of computing devices in use today has increased dramatically over the past years, and with them, the amount of content that is stored in the cloud and shared or used by these devices. Cloud networking was developed to address the need of running these application in data centers, away from the users.

Q03J However, modern, large-scale, data center networks (DCNs) are comprised of thousands of network devices and links. This means that the internal DCN traffic is significant, and the likelihood of bottlenecks is large, which can lead to service interrupts or violation of service-level agreement (SLA) performance guarantees.

00041 The introduction of new technologies such as Network Functions Visualization (NFV) and Software-Defined Networking (SON) has changed the way data is processed and forwarded in a data center. NFV exploits virtualization techniques to decouple network functions from the underlying hardware and SDN simplifies the implementation and management of NFV by separating the network control and data p!anes.

[0006| in current data centers, an example of which is illustrated as center 100 in Figure 1 , a DC or Cloud Controller 102 manages and orchestrates Virtual Network Functions (VNF) 104 and 108. Monitoring functions 105 and 107 associated with VNFs 104 and 106 are also shown being located in physical servers 108 and 1 10. However, the monitoring functions 105 and 107 may be located in other parts of the center and they can be achieved with different methods, for example, sF!ow, passive monitoring of VNF traffic, etc. A network interface 1 12 or 1 14 connects a

corresponding physical server to various network elements 1 16 (e.g., router). The DC controller is responsible for resource allocation associated with the VNFs and also for placement and migration decisions regarding the VNFs based on the available resources. The existing DC controllers try to optimize different objective functions., such as reducing energy consumption in the DC, Figur 1 also shows a network controller 120 (e.g., SDN controller) that monitors and controls the network elements 1 16.

[00061 Network bottlenecks, such as congestion, is currently addressed in a DCN in two different ways: {1 ) the SDN controller 120 updates the routing in DON 100, or (2) DC controller 102 changes the- placement of VNFs 104 and 108. These two approaches are typically performed independently.

[8O07J Network routes in an SD -based DCN can be computed by a Path Computation Element (PCE). According to RFC 4655 (Parrel, A., Vasseur, J., -P. Ash, J. (2008), A Path Computation Element (PCE)-based architecture, internet Engineering Task Force RFC 4655, https^ tools.ietf.org/html/rf 04655), PCE is "an entity {component, application, or network node) that is capable of computing a network path or route base on a network grap and applying computational constraints." PCE can perform Constraint-Based Routing (CBR.) which is a fundamental building block for traffic engineering systems. CBR refers to a set of routing algorithms that select an optima! routing path (to reduce costs, balance network load , etc.) while satisfying some given constraints.

[00081 V F or Virtual Machine (VfVi) placement is typically performed by the DC controller by only considering compute resources and independent of network requirements and routing. Placing V Fs without taking the network into account can lead to inefficient network usage and to the creation of network bottlenecks. To overcome these limitations of the existing orchestration methods, a variety of network-aware orchestration methods have been proposed. These methods mostly aim at placing closer to each other VMs that communicate frequently with each other.

[00091 Recently, methods for joint optimization of V^'lvl placement and routing have received attention and have shown good performance. In such methods, placement and migration decisions are made by considering both compute resources and network load in order to, for example, reduce energy consumption, by avoiding unnecessary network traffic.

[00101 Network- and traffic-aware VM placement solutions aim at placing VMs so that the ones that frequently communicate with each other are placed closer to each other. However, it is challenging to accurately determine the amount of traffic flowing between VMs, particularly in a cloud environment where VMs are dynamically provisioned in the cloud. Moreover, if the placement decision is made only based on the amount of traffic exchanged between two nodes and other metrics are ignored, it can lead to increased latency perceived by a service. Some of these solutions are now discussed.

[0011J Meng et a!. ( mproving the Scalability of Data Center Networks wit Traffic-aware Virtual Machine Placement," IEEE INFOCOM, 2010) defined the

Traffic-aware VM Placement Problem (TVMPP) and used a traffic mairix among VMs and the cost matrix among host machines as inputs to this optimization problem, [00121 Aaron Gember ("Stratos: A Network-Aware Orchestration Layer for Virtual Middleboxes in Clouds/' 2013, ArXsv.) introduced a network-aware orchestration layer for virtual middieboxes in clouds, called Stratos. Stratos performs application-specific, network -aware scaling using three different mechanisms for identifying and addressing network bottlenecks. The initial VM placement is performed by identifying partitions of VMs. Then a lightweight "flow distribution" method is used to address transient network congestions, and migration and scaling are used for addressing more persistent network bottlenecks. Stratos assumes that resource statistics and network utilization statistics are provided by an existing monitoring infrastructure.

[00131 Typically, VM placement and routing for DON are performed separately. Recently, some authors have proposed methods for joint optimization of compute and network resources and have shown that a joint optimization can substantially improve efficiency. Jiang et ai, (Joe Wenjie Jiang, Tian Lan, Sangtae Ha, iVIinghua Chen, Mung Chiang, "Joint Vfvl placement and routing for data center traffic engineering," IEEE !NFOCOM, 2012) has defined an optimization problem as a joint control of routing and VM placement (V PR) for optimizing the network performance and cost.

[0014| The main focus of such joint methods is on the initial placement of V s by performing joint route selection and VM placement in order to avoid congestions and performing dynamic cost-effective VM migrations when congestions happen. However,, the migration of VNFs is costly. In addition, it is difficult to predict the best routing between various VNFs as these functions and their traffic patterns are continuously changing.

[001 S] A large body of work on detecting and localizing bottleneck links across network and services exists. While these methods are focused on finding the locations of performance degradation in networks, the locations change in time as new VNFs are created and older VNFs are suppressed.

[00161 A commonly used technique for localization of link-level bottlenecks is network tomography, where statistical methods are applied to end-to-end

measurements obtained from active probing methods (e.g., trace-route). Many existing tomography methods have only been applied to small networks due to the complexity of their algorithms and the prohibitive cost of the deployed active measurement techniques. Such an example is described In Herodoiou et al

(Rerodotou, Berodotos and Ding, Boiin and Baiakrishnan, Shobana and Outhred, Geoff and Fitter, Percy, "Scalable Near Real-time Failure Localization of Data Center Networks," International Conference on Knowledge Discovery and Data Mining (KDD), 2014).

[G017J Johnsson ef at, (Andreas Johnsson, Catalin eirosu and Christofer Rinta, "Online Network Performance Degradation localization using Probabilistic Inference and Change Detection," Network Operations and Management

Symposium (NOMS), 2014) proposed a fault detection and localization algorithm for network performance degradation. The proposed method uses actsve measurements and probabilistic inference and change detection to identify the location of performance degradation in fixed telecommunication networks.

[001 SI Barford (Barford, P, Outfield N., Ron, A., Sommers, J., "Network Performance Anomaly Detection and Localization," IEEE I FOCOM, 2009) proposed a framework for network performance anomaly detection and localization, which includes three components: an algorithm for detecting performance anomalies on a path, an algorithm for selecting which paths to probe (using active

measurements) , and an algorithm for localizing the problematic links.

[0019] Herodotou et a! use statistical data mining techniques for identifying fault location within a DC or between DCs. The input to the proposed localization algorithm is the topology of the network and ping data, which specifies how many pings succeeded and how many failed. The proposed solution consists of 4 phases, (1) the most likely ping routes and their probabilities are computed, (2) a ranked list of edges and nodes with failure scores, which are obtained from a probabilistic ^■failure modeling, are generated, (3) a statistical hypothesis testing is used for detecting the deviations to filter out the noise, and (4) the produced probabilistic results are correlated with passive monitoring data obtained from e.g., Simple Network Management Protocol (SN P) and NetFlo to localize the device or the link that Is the caus of the problem.

[00201 Stratos detects bottienecked virtual links between middiebox.es by identifying the underlying physical links, for example using traceroute, and monitoring the physical links by gathering metrics from all physical network switches. However, using tools like traceroute is costly and generating tracerouies for covering large-scale datacenters is expensive.

[0021] Problems associated with ali these methods are now discussed., in most existing solutions, routing in DCN is achieved independently, and without communicating with the cloud environment and considering the expected network performance (or guarantees) of VNFs. Joint optimisation of V placement and routing Is performed while only focusing on VM Initial placement and VM migration.

[00221 However, traffic patterns of VNFs dynamically change overtime.

Predicting the actual traffic pattern experienced in operation is extremely difficult, which means that it is necessary to dynamically adapt to the changes and update the optimized placement of VNFs, which can be very costly.

[0023J Solutions for network-aware migration of VMs typically use monitoring data and the traffic matrix to Identify network congestion and determine which VMs should be migrated, so that the distribution of traffic in the network is optimized. However, performing unnecessary migration of VMs leads to inefficient resource usage, and can increase the overall energy consumption in a DC, etc,

[00241 Thus, there is a need to provide a ne method for addressing the network congestion without migrating VMs, or delaying as much as possible the migration of the Vivls. SUMMARY

[80251 As the data centers environments become more complex and are more flexible in terms of their hardware and software configurations, there is a need to prevent network communication degradation due to VNFs pairs or other software entity pairs using the same links of the network.

[0Q2SJ Thus, in one embodiment, a method is presented for re-routing communication, between two software entities, intra- or inter-data center networks.

The method includes a step of receiving, at a route optimizer entity, a list of software entity pairs that experience performance degradation. The list may be sent by a data center controller of the DCNs. The method further includes a step of receiving, at t e route optimizer entity, a topology of the DCNs and routing information associated with the software entity pairs, and a step of calculating, in a path computation element, a new route for a first software entity pair of the software entity pairs from the list, based on the topology and routing information of the software entity pairs, to alleviate the performance degradation. Furthermore, the method includes a step of sending the new route to the data center controller for modifying an existing route of the first software entity pair.

[00271 another embodiment, there is a route optimizing module for rerouting a communication path, between two software entities, intra- or inter-data center networks. The route optimizing module includes a route engine that receives a list of software entity pairs thai experience performance degradation, wherein the list is sent by a data center controller of the DCNs, and a topolog of the DCNs and routing information associated with the software entity pairs. The route optimizing module also includes a path computation element that calculates a ne route for a first software entity pair of the software entity pairs from the list, based on the topology and routing information of the software entity pairs, to alleviate the performance degradation. The route engine sends the new route to the data center controller for modifying an existing route of the first software entity pair.

[0028] In sill! another embodiment, there is a method for calculating, with a path computation element, a new route, between two software entities, intra- or inter- data center networks. The method includes a step of generating at a route engine an overlapping link list that includes Inks between software entity pairs that overlap, a step of defining at the route engine routing constraints using a list of software entity pairs and the overlapping link list, a step of calculating the new route at the path computation element based on information received from the route engine, a step of sending from the path computation element to route engine an indication of whether the new route has been found, and a step of sending from the route engine to a network controller either the new route or a message that no new route has been found.

[QQ29J Thus, it is an object to overcome some of the deficiencies discussed in the Background section and to provide a method and system in which th routes between VNFs are monitored. One or more routes are recalculated if they are violating certain criteria. One or more of the embodiments discussed herein advantageously provides such a mechanism.. BRIEF DESCRIPTION OF TOE DRAWINGS

[00301 The accompanying drawings, which are incorporated in and constitute a part of the specification, iustrate one or more embodiments and, together with the description, explain these embodimenis. In the drawings:

[0031] Figure 1 is a schematic diagram of a data center;

£003 3 Figure 2A is a schematic diagram of a data center having a route optimizer entity that communicates with a DC controller and an SDN controller;

[00331 Figure 2B is a schematic diagram of a data center having a route optimizer entity that communicates with a DC controller;

[00341 Figure 2C is a schematic diagram of a data center having a route optimizer entity and a i computation engine that is not part of the route optimizer entity;

£00363 Figure 3 is a flowchart of a method for calculating a new route for VNFs;

[00361 Figure 4 is a signal diagram of commands exchanged by various entities of the data center for calculating the new route;

[0037} Figure 5A is a schematic diagram of a route optimizer engine;

[00381 Figure 5B is a schematic diagram of another route optimizer engsne;

[0039J Figure 6 is a flowchart of a method for determining overlapping links in the data center;

[00401 Figures 7A-F illustrate how the method for calculating the new route works;

[0041} Figures 8A-J illustrate how a path computation engine calculates the new route and interacts with a route optimizer engine; [00421 Figure 8 is a flowchart of a method for calculating a new route between VNFs;

$043} Figure 10 is a flowchart of another method for calculating a new route between V Fs; and

[0044] Figure 1 1 is a schematic diagram of a computing device.

DETAILED DESCRIPTION

10045] The following description of the embodiments refers to the accompanying drawings, The same reference numbers in different drawings identify the sam or similar elements. The following detailed description does not limit the invention, instead, the scope of the invention is defined by the appended claims. The following embodiments are discussed, for simplicity, with regard to the terminology and structure of a data center and associated data center network and VNfs communications. However, the embodiments discussed herein are not limited to a data center and/or a data center network or to V Fs, but they may be applied to multiple data centers or other large systems that involve hardware running one or more network functionalities and also to communications between other software entities than VNFs (for example, in the SViapReduce system, which is used for parallel and distributed data analysis, the map functions and the reduce functions are the software entities that communicate wth each other; another example is a distributed database application where different database instances communicate with each other ; those skilled in the art would known of other examples of such software entities}.

[0O46J Reference throughout the specification to "one embodiment^1' or "an embodiment^* means that a particular feature, structure or characteristic described in connection with an embodiment is included in at least one embodiment of the present invention. Thus, the appearance of the phrases in one embodiment" or in an embodiment" in various places throughout the specification is not necessarily all referring to the same embodiment. Further, the particular features, structures or characteristics may be combined in any suitable manner in one or more embodiments or claims, [00471 According to an embodiment, there is a method and apparatus that enables inter- VNF route optimization in a DCN by acquiring monitored network performance metrics for VNF pairs from DC controSier/ofchestrator and network topology and route information from SD controller. In one embodiment, the DC controller and the SDN controller are the sam entity. The bottleneck links in the DCN are identified by correlating the monitored data on VNFs with the networking information (actual topology and routing). The detection of bottleneck Sinks— hich can potentially affect VNFs performance guarantees—is used to formulate traffic routing constraints, which in turn trigger the rerouting of affected VNFs' traffic to avoid the bottleneck links.

[0048] According to an embodiment route optimization in DCN is achieved by using both SDN: and Cloud Controllers input. This input ma include monitored network performance metrics (provided by either of the SDN controller or DC controller) and/or expected network performance of VNFs (provided by the DC controller), in one embodiment, the monitored network performance metrics and the expected network performance are provided by the DC controller, in one application, the DC controller may be implemented as a SDN controller. Those skilled in the art would know thai other data may be used for the input of the route optimization. The mute optimization, as discussed later, prevents service interruptions and perfomiance violations, thus reducing the cost by reducing the unnecessary complex and expensive migrations of VMs containers.

[00491 According to another embodiment,, the novel process is "blind" to the measurement method being used in the cloud environment. Regardless of what type of monitoring solution is deployed, e.g. passive or active, the process uses the monitoring data for identifying bottlenecks and formulating routing constraints. The defined constraints can be used for both prevention of violation of expected network performance as well as addressing network performance problems, which are identified by other means.

[0050] The process dynamically adapts to the dynamic changes in the traffic patterns of VNFs, thus achieving the same effect with the existing methods (which are based on VM placement optimization), but without VM placement optimization, which is a challenging and complicated task. The process can be implemented, in one embodiment, as a pluggable module (software, hardware or a combination of the two) for the SDN controller to provide input to PCE.

[0051! In other words, according to an embodiment, the process identifies the links between VNF pairs that are possible bottlenecks and solve the performance degradation problem by re-routing VNF traffic to avoid these Sinks, in a timely manner, before these links cause service interruptions or performance (e.g.. SLA) violations, in one embodiment, it is assumed that if piural VNF pairs are experiencing a similar problem and their traffic passes through a shared Sink, the shared link is a potential bottleneck.

00S2] Figure 2A shows the overall system architecture of a DCN 200 that is similar to that shown in Figure 1 , except for the a new functionality 230. A description of the elements already discussed with regard to Figurel is omitted herein, DCN 200 includes a Route Optimizer (RO) entity 230, which could be implemented as a component, a controller, application, or a network node. The RO entity 230 may be located anywhere in the DCN, and it can be supported by the same hardware that supports the SDN controller 220 and/or DC controller 202. in this embodiment, the RO entity 230 interacts with both the DC controller 202 and SDN controller 220, as illustrated in Figure 2A. However, it Is possible, as illustrated Sn Figure 2B, to have a single controller (DC controller 202/220) that acts as SDN controller 220 and DC controller 202 in Figure 2A. This means thai RO entity 230 receives all the above- discussed information from DC controller 202/220.

[0053] Returning to the embodiment illustrated in Figure 2A, the RO entity 230 receives a VNF list 2Q2A from the DC controiler 202 and topology and VNF routes 220.A from the SDN controller 220, The inputs received from controllers 202 and 220 may include supplemental information, as discussed later. A PCE engine 232 for performing constraint-based routing may be Βύάβ to DCN 200 and this engine may be implemented as a separate entity as illustrated in Figure 2C or as a component of the RO entity 230 as illustrated in Figures 2A-2B.

0S4 In the embodiments of Figures 2A-28, the RO entity 230 includes an RO engine 234 and the PCE engine 232. The RO engine and RO entity are used interchangeably in the following. However, in the embodiment of Figure 2C, the PCE engine 232 is separate from the RO entity 230, and thus, a route optimizing module 233 may be considered to define the RO engine 23 and the PCE engine 232. These engines may be implemented as a set of software commands in a computing device, which is discussed later with reference to Figure 10. irrespective of the embodiment, the PCE engine 232 and the RO entity 230 communicate with each other to obtain a new route for various VNF pairs, as discussed later. The PCE engine is responsible for computing new network routes. The RO engine 234 can request the PCE engine to calculate a new route. Alternatively, the SDN controller can request a new route from the PCE engine, independent from the RO engine, as illustrated by direct link 236 in Figure 2A. [0055] DC Controller 202, also known as Cloud Controller or Cloud

Orchestrator, manages and orchestrates the VNFs In the DC. DC controller makes placement decisions for V'NFs in different service chains, performs resource allocation, and makes orchestration decisions such as scaling and/or migration of VNFs. The DC controller receives monitoring data about sen/ers/VMs/containers and the network performance metrics from a monitoring framework module 203, and can use this data for orchestration purposes,

[0QS8] in the embodiment of Figure 2A, RO entity 230 receives two inputs. Trie first input 202A is from the DC controller 202, and includes (1) a list of VNF pairs (identified, for example, through their IP addresses) and (2) their corresponding network performance metrics (e.g., bandwidth, latency, etc.). The network

performance metrics may be obtained by any know method In the art, for example, through a monitoring framework such as the passive container monitoring framework described in international patent application PCT/EP2015/057206. The minimum Input received from the DC controller in this embodiment is the list of V F pairs. However, in the embodiment illustrated in Figure 2B, the input received from the DC controller further includes data center's topology and VNF routes. The VNF pairs 1st may include info regarding the sensitivity of the pairs to violation of a performance metric and how close they are to exceeding the acceptable thresholds for that metric. One possibility to create the list is to check if the observed metric Is In the 90-percentiie (or another number) of the metric distribution. Alternatively, a prediction algorithm can be used to estimate the probability of metrics exceeding their threshold values.,

[0Οδ7] The second input 220A to the RO entity Is from the SDN controller 220 (for the embodiment of Figure 2A} and includes the data center topology. The data center topology may be in the form of a graph. The second input 220A may also include the actual routing information for the VNF pairs that are being considered. To receive this information, in one embodiment the RO entity queries the SDN controller. The same is true for the first input, in one embodiment, the first and second inputs are automatically sent by the respective controllers. In the embodiment of Figure 2B, the RO entity queries oniy the DC controller 202/220 for receiving the above-discussed information.

QS8J The RO entity (in the embodiments of Figures 2A-2B) may continuously execute a route optimization method using the above inputs, as now discussed with regard to Figure 3. More specifically, in the embodiment illustrated in Figure 3, the RO entit identities In step 300 network Sinks that are possible cause of network bottlenecks and performance problems and formulates in ste 302 constraints for computing new routes. The RO entity then sends the information about VNF pairs (e.g., IP addresses) together with the constraints to the PCE engine, to trigger and calculate (in sfep 304} traffic re-routing according to the formulated constraints. As discussed above, the PCE engine may be part of the RO entity as Illustrated in Figures 2A-2B. or It may be independent of the RO entity, as illustrated in Figure 2C. In step 306, the results of these calculations are sent to the DC controller for implementation. The PCE engine can send this information directl to the DC controller or the RO entity can perform this step.

|00S9J in this way, the RO entity or PCE engine provides re-routing traffic information to the DC controller so that unnecessary,, expensive, VNF migrations are avoided. However, if no new route can satisfy the constraints (according to feedback from the PCE engine), the RO engine can inform the DC controller about the network problem. Thus, a feedback message is sent back to th DC Controller to provide the new route or to inform the DC controller that no new route has been found. This feedback can be used by the orchestrator for making decisions, such as scaling out or migrating the VNFs thai are facing network problems.

[0080] A signaling diagram associated with the method illustrated in Figure 3 is shown in Figure 4 and it depicts the interactions among the elements of the

architectures shown in Figure 2A-2C fo performing route optimization. If the method is applied to the embodiment of Figure 2B, It is understood thai DC controller 220 and SDN controller 202 form a single entity and thus, the RO entity communicates with this single controiier 202/220. More specifically. Figure 4 shows that the list of VNF pairs is sent in step 400 from the DC controiier 220 to the RO entity 230. This list of VNF pairs is assumed to have been already identified as having communication problems. This information is either requested by the RO entity or initiated by the DC controller. In step 402, the RO entity 230 requests the topology and link loads from the SDN controller 202. In one embodiment, this information exchange is Initiated by the SDN controiier. In step 404, the topoiogy and Sinks loads are sent from the SDN controller to the RO entity. In step 406, the RO entity requests from SDN controller 202 routing information for tie VNF pairs exhibiting bottlenecks and in step 408 the SDN controller provides such information. As discussed above, if the embodiment of Figure 2B is considered, all these exchanges are performed between RO entity 230 and DC controiier 202/220. P061J in step 410, the RO engine identities the problematic links in tie network, creates rerouting request for the VNFs exhibiting problems, and formulates

corresponding constraints for determining the new routes. A specific example for illustrating these concepts is discussed later, in step 412, the RO entit 230 sends a message to the PCE engine 232, with the constraint-based path compute request, the PCE engine calculates the new route in step 413 and the PCE engine replies in step 414 with the new route. The computed path response is sent in step 416 to the SDN controller 202 (to DC controller 202/220 for the embodiment of Figure 28} and feedback information is provided to DC controller 220, either Indicating that a new route has been found or that there is no new route.

0062] One possible functional architecture of the RO engine 234 is illustrated in Figure 5A, and it includes two functional blocks 502 and 504, and three interfaces 506, 508, and 510 connected respectively to the SDN controller, PCE engine and DC Controller, Another functional architecture of RO engine 234 is illustrated in Figure 5B, and this architecture is similar to the one illustrated in Figure 5A, except that it does not have SDN interface 508 as the SDN functionality is implemented in the DC controller 202/220. A routing database 512 contains the topology and resource information of the domain which the RO engine can obtain, for example, by accessing the PCE's traffic engineering database (TED), which can be maintained by the PCE engine or a separate node. The two functional blocks 502 and 504 are now discussed.

[0063] Functional Block 502 maps VNF pairs to the network topology and creates an overlapping links list, figures out how traffic between the identified VNF pairs (the VNF pairs identified as having SLA issues by the Cloud Controller) is mapped to the DCN and identifies the potential routing issues (i.e., the overlapping Iinks list). The flowchart in Figure 8 depicts the processes performed by functional block 502 and the steps of this flowchart are now discussed.

[0064] in step 600, IP addresses of each VNF pair are used to map the virtual link between the members of the pair to the network topology. The mapping returns a set consisting of the underlying physical links along which the VNF traffic traverses. In step 602, a subgrap for each virtual link is generated from the mapped physical network links, where a node corresponds to a switch and an edge corresponds to a network link connecting two switch ports. Note that the terms "node" and "edge" are used in the graph theory for representing a physical system, e.g., the data center network that includes switches and real finks.

|008S] After all the VNF pair subgraphs are generated in step 602, a link algorithm (e.g., subgraph matching algorithm) is run in step 604 to identify overlapping links and their loads. As an example, the matching algorithm finds for each edge (in the list of edges corresponding to VNF pairs) all the subgraphs thai contain that edge and assigns a flow-count property (a positive intege number) to the edge based on the number of subgraphs that shared the edge. All the identified edges and their corresponding flow-count values are then saved in the overlapping links (01) list. The matching can also be implemented as Identifying set intersections for subgraph edge sets,

[0O66J A weight is assigned in step 606 to the overlapping links (edges) identified in the previous step. It is intended in this step to assign higher weights to the links contributing more to the routing problem between VNF pairs. The few-count property assigned to each Sink in the previous step (step 604) is a good candidate for this purpose. Nevertheless, it is possible to take into account that the flow-count alone might be misleading, if, for example, the utilization of the considered link is low.

Therefore, in one application, the weight for each link is assigned based on correlating the flow-count and a link load (e.g., the capacity of the link). 10067] For example, the weight for overlapping link oi_: can be calculated as follows: w(ot_t) = « ftowcount(OLd ·*· β χ iinkioad(QL,) + c, where a, jj, and c are predefined constant values. Alternatively, the weight can be defined as a function that equals zero if the Sink load is bellow a given threshold (i.e., the links whose utilization is below a given threshold (e.g., 70%} are given weight zero and omitted from the OL 1st). The result of step 608 is a list of the identified overlapping links and their corresponding weights.

10068] Functional Block 504 identifies routing problems and initiate constraint- based rerouting between selected VNF pairs. One objective of this functional block is to reroute the traffic of VNF pairs to avoid the problematic links, which are populated in the OL list. This objective is formulated as rerouting requests for selected VNF pairs with the constraints of avoiding one or more links from the OL list. That is, the VNF pairs and the OL iist are the inputs of this function. Moreover, the rerouting request can contain a metric or a 1st of metrics to either indicate the metric that needs to be optimized by the path computation algorithm, e.g., hop count, or a boundary on the path cost that must not be exceeded.

[0069} A rerouting strategy is needed to formulate and issue the rerouting requests for problematic VNF pairs. A rerouting strategy could rely on the following principles; (1.) rerouting VNF pairs with high bandwidth to avoid highly utilized links (i.e., higher assigned weight) are more likely to solve the problem in a quicker way, and (2) both lists (VNF pairs list and the OL list) need to be updated after each successful rerouting.

|OO70| Two possible rerouting strategies are now discussed.

Strategy A: Start from the OL list The steps in this strategy are as described below (using pseudo-code language): Input: OL ist, VNF_^pai Jist

Step 0; SET rerouting mfjniir <~ 0, motnjrerotuingjconstraint <~ 0,

other j-erouting_constr ints *~ 0

Step 1 : SET mnin r r nni xanstrnint *- Select the Sink /^* from the OLJist such that the link has the highest weight across OL

Step 2; SET reroutin fnfjmir *- Seiect the VNF„pa/r with the highest bandwidth from VNFjs&irsJist, which has the link / in its subgraph

Step 3; SET otker rerauttngjconstraints «~ Select ail other Sinks from OLJist

Step 4: Formulate a PCE path computation request for the VNFjDair in

rerouting mf pair with the given metric and constraints of strictly excluding the Sink in mainjreroutmgjcenstraint and preferably excluding ali Sinks in

other rerouting..constraints

Step 5; !f a rerouting request in Step 4 is successful, then update VNF jasrsjist and the OLJist accordingly

Step 6: if VNF jpairsjist and OLJist are both NonJEmpty then GoTo Step 0.

[0071] By applying strategy A, the method resolves the problem for the links with the highest weight, which are the links that are highl overloaded and are affecting multiple VNF pair communications. After re-routing the traffic for the VNF pair with the highest bandwidth usage, the weights of the OLs are re-calculated, if the weight of the link becomes lower than a given threshold, the link can be removed from the OL list. Similarly, the VNF pair which has been re-routed is removed from the VNF pairs list. Moreover, the other VNF pairs which do not have overlapping links anymore or their weights became tower than a threshold are also removed from the list. Strategy 8: Start from VNF pairs list

Input: OL ist, VNF_^pai Jist

Step 0; SET rerouting jmfjniir <~ 0, motnjrerotuingjconstraint <~ 0,

other j-erautingjzonstraints *~ 0

Step 1 : SET rerouting ^mf_^p ir *~ Select the VNF _pair with the highest bandwidth from VWF _pairs__iist

Step 2; SET main.jwetrtm0„c0iistratttt <- Select each link / in VNF_pair, if ί e OLJist Step 3: SET oi¾er„ i¾utf«^„cortxi ai^'«fs *- Select ail other Sinks from OLJist

Step 4: Formulate a PCE path computation request for the VN _pair in

with given metric and constraints of strictly excluding the links in main_rerautmg_c&n$traint and preferably excluding all Sinks in

other j-e utingjcamtramts

Step 5: If a rerouting request in Step 4 is successful, then update VNF_pairsJi$t and the OLJist accordingly

Step 6: if VNF_pairsJist and OLJist are both Non_Empty then GoTo Step 0.

[0072] By applying strategy B, the VNF pairs with the highest bandwidth are rerouted first. If a VNF pair has multiple overlapping links with other VNF pairs, it is expected that by re-routing this pair first, for example, the delay experienced by those other VNF pairs is also reduced.

[0073] Alter natively, other methods for different variations of the above strategies can be used for defining the constraints. For example, in strategy A, after selecting the OL with the highest weight, the RO entity can formulate a rerouting request for multiple VNF pairs containing the OL, requesting the PCE engine to find disjoint routes for the VNF pairs with OL being the main constraint. Another method could be to start wiift a VNF pair which is close to violating its expected network performance or performance guarantees. For example, the latency between the VNF pair is consistently increasing and is close to exceeding its acceptable value, and thus, a rerouting request should be formulated for this pair first.

[0074] After formulating the constraints, the constraints can be sent to the PCE engine using a Path Computation Request message (PCReq message, as described in JP. Vasseur, JL Le Roux, "Path Computation Element (PCE) Communication Protocol (PCEPy Internet Engineering Task Force RFC 5440, https://tools;ietf,org/htrril/rfc5440, 2009). A PCReq message may carry more than one path computation request.

Therefore, the RO entity can formulate constraints and request paths for multiple VNF pairs using the same requesi message. The RO entity can also assign priorities to the requests. An extension to PCEP has been defined in RFC 5521 (E. Oki, T. Takeda, A. Parrel, "Extensions to the Path Computation Element Communication Protocol (PCEP) for Route Exclusions," internet Engineering Task Force RFC 5521 ,

ttps://^'too!s.ietf.org/Ti:tml/rfc5521 , 2009), which allows a Path Computation Client to specify the links that are to be explicitly excluded from the computed route. An Explicit Route Exclusion {also defined in the above noted document) defines network elements that must not or should not be used on the path.

[0076] The above noted concepts are now illustrated with respect to a sim le DCN 700, which is schematically illustrated in Figure 7A, DCN 700 has a simplified dafacenter architecture, which consists of a layer 702 of physical servers 702A-F distributed in racks, a layer 720 of local routers at the top of layer 702, a layer 730 of leaf routers that are connected to the local routers, and a layer 740 of spine routers that are connected to the leaf routers. The RO entity receives a list 750 of VNF pairs (see one example of such list in Figure 78), which need rerouting, including the monitoring information, e.g., bandwidth usage, delay, packet loss, jitter, etc., from the DC

controller, in this example, it is assumed that the VNF pairs are sorted with respect to the bandwidth value in increasing order, i.e., B_7S > ... > s_ts. Additionally, the RO entity receives the topology graph 752 (see Figure 7C) of the DCN 700 and the associated link ioads from the SDN conirolier or from the DC controller. The RO entity then queries the SDN controlier or the DC controller to obtain the routing information for the VNF pairs of list 750.

00763 T e input from the SDN controller is then used by the functional block 502 {Map VNF pairs to Topology and Create Overlapping Links List) in the RO engine to map the virtual links between the VNF pairs to the physica! links 754, as illustrated in Figure 7D. Then, the overlapping links 756 are identified using a subgraph matching algorithm or set intersection or any other equivalent algorithm. For each identified overlapping iink 756, a weight "w" is calculated based on (1 } the flow-count, i.e., number of VNF pairs that use the Sink, and (2) the load of the iink which is obtained from the SDN controlier or the DC controlier, as illustrated in Figure 7E. Note that Figure 7E shows only three overlapping links for simplicity and both the flow-count value and the link load value for these three links. Further, Figure 7E shows that each of the flow-count value and the link load value contributed 0.5 to the iota! weight w. However, these values can be changed as required. The overlapping links that have a weight w higher than a given threshold value, e.g., thresholds, are then added to the 01 list 758, which is illustrated in Figure 7F.

0077] After populating the OL list 758 wit overlapping links 756, this list together wit the V F pairs list 750 are used as inputs to the functional biock 504, Identify routing problems and Initiate constraint-based rerouting between selected VNF pairs. Depending on the routing strategy being applied (A, B or any other strategy), the rerouting requests for VNF pairs are formulated.

|0078J For this specific embodiment, strategy A is applied. In this strategy, the RO entity starts from the OL fist 758 and selects the overlapping link with the highest weight, i.e., OL, _W& in Figure ?E. The high weight value (3.5 in this case) shows that this link is overloaded and is affecting multiple VNF pairs. Then, from the VNF pairs 1st 750, the pair which has the highest bandwidth usage and contains the link oi_{i n} is selected for rerouting, i.e., V7 - ve. if is believed that this pair has contributed more to the overloading of the link, which has also affected other VNF pairs, so it is expected that by rerouting the traffic between the members of this pair, the bottleneck experienced by other pairs sharing this link also gets resolved,

[0079] The RO entity formulates a rerouting request for vi - vs and adds o½ ₄ to the set of main-constraints which has to be strictly excfuded (step 4 in Approach A). The RO entity then adds the other overlapping links, from the OL list, to the set of other-constraints which should preferably be avoided (step 4 in Approac A). This is required to avoid shifting the problem from one link to the other links. Additionally, the metric (e.g., hop-count) which should be optimized can also be added to the rerouting request message 802, which is sent to the PCE engine (see Figure 8A).

[0080] Assuming that the PCE engine finds a new route 804 for VNF pair v? - (see Figure 8B}_« this information is sent to the RO entity, which removes this VNF pair from the VNF pairs list 750 (see Figure 8C) and the overlapping links and their weights are recalculated in the RO entity, as illustrated in Figure 8D, and the OL list 758 is updated, as illustrated in Figure 8E. [0081] in the next round, illustrated by Figures 8F-J, a new OL with the highest weight is selected from the OL list and the above steps are repeated for formulating the rerouting request and updating the lists. This process continues until there are no more link overlaps or until a certain number of links have been rerouted.

[0082] In one application, the VNF pairs list 750 can be dynamically updated by the DC controller if, for example, new VNF pairs are having problems such as increased delay or if som existing pairs do not have delay problems anymore because of rerouting of the other pairs.

[00831 The methods and processes discussed above show one or more advantages over th existing solutions, as now discussed. In one embodiment, route optimization in DON is achieved by using both SDN and Cloud Controllers input (or only Cloud Controller input), which includes monitored network performance metrics and required performance guarantees of VNFs. By preventing service interruptions and performance violations, this embodiment reduces the cost by reducing the unnecessary complex and expensive migration of V s/containers.

[00841 in another embodiment, the rerouting process is oblivious to the measurement method being used in the cloud environment. Regardless of what type of monitoring solution is deployed, e,g. passive or active, the rerouting method uses the monitoring data for identifying bottlenecks and formulating routing constraints, [00861 The defined constraints used in one or more methods can be used for both prevention of violation of expected performance as well as for addressing network performance problems, which are identified by other means.,

[00861 in another embodiment, the method dynamically adapts to the dynamic changes in the traffic patterns of VMFs. One or more methods discussed above can be implemented as a pluggable module for the SDN controller to provide input to PCE engine.

[0OS7J As one skilled in the art would appreciate, the above embodiments describe many novel features that can work together In various implementations. Two such implementations are now discussed. According to an embodiment illustrated in Figure 9, a method for re-routing communication, between two software entities (e.g., two VNFs), intra- or infer-DCNs includes a step 900 of receiving at a route optimizer entity 230 a list 750 of software entity pairs that experience performance degradation.. The list 750 is sent by a data center controi!er 202/220 of the DCNs. The method further includes a step 902 of receiving at the route optimizer entity 230 a topology of the DCNs and routing information associated with the software entity pairs, a step 904 of calculating, in a path computation element 232, a new route for a first software entity pair of the software entity pairs from the list 750, based on the topology and routing information of the software entity pairs, to alleviate the performance degradation, and a step 908 of sending the new route to the data center controller 202/220 for modifying an existing route of the first software entity pair,

[00081 The topology and routing information for the software entity pairs may be sent by a SDN controller 202 of the DCNs. in one embodiment, the path computation element 232 is part of the route optimizer entity. The method may include sending the new route from the path computation element to the data center controller. The step of calculating may include defining routing constraints, in the route optimizer entity, for the first software entit pair to prevent violation of expected network performance guarantees. The method ma further include locating links in the DCNs that are becoming bottlenecks, using (1 } software entity monitoring data, (2) the network topology, and (3) the routing information. The monitoring data includes at least one of bandwidth usage, or delay, jitter packet loss.

[0083] The method may further include mapping virtual links between the software entity pairs to physical links found in the routing information, applying a subgrap matching algorithm: or a set intersection algorithm to the physical links to find overlapping links, and generating an overlapping link list by including only those overlapping links having a specific metric, e.g., a weight larger than a predefined threshold. Optionally, the method may also include using the list of software entity pairs and the overlapping link list to calculate the new route, updating the list of software entity pairs that experience performance degradation after the new route has been calculated, and calculating another new route for a second software entity pair from the updated list of software entity pairs, based on the topology and routing information of the software entity pairs.

[00901 in another embodiment, a method is discussed with regard to Figure 10 for calculating, with a path computation element 232, a new route between two VNFs (or two software entity pairs) intra- or in er-DCNs, The method includes a step 1000 of generating at a route engine 234 an overlapping link list that includes links between software entity pairs that overlap, a step 1002 of defining, at the route engine 234, routing constraints using a list of software entity pairs and the overlapping link list, a step 1004 o! calculating the new route at the path computation element based on information received from the route engine, a step 1008 of sending from the path computation element to route engine an indication of whether the new route has been found, and a step 1008 of sending from the route engine to a network controller either the new route or a message that no new route has been found.

[00011 The method may also include a step of receiving, at the route engine, the list of software entity pairs and a topology of the network from the network controller, a step of sending from the route engine to the path computation element a re-routing request when one or more links of the software entity pairs experience communication degradation, and a step of modifying an existing route between the two software entities instead of migrating one o both software entities,

[Qui 2| The above discussed methods may be Implemented in software, hardware or a combination of the two in a computing device of the DCN. Also, a ty of the engines or entities or modules or elements discussed above, may be implemented in such computing device. Any of the controllers discussed above may have the structure of the computing device now discussed. The computing device 1100 is illustrated In Figure 1 1 and includes a processor 1 102 connected through a bus 1 104 to a storage device 1108. Computing device 1 100 may also include an input/output interface 1108 through which data can be exchanged with the processor and/or storage device. For example, a keyboard, mouse or other device may be connected to th input/output interface 1108 to send commands to the processor and/or to collect dat stored in storage device or to provide data necessary to the processor. A monitor 1 110 may be attached to bus 1104 for displaying the VNF pairs list or other data related to the network traffic or topology,

[0093J The disclosed embodiments provide a method and system for calculating a new route between members of a VNF pair in a data center. As discussed earlier, the methods and systems discussed herein can be implemented for other software entit pairs that communicate among themselves, not only V F pairs, ft should be understood that this description is not intended to limit the invention. On the contrary, the embodiments are intended to cover alternatives, modifications and equivalents, which are included in the spirit and scope of the invention as defined by the appended claims. Further, in the detailed description of the embodiments, numerous specific details are set forth in order to provide a comprehensive understanding of the claimed invention. However, one skilled in the art would understand fiat various embodiments may be practiced without such specific details.

[00S4J As also will be appreciated by one skilled in the art, the embodiments may be embodied in a wireless communication device, a telecommunication network, as a method or in a computer program product. Accordingly, the embodiments may fake the form of an entirely hardware embodiment or an embodiment combining hardware and software aspects. Further, the embodiments may take the form of a computer program product stored on a computer-readable storage medium having computer-readable instructions embodied in the medium. Any suitable computer- readable medium may be utilized, including hard disks, CD-ROMs, digital versatile disc (DVD , optical storage devices, or magnetic storage devices such a floppy disk or magnetic tape. Other non-limiting examples of computer readable media include flash- type memories or other known memories.

pSG&S] Although the features and elements of the presen embodiments ar described in the embodiments in particular combinations, each feature or element can be used alone without the other features and elements of the embodiments or in various combinations with or without other features and elements disclosed herein. The methods or flow charts provided in the present application may be implemented in a computer program, software or firmware tangibly embodied in a computer-readable storage medium for execution by a specifically programmed computer or processor.

Abbreviation Explanation

DC Data Center

DC Data center network

SON Software defined networking

NFV Network function visualization

VNF Virtual network function

VM Virtual machine

PCE Path computation element

PCEP Path computation element protoco!

CBR Constraint based routing

OL Overlapping Sink

RO Route optimizer

SLA Service Level Agreement

Claims

WHAT IS CLAMED IS:

1 . Λ method for re-routing communication, between two software entities, intra- or inter-data center networks (DCNs), t e method comprising:

receiving (400) at a route optimizer entity (230) a list (750) of software entity pairs that experience performance degradation, wherein the list (750) is sent by a data center controller (202/220) of the DCNs;

receiving (404) at the route optimizer entity (230) a topology of the DCNs and routing information associated with the software entity pairs- calculating (413), in a path computation element (232), a new route for a first software entity pair of the software entity pairs from the list (750), based on the topology and routing information of the software entity pairs, to alleviate the performance degradation: and

sending the new route to the data center controller (202/220) for modifying an existing route of the first software entity pair.

2. The method of Claim 1 , wherein tie topology and routing information for the software entity pairs are sent by a software-defined network. (SDN) controiler (202) of the DCNs,

3. The method of Claim 1 , wherein the path computation element (232) is part of the route optimizer entity.

4. The method of Claim 1 , wherein the path computation element sends the new route to data center controller

5. The method of Claim 1 , where n the step of calculating comprises:

defining {410} routing constraints, in the route optimizer entity., fo the first software entity pair to prevent violation of expected network performance.

6. The method of Claim 5, further comprising:

locating links in the DCNs that are becoming bottlenecks, using software entity monitoring data, the network topology, and the routing information.

7. The method of Claim 6, wherein the monitoring data includes at least one of bandwidth usage, delay, jitter packet toss,

8. The method of Claim 8, further comprising:

mapping virtual links between the software entity pairs to physical links found in the routing information;

applying a link algorithm to the physical links to find overlapping links; and generating an overlapping link fist by including only those overlapping links having a weight larger than a predefined threshold.

9. The method of Claim 8, further comprising:

using the list of software entity pairs and the overlapping link list to calculate the new route.

10. The method of Claim 1, further comprising:

updating the list of software entity pairs that experience performance degradation after the new route has been calculated; and

calculating another new route for a second software entity pair from the updated list of software entity pairs, based on the topoiogy and routing information of the software entity pairs,

1 1. A route optimizing module (233) for re-routing a communication path, between two software entities, intra- or inter-data center networks (DCNs), the route optimizing module comprising:

a route engine (234) that receives (4Q0):

0) a list {750} of software entity pairs that experience performance degradation, wherein the list (750) is sent by a data center controller (202/220) of the DCNs, and

(ii) a topology of the DCNs and routing information associated with the software entity pairs; and

a path computation element (232) that calculates (413) a new route for a first software entity pair of the software entity pair from the list {750}, based on the topology and routing information of the software entity pairs, to alleviate the performance degradation,

wherein the route engine sends the new route to the data center controller (202/220) for modifying an existing route of the first software entity pair.

12. The route optimizing module of Claim 11 , wherein the route engine defines (410) routing constraints for the first software entity pair to prevent violation of expected network performance.

13. The route optimizing module of Claim 12, wherein the route engine locates links in the DCNs that are becoming bottlenecks, using software entity monitoring data, the network topology, and the routing Information.

14. The route optimizing module of Claim 13, wherein the route engine sends the routing constraints for the first software entity pair and the finks thai are becoming bottlenecks to the path computation element for calculating the new route.

15. The route optimizing module of Claim 13, wherein the route engine maps virtual links between the software entity pairs to physical links found in the routing information,

the route engine applies a subgraph matching or a set intersection algorithm to the physical links to find overlapping links, and

the mute engine generates an overlapping link list by Including only those overlapping links having a specific metric,

18. The route optimizing module of Claim 11 , wherein the route engine updates the list of software entity pairs that experience performance degradation and sends constraints, for anothe new route, to the path computation element, and the path computation element calculates the another new route for a second software entity pair from the updated list of software entity pairs, based on the topology and routing information of the software entity pairs.

17. A method for calculating, with a path computation element {232), a new route, between two software entities, intra- or inter-data center networks {DCNs), the method comprising:

generating at a route engine (234) an overlapping Sink list that includes links between software entity pairs that overlap;

defining at the route engine (234) routing constraints using a list of software entity pairs and the overlapping link list:

calculating the new route at the path computation element based on information received from the route engine;

sending from the path computation element to route engine an indication of whether the new route has been found; and

sending from the route engine to a network controller either the new route or a message that no new route has been found.

13. The method of Claim 17, further comprising:

receiving, at the route engine, the list of software entity pairs and a topology of the network from the network controller.

19. The method of Claim 1 , further comprising: sending from the route engine to the path computation element a re-routing request when one or more links of the software entity pairs experience communication degradation.

20. The method of Claim 17, further comprising:

modifying an existing route between the to software entities instead of migrating one or both software entities.