EP2531917A1 - Network performance monitor for virtual machines - Google Patents

Network performance monitor for virtual machines

Info

Publication number
EP2531917A1
EP2531917A1 EP10845355A EP10845355A EP2531917A1 EP 2531917 A1 EP2531917 A1 EP 2531917A1 EP 10845355 A EP10845355 A EP 10845355A EP 10845355 A EP10845355 A EP 10845355A EP 2531917 A1 EP2531917 A1 EP 2531917A1
Authority
EP
European Patent Office
Prior art keywords
monitor
network
virtual machines
network performance
performance
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP10845355A
Other languages
German (de)
French (fr)
Other versions
EP2531917B1 (en
EP2531917A4 (en
Inventor
Ayodele Damola
Andreas Johnsson
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Telefonaktiebolaget LM Ericsson AB
Original Assignee
Telefonaktiebolaget LM Ericsson AB
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Telefonaktiebolaget LM Ericsson AB filed Critical Telefonaktiebolaget LM Ericsson AB
Priority to EP19208540.5A priority Critical patent/EP3699758B1/en
Publication of EP2531917A1 publication Critical patent/EP2531917A1/en
Publication of EP2531917A4 publication Critical patent/EP2531917A4/en
Application granted granted Critical
Publication of EP2531917B1 publication Critical patent/EP2531917B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
    • G06F11/3419—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment by assessing time
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
    • G06F11/3433—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment for load management
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466—Performance evaluation by tracing or monitoring
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466—Performance evaluation by tracing or monitoring
    • G06F11/349—Performance evaluation by tracing or monitoring for interfaces, buses
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00—Arrangements for program control, e.g. control units
    • G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44—Arrangements for executing specific programs
    • G06F9/455—Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533—Hypervisors; Virtual machine monitors
    • G06F9/45558—Hypervisor-specific management and integration aspects
    • G06F2009/45591—Monitoring or debugging support
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00—Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/815—Virtual

Definitions

  • the present invention generally relates to methods, systems and nodes to monitor network performance in a computer.
  • Virtualization of networks and computers is a reality today.
  • One prominent example for computer virtualization which can be utilized by more or less anyone, is the Amazon Elastic Compute Cloud (Amazon EC2) . It is a web service that provides dynamic computer capacity to their customers.
  • Computer virtualization is common in data centres where resources can be shared in order to increase utilization of hardware.
  • Developed prototypes running in virtualized environments have been described in the document "Managing Virtualized Systems - Pinpointing performance problems in the virtual infrastructure, Bernd Harzog, April 2008".
  • Several virtualization implementations exist including VMWare, see “http://www.vmware.com/”, Sun VirtualBox, see “http://www.virtualbox.com” and XEN, see “http://xen.org/” [2010-01-14] .
  • Figure 1 belongs to the prior art and discloses a simple example of a virtualized computer. Instead of running one operating system on top of the hardware, a hypervisor 2 runs as a middleware between the operating systems 5 and the hardware 3.
  • the hypervisor acts as a hardware controller that manages the shared hardware recourses among the virtual machines (in which operating systems can run) .
  • Applications 4 such as for example a web server, represent server software that can be run by a virtual machine.
  • the main intent of the hypervisor is to instantiate the virtual machines, provision the resources and make sure that the resources are shared in a manner decided by an operator, among the executing operating systems.
  • the operating systems depicted in Figure 1 can be anything ranging from ordinary servers (e.g. Linux running an Apache web server) to software-based routers and switches.
  • Network communication between a virtual machine on top of a hypervisor and an outside client can of course experience congestion.
  • the congestion or in other words, the observed network performance degradation can originate from either real congestion on the physical network outside the physical computer or due to poor resource allocation and sharing of the common resources within the physical computer.
  • An object of the invention is to overcome above identified limitations of the prior art when it comes to poor resource allocation and sharing of the common resources.
  • a simple and non-intrusive mechanism assisting the resource allocation and performance monitoring tools is hereby required.
  • the basic concept of the invention is an apparatus for measuring performance parameters such as for example packet delay and jitter between two measurements points within a physical entity. By using the invention, it is possible to determine where the source of performance degradation resides.
  • the technique can be used e.g. as an indicator for migration of a virtual machine to another host - with spare resources.
  • a method to monitor network performance in a computer that comprises at least one physical server having a hardware machine attached via a hypervisor to a set of virtual machines.
  • the method includes measuring network performance parameters for selected packet flows in transition between the hardware machine and the virtual machines.
  • the measured performance parameters are analyzed and a source of network degradation is identified.
  • there is a performance monitor arrangement for monitoring network performance The arrangement comprises a Flow allocator to select network flows to monitor and interface means to receive measured performance parameters related to a hardware machine and virtual machines.
  • the performance monitor arrangement further comprises an Analyzer to analyze the measured performance parameters and to identify a source of network degradation.
  • the arrangement comprises a performance monitor and a hardware machine attached to a set of virtual machines.
  • the physical arrangement further comprises means in the performance monitor to measure network performance parameters for selected packet flows in transition between the hardware machine and the virtual machines, means to analyze the measured performance parameters and means to identify a source of network degradation.
  • An object of the invention is to avoid poor resource allocation and sharing of the common resources. This object and others are achieved by methods, arrangements, nodes, systems and articles of manufacture. Some examples of advantages of the invention are as follows:
  • the invention provides an apparatus that monitors e.g. jitter and delay in virtualized environments.
  • the invention provides a way to detect when a physical server is in need of load balancing.
  • This invention proposes a mechanism of load balancing a physical node running several Virtual Machines by VM migration triggered when an overload is detected based on observed delay within the physical node.
  • the invention provides a way of load balancing Virtual Machines across physical servers in a computer depending on network performance parameters.
  • the invention will now be described more in detail with the aid of preferred embodiments in connection with the enclosed drawings .
  • Figure 1 is part of tihe prior art and discloses a block schematic illustration of a virtualized computer.
  • Figure 2 is a block schematic illustration of a computer having a physical server that comprises a performance monitor and a hardware machine that is attached via a hypervisor to a set of virtual machines.
  • Figure 3 is a block schematic illustration of a performance monitor located within the same machine as the hypervisor.
  • Figure 4 discloses a signal sequence diagram representing a method to monitor network performance.
  • Figure 5 discloses a flowchart illustrating a matrix wherein measured delay values or loss rates are stored.
  • Figure 6 discloses a flowchart illustrating a migration decision algorithm.
  • Figure 7 discloses an example of a computer system comprising several physical servers and a migration engine.
  • FIG. 2 is a block schematic illustration of a computer 40 comprising a physical server 10 having a performance monitor 13, and a hardware machine 12 attached via a hypervisor 14 to a set of virtual machines (VM1, VM2, VM3) 15,16,17.
  • the hardware machine comprises a Hardware Network Interface Card (H NIC) 24 and the virtual machines each one comprises a Virtual Machine Network Interface Card (VM NIC) 18,19,20.
  • a hardware clock 25 is in this example located at the HW NIC.
  • a client 11 is attached to the HW NIC and a Service Level Agreement Data Base SLA DB 43 is attached to the performance monitor 13. Packet flows 21,22,23 are illustrated in the figure .
  • the performance monitor 13 is responsible for selecting network flows to monitor, signalling monitoring requests to various measurement points, collecting measurement data and for estimating e.g. packet delay and jitter.
  • the performance monitor components will be shown more in detail later in the description together with Figure 3.
  • SLA module SLA DB 43 contains predefined SLA parameters for jitter, delay etc. HW NIC.
  • the hardware NIC 24 must be able to time stamp in- and outgoing packets at line speed. It must also be able to accept requests on which packets to time stamp from the performance monitor as well as signal back measurement results.
  • the time stamps should preferably be made using the physical hardware clock 25. (NICs are capable of this type of operations.) To be noted is that this is an example and the location of the hardware clock is an implementation issue.
  • the hypervisor must be able to time stamp in- and outgoing packets at the virtual NICs 18,19,20 at line speed.
  • a VM NIC must also be able to accept requests on which packets to time stamp from the performance monitor as well as signal back measurement results.
  • the time stamp should preferably be made using the physical HW clock 25.
  • the hypervisor 14 is software capable of running different virtual machines at the same time. Each virtual machine can run an operating system or server software. In order to utilize measurement results produced by the performance monitor 13, an interface between the hypervisor and the performance monitor must be in place. This is an implementation issue .
  • a virtual machine 15,16,17 is a slot in the hypervisor 14 that can run any server software.
  • the VM could be a web server providing certain functionality to a set of users.
  • the client 11 is a network server or other element that communicates via the hardware machine 12 with a virtual machine in the hypervisor.
  • Figure 3 discloses the Performance monitor 13 arrangement more in detail.
  • the Performance monitor could be either a standalone node or constitute separate entities within the physical server 10, which operates in collaboration with the hypervisor 14.
  • a Flow allocator 27 makes choices based on information in a DB 29 (e.g. source/destination IP, SLA parameters) on which traffic to monitor.
  • the DB comprises storages 30-32 with information of active Virtual Machines.
  • An Analyzer 28 estimates e.g. jitter and delay for the selected flows.
  • the Flow allocator 27, the Analyzer 28 and the DB 29 will be further clarified in the description of a method according to the invention.
  • the performance monitor 13 have interfaces (HW NIC IF, VM NIC IF, Hypervisor IF) 34,35,36 towards the Hardware NIC, the virtual machine NICs as well as to the hypervisor.
  • Figure 4 discloses the measurement process according to the invention.
  • Figure 4 is a signal sequence diagram wherein the signalling points, i.e. the Flow allocator 27, the Analyzer 28, the data base 29, the HW- and VM NIC IFs 34 and 35, and the Hypervisor 14 that were explained earlier together with figure 2 and 3, have been disclosed.
  • the method will show the process for setting up monitoring and estimating delay and jitter. The method comprises the following steps:
  • the Analyzer requests 51 information of a set of active Virtual Machines VMs from the Hypervisor 14, through the Hypervisor IF.
  • VM1-VM3 are all active.
  • the received information contains for example: > IP addresses
  • the Analyzer inserts 53 identifications of active VMs (VM1, VM2 and VM3 in this example) and the received information, in the data base DB 29.
  • the Flow allocator requests 54 the information of active VMs from the data base 29 to select flows to monitor.
  • a flow can be defined e.g. by source/destination IP and source/destination TCP/UDP port
  • the Information about active VMs is received 55 from the data base to the Flow allocator.
  • the Flow allocator selects which flows related to the VMs to monitor. Flows could for example be randomly selected. Other selection criteria could be based on critical application data such as voice or streaming video packets .
  • Each HW and VM NIC hereby time stamps all packets, in- and outgoing, that are part of the selected network flow. In this example the same hardware clock is used for the time stamping. Time stamps for packets leaving the HW NIC are so called first time stamps TS1. All first time stamps are reported 57 from HW NIC to the analyzer.
  • the V NIC When a packet that has been time stamped at the HW NIC arrives to the VM NIC, the V NIC will report this arrival to the hypervisor that in turn will report to the hardware clock and the packet will be time stamped by using the hardware clock.
  • This time stamp is a so called second time stamp TS2.
  • all time stamping is done using the same hardware clock 25.
  • the Hypervisor When a packet arrives to the physical server 10 i.e. to the HW NIC as well as when a virtual machine reads the packet from its VM NIC, the Hypervisor will instruct the hardware clock 25 to record the time. The operating system does not have to be involved. This is an implementation issue in the hypervisor. All second time stamps are reported 58 to the analyzer. The interaction during time stamping between the NICs and the Hypervisor has not been disclosed in figure 4.
  • Delay and jitter for the packets transmitted from the HW NIC to the VM NIC is estimated 59 in the analyzer 28.
  • the delay and jitter between these two can be estimated.
  • the process of measuring jitter is made in the same manner as measuring delay. That is, the packets are time stamped using the same hardware clock at the HW NIC and at the VM NIC. Estimation of jitter requires time stamps of consecutive packets. From this the inter-packet separation can be calculated, which is the basis for jitter calculation. This is done at both the HW NIC and the VM NICs. Measurement results are stored in a similar fashion as described above for delay.
  • the estimated jitter on the HW NIC is compared to the VM NIC. If larger on the VM NIC, jitter has been introduced by the hypervisor. A comparison to Service level Agreement SLA if acceptable can be done. If the jitter measured when the packets arrive at the VM NIC is larger than when measured on the HW NIC, the jitter originates from the virtualized environment. That is, not from the network.
  • Figure 5 discloses a matrix 65 wherein measured delay values X, i.e. TS2-TS1 (mentioned above), are stored.
  • the matrix is in this example part of the data base 29.
  • the rows represent measurements at time T for each Virtual Machine VMl-VMk.
  • Delay average values A for the different times T are calculated and stored in the matrix.
  • Figure 6 discloses a flowchart illustrating a process of determining when migration of a VM from a physical host shall be done. The process comprises the following steps:
  • a baseline for delay is estimated pre-service and is estimated as an average of the delay between the HW NIC and each VM NIC. This step is illustrated in the figure with a block 101.
  • Delay estimates are collected from the monitored flows, this is added into the measurement matrix.
  • This step is illustrated in the figure with a block 102.
  • the average is estimated for a time window where the window is set to represent a number of rows back in time in the measurement matrix. For example the time window could be set to 10 seconds; in that case if each row represents measurements done in an interval of one second, 10 rows will be considered.
  • This step is illustrated in the figure with a block 103.
  • Determination of which VM to migrate This step is illustrated in the figure with a block 106.
  • VM that was found would be migrated to a host with more resources.
  • a similar approach can be used for measured jitter. However, for jitter there is no need of a pre-service estimated baseline. Instead, the jitter estimated at the HW NIC is compared to the jitter estimated at the VM NICs. If the average jitter calculated based on the jitter experienced by all the V s, for a time window is increasing (or is above a predefined SLA) the host is considered overloaded and hence migration can take place.
  • Figure 7 discloses a computer usage scenario.
  • a computer 40 can be seen that comprises three physical servers 10, 70 and 80.
  • the physical server 10 comprising the entities 13, 15, 16 has been explained earlier together with figure 2.
  • a similar entity approach is assumed for the entities 70 and 80.
  • a migration engine 42 and a policy engine 41 can be seen in figure 7.
  • One of the usages of the performance monitoring described in this invention is to utilize Service Level Agreements SLA between a computer operator and customers.
  • the SLA could outline parameters such as minimum delay or minimum jitter acceptable by the applications run in the computer by the customers.
  • the performance monitoring tool could also serve as the trigger mechanism to the migration of VM running customer applications when the performance parameters drop below a threshold level as defined by the SLA.
  • the policy engine gets performance monitoring input for the performance monitoring modules embedded across the physical servers in the computer. By comparing the input with the values defined in the SLA, a decision is made if a VM should be moved to another physical server where more resources are available. The movement or migration is done by the migration engine.
  • An example of a migration engine is VMWare' s VMotion, see http: //www. vmware . com/pdf/vmotion
  • the program storage medium includes data signal embodied in one or more of a carrier wave, a computer disk (magnetic, or optical (e.g., CD or DVD, or both), non-volatile memory, tape, a system memory, and a computer hard drive.
  • the invention is not limited to the above described and in the drawings shown embodiments but can be modified within the scope of the enclosed claims.
  • the systems and methods of the present invention may be implemented for example on any of the Third Generation Partnership Project (3GPP), European Telecommunications Standards Institute (ETSI), American National Standards Institute (ANSI) or other standard telecommunication network architecture.
  • 3GPP Third Generation Partnership Project
  • ETSI European Telecommunications Standards Institute
  • ANSI American National Standards Institute
  • Other examples are the Institute of Electrical and Electronics Engineers (IEEE) or The Internet Engineering Task Force (IETF) .

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Software Systems (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Data Exchanges In Wide-Area Networks (AREA)

Abstract

The present invention relates to a method to monitor network performance in a computer (40) comprising at least one physical server (10) having a hardware machine (12) attached via a hypervisor (14) to a set of virtual machines (15,16,17). The method comprises steps like: Measuring network performance parameters for selected packet flows (21,22,23) in transition between the hardware machine and the virtual machines. Analyzing (59) the measured performance parameters. - Identifying a source of network degradation.

Description

NETWORK PERFORMANCE MONITOR FOR VIRTUAL MACHINES
TECHNICAL FIELD The present invention generally relates to methods, systems and nodes to monitor network performance in a computer.
BACKGROUND
Virtualization of networks and computers is a reality today. One prominent example for computer virtualization, which can be utilized by more or less anyone, is the Amazon Elastic Compute Cloud (Amazon EC2) . It is a web service that provides dynamic computer capacity to their customers. Also, computer virtualization is common in data centres where resources can be shared in order to increase utilization of hardware. Developed prototypes running in virtualized environments have been described in the document "Managing Virtualized Systems - Pinpointing performance problems in the virtual infrastructure, Bernd Harzog, April 2008". Several virtualization implementations exist including VMWare, see "http://www.vmware.com/", Sun VirtualBox, see "http://www.virtualbox.com" and XEN, see "http://xen.org/" [2010-01-14] .
Figure 1 belongs to the prior art and discloses a simple example of a virtualized computer. Instead of running one operating system on top of the hardware, a hypervisor 2 runs as a middleware between the operating systems 5 and the hardware 3. The hypervisor acts as a hardware controller that manages the shared hardware recourses among the virtual machines (in which operating systems can run) . Applications 4, such as for example a web server, represent server software that can be run by a virtual machine. The main intent of the hypervisor is to instantiate the virtual machines, provision the resources and make sure that the resources are shared in a manner decided by an operator, among the executing operating systems. The operating systems depicted in Figure 1 can be anything ranging from ordinary servers (e.g. Linux running an Apache web server) to software-based routers and switches. This provides a very flexible platform for performing various tasks. One fundamental problem with respect to virtualization of networks and computers is how to manage the shared resources as well as how to handle dynamic changes in the provisioning, see "Challenges in Resource Allocation in Network Virtualization, Aun Haider and Richard Potter, in 20th ITC Specialist Seminar, May 2009, Vietnam". Another important issue to address is performance management. For example, performance management tools that are used today do not work satisfactory in a virtualized environment. It is elaborated on this issue in "Timekeeping in VMware Virtual Machines, VMware Inc., 2008" and "Managing Virtualized Systems - Pinpointing performance problems in the virtual infrastructure, Bernd Harzog, April 2008".
Network communication between a virtual machine on top of a hypervisor and an outside client can of course experience congestion. In the virtualized world the congestion or in other words, the observed network performance degradation, can originate from either real congestion on the physical network outside the physical computer or due to poor resource allocation and sharing of the common resources within the physical computer. SUMMARY
An object of the invention is to overcome above identified limitations of the prior art when it comes to poor resource allocation and sharing of the common resources. A simple and non-intrusive mechanism assisting the resource allocation and performance monitoring tools is hereby required.
The basic concept of the invention is an apparatus for measuring performance parameters such as for example packet delay and jitter between two measurements points within a physical entity. By using the invention, it is possible to determine where the source of performance degradation resides. The technique can be used e.g. as an indicator for migration of a virtual machine to another host - with spare resources.
According to one exemplary embodiment of the invention, there is a method to monitor network performance in a computer that comprises at least one physical server having a hardware machine attached via a hypervisor to a set of virtual machines. The method includes measuring network performance parameters for selected packet flows in transition between the hardware machine and the virtual machines. The measured performance parameters are analyzed and a source of network degradation is identified. According to another exemplary embodiment, there is a performance monitor arrangement for monitoring network performance. The arrangement comprises a Flow allocator to select network flows to monitor and interface means to receive measured performance parameters related to a hardware machine and virtual machines. The performance monitor arrangement further comprises an Analyzer to analyze the measured performance parameters and to identify a source of network degradation. According to yet another exemplary embodiment, there is a Physical server arrangement for monitoring network performance. The arrangement comprises a performance monitor and a hardware machine attached to a set of virtual machines. The physical arrangement further comprises means in the performance monitor to measure network performance parameters for selected packet flows in transition between the hardware machine and the virtual machines, means to analyze the measured performance parameters and means to identify a source of network degradation.
An object of the invention is to avoid poor resource allocation and sharing of the common resources. This object and others are achieved by methods, arrangements, nodes, systems and articles of manufacture. Some examples of advantages of the invention are as follows:
• The invention provides an apparatus that monitors e.g. jitter and delay in virtualized environments.
• Using the invention it is possible to find out whether network degradation, in terms of e.g. delay and jitter, is due to network congestion or due to a hypervisor that distributes and allocates resources to Virtual Machines.
• The invention provides a way to detect when a physical server is in need of load balancing.
• This invention proposes a mechanism of load balancing a physical node running several Virtual Machines by VM migration triggered when an overload is detected based on observed delay within the physical node.
• The invention provides a way of load balancing Virtual Machines across physical servers in a computer depending on network performance parameters. The invention will now be described more in detail with the aid of preferred embodiments in connection with the enclosed drawings .
BRIEF DESCRIPTION OF THE DRAWINGS
Figure 1 is part of tihe prior art and discloses a block schematic illustration of a virtualized computer.
Figure 2 is a block schematic illustration of a computer having a physical server that comprises a performance monitor and a hardware machine that is attached via a hypervisor to a set of virtual machines.
Figure 3 is a block schematic illustration of a performance monitor located within the same machine as the hypervisor.
Figure 4 discloses a signal sequence diagram representing a method to monitor network performance.
Figure 5 discloses a flowchart illustrating a matrix wherein measured delay values or loss rates are stored.
Figure 6 discloses a flowchart illustrating a migration decision algorithm. Figure 7 discloses an example of a computer system comprising several physical servers and a migration engine.
DETAILED DESCRIPTION
In the following description, for purposes of explanation and not limitation, specific details are set forth, such as particular circuits, circuit components, techniques, etc. in order to provide a thorough understanding of the present invention. However, it will be apparent to one skilled in the art that the present invention may be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well known methods, devices, and circuits are omitted so as not to obscure the description of the present invention with unnecessary detail.
Figure 2 is a block schematic illustration of a computer 40 comprising a physical server 10 having a performance monitor 13, and a hardware machine 12 attached via a hypervisor 14 to a set of virtual machines (VM1, VM2, VM3) 15,16,17. The hardware machine comprises a Hardware Network Interface Card (H NIC) 24 and the virtual machines each one comprises a Virtual Machine Network Interface Card (VM NIC) 18,19,20. A hardware clock 25 is in this example located at the HW NIC. A client 11 is attached to the HW NIC and a Service Level Agreement Data Base SLA DB 43 is attached to the performance monitor 13. Packet flows 21,22,23 are illustrated in the figure .
The functionality and requirements for each component is described below. In Figure 2 interaction between different components is illustrated by dashed lines.
Performance monitor. The performance monitor 13 is responsible for selecting network flows to monitor, signalling monitoring requests to various measurement points, collecting measurement data and for estimating e.g. packet delay and jitter. The performance monitor components will be shown more in detail later in the description together with Figure 3. - Service Level Agreement SLA parameters. The SLA module SLA DB 43 contains predefined SLA parameters for jitter, delay etc. HW NIC. The hardware NIC 24 must be able to time stamp in- and outgoing packets at line speed. It must also be able to accept requests on which packets to time stamp from the performance monitor as well as signal back measurement results. The time stamps should preferably be made using the physical hardware clock 25. (NICs are capable of this type of operations.) To be noted is that this is an example and the location of the hardware clock is an implementation issue.
VM NIC. The hypervisor must be able to time stamp in- and outgoing packets at the virtual NICs 18,19,20 at line speed. A VM NIC must also be able to accept requests on which packets to time stamp from the performance monitor as well as signal back measurement results. The time stamp should preferably be made using the physical HW clock 25.
Hypervisor. The hypervisor 14 is software capable of running different virtual machines at the same time. Each virtual machine can run an operating system or server software. In order to utilize measurement results produced by the performance monitor 13, an interface between the hypervisor and the performance monitor must be in place. This is an implementation issue .
VM. A virtual machine 15,16,17 is a slot in the hypervisor 14 that can run any server software. For example, the VM could be a web server providing certain functionality to a set of users.
Client. The client 11 is a network server or other element that communicates via the hardware machine 12 with a virtual machine in the hypervisor. Figure 3 discloses the Performance monitor 13 arrangement more in detail. The Performance monitor could be either a standalone node or constitute separate entities within the physical server 10, which operates in collaboration with the hypervisor 14. A Flow allocator 27 makes choices based on information in a DB 29 (e.g. source/destination IP, SLA parameters) on which traffic to monitor. The DB comprises storages 30-32 with information of active Virtual Machines. An Analyzer 28 estimates e.g. jitter and delay for the selected flows. The Flow allocator 27, the Analyzer 28 and the DB 29 will be further clarified in the description of a method according to the invention. The performance monitor 13 have interfaces (HW NIC IF, VM NIC IF, Hypervisor IF) 34,35,36 towards the Hardware NIC, the virtual machine NICs as well as to the hypervisor.
Figure 4 discloses the measurement process according to the invention. Figure 4 is a signal sequence diagram wherein the signalling points, i.e. the Flow allocator 27, the Analyzer 28, the data base 29, the HW- and VM NIC IFs 34 and 35, and the Hypervisor 14 that were explained earlier together with figure 2 and 3, have been disclosed. The method will show the process for setting up monitoring and estimating delay and jitter. The method comprises the following steps:
• The Analyzer requests 51 information of a set of active Virtual Machines VMs from the Hypervisor 14, through the Hypervisor IF.
• Information about the set of active VMs is received 52 from the Hypervisor. In this example VM1-VM3 are all active. The received information contains for example: > IP addresses
> Open ports
> SLA parameters. Other VM specific data
• The Analyzer inserts 53 identifications of active VMs (VM1, VM2 and VM3 in this example) and the received information, in the data base DB 29.
• The Flow allocator requests 54 the information of active VMs from the data base 29 to select flows to monitor. A flow can be defined e.g. by source/destination IP and source/destination TCP/UDP port
• The Information about active VMs is received 55 from the data base to the Flow allocator. The Flow allocator selects which flows related to the VMs to monitor. Flows could for example be randomly selected. Other selection criteria could be based on critical application data such as voice or streaming video packets .
• According to the invention, the Flow allocator signals
56 via the interfaces 34 and 35 to the HW NIC 24 and to the VM NICs 18-20 acting as endpoints for selected flows, which packets (that constitutes a flow) to time stamp. The NIC addressing can be based on source/destination IP, UDP/TCP port numbers etc. Each HW and VM NIC hereby time stamps all packets, in- and outgoing, that are part of the selected network flow. In this example the same hardware clock is used for the time stamping. Time stamps for packets leaving the HW NIC are so called first time stamps TS1. All first time stamps are reported 57 from HW NIC to the analyzer.
When a packet that has been time stamped at the HW NIC arrives to the VM NIC, the V NIC will report this arrival to the hypervisor that in turn will report to the hardware clock and the packet will be time stamped by using the hardware clock. This time stamp is a so called second time stamp TS2. In this exemplified embodiment, all time stamping is done using the same hardware clock 25. When a packet arrives to the physical server 10 i.e. to the HW NIC as well as when a virtual machine reads the packet from its VM NIC, the Hypervisor will instruct the hardware clock 25 to record the time. The operating system does not have to be involved. This is an implementation issue in the hypervisor. All second time stamps are reported 58 to the analyzer. The interaction during time stamping between the NICs and the Hypervisor has not been disclosed in figure 4.
Delay and jitter for the packets transmitted from the HW NIC to the VM NIC is estimated 59 in the analyzer 28. By observing network traffic between a set of virtual machines and outside clients at two different measurement points within the physical server, the delay and jitter between these two can be estimated.
Measuring of delay can be done according to the following algorithm: Packets arrive at HW NIC. Time stamp using HW clock <TS1) . Packet put in VM NIC queue. Packet read from V NIC by the operating system running on top of the virtual machine. Time stamp using HW clock (TS2) . Delay = TS2 - TS1. Delay is compared to a baseline (or SLA) set for a specific HW NIC - VM NIC path.
Trend analysis on delay to determine whether the Hypervisor adds to network performance degradation, increasing delay means lack of resources allocated to a Virtual Machine VM running in the Hypervisor. Thus, if an application in the VM experience network degradation at the same time as the analysis show increasing trend the Hypervisor is probably overloaded which causes overall network degradation to the VM. I.e. if packet delay measured as the time for a packet to travel from the HW NIC to the VM NIC is increasing, or is above a predefined base line (e.g. an SLA parameter or according to application requirements) , the additional delay originates from the hypervisor rather than from the network. This is valuable information when optimizing.
The process of measuring jitter is made in the same manner as measuring delay. That is, the packets are time stamped using the same hardware clock at the HW NIC and at the VM NIC. Estimation of jitter requires time stamps of consecutive packets. From this the inter-packet separation can be calculated, which is the basis for jitter calculation. This is done at both the HW NIC and the VM NICs. Measurement results are stored in a similar fashion as described above for delay.
The estimated jitter on the HW NIC is compared to the VM NIC. If larger on the VM NIC, jitter has been introduced by the hypervisor. A comparison to Service level Agreement SLA if acceptable can be done. If the jitter measured when the packets arrive at the VM NIC is larger than when measured on the HW NIC, the jitter originates from the virtualized environment. That is, not from the network.
• Signal information is sent 60 to the hypervisor that can use the information for: SLA monitoring > Make a decision to migrate a VM
Figure 5 discloses a matrix 65 wherein measured delay values X, i.e. TS2-TS1 (mentioned above), are stored. The matrix is in this example part of the data base 29. The rows represent measurements at time T for each Virtual Machine VMl-VMk. Delay average values A for the different times T are calculated and stored in the matrix.
Figure 6 discloses a flowchart illustrating a process of determining when migration of a VM from a physical host shall be done. The process comprises the following steps:
A baseline for delay is estimated pre-service and is estimated as an average of the delay between the HW NIC and each VM NIC. This step is illustrated in the figure with a block 101.
Delay estimates are collected from the monitored flows, this is added into the measurement matrix.
This step is illustrated in the figure with a block 102. The average is estimated for a time window where the window is set to represent a number of rows back in time in the measurement matrix. For example the time window could be set to 10 seconds; in that case if each row represents measurements done in an interval of one second, 10 rows will be considered. This step is illustrated in the figure with a block 103.
Compare time window average with the base line. This step is illustrated in the figure with a block 104. - If the average for a specific time window is above the baseline the node is considered overloaded. The average value has increased and thus one or possibly several of the VMs experience insufficient resources. This step is illustrated in the figure with a block 105.
Determination of which VM to migrate. This step is illustrated in the figure with a block 106. a. One way is to migrate any VM. This will free resources from the physical host which can be utilized by the remaining VMs. b. A second way is to find which VM first experienced increased delay which then lead to an increase in resource utilization of the physical node and a subsequent delay for other VMs; this is done by inspecting the measurement matrix. The
VM that was found would be migrated to a host with more resources.
Actual migration is done using state-of-the-art methods known to a person skilled in the art. This step is illustrated in the figure with a block 107.
A similar approach can be used for measured jitter. However, for jitter there is no need of a pre-service estimated baseline. Instead, the jitter estimated at the HW NIC is compared to the jitter estimated at the VM NICs. If the average jitter calculated based on the jitter experienced by all the V s, for a time window is increasing (or is above a predefined SLA) the host is considered overloaded and hence migration can take place.
Figure 7 discloses a computer usage scenario. In Figure 7 a computer 40 can be seen that comprises three physical servers 10, 70 and 80. The physical server 10 comprising the entities 13, 15, 16 has been explained earlier together with figure 2. A similar entity approach is assumed for the entities 70 and 80. A migration engine 42 and a policy engine 41 can be seen in figure 7. One of the usages of the performance monitoring described in this invention is to utilize Service Level Agreements SLA between a computer operator and customers. The SLA could outline parameters such as minimum delay or minimum jitter acceptable by the applications run in the computer by the customers. The performance monitoring tool could also serve as the trigger mechanism to the migration of VM running customer applications when the performance parameters drop below a threshold level as defined by the SLA.
The policy engine gets performance monitoring input for the performance monitoring modules embedded across the physical servers in the computer. By comparing the input with the values defined in the SLA, a decision is made if a VM should be moved to another physical server where more resources are available. The movement or migration is done by the migration engine. An example of a migration engine is VMWare' s VMotion, see http: //www. vmware . com/pdf/vmotion
[2010-01-20] . A situation could arise where there is not sufficient resources for the VMs in an existing physical server, in this case the VM could be migrated to another physical server site under the control of the computer operator .
Items are shown in the figures as individual elements. In actual implementations of the invention however, they may be inseparable components of other electronic devices such as a digital computer. Thus, actions described above may be implemented in software that may be embodied in an article of manufacture that includes a program storage medium. The program storage medium includes data signal embodied in one or more of a carrier wave, a computer disk (magnetic, or optical (e.g., CD or DVD, or both), non-volatile memory, tape, a system memory, and a computer hard drive.
The invention is not limited to the above described and in the drawings shown embodiments but can be modified within the scope of the enclosed claims. The systems and methods of the present invention may be implemented for example on any of the Third Generation Partnership Project (3GPP), European Telecommunications Standards Institute (ETSI), American National Standards Institute (ANSI) or other standard telecommunication network architecture. Other examples are the Institute of Electrical and Electronics Engineers (IEEE) or The Internet Engineering Task Force (IETF) .
Individual function blocks are shown in one or more figures. Those skilled in the art will appreciate that functions may be implemented using discrete components or multi-function hardware. Processing functions may be implemented using a programmed microprocessor or general-purpose computer. The invention is in other words not limited to the above described and in the drawings shown embodiments but can be modified within the scope of the enclosed claims.
The invention is of course not limited to the above described and in the drawings shown embodiments but can be modified within the scope of the enclosed claims.

Claims

Method to monitor network performance in a computer (40) comprising at least one physical server (10,70,80) having a hardware machine (12) attached via a hypervisor (14) to a set of virtual machines (15,16,17), which method is characterized by the following steps: measuring network performance parameters for selected packet flows (21,22,23) in transition between the hardware machine and the virtual machines; analyzing (59) the measured performance parameters; identifying a source of network degradation.
Method to monitor network performance according to claim 1, which method comprises the following further step : optimizing the prerequisite for packet transition, with respect to the identified source of network degradation.
Method to monitor network performance according to claim 1 or 2, which performance parameters are measured by utilizing time stamps obtained in the hardware machine and the virtual machines for in and/or outgoing packets.
A . Method to monitor network performance according to claim 3, which time stamps are obtained by the same physical hardware clock.
Method to monitor network performance according to any of the claims 1-4, whereby the optimization is performed by migration across physical servers (10,70,80) of virtual machines contributing to or experiencing resource depletion in the computer (40) .
6. Method to monitor network performance according to any of the claims 1-5, whereby the optimization is performed in accordance to Service Level Agreements or application requirements.
Method to monitor network performance according to any of the previous claims, whereby the source of network degradation is determined by observing jitter and/or by observing delay.
Method to monitor network performance according to any of the previous claims whereby determining when migration of a virtual machine shall be done comprises the following steps: estimation of a baseline for delay; collection of delay estimates from monitored flows; estimating average value for a specified time window; comparing average value with baseline; triggering of a virtual machine migration average > baseline.
Arrangement adapted to monitor network performance, which arrangement {13) comprises: means (27) to select network flows to monitor; means (34,35,36) to receive measured performance parameters related to a hardware machine and virtual machines; means (28) to analyze the measured performance parameters and to identify a source of network degradation.
Arrangement adapted to monitor network performance, which arrangement (13) comprises: a Flow allocator (27) to select network flows to monitor; interfaces (34,35,36) to receive measured performance parameters related to a hardware machine and virtual machines; an analyzer (28) to analyze the measured performance parameters and to identify a source of network degradation.
Arrangement adapted to monitor network performance, which arrangement (10) comprises a performance monitor (13) , and a hardware machine (12) attached to a set of virtual machines (15,16,17), which arrangement further comprises : means in the performance monitor (13) to measure network performance parameters for selected packet flows (21,22,23) in transition between the hardware machine
(12) and the virtual machines (15,16,17) ; means in the performance monitor (13) to analyze the measured performance parameters; means in the performance monitor (13) to identify a source of network degradation.
Arrangement adapted to monitor network performance according to claim 11, which arrangement further comprises means to optimize the prerequisite for packet transition with respect to the identified source of network degradation.
13. Arrangement adapted to monitor network performance according to claim 11 or 12, which arrangement further comprises means to utilize obtained time stamps.
1 . Arrangement adapted to monitor network performance according to any of claims 11-13, which arrangement further comprises : means to estimate a baseline for delay; means to collect delay estimates from monitored flows; means to estimate average value for a specified time window; - means to compare average value with baseline;
15. Article of manufacturing comprising a program storage medium having computer readable code embodied therein for monitoring network performance, which computer readable code comprises: computer readable program code to select network flows to monitor; computer readable program code to receive measured performance parameters related to a hardware machine and virtual machines; computer readable program code to analyze the measured performance parameters and to identify a source of network degradation.
EP10845355.6A 2010-02-04 2010-02-04 Network performance monitor for virtual machines Active EP2531917B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
EP19208540.5A EP3699758B1 (en) 2010-02-04 2010-02-04 Network performance monitor for virtual machines

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/SE2010/050132 WO2011096859A1 (en) 2010-02-04 2010-02-04 Network performance monitor for virtual machines

Related Child Applications (1)

Application Number Title Priority Date Filing Date
EP19208540.5A Division EP3699758B1 (en) 2010-02-04 2010-02-04 Network performance monitor for virtual machines

Publications (3)

Publication Number Publication Date
EP2531917A1 true EP2531917A1 (en) 2012-12-12
EP2531917A4 EP2531917A4 (en) 2017-08-09
EP2531917B1 EP2531917B1 (en) 2019-11-27

Family

ID=44355651

Family Applications (2)

Application Number Title Priority Date Filing Date
EP19208540.5A Active EP3699758B1 (en) 2010-02-04 2010-02-04 Network performance monitor for virtual machines
EP10845355.6A Active EP2531917B1 (en) 2010-02-04 2010-02-04 Network performance monitor for virtual machines

Family Applications Before (1)

Application Number Title Priority Date Filing Date
EP19208540.5A Active EP3699758B1 (en) 2010-02-04 2010-02-04 Network performance monitor for virtual machines

Country Status (3)

Country Link
US (1) US9063769B2 (en)
EP (2) EP3699758B1 (en)
WO (1) WO2011096859A1 (en)

Families Citing this family (67)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8607229B2 (en) * 2010-02-05 2013-12-10 Riverbed Technology, Inc. Correcting packet timestamps in virtualized environments
US8234400B2 (en) * 2010-03-16 2012-07-31 Microsoft Corporation Shaping virtual machine communication traffic
US8489733B2 (en) * 2010-05-21 2013-07-16 Red Hat, Inc. Automatic enterprise service bus deployment at the level of individual services
JP5458308B2 (en) * 2010-06-11 2014-04-02 株式会社日立製作所 Virtual computer system, virtual computer system monitoring method, and network device
CN103154896A (en) * 2010-10-19 2013-06-12 株式会社日立制作所 Method and device for deploying virtual computers
US8756602B2 (en) * 2010-11-14 2014-06-17 Brocade Communications Systems, Inc. Virtual machine and application migration over local and wide area networks without timeout
EP2500823A1 (en) 2011-03-15 2012-09-19 Siemens Aktiengesellschaft Operation of a data processing network with multiple geographically decentralised data centres
EP2693704B1 (en) * 2011-04-28 2018-06-06 Huawei Technologies Co., Ltd. Method, regional management entity and system for virtual network migration
US8856784B2 (en) 2011-06-14 2014-10-07 Vmware, Inc. Decentralized management of virtualized hosts
US8701107B2 (en) 2011-06-14 2014-04-15 Vmware, Inc. Decentralized management of virtualized hosts
US9026630B2 (en) 2011-06-14 2015-05-05 Vmware, Inc. Managing resources in a distributed system using dynamic clusters
CN103649910A (en) * 2011-07-11 2014-03-19 惠普发展公司，有限责任合伙企业 Virtual machine placement
US8732693B2 (en) * 2011-08-04 2014-05-20 Microsoft Corporation Managing continuous software deployment
US8943220B2 (en) 2011-08-04 2015-01-27 Microsoft Corporation Continuous deployment of applications
US9038055B2 (en) 2011-08-05 2015-05-19 Microsoft Technology Licensing, Llc Using virtual machines to manage software builds
US9785459B2 (en) * 2011-08-22 2017-10-10 Red Hat Israel, Ltd. Managed network filter/forward programming in a virtualization system by an agent more privileged than the hypervisor
US9098431B2 (en) * 2011-11-23 2015-08-04 Red Hat, Inc. USB redirection for interrupt transactions
JP5709735B2 (en) * 2011-12-08 2015-04-30 三菱電機株式会社 Virtual computer system
JP5825360B2 (en) * 2012-01-27 2015-12-02 富士通株式会社 Information processing apparatus, information processing system, communication data output method, and communication data output program
US9218196B2 (en) * 2012-05-17 2015-12-22 International Business Machines Corporation Performing pre-stage replication of data associated with virtual machines prior to migration of virtual machines based on resource usage
US20140092753A1 (en) * 2012-09-28 2014-04-03 Cisco Technology, Inc. Traffic-based quality of service (qos) monitoring in highly constrained networks
US9762471B2 (en) * 2013-01-26 2017-09-12 F5 Networks, Inc. Methods and systems for estimating and analyzing flow activity and path performance data in cloud or distributed systems
JP2014164425A (en) * 2013-02-22 2014-09-08 Sony Corp Information processing apparatus, resource control method, and program
US9430259B2 (en) 2013-03-15 2016-08-30 Centurylink Intellectual Property Llc Virtualization congestion control framework for modifying execution of applications on virtual machine based on mass congestion indicator in host computing system
US9407519B2 (en) * 2013-03-15 2016-08-02 Vmware, Inc. Virtual network flow monitoring
US9141416B2 (en) 2013-03-15 2015-09-22 Centurylink Intellectual Property Llc Virtualization congestion control framework for modifying execution of applications on virtual machine based on mass congestion indicator in host computing system
US9940151B2 (en) 2013-03-15 2018-04-10 Telefonaktiebolaget Lm Ericsson (Publ) Hypervisor and physical machine and respective methods therein for performance measurement
US9363190B2 (en) * 2013-07-31 2016-06-07 Manjrasoft Pty. Ltd. System, method and computer program product for energy-efficient and service level agreement (SLA)-based management of data centers for cloud computing
US10389577B2 (en) 2013-08-14 2019-08-20 Centurylink Intellectual Property Llc Ethernet carrier group alarm (CGA)
WO2015032430A1 (en) * 2013-09-04 2015-03-12 Telefonaktiebolaget L M Ericsson (Publ) Scheduling of virtual machines
US20150071091A1 (en) * 2013-09-12 2015-03-12 Alcatel-Lucent Usa Inc. Apparatus And Method For Monitoring Network Performance
JP2015069384A (en) * 2013-09-27 2015-04-13 富士通株式会社 Information processing system, control method for information processing system, and control program for information processor
US9864623B2 (en) 2013-11-21 2018-01-09 Centurylink Intellectual Property Llc Physical to virtual network transport function abstraction
US10110710B2 (en) 2014-04-03 2018-10-23 Centurylink Intellectual Property Llc System and method for implementing extension of customer LAN at provider network service point
US9998320B2 (en) 2014-04-03 2018-06-12 Centurylink Intellectual Property Llc Customer environment network functions virtualization (NFV)
US9465715B2 (en) * 2014-06-12 2016-10-11 Oracle International Corporation Optimizing the number of shared processes executing in a computer system
US10225327B2 (en) 2014-08-13 2019-03-05 Centurylink Intellectual Property Llc Remoting application servers
US9898318B2 (en) 2014-08-15 2018-02-20 Centurylink Intellectual Property Llc Multi-line/multi-state virtualized OAM transponder
WO2016048382A1 (en) * 2014-09-26 2016-03-31 Hewlett Packard Enterprise Development Lp Storage of hypervisor messages in network packets generated by virtual machines
US10867264B2 (en) * 2014-10-31 2020-12-15 Xerox Corporation Methods and systems for estimating lag times in a cloud computing infrastructure
EP3278506B1 (en) * 2015-04-01 2020-02-12 Telefonaktiebolaget LM Ericsson (publ) Methods and devices for monitoring of network performance for container virtualization
WO2016173615A1 (en) * 2015-04-27 2016-11-03 Telefonaktiebolaget Lm Ericsson (Publ) Compute infrastructure resource monitoring method and entities
US10481938B2 (en) 2015-05-06 2019-11-19 Centurylink Intellectual Property Llc System and method for implementing network experience shifting
US10673978B2 (en) 2015-05-06 2020-06-02 Centurylink Intellectual Property Llc Method and system for implementing network experience shifting using shared objects
US20170005888A1 (en) * 2015-06-30 2017-01-05 Tieto Oyj Network element monitoring
WO2017030604A1 (en) * 2015-08-20 2017-02-23 Ruckus Wireless, Inc. Virtual-machine dataplane having fixed interpacket time
US20170093677A1 (en) * 2015-09-25 2017-03-30 Intel Corporation Method and apparatus to securely measure quality of service end to end in a network
US9882833B2 (en) 2015-09-28 2018-01-30 Centurylink Intellectual Property Llc Intent-based services orchestration
US10078528B2 (en) 2015-10-06 2018-09-18 Centurylink Intellectual Property Llc Virtual machine-to-port peripheral device driver for implementing communications between virtual machines and client devices
EP3433735A1 (en) * 2016-03-23 2019-01-30 Intel IP Corporation Method and system to perform performance measurements job operations
CN105975385A (en) * 2016-04-28 2016-09-28 浪潮(北京)电子信息产业有限公司 Fuzzy neural network-based virtual machine energy consumption prediction method and system
WO2017189015A1 (en) * 2016-04-29 2017-11-02 Intel IP Corporation Network function virtualization
US10271352B2 (en) * 2016-07-19 2019-04-23 Realtek Semiconductor Corp. Wireless communication system and associated wireless communication method and wireless device having efficient polling mechanism in overlapping network environments
US11088988B2 (en) 2016-08-30 2021-08-10 Arris Enterprises Llc Virtual-machine dataplane with DHCP-server functionality
US20180088977A1 (en) * 2016-09-28 2018-03-29 Mark Gray Techniques to determine and mitigate latency in virtual environments
US10628233B2 (en) * 2016-12-30 2020-04-21 Samsung Electronics Co., Ltd. Rack-level scheduling for reducing the long tail latency using high performance SSDS
US20190007292A1 (en) * 2017-06-28 2019-01-03 Cpacket Networks Inc. Apparatus and method for monitoring network performance of virtualized resources
CN109426911B (en) * 2017-08-31 2022-04-05 华为技术有限公司 Method and device for evaluating software running environment quality of equipment
TWI703835B (en) * 2019-01-25 2020-09-01 中華電信股份有限公司 System and method for provisioning and monitoring virtual machine virtual network interface controller
KR102269647B1 (en) * 2019-03-20 2021-06-25 주식회사 팀스톤 Server performance monitoring apparatus
US12107670B2 (en) * 2019-04-29 2024-10-01 Comcast Cable Communications, Llc Container based time differentials
US11256624B2 (en) 2019-05-28 2022-02-22 Micron Technology, Inc. Intelligent content migration with borrowed memory
CN110932931A (en) * 2019-11-13 2020-03-27 苏宁云计算有限公司 Detection method and device for network delay of data center
US11165676B1 (en) * 2020-11-11 2021-11-02 Vmware, Inc. Generating network flow profiles for computing entities
US11765057B2 (en) * 2020-12-30 2023-09-19 Oracle International Corporation Systems and methods for performing end-to-end link-layer and IP-layer health checks between a host machine and a network virtualization device
CN114070754B (en) * 2021-11-15 2023-06-23 迈普通信技术股份有限公司 Method and system for testing migration performance of Overlay network virtual machine
US11816364B2 (en) * 2022-01-13 2023-11-14 Microsoft Technology Licensing, Llc Performance evaluation of an application based on detecting degradation caused by other computing processes

Family Cites Families (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6317775B1 (en) * 1995-11-03 2001-11-13 Cisco Technology, Inc. System for distributing load over multiple servers at an internet site
US8095640B2 (en) * 2003-12-12 2012-01-10 Alcatel Lucent Distributed architecture for real-time flow measurement at the network domain level
US7543161B2 (en) * 2004-09-30 2009-06-02 International Business Machines Corporation Method and apparatus for tracking variable speed microprocessor performance caused by power management in a logically partitioned data processing system
US7610266B2 (en) * 2005-05-25 2009-10-27 International Business Machines Corporation Method for vertical integrated performance and environment monitoring
US8145760B2 (en) * 2006-07-24 2012-03-27 Northwestern University Methods and systems for automatic inference and adaptation of virtualized computing environments
US8799918B2 (en) * 2006-09-11 2014-08-05 Microsoft Corporation Dynamic network load balancing using roundtrip heuristic
US8627335B2 (en) * 2006-11-13 2014-01-07 Oracle America, Inc. Method and apparatus for data space profiling of applications across a network
US8136124B2 (en) * 2007-01-18 2012-03-13 Oracle America, Inc. Method and apparatus for synthesizing hardware counters from performance sampling
US8209683B2 (en) * 2007-04-17 2012-06-26 International Business Machines Corporation System and method for probing hypervisor tasks in an asynchronous environment
US7853928B2 (en) * 2007-04-19 2010-12-14 International Business Machines Corporation Creating a physical trace from a virtual trace
US9459984B2 (en) * 2007-10-29 2016-10-04 Intel Corporation Method and systems for external performance monitoring for virtualized environments
GB2458157B (en) * 2008-03-07 2012-04-25 Hewlett Packard Development Co Virtual machine liveness check
US8832682B2 (en) * 2008-03-28 2014-09-09 Vmware, Inc. Trace collection for a virtual machine
US8521868B2 (en) * 2008-10-15 2013-08-27 International Business Machines Corporation Platform-level indicators of application performance
US7953092B2 (en) * 2009-04-08 2011-05-31 Ixia Traffic receiver using parallel capture engines
KR101094628B1 (en) * 2008-12-23 2011-12-15 주식회사 케이티 Network apparatus and method for monitoring real time service using timestamp
CN102334112B (en) * 2009-02-27 2014-06-11 美国博通公司 Method and system for virtual machine networking
US8635057B2 (en) * 2009-03-30 2014-01-21 Microsoft Corporation Enlightenment for low overhead hardware access
CN102460408B (en) * 2009-06-01 2014-08-06 惠普开发有限公司 System and method for collecting application performance data
US20120128011A1 (en) * 2009-06-16 2012-05-24 Holmeide Oeyvind Method on a network element for the purpose of synchronization of clocks in a network
US9077590B2 (en) * 2009-06-22 2015-07-07 Citrix Systems, Inc. Systems and methods for providing link management in a multi-core system
US8219676B2 (en) * 2009-06-22 2012-07-10 Citrix Systems, Inc. Systems and methods for web logging of trace data in a multi-core system
US8289960B2 (en) * 2009-06-22 2012-10-16 Citrix Systems, Inc. Systems and methods for N-core tracing
US8554892B2 (en) * 2009-06-22 2013-10-08 Citrix Systems, Inc. Systems and methods for n-core statistics aggregation
US8189487B1 (en) * 2009-07-28 2012-05-29 Sprint Communications Company L.P. Determination of application latency in a network node
US11132237B2 (en) * 2009-09-24 2021-09-28 Oracle International Corporation System and method for usage-based application licensing in a hypervisor virtual execution environment
US8504686B2 (en) * 2009-11-02 2013-08-06 InMon Corp. Method and apparatus for combining data associated with hardware resources and network traffic
US8311032B2 (en) * 2009-12-03 2012-11-13 International Business Machines Corporation Dynamically provisioning virtual machines
US8644352B1 (en) * 2010-03-12 2014-02-04 Marvell International Ltd. System and method for accurate time sampling in presence of output delay
US8930525B2 (en) * 2010-04-27 2015-01-06 Hewlett-Packard Development Company, L.P. Method and apparatus for measuring business transaction performance

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO2011096859A1 *

Also Published As

Publication number Publication date
EP3699758B1 (en) 2021-08-18
EP2531917B1 (en) 2019-11-27
WO2011096859A1 (en) 2011-08-11
US9063769B2 (en) 2015-06-23
US20120304175A1 (en) 2012-11-29
EP3699758A1 (en) 2020-08-26
EP2531917A4 (en) 2017-08-09

Similar Documents

Publication Publication Date Title
EP3699758B1 (en) Network performance monitor for virtual machines
EP3278506B1 (en) Methods and devices for monitoring of network performance for container virtualization
US9426072B2 (en) Traffic flow classification
US10387185B2 (en) Performing monitoring and migration of virtual machines in a data center to prevent service level degradation due to the load imposed by the monitoring
CN110865867B (en) Method, device and system for discovering application topological relation
US9183033B2 (en) Method and system for analyzing root causes of relating performance issues among virtual machines to physical machines
US8364820B2 (en) Data center resource usage and cost determination
Hu et al. Net-cohort: Detecting and managing vm ensembles in virtualized data centers
Huang et al. Prediction-based dynamic resource scheduling for virtualized cloud systems
US10938688B2 (en) Network costs for hyper-converged infrastructures
JP2013150134A (en) Computer system, virtual server arrangement method, and arrangement control device
Mogul et al. Inferring the network latency requirements of cloud tenants
US20130282895A1 (en) Correlation based adaptive system monitoring
US20230327967A1 (en) Generating network flow profiles for computing entities
Huang et al. Migration-based elastic consolidation scheduling in cloud data center
Maziku et al. Towards a network aware VM migration: Evaluating the cost of VM migration in cloud data centers
Hovestadt et al. Evaluating adaptive compression to mitigate the effects of shared i/o in clouds
US11477095B2 (en) Determining a service impact score for a metric according to a scope of the metric
JP2017182346A (en) Analysis program, analysis device, and analysis method
US10568112B1 (en) Packet processing in a software defined datacenter based on priorities of virtual end points
Hovestadt et al. Adaptive online compression in clouds—making informed decisions in virtual machine environments
WO2017218149A1 (en) Systems and methods for detecting a zombie server
US10904127B2 (en) Systems and methods for detecting a zombie server
Basta et al. Logically Isolated, Actually Unpredictable? Measuring Hypervisor Performance in Multi-Tenant SDNs
Xu et al. On the interplay between network traffic and energy consumption in virtualized environment: An empirical study

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20120624

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

DAX Request for extension of the european patent (deleted)
RA4 Supplementary search report drawn up and despatched (corrected)

Effective date: 20170707

RIC1 Information provided on ipc code assigned before grant

Ipc: G06F 9/455 20060101AFI20170704BHEP

Ipc: G06F 11/34 20060101ALI20170704BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20180208

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20190621

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1207455

Country of ref document: AT

Kind code of ref document: T

Effective date: 20191215

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602010062190

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: NL

Ref legal event code: FP

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200228

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200227

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200227

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200327

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200419

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602010062190

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20200229

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200204

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

26N No opposition filed

Effective date: 20200828

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200229

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200229

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200229

REG Reference to a national code

Ref country code: AT

Ref legal event code: UEP

Ref document number: 1207455

Country of ref document: AT

Kind code of ref document: T

Effective date: 20191127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20191127

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230523

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20240226

Year of fee payment: 15

Ref country code: IE

Payment date: 20240227

Year of fee payment: 15

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: AT

Payment date: 20240119

Year of fee payment: 15

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240228

Year of fee payment: 15

Ref country code: GB

Payment date: 20240227

Year of fee payment: 15

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20240226

Year of fee payment: 15