WO2013176918A1 - Cross-layer troubleshooting of application delivery - Google Patents
Cross-layer troubleshooting of application delivery Download PDFInfo
- Publication number
- WO2013176918A1 WO2013176918A1 PCT/US2013/040797 US2013040797W WO2013176918A1 WO 2013176918 A1 WO2013176918 A1 WO 2013176918A1 US 2013040797 W US2013040797 W US 2013040797W WO 2013176918 A1 WO2013176918 A1 WO 2013176918A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- application delivery
- visualization
- test results
- troubleshooting
- layer
- Prior art date
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0852—Delays
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/04—Network management architectures or arrangements
- H04L41/042—Network management architectures or arrangements comprising distributed management centres cooperatively managing the network
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/079—Root cause analysis, i.e. error or fault diagnosis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/22—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks comprising specially adapted graphical user interfaces [GUI]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/04—Processing captured monitoring data, e.g. for logfile generation
- H04L43/045—Processing captured monitoring data, e.g. for logfile generation for graphical visualisation of monitoring data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/06—Generation of reports
- H04L43/062—Generation of reports related to network traffic
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
Definitions
- Cloud computing generally refers to the use of computing resources (e.g., hardware and software) that are delivered as a service over a network (e.g., typically, the Internet). Cloud computing includes using remote services to provide a user's data, software, and computation.
- computing resources e.g., hardware and software
- a network e.g., typically, the Internet
- Distributed applications can generally be delivered using cloud computing techniques.
- distributed applications can be provided using a cloud computing model, in which users are provided access to application software and databases over a network.
- the cloud providers generally manage the infrastructure and platforms on which the applications run.
- Various types of distributed applications can be provided as a software as a service (SaaS).
- Figure 1 illustrates a functional block diagram of a platform for cross-layer troubleshooting of application delivery in accordance with some embodiments.
- Figure 2 illustrates a network performance visualization in accordance with some embodiments.
- Figure 3 illustrates a graph of a visualization of a data path to a destination site in accordance with some embodiments.
- Figure 4A illustrates a graph of a full connectivity visualization of a data path to a destination site in accordance with some embodiments.
- Figure 4B illustrates a graph providing a simplified view of a visualization of a data path to a destination site in accordance with some embodiments.
- Figure 4C illustrates a graph providing a simplified view of a visualization of a data path to a destination site with problem nodes in accordance with some embodiments.
- Figure 4D illustrates a graph providing a selective exploration expanding a data path from selected source/agent site in accordance with some embodiments.
- Figure 5A is screen shot illustrating an HTTP availability drop during a particular time interval in accordance with some embodiments.
- Figure 5B is screen shot illustrating a network view that shows packet loss from the agents shown in Figure 5A in accordance with some embodiments.
- Figure 5C is screen shot illustrating a visualization of a problem zone shown in Figure 5B in accordance with some embodiments.
- Figure 6 illustrates a graph of a visualization of routing paths in accordance with some embodiments.
- Figure 7 illustrates a graph of a visualization of routing paths including a series of nodes indicating data loss in accordance with some embodiments.
- Figure 8 A and Figure 8B illustrate views for showing Border Gateway
- Figure 9 illustrates a visualization of DNS data in accordance with some embodiments.
- Figure 10 illustrates Anycast debugging, to identify any relevant DNS problems, in accordance with some embodiments.
- Figure 11 illustrates a path visualization that shows all problem agents routing to an Ashburn, Virginia instance in accordance with some embodiments.
- Figure 12 illustrates a visualization of HTTP data in accordance with some embodiments.
- Figure 13 illustrates a visualization of a web page load performance in accordance with some embodiments.
- Figure 14 illustrates a bottleneck analysis by provider for page loads in accordance with some embodiments.
- Figure 15 illustrates a visualization of transactions (e.g., web transactions) in accordance with some embodiments.
- Figure 16 shows a summary of transaction steps and identifying bottlenecks in accordance with some embodiments.
- Figure 17 illustrates a flow diagram for cross-layer troubleshooting of application delivery in accordance with some embodiments.
- Figure 18 illustrates another flow diagram for cross-layer troubleshooting of application delivery in accordance with some embodiments.
- the invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor.
- these implementations, or any other form that the invention may take, may be referred to as techniques.
- the order of the steps of disclosed processes may be altered within the scope of the invention.
- a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task.
- the term 'processor' refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
- Cloud computing generally refers to the use of computing resources (e.g., hardware and software) that are delivered as a service over a network (e.g., typically, the Internet). Cloud computing includes using remote services to provide a user's data, software, and computation.
- computing resources e.g., hardware and software
- a network e.g., typically, the Internet
- Distributed applications can generally be delivered using cloud computing techniques.
- distributed applications can be provided using a cloud computing model, in which users are provided access to application software and databases over a network.
- the cloud providers generally manage the infrastructure and platforms on which the applications run.
- Various types of distributed applications can be provided as a software as a service (SaaS).
- Cloud-based distributed applications e.g., distributed applications
- a web browser e.g., a light-weight desktop
- mobile application e.g., mobile app
- cloud-based distributed applications can allow enterprises to get their applications up and running faster, with improved manageability and less maintenance, and can enable enterprise IT to more rapidly adjust resources to meet fluctuating and unpredictable business demand.
- distributed applications can allow a business to reduce Information Technology (IT) operational costs by outsourcing hardware and software maintenance and support to the cloud provider.
- IT Information Technology
- various techniques for cross-layer troubleshooting of application delivery are disclosed.
- various techniques are provided for cross-layer visualization and troubleshooting of application delivery, such as for performance problems associated with distributed applications.
- a platform for identifying and/or determining performance problems associated with a distributed application(s) can be provided.
- the platform can generate reports that include various cross- layer visualizations that facilitate identifying and/or determining performance problems associated with a distributed application(s).
- various techniques described herein can be used to diagnose application deliver problems from cloud service providers, such as SaaS and/or other network delivered based applications (e.g,. web sites, online stores, cloud based software, and/or other such network based applications and/or services) to determine the causes or sources of the application delivery performance issues or problems.
- cloud service providers such as SaaS and/or other network delivered based applications (e.g,. web sites, online stores, cloud based software, and/or other such network based applications and/or services) to determine the causes or sources of the application delivery performance issues or problems.
- cross-layer troubleshooting of application delivery includes collecting test results from a plurality of distributed agents for a plurality of application delivery layers; and generating a graphical visualization of an application delivery state based on the test results for the plurality of application delivery layers (e.g., different application delivery layers).
- the graphical visualization of the application delivery state can facilitate cross-layer troubleshooting of problems (e.g., associated with application delivery of a distributed application).
- the graphical visualization of the application delivery state can facilitate cross-layer visualization and troubleshooting of application delivery (e.g., associated with application delivery of a distributed application, which can identify and/or facilitate diagnosing of causes of application delivery problems).
- cross-layer troubleshooting of application delivery further includes outputting the graphical visualization of the application delivery state based on the test results for the plurality of application delivery layers.
- the graphical visualization of the application delivery state based on the test results for the plurality of application delivery layers facilitates cross- layer visualization and troubleshooting of problems (e.g., associated with application delivery of a distributed application).
- the graphical visualization of the application delivery state based on the test results for the plurality of application delivery layers facilitates cross- layer troubleshooting of application delivery by providing for correlation of the test results across a plurality of layers (e.g., application delivery layers).
- the graphical visualization of the application delivery state based on the test results for the plurality of layers facilitates cross-layer troubleshooting of application delivery by providing for correlation of the test results across a plurality of layers using a space domain and/or a time domain.
- the plurality of distributed agents are controlled by an agent controller.
- the graphical visualization of network performance is generated by a platform for cross-layer visibility and troubleshooting of distributed applications 100, such as shown in Figure 1.
- the network performance test results from the plurality of distributed agents for the plurality of layers are stored in a storage tier (e.g., which can include a database or another type of data store).
- cross-layer visibility and troubleshooting of distributed applications includes using software agents to collect information from different points in a network across different layers in the application delivery (e.g., of a distributed application), as further described herein with respect to various embodiments.
- information can be collected and aggregated by centralized collectors and aggregators and presented to a user as a Software as a Service (SaaS).
- SaaS Software as a Service
- different layers are correlated using one or more of the following: a monitored object, time, and location.
- various techniques described herein allow users to drill down to identify problems and locations whether at different layers (e.g., a network, transport, and/or application layer) to correlate across application delivery layers to determine whether, for example, network issues are affecting the performance of a distributed application(s), as further described herein with respect to various embodiments.
- such techniques can be applied to both internal network diagnostics (e.g., for an organization's internal network) and/or external network diagnostics (e.g., for a web site accessed across the Internet, such as for a cloud-based distributed application).
- various techniques described herein allow for determination of a network topology to indicate, for example, anycast routing issues used by DNS providers (e.g., such as when there are multiple DNS servers sharing the same IP address) as well as structure and/or activity of a network topology, as further described herein with respect to various embodiments.
- DNS providers e.g., such as when there are multiple DNS servers sharing the same IP address
- structure and/or activity of a network topology as further described herein with respect to various embodiments.
- such techniques can be applied to both internal network diagnostics (e.g., for an organization's internal network) and/or external network diagnostics (e.g., for a web site accessed across the Internet, such as for a cloud- based distributed application).
- various techniques described herein allow for diagnosing SaaS applications, such as for cloud-based distributed applications, using, for example, a lightweight agent, as further described herein with respect to various embodiments.
- cross-layer visibility and troubleshooting of distributed applications includes a distributed framework to distribute tests across different agents in the Internet.
- agents can be executed on hosted providers using cloud computing distributed across multiple ISPs, which are controlled by agent controllers to perform one or more tests as further described herein, in which the test results can be collected for correlation and analysis, as further described herein with respect to various embodiments.
- agents are computing resources that are controlled, and, for example, can be either virtual or dedicated servers.
- Agents can be distributed across different geographies and networks, for example, distributed agents can be distributed to mostly Tier-1 and Tier-2 networks to avoid the noise of bad connectivity of last mile connections.
- FIG. 1 An example of a system architecture for providing cross-layer visibility and troubleshooting of distributed applications is shown in Figure 1 as described below.
- Figure 1 illustrates a functional block diagram of a platform for cross-layer troubleshooting of application delivery in accordance with some embodiments.
- Figure 1 illustrates an environment in which a platform for cross-layer visibility and troubleshooting of distributed applications 100 includes distributed agents 116-120 (e.g., which can be distributed across various geographies and/or devices for performing different types of tests and/or targeting different sites, locations, and/or metrics) that collect data based on configured tests, and the distributed agents 116-120 send this data to a controller(s) 114 (e.g., agent controllers).
- distributed agents 116-120 e.g., which can be distributed across various geographies and/or devices for performing different types of tests and/or targeting different sites, locations, and/or metrics
- controller(s) 114 e.g., agent controllers
- the controller 114 stores the data in a storage tier 112 (e.g., providing permanent storage) that can be used by a web tier 104 to generate visualizations and reports to users accessing the platform 100 using client devices (e.g., computers, laptops, smartphones, and/or various other computing devices).
- a storage tier 112 e.g., providing permanent storage
- client devices e.g., computers, laptops, smartphones, and/or various other computing devices.
- a report can be output to a user to present the collected and analyzed cross-layer application delivery information of a distributed application.
- Example reports can include various visualizations and/or diagnostic information as further described herein with respect to various embodiments.
- the report can facilitate troubleshooting application delivery associated with the distributed application to determine whether performance problems are the result of the cloud provider of the distributed application, the customer's own internal IT network, a user's client device, and/or intermediate network providers between the user's client device and the cloud provider.
- the report can also include recommendations to the user to resolve any such determined application delivery problems associated with the distributed application.
- the report can also be provided to a third party, such as the SaaS provider of the distributed application and/or a network provider, which can be provided as information to indicate the source of such determined application delivery problems associated with the distributed application.
- the user of client device 106 (hereinafter referred to as
- Bob is employed as an IT manager of a distributed application company (“SaaS Company”).
- SaaS Company distributed application company
- Alice is employed as an IT manager of a national company (“ACME Company”).
- CAME Company a national company
- Bob and Alice can each access the services of platform 100 (e.g., platform for cross-layer visibility and troubleshooting of distributed applications) via web tier 104 over a network, such as the Internet.
- platform 100 e.g., platform for cross-layer visibility and troubleshooting of distributed applications
- the techniques described herein can work with a variety of client devices 106-108 including, but not limited to personal computers, tablet computers, smartphones, and/or other computing devices.
- platform 100 generates various reports based on results of the network performance tests to facilitate cross-layer visibility and troubleshooting of application delivery associated with a distributed application(s), as further described herein.
- platform 100 includes a data store, such as storage tier 112 for storing results of the network performance tests and/or the reports.
- a set of agent controllers 114 is provided as shown to send various tests (e.g., such as the various test described herein with respect to various embodiments) to the distributed agents for execution by the distributed agents.
- agents can be executed on hosted providers using cloud computing distributed across multiple ISPs, which are controlled by agent controllers to perform one or more tests as further described herein, in which the test results can be collected for correlation and analysis, as further described herein with respect to various embodiments.
- the tests are configured through a web interface by a user.
- typical parameters can include the frequency of various tests, the target of the tests, and the agents (e.g., or locations) where the tests are to be performed.
- the test parameters can be sent from the controller (e.g., agent controllers 114) to the distributed agents after an agent checks-in (e.g., using a pull mechanism). After an agent executes a test, the agent can export the test result(s) back to the controller. The controller can then provide the results back to a data store (e.g., storage tier 112) for permanent storage (e.g., or temporary storage).
- a controller can also send on-demand tests to an agent(s) through, for example, a Remote Procedure Call (RPC) call for immediate or on- demand execution.
- RPC Remote Procedure Call
- platform 100 is a scalable, elastic architecture and may comprise several distributed components, including components provided by one or more third parties. Further, when platform 100 is referred to as performing a task, such as storing data or processing data, it is to be understood that a sub-component or multiple sub- components of platform 100 (whether individually or in cooperation with third party components) may cooperate to perform that task.
- tests include various types of tests to facilitate cross- layer visibility and troubleshooting of application delivery associated with a distributed application(s), as further described herein.
- Example network tests include data path measurement tests, routing path measurement tests, and end-to-end network metrics tests.
- Example DNS tests include per name server testing and Domain Name System Security Extensions (DNSSEC) bottom-up validation tests.
- Example HTTP tests include testing of steps of a Uniform Resource Locator (URL) fetch.
- Example page load tests include testing of a load of an entire web page using a web browser (e.g., a typical web browser).
- Example transaction tests include performing a multi-step scripted transaction from a web browser (e.g., a typical web browser). These and various other tests are discussed in greater detail below.
- platform 100 can provide users with a natural flow of action for root cause identification of problems as further described herein with respect to various embodiments.
- tests aggregate data over certain time intervals (e.g., small time intervals), referred to as rounds.
- a round includes one sample of data from each agent (e.g., each agent can be placed in a different geographical location) for a given test.
- data over each round is aggregated for a set of agents (e.g., all the agents) and, for example, can be shown on a timeline.
- a timeline is implemented as a navigation widget that enables users to see the aggregate behavior of all of the agents over time (e.g., or just an individual agent) and to click on a specific point in time for a detailed drill down (e.g., to view specific cross-layer network test related data at a specified time or time window/interval).
- a user can be provided with several metrics to choose from, and a timeline can be associated with one or more metrics.
- some tests are composed of multiple subtests that are run at the same time.
- the subtests can each be different and associated with different layers of visibility. Performing different active tests at the same time for different layers allows for switching between subtests while keeping the same test and round (e.g., and eventually the same agent location).
- a test can include an HTTP subtest and a network subtest at the same time. Then the user can navigate from HTTP metrics, to end-to- end network metrics, all the way down to individual router hop metrics (e.g., by inspecting the data path). For example, this can be useful to identify/classify problems that are network induced versus problems that are back-end related.
- each layer L0 has an associated vector of features, referred to as the context vector, C LO , that includes all dimensions that uniquely identify a view in layer L0.
- C LO context vector
- a function is applied to C LO to provide C LI , such as for example, the following:
- Ci_i FLO->LI(CI_O)-
- GUI Graphical User Interface
- a user e.g., Bob or Alice
- GUI Graphical User Interface
- application delivery layers e.g., path visualization, BGP metrics, BGP route visualization, etc.
- web layers e.g., basic HTTP, etc.
- a set of layers are correlated, and reports including visualizations that present results of test performed at the different correlated layers can be accessed, such as using hyperlinks in a GUI menu that can encode the context vectors as GET parameters in the URLs.
- users can jump to Path Visualization, BGP Metrics, BGP Route Visualization, and Basic HTTP views while maintaining the context as discussed above.
- Table 1 shows a transition table between different layers, indicating from which layers (rows) is it possible to jump to (columns) in accordance with some embodiments.
- an approach for measuring response time of network level metrics from a network server is provided using a train of TCP SYN packets (synchronise packets in transmission control protocol). For example, providing such a train of TCP SYN packets appears to the network server to be like any normal connection that the network server will normally respond to; and while the network server may throttle such connections, even if throttling occurs, that would still generally provide an accurate reflection of a typical user connection response time as that network server is throttling for other such user connections as well.
- TCP SYN packets synchronise packets in transmission control protocol
- ICMP Internet Control Message Protocol
- ISPs Internet Service Providers
- active measurements from agents e.g., distributed agents
- servers can be identified by host name (or IP address) and TCP port number.
- host name or IP address
- TCP port number Periodically, a train of N TCP SYN packets is sent to each server from each agent. A full TCP connection is not established, because the client sends a TCP RST (TCP reset) right after receiving the SYN ACK response from the server.
- N points allow for measuring, for example, the following: (1) packet loss (e.g., ratio of sent packets that were ACKed), (2) average network delay (e.g., time between SYN and ACK), and (3) network jitter (e.g., average delta between consecutive delays).
- packet loss e.g., ratio of sent packets that were ACKed
- average network delay e.g., time between SYN and ACK
- network jitter e.g., average delta between consecutive delays.
- Figure 2 illustrates a network performance visualization in accordance with some embodiments.
- Figure 2 is a screen shot 200 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- users can select the target (e.g., server) that they want to look at, such as shown in this example, www.example.com, port 443 (SSL).
- target e.g., server
- SSL port 443
- there is also an option to select the network metric such as shown, in this example; "loss" (or packet loss) is selected.
- a timeline is provided that shows the time-series of the metric.
- the timeline allows a user to select an instant in or interval of time to drill down to (e.g., interactive timeline that allows users to select a specified time or time interval for drill down for further analysis, such as to look at a window of time of high packet loss to try to determine root cause(s) of such high packet loss).
- a world map depicts the locations of the distributed agents as well as a visual indication of their status according to the selected metric 204.
- the red dots e.g., shown as completely shaded circles or another graphical indicator to differentiate these icons
- the green dots e.g., shown as completely partially shaded or hashed circles or another graphical indicator to differentiate these icons
- a summary box 210 provides a concise report of the metrics.
- a table 212 shows a more detailed breakdown of the metrics as measured by each agent. For example, the table can be sorted by any column to facilitate the analysis of the data.
- a "Run Test" button 214 is provided that allows users to run on-demand tests to the target (e.g., target server). For example, this can be useful to verify if a problem reported by an automated test is still occurring.
- Traceroute is generally a computer network diagnostic tool for displaying the route (path) and measuring transit delays of packets across an Internet Protocol (IP) network. While a traceroute diagnostic using ICMP packets can be used to measure data paths, such an approach may not be effective on some networks as many ISPs block ICMP packets. Accordingly, in some embodiments, in order to collect data paths, Time To Live (TTL) limited TCP SYN packets are sent to a specific destination server.
- TTL Time To Live
- TTL Time Exceeded
- probes are sent with increasing TTL to collect the source IP addresses of the ICMP packets to reconstruct the path packets are taking.
- special precaution can be taken to avoid issues with load balancing.
- TCP if the same 5 tuple (e.g., source IP, destination IP, source port, destination port, and protocol) is kept between probes, balancers will send packets in that flow through the same interfaces.
- vReply SendTCPPacket (vTTL, vDestination, vSourcePort, vDestPort) ;
- Algorithm 1 Data path measurements.
- data delivery is represented in the network in a per-hop model in which each hop represents an interface of a physical router.
- a graph of a data path visualization of a data path to a destination site e.g., a destination, such as port 443 (SSL) of www.example.com, or some other destination
- SSL port 443
- Figure 3 illustrates a graph of a visualization of a data path to a destination site in accordance with some embodiments.
- Figure 3 is a screen shot 300 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- users can select the test name (e.g., test to be performed by distributed agents).
- the network metric such as shown, in this example; "loss" (or packet loss) is selected.
- various agent filters e.g., configurations or settings
- a graph 308 is presented to provide a data path visualization of a data path to a destination site in accordance with some embodiments.
- each path of the graph starts with an agent node in the left side that is shown with icon indicators (e.g., shown as colored circles, completely shaded circles, or another graphical indicator to differentiate these icons) according to each agent's results or status relative to the selected metric, in this case, packet loss.
- icon indicators e.g., shown as colored circles, completely shaded circles, or another graphical indicator to differentiate these icons
- packet loss e.g., packet loss metric
- nodes with a red outline can represent points in the network dropping packets.
- such nodes can indicate the last responsive hops in the path, such that the packet loss is indicated to most likely be happening in the immediate hop.
- the data path visualization can still display hops with packet loss to illustrate potential problematic hops (e.g., regardless of the number of hops to be shown, all bad nodes with losses can be presented in the visualization graph 308).
- the network delay between consecutive hops can also be determined (e.g., approximated or estimated) using this technique.
- agent filter 306 a control "Color links with delay > x ms" is provided that can provide an icon indicator (shown as colored circles such as red circles, completely shaded circles, or another graphical indicator to differentiate these icons) for links that have more than a selected number of milliseconds (ms) (e.g., 100 ms as shown).
- ms milliseconds
- the threshold of link delay can be dynamically changed using agent filter 306, which then automatically updates the data path visualization accordingly.
- a hop expansion control of agent filter 306 allows the topology to be expanded any number of hops from any side (e.g., agent or server) of the topology. For example, this can provide a useful agent filter control from the server side, because that is typically a customer actionable zone.
- the links with a number on top are aggregation links that are expanded as the number of hops in the filter increases or just by clicking in the number.
- another useful feature is the ability to select nodes and links in the topology shown in graph 308. For example, selected nodes and links can be inserted into a selection box, and their position in the topology can be tracked over time by clicking in different points in the timeline. In this manner, routing changes can be tracked over time. In some embodiments, double clicking in a specific node selects all nodes and links belonging to routes going through that node, making it easier to track entire routes over time.
- the metric selection affects timeline
- the users can select from one of three end-to-end metrics, such as packet loss, latency, and jitter.
- the values of these metrics are computed from a train of n TCP SYN packets sent to the destination about the same time the data path measurements take place to preserve time correlation between the end-to-end effect and the per-hop effect.
- agent x is experiencing a very high end-to-end latency to reach the destination
- very high end-to-end latency to reach the destination can be displayed with a color red (e.g., coloring of the agent icons or some other icon indicator) in the graph 308, and by looking at the breakdown per hop, where the bottleneck is located can be determined.
- red e.g., coloring of the agent icons or some other icon indicator
- the visualization layout (e.g., using a version of the Sugiyama's algorithm or another algorithm) can display a graph hop by hop and minimize the link crossings between nodes, making the graph more readable.
- Figure 4A illustrates a graph of a full connectivity visualization of a data path to a destination site in accordance with some embodiments.
- Figure 4 A is a screen shot 400 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- various agent filters e.g., configurations or settings
- a graph 408 is presented to provide a data path visualization of a data path to a destination site in accordance with some embodiments.
- Figure 4A shows a hop- by-hop data path view from the agents (e.g., the agents as shown and discussed below with respect to Figures 5A-5C) to a target site, such as example web site (e.g., www.example.com).
- agents e.g., the agents as shown and discussed below with respect to Figures 5A-5C
- target site such as example web site (e.g., www.example.com).
- Figure 4B illustrates a graph providing a simplified view of a visualization of a data path to a destination site in accordance with some embodiments.
- Figure 4B is a screen shot 420 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- various agent filters e.g., configurations or settings
- a graph 428 is presented to provide a data path visualization of a data path to a destination site in accordance with some embodiments.
- Figure 4B is a simplified view of the Figure 4A view of the hop- by-hop data path from agents to a target site, such as example web site (e.g., www.example.com), in which only show two hops away from the web site are shown (e.g., each IP address associated with the target site (e.g., example web site), which is associated with these three IP addresses, and hiding/consolidating all other hops except that all nodes with losses (e.g., nodes with losses exceeding a threshold) are shown in this view. For example, if a user selects " 3 nodes with losses" as shown at component (e.g., a sub-window) 426 of Figure 4B, then Figure 4C is shown in accordance with some embodiments, as discussed below.
- a target site such as example web site (e.g., www.example.com)
- each IP address associated with the target site e.g., example web site
- hiding/consolidating all other hops except that all nodes
- Figure 4C illustrates a graph providing a simplified view of a visualization of a data path to a destination site with problem nodes in accordance with some embodiments.
- Figure 4C is a screen shot 440 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- Figure 14C illustrates a simplified view of a graph 448 with problem nodes selected in accordance with some embodiments.
- Figure 14C shows the result of a user selecting "3 nodes with losses" as shown at component (e.g., a sub- window) 426 of Figure 4B. For example, this can allow the user to drill down to see more information regarding selected problem nodes.
- component e.g., a sub- window
- each of the problem nodes is associated with a common provider that would be listed in the details window for each of these three problem nodes. If the user selects link 446 shown as (8) hops between Amsterdam, Netherlands, then the next hop/node is shown with losses as shown in Figure 14D in accordance with some embodiments, as discussed below.
- Figure 4D illustrates a graph providing a selective exploration expanding a data path from selected source/agent site in accordance with some embodiments.
- Figure 4D is a screen shot 460 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- Figure 4D illustrates a graph 468 providing a selective exploration expanding a data path from Amsterdam in accordance with some embodiments.
- Figure 4D illustrates an expanded data path if a user selects the link 446 shown as (8) hops between Amsterdam, Netherlands and the next hop/node shown with losses as shown in Figure 4C, which thereby allowing for an expanded view of the intermediate hops on the data path from Amsterdam to that problem node as shown in Figure 4C.
- Figures 5A-5C illustrate examples of using cross-layer visibility of application delivery to determine, at different layers of a network, sources of one or more application delivery problems or performance issues in accordance with some embodiments, as discussed below.
- Figure 5 A is screen shot illustrating an HTTP availability drop during a particular time interval in accordance with some embodiments.
- Figure 5A is a screen shot 500 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- Figure 5 A shows a drop in HTTP availability at time interval
- a window is provided that shows a number of error by type, including the following: DNS, Connect (2 errors as shown), SSL, Send, Receive (1 error), and HTTP.
- the details of these errors are also shown in a table format, in which a first column is a location of agent, date is a time of measurement, server IP is the destination address (e.g., target for HTTP availability testing), response code (e.g., HTTP response code, which is 200 if there are no errors, or empty if no response code received), and a number of redirects, error type (e.g., as discussed above), and error details (e.g., providing a description of the particular error, and a run test option can be provided to further diagnose this problem and to check if the problem still persists).
- Figure 5B is screen shot illustrating a network view that shows packet loss from the agents shown in Figure 5A in accordance with some embodiments.
- Figure 5B is a screen shot 520 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- Figure 5B illustrates a network view that shows packet loss, from the same agents as shown in Figure 5A, in accordance with some embodiments. More specifically, Figure 5B is a network view of the same destination at the same time interval for same destination, as shown in Figure 5A. However, Figure 5B provides a different view, a network view as shown, which indicates that the problem is a packet loss issue at the network layer. Figure 5B illustrates that the cause of the HTTP availability drop shown in Figure 5A is revealed by three agents that each have significant packet loss, such as shown in the details of the packet losses of these three agents in table 526, which are the same agents that were shown in Figure 5A with problems on the HTTP tests/measurements.
- Figure 5B reveals that the problem is not at the HTTP layer, but rather is a problem at the network layer at this particular time interval. Accordingly, this illustrates an example of various techniques disclosed herein for cross-layer visibility and troubleshooting of application delivery in accordance with some embodiments.
- Figure 5C is screen shot illustrating a visualization of a problem zone shown in Figure 5B in accordance with some embodiments.
- Figure 5C is a screen shot 540 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- Figure 5C illustrates a path visualization 546 highlighting a problem zone as shown at 544 (e.g., in which an earlier time as shown at 542 indicates that they share a common next hop, indicating the possible problem location or problem zone) in accordance with some embodiments. More specifically, Figure 5C illustrates a hop-by-hop path visualization to determine the problem zone causing the packet losses, as shown in Figure 5B.
- the dashed circled nodes e.g., some other icon indicator
- Border Gateway Protocol is a standard protocol used to exchange routing information between different Autonomous Systems (AS) (e.g., which is the control plane between different networks, such as between Verizon networks and Time Warner networks).
- AS Autonomous Systems
- An AS is an independently managed domain (e.g., an organization, such as Dell, which is associated with a particular AS number, such as AS number 3614, which is a unique identifier for Dell's network), typically having a one-to-one mapping to an organization.
- BGP messages carry routing information for individual prefixes (e.g., or group of IP addresses), and originate from the traffic destination outwards (e.g., BGP message propagation follows the opposite direction of traffic propagation).
- routing information is collected from public BGP data repositories that archive routing information from hundreds of routers across the Internet. For example, by looking at the AS PATH attribute of each message sent from router R, the routing path R was using at each time can be determined, and this information can also be used to determine when a certain destination IP address (e.g., or prefix) is/was not reachable from R.
- a certain destination IP address e.g., or prefix
- reachability refers to the fraction of time that the router can reach a certain destination prefix.
- Path changes refers to the number of times the attribute AS PATH changed for a certain destination prefix.
- Updates refers to the plain count of BGP update messages received during the time interval.
- the BGP route information can be collected from Route Views and/or RIPE-RIS (Routing Information Service), which publish BGP route information.
- the BGP route information can be correlated with various other layers of network information to allow for visualization of this data and analysis of this data in a manner that facilitates cross-layer visibility and troubleshooting of application delivery (e.g., to determine if a problem is caused by a BGP routing information related problem, data path packet loss related problem, HTTP related problem, DNS related problem, and/or some other problem).
- a BGP routing issue at a particular time interval may be a cause of various network problems during that time interval (e.g., HTTP measurement errors may actually be a result of this BGP routing issue, such as when looking at data path loss errors in which there are random drops at random locations, a user can then look to see if problem is at BGP level in which there may be no announced route to get to that network AS number, as that will cause the routers to just drop such packets for which it has no routing information).
- HTTP measurement errors may actually be a result of this BGP routing issue, such as when looking at data path loss errors in which there are random drops at random locations, a user can then look to see if problem is at BGP level in which there may be no announced route to get to that network AS number, as that will cause the routers to just drop such packets for which it has no routing information).
- Figure 6 illustrates a graph of a visualization of routing paths in accordance with some embodiments.
- Figure 6 is a screen shot 600 of a GUI presented via an interface (e.g., using platform 10002) that provides a visualization of a network performance using various techniques described herein.
- Figure 6 illustrates similar components as in previous visualizations, including metric selector 204, timeline 206, graph 608, and summary components 210.
- a force-directed layout is used to represent the origin of the routes in the center and the connected ASes (e.g., each node is an AS) laid out in a radial way as shown in the graph 608.
- Routers are shown as the leaf nodes of the graph 608 and are visually indicated according to the selected metric (e.g., visual indicators, such as colors and/or other visual indicators of such icons, can be used to indicate a selected metric).
- the yellow/orange nodes can correspond to routers that had path changes
- the green nodes e.g., or some other visual indicator(s)
- the green nodes can correspond to routers that did not have any path changes.
- the timeline 206 shows increments of 5 minutes
- the red dashed links (e.g., or some other visual indicator(s)) in the graph 608 corresponds to paths that were used at some point during the round, but were not in use at the end of the round, which, for example, can be useful to detect path exploration in BGP.
- Figure 7 illustrates a graph of a visualization of routing paths including a series of nodes indicating data loss in accordance with some embodiments.
- some of these white nodes 710 e.g., or other visual indicators can be used to indicate such nodes
- These nodes can be referred to as real stars.
- Other nodes usually reply but, due to loss, occasionally the reply is not received.
- These nodes can be referred to as spurious stars. Distinguishing real stars from spurious stars is a challenging problem. For example, the challenge is that there is no other indication of packet loss, as both real stars and spurious stars look the same.
- a technique that can be used to address this issue by using a set of paths from multiple sources to a given destination is described below.
- the paths can be merged into path 3-4-5-6.
- the loss percentage on link 3-4 can be computed as L/(N+L). More generally, the loss of a node n can be determined by computing as the total number of packets lost in next hops divided by the total number of packets forwarded to next hops of n, as follows:
- Loss(n) Sum L(i) / Sum L(i) + Sum N(i)
- the loss is always 100%. In some cases, it can also happen that a node is terminal for some routes, but not for other routes. In such cases, the loss can be computed as if there were a next hop on the terminal route where all N packets sent to it were lost. For these next hops, N is equal to the threshold of consecutive white nodes used in the terminal condition to stop probing a path.
- the graph visualization of such nodes marks loss using a visual indicator(s) (e.g., as a red circle on a node, the thickness of which is proportional to the loss percentage, or some other visual indicator(s) can be used).
- a slider to control the amount of loss visible is provided (e.g., "Mark nodes with more than x% loss").
- the destination is treated as a special case.
- the destination will reply to any TCP SYN packet it receives with a TCP SYNACK, which means that it will respond to any packet with TTL equal to or greater than the true number of hops to the destination.
- TCP SYNACK a TCP SYNACK
- an algorithm that can be used for the destination case is similar to the path merging algorithm, but in this case paths are not required to be of the same length. For example, for the following paths:
- pathl and path2 are of different lengths. Then, if pathl and path2 are mergeable, the path can still be reduced, such as follows:
- condition to filter cases with real stars is as follows: if we have
- the "?” link is used in this example, because there is not enough information is available to determine the true number of hops (e.g., or true number of distinct paths) between A and B with sufficient certainty.
- agent2-A-B-C-F dominant
- agent 1 -A-B-C-F were also present in the above example, the path with the star could be merged into A-B-C-F according to the default rules as discussed above.
- information collected from multiple sources can be used to infer the reality of a path from a single source. Similar techniques can be used across the time dimension. For example, by looking at the data on the preceding and subsequent intervals, the reality of the path can be inferred at the current interval.
- space- based approach discussed above can be more convenient in some cases, such as if the necessary information is readily available at the time of visualization, in other cases, a time- based approach can also be used, such as for cases where not enough data in one time slice is available to get an accurate representation of a path.
- Figure 8A and Figure 8B illustrate views for showing Border Gateway Protocol (BGP) and packet loss in accordance with some embodiments.
- Figure 8A and 8B are screen shots 800 and 810, respectively, of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- BGP Border Gateway Protocol
- Figure 8A illustrates a data path view in which a data path from source site (e.g., Chicago node) to the lowest listed destination does not have an IP address and is shown as having packet loss (e.g., using a visual indicator for that destination node, such as a color coded indicator or other graphical user interface related visual indicator(s)).
- Figure 5B illustrates a BGP view, in which a window 812 shows an IP address destination for which there are no routes to reach that destination, which corresponds to the path from Chicago for which there is packet loss on the path as shown in the data path view of Figure 8A.
- Figures 5A and 5B illustrate the user of cross-layer visibility and troubleshooting into application delivery in which a root cause of a packet loss seen on the Chicago node in Figure 8A can be illustrated by navigating to a BGP view for that same time interval to determine that the Chicago node was unreachable due to BGP route information related problems.
- Figure 8B provides a visual representation that explains that the packet loss seen on the Chicago node in Figure 8A is a result of the unavailability to a certain BGP prefix.
- Figure 9 illustrates a visualization of DNS data in accordance with some embodiments.
- Figure 9 is a screen shot 900 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- a DNS test (e.g., that can be performed by one or more distributed agents and controlled by agent controllers, such as agent controllers 114) includes several authoritative name servers that serve a specific domain name. For example, DNS queries from agents can be sent to the specified DNS servers targeting the specified domain name (e.g., one or more domain names that an entity wants to test for, such as www.domain- name.com). For example, the following cases can be marked as errors:
- This technique can be used for identifying and resolving application level problems (e.g., application layer response time), as DNS tests can be correlated to identify which DNS name server(s) may be the root of the problem. For example, if a problem is identified with a particular DNS name server, then a user can drill down to the network layer view to determine if it is a network problem to access that DNS name server from a particular location (e.g., a location at which one or more agents are present for such testing).
- a particular location e.g., a location at which one or more agents are present for such testing.
- metric selector 204 can be used to choose between Availability and Resolution Time.
- Timeline 206 can be used to select an instant in time to drill down to.
- World map 208 shows a geographical representation of the problem.
- Summary window 210 shows a small summary of the metrics/ number of errors.
- Table 212 shows the breakdown of the DNS tests per location.
- the view can show aggregated metrics (e.g., average for the case of Availability and Minimum value for the case of Resolution Time).
- a minimum can be selected as this is the most likely the value a DNS resolver would get if it had to query multiple authoritative servers for a domain.
- a similar visualization for DNSSEC is provided, such as for the DNS extension that adds authentication and data integrity to DNS.
- DNSSEC delegation chains a test from each agent is provided in which the test can start at the bottom of the chain and verify signatures all the way to the top of the chain. An error is triggered if there is any step in the resolution chain that fails, either because a resource record is missing or because a signature does not match.
- Figure 10 illustrates Anycast troubleshooting, to identify any relevant DNS problems, in accordance with some embodiments.
- Figure 10 is a screen shot 1000 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- Figure 10 illustrates a DNS view, illustrating DNS resolutions to the DNS servers serverl .com (e.g., showing four errors in this example) and server2.com (e.g., showing 7 errors in this example) in table 1010, which are geographically represented by location in red on the world map view 1008, and the time interval is 6:00-6:00:47 on 3-8- 2012.
- serverl .com e.g., showing four errors in this example
- server2.com e.g., showing 7 errors in this example
- Figure 11 illustrates a path visualization that shows all problem agents routing to an Ashburn, Virginia instance in accordance with some embodiments.
- Figure 11 is a screen shot 1100 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- Figure 11 illustrates a path visualization to server2.com in path visualization graph 1102.
- DNS Anycast is used, which can announce its IP from several different locations (e.g., five different locations). This shows that a commonality among the agents that have DNS errors is that all of these agents are performing their DNS lookup with this particular Anycast server in Ashburn Virginia, which can be inferred after looking at the location of the hops immediately preceding the server in path visualization graph 1102 as shown.
- Figure 12 illustrates a visualization of HTTP data in accordance with some embodiments.
- Figure 12 is a screen shot 1200 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- the various other components as shown in Figure 12 are similar to components shown in previously discussed views, including timeline 206, world map 208, summary 210, and table 212 components of the screen shot 1200.
- the URL to be tested is fetched from each agent location, the time to complete each step is measured, and whether there was an error at any particular step is determined and recorded.
- an HTTP fetch can include the following steps: DNS resolution, TCP connection establishment, SSL certificate exchange, HTTP GET request, and/or HTTP response.
- the HTTP tests can capture the following metrics: Availability: an indication of errors, the aggregate availability is the fraction of agents that complete the test without any error; Response Time: or Time to First Byte (TTFB) is the time it takes for the client to start receiving HTTP data from the server; this metric is useful to understand if delays are coming from network or from backend; typically slow response times are caused by slow networks; and/or Fetch Time: this is the time it takes to actually receive all the HTTP data.
- Availability an indication of errors
- the aggregate availability is the fraction of agents that complete the test without any error
- Response Time or Time to First Byte (TTFB) is the time it takes for the client to start receiving HTTP data from the server
- TTFB Time to First Byte
- Figure 13 illustrates a visualization of a web page load performance in accordance with some embodiments.
- Figure 13 is a screen shot 1300 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- real web browsers can be used by agents (e.g., web testing agents that can be distributed and controlled by agent controllers, such as agent controllers 114) to load the web page(s) from different locations and extract the performance for each of the components of the web page(s).
- agents e.g., web testing agents that can be distributed and controlled by agent controllers, such as agent controllers 11
- the components can be grouped per domain and per provider to rank Response Time and Throughput, and to better identify bottlenecks.
- Example metrics can include the following: DOM time: the time it takes for the document object model (DOM) to build; after the DOM is created, the user can start interacting with the page; and/or Page load time: the time it takes for the entire page to load, including style sheets and images.
- the waterfall at 1312 is updated.
- the waterfall view 1312 provides a breakdown of the individual performance of each component of the page, for example, including time measurements for each of the following: Blocked: the time the browser is waiting for a TCP connection to be available to process the request; DNS: the time it takes to do a DNS resolution for the name of the resource; Connect: the time it takes to establish a TCP connection to the server; SSL/TSL: the time it takes to do the SSL/TSL handshake; Wait: the time between sending the HTTP request and getting the first byte of the reply back from the server; and Receive: the time it takes from the first byte of the answer until the last byte is received.
- Figure 14 illustrates a bottleneck analysis by provider for page loads in accordance with some embodiments.
- Figure 14 is a screen shot 1400 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- Figure 14 illustrates a page load view that indicates detailed views of page load by provider (e.g., each object can provide from different providers, such as Dell, Adobe, and Akamai for www.example.com), for a particular period of time and an average response time and throughput, using browser measurements (e.g., using web testing agents, as similarly discussed above).
- provider e.g., each object can provide from different providers, such as Dell, Adobe, and Akamai for www.example.com
- browser measurements e.g., using web testing agents, as similarly discussed above.
- Figure 15 illustrates a visualization of transactions (e.g., web transactions) in accordance with some embodiments.
- Figure 15 is a screen shot 1500 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
- the platform for cross-layer visibility for application delivery also supports transaction tests, such as for web-based transactions.
- a transaction can refer to a series of scripted steps that are automatically executed in a browser.
- a transaction can include going to a homepage of a web site and adding an item into a shopping cart.
- a transaction can include going to a home page of a web site, logging into the web site with user credentials, and downloading a first home login page associated with that user's credentials.
- transactions are used to refer to extensions of single page loads, and, as shown in Figure 15, the visualization of transactions can similarly be provided as for single page loads (e.g., simple, single web page load transactions).
- single page loads e.g., simple, single web page load transactions.
- users are allowed to define an initial step and a final step. For example, these steps can be used to define the boundaries of interest in the transaction and to define the boundaries for computing the transaction time for the measured transaction.
- waterfall 1512 all the steps of the transaction are presented as vertical colored lines or using another visual indicator(s) (e.g., the x-offset corresponds to elapsed time since step 0).
- the individual components that are outside of the interval defined between the initial and final step can be faded out to visually emphasize the relevant steps.
- a breakdown of the time spent on each step and the time each page loaded during a transaction took to load is provided.
- metrics for transactions can include: Completion: the fraction of transaction steps that were completed during the transaction; and/or Transaction time: the time between the initial step and the final step of the transaction.
- Figure 16 shows a summary of transaction steps and identifying bottlenecks in accordance with some embodiments.
- Figure 16 illustrates web transaction screen shot 1600 of a summary including a time per step and time per page for an example web transaction test results.
- step 6 for page load #1 was measured as using a relatively significant/longer amount of time. Thus, this indicates that step 6 for page load #1 may be the cause of the bottleneck for these web transactions.
- Performance problems can be challenging and time consuming to diagnose when they cannot be reproduced.
- the providers often use data from a nearby city from an external monitoring system to cross check. However, this is problematic, because it does not collect data from the end user reporting the problem.
- a new browser extension is provided
- a web browser which is a lightweight extension that can be installed by any users that desire to report their experience to a service provider or Information Technology (IT) department.
- IT Information Technology
- the add-on starts recording errors and timings for each object received by the browser, and also perform active measurements such as traceroutes at the same instant of time.
- the extension then exports the data collected to a collector of a service provider or IT department (e.g., cloud based network performance service provider or IT), which channels the data to the relevant customer of the service provider/IT based on the domain that was being tested. For example, data on a test to store.apple.com can be made available on the Apple users (e.g., in the service provider's data for the customer) if they exist.
- Figure 17 illustrates a flow diagram for cross-layer troubleshooting of application delivery in accordance with some embodiments.
- process 1700 is performed using platform 100 as shown in Figure 1.
- collecting test results from a plurality of distributed agents for a plurality of application delivery layers is performed.
- generating a graphical visualization of an application delivery state for the plurality of application delivery layers based on the test results is performed.
- the graphic visualization is output.
- Figure 18 illustrates another flow diagram for cross-layer troubleshooting of application delivery in accordance with some embodiments.
- process 1800 is performed using platform 100 as shown in Figure 1.
- sending tests to distributed agents e.g., application delivery tests
- different application delivery layers e.g., to perform various active measurements across different application delivery layers
- collecting test results from the distributed agents for the different application delivery layers is performed.
- correlating the collected test results for the different application delivery layers is performed.
- generating a graphical visualization based on the correlated test results for the different application delivery layers e.g., generating a graphical visualization of application delivery results for the different layers and enabling layer correlation
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Data Mining & Analysis (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Health & Medical Sciences (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Physics & Mathematics (AREA)
- Quality & Reliability (AREA)
- Biomedical Technology (AREA)
- Debugging And Monitoring (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Computer And Data Communications (AREA)
Abstract
Techniques for cross-layer troubleshooting of application delivery are disclosed. In some embodiments, cross-layer troubleshooting of application delivery includes collecting test results from a plurality of distributed agents for a plurality of application delivery layers; and generating a graphical visualization of an application delivery state based on the test results for the plurality of application delivery layers (e.g., different application delivery layers).
Description
CROSS-LAYER TROUBLESHOOTING OF APPLICATION DELIVERY
CROSS REFERENCE TO OTHER APPLICATIONS
[0001] This application claims priority to U.S. Provisional Patent Application No.
61/649,473 (Attorney Docket No. THOUP001+) entitled CROSS-LAYER VISIBILITY OF APPLICATION DELIVERY filed May 21, 2012, which is incorporated herein by reference for all purposes.
BACKGROUND OF THE INVENTION
[0002] Cloud computing generally refers to the use of computing resources (e.g., hardware and software) that are delivered as a service over a network (e.g., typically, the Internet). Cloud computing includes using remote services to provide a user's data, software, and computation.
[0003] Distributed applications can generally be delivered using cloud computing techniques. For example, distributed applications can be provided using a cloud computing model, in which users are provided access to application software and databases over a network. The cloud providers generally manage the infrastructure and platforms on which the applications run. Various types of distributed applications can be provided as a software as a service (SaaS).
BRIEF DESCRIPTION OF THE DRAWINGS
[0004] Various embodiments of the invention are disclosed in the following detailed description and the accompanying drawings.
[0005] Figure 1 illustrates a functional block diagram of a platform for cross-layer troubleshooting of application delivery in accordance with some embodiments.
[0006] Figure 2 illustrates a network performance visualization in accordance with some embodiments.
[0007] Figure 3 illustrates a graph of a visualization of a data path to a destination site in accordance with some embodiments.
[0008] Figure 4A illustrates a graph of a full connectivity visualization of a data path to a destination site in accordance with some embodiments.
[0009] Figure 4B illustrates a graph providing a simplified view of a visualization of a data path to a destination site in accordance with some embodiments.
[0010] Figure 4C illustrates a graph providing a simplified view of a visualization of a data path to a destination site with problem nodes in accordance with some embodiments.
[0011] Figure 4D illustrates a graph providing a selective exploration expanding a data path from selected source/agent site in accordance with some embodiments.
[0012] Figure 5A is screen shot illustrating an HTTP availability drop during a particular time interval in accordance with some embodiments.
[0013] Figure 5B is screen shot illustrating a network view that shows packet loss from the agents shown in Figure 5A in accordance with some embodiments.
[0014] Figure 5C is screen shot illustrating a visualization of a problem zone shown in Figure 5B in accordance with some embodiments.
[0015] Figure 6 illustrates a graph of a visualization of routing paths in accordance with some embodiments.
[0016] Figure 7 illustrates a graph of a visualization of routing paths including a series of nodes indicating data loss in accordance with some embodiments.
[0017] Figure 8 A and Figure 8B illustrate views for showing Border Gateway
Protocol (BGP) and packet loss in accordance with some embodiments.
[0018] Figure 9 illustrates a visualization of DNS data in accordance with some embodiments.
[0019] Figure 10 illustrates Anycast debugging, to identify any relevant DNS problems, in accordance with some embodiments.
[0020] Figure 11 illustrates a path visualization that shows all problem agents routing to an Ashburn, Virginia instance in accordance with some embodiments.
[0021] Figure 12 illustrates a visualization of HTTP data in accordance with some embodiments.
[0022] Figure 13 illustrates a visualization of a web page load performance in accordance with some embodiments.
[0023] Figure 14 illustrates a bottleneck analysis by provider for page loads in accordance with some embodiments.
[0024] Figure 15 illustrates a visualization of transactions (e.g., web transactions) in accordance with some embodiments.
[0025] Figure 16 shows a summary of transaction steps and identifying bottlenecks in accordance with some embodiments.
[0026] Figure 17 illustrates a flow diagram for cross-layer troubleshooting of application delivery in accordance with some embodiments.
[0027] Figure 18 illustrates another flow diagram for cross-layer troubleshooting of application delivery in accordance with some embodiments.
DETAILED DESCRIPTION
[0028] The invention can be implemented in numerous ways, including as a process; an apparatus; a system; a composition of matter; a computer program product embodied on a computer readable storage medium; and/or a processor, such as a processor configured to execute instructions stored on and/or provided by a memory coupled to the processor. In this specification, these implementations, or any other form that the invention may take, may be referred to as techniques. In general, the order of the steps of disclosed processes may be altered within the scope of the invention. Unless stated otherwise, a component such as a processor or a memory described as being configured to perform a task may be implemented as a general component that is temporarily configured to perform the task at a given time or a specific component that is manufactured to perform the task. As used herein, the term 'processor' refers to one or more devices, circuits, and/or processing cores configured to process data, such as computer program instructions.
[0029] A detailed description of one or more embodiments of the invention is
provided below along with accompanying figures that illustrate the principles of the invention. The invention is described in connection with such embodiments, but the invention is not limited to any embodiment. The scope of the invention is limited only by the claims and the invention encompasses numerous alternatives, modifications and equivalents. Numerous specific details are set forth in the following description in order to provide a thorough understanding of the invention. These details are provided for the purpose of example and the invention may be practiced according to the claims without some or all of these specific details. For the purpose of clarity, technical material that is known in the technical fields related to the invention has not been described in detail so that the invention is not unnecessarily obscured.
[0030] Cloud computing generally refers to the use of computing resources (e.g., hardware and software) that are delivered as a service over a network (e.g., typically, the Internet). Cloud computing includes using remote services to provide a user's data, software, and computation.
[0031] Distributed applications can generally be delivered using cloud computing techniques. For example, distributed applications can be provided using a cloud computing model, in which users are provided access to application software and databases over a network. The cloud providers generally manage the infrastructure and platforms on which the applications run. Various types of distributed applications can be provided as a software as a service (SaaS).
[0032] Users typically access cloud-based distributed applications (e.g., distributed applications) through a web browser, a light-weight desktop, and/or mobile application (e.g., mobile app) while the enterprise software and user's data are typically stored on servers at a remote location. Such cloud-based distributed applications can allow enterprises to get their applications up and running faster, with improved manageability and less maintenance, and can enable enterprise IT to more rapidly adjust resources to meet fluctuating and unpredictable business demand. Thus, distributed applications can allow a business to reduce Information Technology (IT) operational costs by outsourcing hardware and software maintenance and support to the cloud provider.
[0033] However, a significant drawback of distributed applications is that troubleshooting performance problems can be very challenging and time consuming. For
example, determining whether performance problems are the result of the cloud provider of the distributed application, the customer's own internal IT network, a user's client device, and/or intermediate network providers between the user's client device and the cloud provider can present significant challenges.
[0034] What are needed are new techniques to visualize and troubleshoot the performance of distributed applications.
[0035] Accordingly, techniques for cross-layer troubleshooting of application delivery are disclosed. In some embodiments, various techniques are provided for cross-layer visualization and troubleshooting of application delivery, such as for performance problems associated with distributed applications. For example, a platform for identifying and/or determining performance problems associated with a distributed application(s) can be provided. As another example, the platform can generate reports that include various cross- layer visualizations that facilitate identifying and/or determining performance problems associated with a distributed application(s). As yet another example, various techniques described herein can be used to diagnose application deliver problems from cloud service providers, such as SaaS and/or other network delivered based applications (e.g,. web sites, online stores, cloud based software, and/or other such network based applications and/or services) to determine the causes or sources of the application delivery performance issues or problems.
[0036] In some embodiments, cross-layer troubleshooting of application delivery includes collecting test results from a plurality of distributed agents for a plurality of application delivery layers; and generating a graphical visualization of an application delivery state based on the test results for the plurality of application delivery layers (e.g., different application delivery layers).
[0037] For example, the graphical visualization of the application delivery state can facilitate cross-layer troubleshooting of problems (e.g., associated with application delivery of a distributed application). As another example, the graphical visualization of the application delivery state can facilitate cross-layer visualization and troubleshooting of application delivery (e.g., associated with application delivery of a distributed application, which can identify and/or facilitate diagnosing of causes of application delivery problems).
[0038] In some embodiments, cross-layer troubleshooting of application delivery further includes outputting the graphical visualization of the application delivery state based on the test results for the plurality of application delivery layers.
[0039] In some embodiments, the graphical visualization of the application delivery state based on the test results for the plurality of application delivery layers facilitates cross- layer visualization and troubleshooting of problems (e.g., associated with application delivery of a distributed application).
[0040] In some embodiments, the graphical visualization of the application delivery state based on the test results for the plurality of application delivery layers facilitates cross- layer troubleshooting of application delivery by providing for correlation of the test results across a plurality of layers (e.g., application delivery layers).
[0041] In some embodiments, the graphical visualization of the application delivery state based on the test results for the plurality of layers facilitates cross-layer troubleshooting of application delivery by providing for correlation of the test results across a plurality of layers using a space domain and/or a time domain.
[0042] In some embodiments, the plurality of distributed agents are controlled by an agent controller. In some embodiments, the graphical visualization of network performance is generated by a platform for cross-layer visibility and troubleshooting of distributed applications 100, such as shown in Figure 1. In some embodiments, the network performance test results from the plurality of distributed agents for the plurality of layers are stored in a storage tier (e.g., which can include a database or another type of data store).
[0043] Overview of Techniques for Cross-Layer Visibility and Troubleshooting of Distributed Applications
[0044] In some embodiments, cross-layer visibility and troubleshooting of distributed applications includes using software agents to collect information from different points in a network across different layers in the application delivery (e.g., of a distributed application), as further described herein with respect to various embodiments. For example, such information can be collected and aggregated by centralized collectors and aggregators and presented to a user as a Software as a Service (SaaS). In some embodiments, different layers are correlated using one or more of the following: a monitored object, time, and location.
[0045] In some embodiments, various techniques described herein allow users to drill down to identify problems and locations whether at different layers (e.g., a network, transport, and/or application layer) to correlate across application delivery layers to determine whether, for example, network issues are affecting the performance of a distributed application(s), as further described herein with respect to various embodiments. For example, such techniques can be applied to both internal network diagnostics (e.g., for an organization's internal network) and/or external network diagnostics (e.g., for a web site accessed across the Internet, such as for a cloud-based distributed application).
[0046] In some embodiments, various techniques described herein allow for determination of a network topology to indicate, for example, anycast routing issues used by DNS providers (e.g., such as when there are multiple DNS servers sharing the same IP address) as well as structure and/or activity of a network topology, as further described herein with respect to various embodiments. For example, such techniques can be applied to both internal network diagnostics (e.g., for an organization's internal network) and/or external network diagnostics (e.g., for a web site accessed across the Internet, such as for a cloud- based distributed application).
[0047] In some embodiments, various techniques described herein allow for diagnosing SaaS applications, such as for cloud-based distributed applications, using, for example, a lightweight agent, as further described herein with respect to various embodiments.
[0048] A Distributed Testing Framework
[0049] In some embodiments, cross-layer visibility and troubleshooting of distributed applications includes a distributed framework to distribute tests across different agents in the Internet. For example, agents can be executed on hosted providers using cloud computing distributed across multiple ISPs, which are controlled by agent controllers to perform one or more tests as further described herein, in which the test results can be collected for correlation and analysis, as further described herein with respect to various embodiments. In some embodiments, agents are computing resources that are controlled, and, for example, can be either virtual or dedicated servers. Agents can be distributed across different geographies and networks, for example, distributed agents can be distributed to mostly Tier-1 and Tier-2 networks to avoid the noise of bad connectivity of last mile connections.
[0050] An example of a system architecture for providing cross-layer visibility and troubleshooting of distributed applications is shown in Figure 1 as described below.
[0051] Figure 1 illustrates a functional block diagram of a platform for cross-layer troubleshooting of application delivery in accordance with some embodiments. In particular, Figure 1 illustrates an environment in which a platform for cross-layer visibility and troubleshooting of distributed applications 100 includes distributed agents 116-120 (e.g., which can be distributed across various geographies and/or devices for performing different types of tests and/or targeting different sites, locations, and/or metrics) that collect data based on configured tests, and the distributed agents 116-120 send this data to a controller(s) 114 (e.g., agent controllers). The controller 114 stores the data in a storage tier 112 (e.g., providing permanent storage) that can be used by a web tier 104 to generate visualizations and reports to users accessing the platform 100 using client devices (e.g., computers, laptops, smartphones, and/or various other computing devices).
[0052] For example, a report can be output to a user to present the collected and analyzed cross-layer application delivery information of a distributed application. Example reports can include various visualizations and/or diagnostic information as further described herein with respect to various embodiments. For example, the report can facilitate troubleshooting application delivery associated with the distributed application to determine whether performance problems are the result of the cloud provider of the distributed application, the customer's own internal IT network, a user's client device, and/or intermediate network providers between the user's client device and the cloud provider. The report can also include recommendations to the user to resolve any such determined application delivery problems associated with the distributed application. In some cases, the report can also be provided to a third party, such as the SaaS provider of the distributed application and/or a network provider, which can be provided as information to indicate the source of such determined application delivery problems associated with the distributed application.
[0053] In the example shown, the user of client device 106 (hereinafter referred to as
"Bob") is employed as an IT manager of a distributed application company ("SaaS Company"). The user of client device 108 (hereinafter referred to as "Alice") is employed as an IT manager of a national company ("ACME Company"). As will be described in more detail below, Bob and Alice can each access the services of platform 100 (e.g., platform for
cross-layer visibility and troubleshooting of distributed applications) via web tier 104 over a network, such as the Internet. The techniques described herein can work with a variety of client devices 106-108 including, but not limited to personal computers, tablet computers, smartphones, and/or other computing devices.
[0054] In some embodiments, platform 100 generates various reports based on results of the network performance tests to facilitate cross-layer visibility and troubleshooting of application delivery associated with a distributed application(s), as further described herein. In some embodiments, platform 100 includes a data store, such as storage tier 112 for storing results of the network performance tests and/or the reports.
[0055] In some embodiments, a set of agent controllers 114 is provided as shown to send various tests (e.g., such as the various test described herein with respect to various embodiments) to the distributed agents for execution by the distributed agents. For example, agents can be executed on hosted providers using cloud computing distributed across multiple ISPs, which are controlled by agent controllers to perform one or more tests as further described herein, in which the test results can be collected for correlation and analysis, as further described herein with respect to various embodiments.
[0056] In some embodiments, the tests are configured through a web interface by a user. For example, typical parameters can include the frequency of various tests, the target of the tests, and the agents (e.g., or locations) where the tests are to be performed. The test parameters can be sent from the controller (e.g., agent controllers 114) to the distributed agents after an agent checks-in (e.g., using a pull mechanism). After an agent executes a test, the agent can export the test result(s) back to the controller. The controller can then provide the results back to a data store (e.g., storage tier 112) for permanent storage (e.g., or temporary storage). Besides periodic tests, a controller can also send on-demand tests to an agent(s) through, for example, a Remote Procedure Call (RPC) call for immediate or on- demand execution.
[0057] In various embodiments, platform 100 is a scalable, elastic architecture and may comprise several distributed components, including components provided by one or more third parties. Further, when platform 100 is referred to as performing a task, such as storing data or processing data, it is to be understood that a sub-component or multiple sub-
components of platform 100 (whether individually or in cooperation with third party components) may cooperate to perform that task.
[0058] In some embodiments, tests include various types of tests to facilitate cross- layer visibility and troubleshooting of application delivery associated with a distributed application(s), as further described herein. Example network tests include data path measurement tests, routing path measurement tests, and end-to-end network metrics tests. Example DNS tests include per name server testing and Domain Name System Security Extensions (DNSSEC) bottom-up validation tests. Example HTTP tests include testing of steps of a Uniform Resource Locator (URL) fetch. Example page load tests include testing of a load of an entire web page using a web browser (e.g., a typical web browser). Example transaction tests include performing a multi-step scripted transaction from a web browser (e.g., a typical web browser). These and various other tests are discussed in greater detail below.
[0059] Cross-layer Correlation
[0060] In the next sections, various techniques for allowing users (e.g., Bob and
Alice) to easily navigate between different layers of data are described in accordance with some embodiments. For example, platform 100 can provide users with a natural flow of action for root cause identification of problems as further described herein with respect to various embodiments.
[0061] Slicing Data Across Time - Rounds and Timeline
[0062] In some embodiments, tests aggregate data over certain time intervals (e.g., small time intervals), referred to as rounds. In some embodiments, a round includes one sample of data from each agent (e.g., each agent can be placed in a different geographical location) for a given test. In some embodiments, data over each round is aggregated for a set of agents (e.g., all the agents) and, for example, can be shown on a timeline. In some embodiments, a timeline is implemented as a navigation widget that enables users to see the aggregate behavior of all of the agents over time (e.g., or just an individual agent) and to click on a specific point in time for a detailed drill down (e.g., to view specific cross-layer network test related data at a specified time or time window/interval). For example, a user can be provided with several metrics to choose from, and a timeline can be associated with one or more metrics.
[0063] Compound Tests
[0064] In some embodiments, some tests are composed of multiple subtests that are run at the same time. For example, the subtests can each be different and associated with different layers of visibility. Performing different active tests at the same time for different layers allows for switching between subtests while keeping the same test and round (e.g., and eventually the same agent location). For instance, a test can include an HTTP subtest and a network subtest at the same time. Then the user can navigate from HTTP metrics, to end-to- end network metrics, all the way down to individual router hop metrics (e.g., by inspecting the data path). For example, this can be useful to identify/classify problems that are network induced versus problems that are back-end related.
[0065] Correlating Different Layers
[0066] In some embodiments, when switching between views (e.g., or layers), the context is maintained, such as the following: (1) the test, (2) the round, and (3) the location (e.g., the agent ID or the agent location). More formally, each layer L0 has an associated vector of features, referred to as the context vector, CLO, that includes all dimensions that uniquely identify a view in layer L0. When moving between layer L0 and LI, a function is applied to CLO to provide CLI, such as for example, the following:
Ci_i = FLO->LI(CI_O)- We can think of F as a matrix of functions, and the indexes are the current layer, and the next layer the function is mapping to.
[0067] In some embodiments, a Graphical User Interface (GUI) is provided that can allow a user (e.g., Bob or Alice) to navigate between different layers, such as to jump to different application delivery layers (e.g., path visualization, BGP metrics, BGP route visualization, etc.) and web layers (e.g., basic HTTP, etc.). In some embodiments, a set of layers are correlated, and reports including visualizations that present results of test performed at the different correlated layers can be accessed, such as using hyperlinks in a GUI menu that can encode the context vectors as GET parameters in the URLs. For example, users can jump to Path Visualization, BGP Metrics, BGP Route Visualization, and Basic HTTP views while maintaining the context as discussed above.
[0068] Table 1 (below) shows a transition table between different layers, indicating from which layers (rows) is it possible to jump to (columns) in accordance with some embodiments.
Page Load - - - - Yes* -
Table 1: Transitions between layers, * indicates transition is only available for compound tests.
[0069] Application Delivery Layers
[0070] The next sections describe the layers involved in application delivery and how data is collected and analyzed at each step in accordance with some embodiments.
[0071] Network Performance
[0072] Measuring Network Performance
[0073] In some embodiments, an approach for measuring response time of network level metrics from a network server is provided using a train of TCP SYN packets (synchronise packets in transmission control protocol). For example, providing such a train of TCP SYN packets appears to the network server to be like any normal connection that the network server will normally respond to; and while the network server may throttle such connections, even if throttling occurs, that would still generally provide an accurate reflection of a typical user connection response time as that network server is throttling for other such user connections as well. Other approaches for measuring response time of network level metrics from a network server can include using ping/Internet Control Message Protocol (ICMP) techniques, but some servers and/or Internet Service Providers (ISPs) (e.g., gateway s/firewalls) often block such ICMP traffic.
[0074] In some embodiments, in order to measure end-to-end network properties, active measurements from agents (e.g., distributed agents) to destination servers are performed. For example, servers can be identified by host name (or IP address) and TCP port number. Periodically, a train of N TCP SYN packets is sent to each server from each agent.
A full TCP connection is not established, because the client sends a TCP RST (TCP reset) right after receiving the SYN ACK response from the server. These N points allow for measuring, for example, the following: (1) packet loss (e.g., ratio of sent packets that were ACKed), (2) average network delay (e.g., time between SYN and ACK), and (3) network jitter (e.g., average delta between consecutive delays).
[0075] Visualizing Network Performance
[0076] Figure 2 illustrates a network performance visualization in accordance with some embodiments. In particular, Figure 2 is a screen shot 200 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein. As shown at 202, users can select the target (e.g., server) that they want to look at, such as shown in this example, www.example.com, port 443 (SSL). As shown at 204, there is also an option to select the network metric, such as shown, in this example; "loss" (or packet loss) is selected.
[0077] As shown at 206, a timeline is provided that shows the time-series of the metric. In some embodiments, as described further below, the timeline allows a user to select an instant in or interval of time to drill down to (e.g., interactive timeline that allows users to select a specified time or time interval for drill down for further analysis, such as to look at a window of time of high packet loss to try to determine root cause(s) of such high packet loss).
[0078] As shown at 208, a world map depicts the locations of the distributed agents as well as a visual indication of their status according to the selected metric 204. In this example, the red dots (e.g., shown as completely shaded circles or another graphical indicator to differentiate these icons) are agents that are experiencing heavy packet loss, and the green dots (e.g., shown as completely partially shaded or hashed circles or another graphical indicator to differentiate these icons) are agents without packet loss.
[0079] A summary box 210 provides a concise report of the metrics. A table 212 shows a more detailed breakdown of the metrics as measured by each agent. For example, the table can be sorted by any column to facilitate the analysis of the data. On the right end of each row, a "Run Test" button 214 is provided that allows users to run on-demand tests to the target (e.g., target server). For example, this can be useful to verify if a problem reported by an automated test is still occurring.
[0080] Data Paths
[0081] Measuring Data Paths
[0082] Traceroute is generally a computer network diagnostic tool for displaying the route (path) and measuring transit delays of packets across an Internet Protocol (IP) network. While a traceroute diagnostic using ICMP packets can be used to measure data paths, such an approach may not be effective on some networks as many ISPs block ICMP packets. Accordingly, in some embodiments, in order to collect data paths, Time To Live (TTL) limited TCP SYN packets are sent to a specific destination server. Routers reply with a TTL Time Exceeded message every time they receive a packet with TTL=1 (e.g., set counter to 1, which is decremented to 0 in IPv4 by routers each time they are processed by router; add one to the TTL to keep extending the path an extra hop; repeat 3 times for destination to map out path, as shown in the below pseudo code sample). Thus, probes are sent with increasing TTL to collect the source IP addresses of the ICMP packets to reconstruct the path packets are taking. In some cases, special precaution can be taken to avoid issues with load balancing. In the case of TCP, if the same 5 tuple (e.g., source IP, destination IP, source port, destination port, and protocol) is kept between probes, balancers will send packets in that flow through the same interfaces.
[0083] As shown below in Algorithm 1, a sample pseudo code scheme is provided in accordance with some embodiments for measuring data paths using TCP SYN packets as discussed above.
MAX_ROUNDS=3;
For vRound=l to MAX_ROUNDS
For vTTL=l to 255
vReply = SendTCPPacket (vTTL, vDestination, vSourcePort, vDestPort) ;
If vReply!=null
push (vHops [vRound] , vReply) ;
End If
End For
//distinguishing nodes with losses from non-responding nodes
If vDestination not in vReply
Mark last responding node as lossy
End If
End For
Algorithm 1: Data path measurements.
[0084] Because the final hop is a TCP server, we should always expect a TCP SYN
ACK packet back from the server. If that does not happen, then it either means that the server is not reachable at layer 3 or that the application stopped at the server. Note that some hops in the path might not send ICMP TTL Exceeded messages, so the server SYN ACK is used as a reference. If there are non-responsive hops after a certain point on the way to the server and if the destination server replies with a TCP SYN ACK, then we assume that those hops do not send ICMP TTL Exceeded (so there is no packet loss). On the other hand, if a SYN ACK is not received from the server, and we have non-responsive hops after hop X in the path, then we assume X is one hop way from the point where packets are being dropped - that is, the last known good hop in the data path.
[0085] Below is an example illustrating hops in a data path and whether or not a response is received on such hops on a path between a start (e.g., a Source Node) and a Destination Node.
START -> [IP-1 - responsive] -> [no response] -> ... -> [no response] -> [Destination Node]
[0086] In this example, if we do not receive a response from the Destination
Node, then in this case we would identify the hop at IP-1, which did respond (the last known good hop) as the path termination point.
[0087] Visualization of Data Paths
[0088] In some embodiments, various techniques for visualization of data paths are provided. In some embodiments, data delivery is represented in the network in a per-hop model in which each hop represents an interface of a physical router. In some embodiments, a graph of a data path visualization of a data path to a destination site (e.g., a destination, such as port 443 (SSL) of www.example.com, or some other destination) is provided, such as shown in Figure 3 as discussed below.
[0089] Figure 3 illustrates a graph of a visualization of a data path to a destination site in accordance with some embodiments. In particular, Figure 3 is a screen shot 300 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a
network performance using various techniques described herein. As shown at 302, users can select the test name (e.g., test to be performed by distributed agents). As shown at 204, there is also an option to select the network metric, such as shown, in this example; "loss" (or packet loss) is selected. As shown at 306, various agent filters (e.g., configurations or settings) can be set for the agents. Based on the test results received from the distributed agents for the selected test(s) and agent filter input, a graph 308 is presented to provide a data path visualization of a data path to a destination site in accordance with some embodiments.
[0090] In some embodiments, each path of the graph starts with an agent node in the left side that is shown with icon indicators (e.g., shown as colored circles, completely shaded circles, or another graphical indicator to differentiate these icons) according to each agent's results or status relative to the selected metric, in this case, packet loss. For example, nodes with a red outline can represent points in the network dropping packets. In particular, such nodes can indicate the last responsive hops in the path, such that the packet loss is indicated to most likely be happening in the immediate hop. As another example of screen visualization of a data path to a destination site, even if view of visualization of data paths is selected to show zero hops using agent filter 306, the data path visualization can still display hops with packet loss to illustrate potential problematic hops (e.g., regardless of the number of hops to be shown, all bad nodes with losses can be presented in the visualization graph 308).
[0091] For example, the network delay between consecutive hops can also be determined (e.g., approximated or estimated) using this technique. As shown as agent filter 306, a control "Color links with delay > x ms" is provided that can provide an icon indicator (shown as colored circles such as red circles, completely shaded circles, or another graphical indicator to differentiate these icons) for links that have more than a selected number of milliseconds (ms) (e.g., 100 ms as shown). The threshold of link delay can be dynamically changed using agent filter 306, which then automatically updates the data path visualization accordingly.
[0092] In some embodiments, a hop expansion control of agent filter 306 allows the topology to be expanded any number of hops from any side (e.g., agent or server) of the topology. For example, this can provide a useful agent filter control from the server side, because that is typically a customer actionable zone. As shown in graph 308, the links with a
number on top are aggregation links that are expanded as the number of hops in the filter increases or just by clicking in the number.
[0093] In some embodiments, another useful feature is the ability to select nodes and links in the topology shown in graph 308. For example, selected nodes and links can be inserted into a selection box, and their position in the topology can be tracked over time by clicking in different points in the timeline. In this manner, routing changes can be tracked over time. In some embodiments, double clicking in a specific node selects all nodes and links belonging to routes going through that node, making it easier to track entire routes over time.
[0094] In some embodiments, the metric selection, as shown at 204, affects timeline
206 and the agent node status indicators (e.g., coloring of the agent icons or some other icon indicator) in the graph 308. For example, the users can select from one of three end-to-end metrics, such as packet loss, latency, and jitter. In some embodiments, the values of these metrics are computed from a train of n TCP SYN packets sent to the destination about the same time the data path measurements take place to preserve time correlation between the end-to-end effect and the per-hop effect. As an example, if agent x is experiencing a very high end-to-end latency to reach the destination, such very high end-to-end latency to reach the destination can be displayed with a color red (e.g., coloring of the agent icons or some other icon indicator) in the graph 308, and by looking at the breakdown per hop, where the bottleneck is located can be determined.
[0095] For example, the visualization layout (e.g., using a version of the Sugiyama's algorithm or another algorithm) can display a graph hop by hop and minimize the link crossings between nodes, making the graph more readable.
[0096] Case Studies
[0097] Figure 4A illustrates a graph of a full connectivity visualization of a data path to a destination site in accordance with some embodiments. In particular, Figure 4 A is a screen shot 400 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein. As shown at 306, various agent filters (e.g., configurations or settings) can be set for the agents. Based on the test results received from the distributed agents for the selected test(s) and agent filter input, a graph 408 is presented to provide a data path visualization of a data path to a
destination site in accordance with some embodiments. In particular, Figure 4A shows a hop- by-hop data path view from the agents (e.g., the agents as shown and discussed below with respect to Figures 5A-5C) to a target site, such as example web site (e.g., www.example.com).
[0098] Figure 4B illustrates a graph providing a simplified view of a visualization of a data path to a destination site in accordance with some embodiments. In particular, Figure 4B is a screen shot 420 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein. As shown at 306, various agent filters (e.g., configurations or settings) can be set for the agents. Based on the test results received from the distributed agents for the selected test(s) and agent filter input, a graph 428 is presented to provide a data path visualization of a data path to a destination site in accordance with some embodiments.
[0099] In particular, Figure 4B is a simplified view of the Figure 4A view of the hop- by-hop data path from agents to a target site, such as example web site (e.g., www.example.com), in which only show two hops away from the web site are shown (e.g., each IP address associated with the target site (e.g., example web site), which is associated with these three IP addresses, and hiding/consolidating all other hops except that all nodes with losses (e.g., nodes with losses exceeding a threshold) are shown in this view. For example, if a user selects " 3 nodes with losses" as shown at component (e.g., a sub-window) 426 of Figure 4B, then Figure 4C is shown in accordance with some embodiments, as discussed below.
[00100] Figure 4C illustrates a graph providing a simplified view of a visualization of a data path to a destination site with problem nodes in accordance with some embodiments. In particular, Figure 4C is a screen shot 440 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
[00101] As shown, Figure 14C illustrates a simplified view of a graph 448 with problem nodes selected in accordance with some embodiments. In particular, Figure 14C shows the result of a user selecting "3 nodes with losses" as shown at component (e.g., a sub- window) 426 of Figure 4B. For example, this can allow the user to drill down to see more information regarding selected problem nodes. As shown in this example, each of the
problem nodes is associated with a common provider that would be listed in the details window for each of these three problem nodes. If the user selects link 446 shown as (8) hops between Amsterdam, Netherlands, then the next hop/node is shown with losses as shown in Figure 14D in accordance with some embodiments, as discussed below.
[00102] Figure 4D illustrates a graph providing a selective exploration expanding a data path from selected source/agent site in accordance with some embodiments. In particular, Figure 4D is a screen shot 460 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
[00103] As shown, Figure 4D illustrates a graph 468 providing a selective exploration expanding a data path from Amsterdam in accordance with some embodiments. In particular, Figure 4D illustrates an expanded data path if a user selects the link 446 shown as (8) hops between Amsterdam, Netherlands and the next hop/node shown with losses as shown in Figure 4C, which thereby allowing for an expanded view of the intermediate hops on the data path from Amsterdam to that problem node as shown in Figure 4C.
[00104] Figures 5A-5C illustrate examples of using cross-layer visibility of application delivery to determine, at different layers of a network, sources of one or more application delivery problems or performance issues in accordance with some embodiments, as discussed below.
[00105] Figure 5 A is screen shot illustrating an HTTP availability drop during a particular time interval in accordance with some embodiments. In particular, Figure 5A is a screen shot 500 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
[00106] In particular, Figure 5 A shows a drop in HTTP availability at time interval
21 :45 - 21 :48 on April 28 as shown at 502. As shown at 504, a window is provided that shows a number of error by type, including the following: DNS, Connect (2 errors as shown), SSL, Send, Receive (1 error), and HTTP. At table 506, the details of these errors are also shown in a table format, in which a first column is a location of agent, date is a time of measurement, server IP is the destination address (e.g., target for HTTP availability testing), response code (e.g., HTTP response code, which is 200 if there are no errors, or empty if no response code received), and a number of redirects, error type (e.g., as discussed above), and
error details (e.g., providing a description of the particular error, and a run test option can be provided to further diagnose this problem and to check if the problem still persists).
[00107] Figure 5B is screen shot illustrating a network view that shows packet loss from the agents shown in Figure 5A in accordance with some embodiments. In particular, Figure 5B is a screen shot 520 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
[00108] Specifically, Figure 5B illustrates a network view that shows packet loss, from the same agents as shown in Figure 5A, in accordance with some embodiments. More specifically, Figure 5B is a network view of the same destination at the same time interval for same destination, as shown in Figure 5A. However, Figure 5B provides a different view, a network view as shown, which indicates that the problem is a packet loss issue at the network layer. Figure 5B illustrates that the cause of the HTTP availability drop shown in Figure 5A is revealed by three agents that each have significant packet loss, such as shown in the details of the packet losses of these three agents in table 526, which are the same agents that were shown in Figure 5A with problems on the HTTP tests/measurements. Accordingly, Figure 5B reveals that the problem is not at the HTTP layer, but rather is a problem at the network layer at this particular time interval. Accordingly, this illustrates an example of various techniques disclosed herein for cross-layer visibility and troubleshooting of application delivery in accordance with some embodiments.
[00109] Figure 5C is screen shot illustrating a visualization of a problem zone shown in Figure 5B in accordance with some embodiments. In particular, Figure 5C is a screen shot 540 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
[00110] Specifically, Figure 5C illustrates a path visualization 546 highlighting a problem zone as shown at 544 (e.g., in which an earlier time as shown at 542 indicates that they share a common next hop, indicating the possible problem location or problem zone) in accordance with some embodiments. More specifically, Figure 5C illustrates a hop-by-hop path visualization to determine the problem zone causing the packet losses, as shown in Figure 5B. For example, the dashed circled nodes (e.g., some other icon indicator) can indicate the last known good hops on these paths, which indicates that the subsequent
node(s), in this case it is a shared common next hop, thereby revealing that this hop is the likely root cause of the network packet loss on this data path. Accordingly, this illustrates an example of various techniques disclosed herein for cross-layer visibility and troubleshooting of application delivery in accordance with some embodiments.
[00111] Routing Paths
[00112] Collecting Routing Information
[00113] The Border Gateway Protocol (BGP) is a standard protocol used to exchange routing information between different Autonomous Systems (AS) (e.g., which is the control plane between different networks, such as between Verizon networks and Time Warner networks). An AS is an independently managed domain (e.g., an organization, such as Dell, which is associated with a particular AS number, such as AS number 3614, which is a unique identifier for Dell's network), typically having a one-to-one mapping to an organization. BGP messages carry routing information for individual prefixes (e.g., or group of IP addresses), and originate from the traffic destination outwards (e.g., BGP message propagation follows the opposite direction of traffic propagation).
[00114] In some embodiments, routing information is collected from public BGP data repositories that archive routing information from hundreds of routers across the Internet. For example, by looking at the AS PATH attribute of each message sent from router R, the routing path R was using at each time can be determined, and this information can also be used to determine when a certain destination IP address (e.g., or prefix) is/was not reachable from R.
[00115] In some embodiments, three different metrics for BGP visualization are provided: (1) reachability, (2) number of path changes, and (3) number of BGP updates. From the point of view of a router (or monitor), reachability refers to the fraction of time that the router can reach a certain destination prefix. Path changes refers to the number of times the attribute AS PATH changed for a certain destination prefix. Updates refers to the plain count of BGP update messages received during the time interval.
[00116] For example, the BGP route information can be collected from Route Views and/or RIPE-RIS (Routing Information Service), which publish BGP route information. As described herein with respect to various embodiments (see, e.g., Figures 8A-8B, which are
described below), the BGP route information can be correlated with various other layers of network information to allow for visualization of this data and analysis of this data in a manner that facilitates cross-layer visibility and troubleshooting of application delivery (e.g., to determine if a problem is caused by a BGP routing information related problem, data path packet loss related problem, HTTP related problem, DNS related problem, and/or some other problem). As an example, by correlating cross-layer network data and visualizing such cross- layer network data, users can make better sense of such information and the correlation and presentation of such information can more clearly reveal if a BGP routing issue at a particular time interval may be a cause of various network problems during that time interval (e.g., HTTP measurement errors may actually be a result of this BGP routing issue, such as when looking at data path loss errors in which there are random drops at random locations, a user can then look to see if problem is at BGP level in which there may be no announced route to get to that network AS number, as that will cause the routers to just drop such packets for which it has no routing information).
[00117] Visualizing Routing Paths
[00118] Figure 6 illustrates a graph of a visualization of routing paths in accordance with some embodiments. In particular, Figure 6 is a screen shot 600 of a GUI presented via an interface (e.g., using platform 10002) that provides a visualization of a network performance using various techniques described herein.
[00119] As shown, Figure 6 illustrates similar components as in previous visualizations, including metric selector 204, timeline 206, graph 608, and summary components 210. In some embodiments, a force-directed layout is used to represent the origin of the routes in the center and the connected ASes (e.g., each node is an AS) laid out in a radial way as shown in the graph 608. Routers are shown as the leaf nodes of the graph 608 and are visually indicated according to the selected metric (e.g., visual indicators, such as colors and/or other visual indicators of such icons, can be used to indicate a selected metric). For example, the yellow/orange nodes (e.g., or some other visual indicator(s)) can correspond to routers that had path changes, while the green nodes (e.g., or some other visual indicator(s)) can correspond to routers that did not have any path changes.
[00120] As also shown in Figure 6, the timeline 206 shows increments of 5 minutes
(e.g., other time intervals/increments can be used or configured to be used), which are
referred to as rounds. Data is visualized per round, and in each round, the paths are shown, using dashed style, and the final paths of the rounds are shown in solid style. The red dashed links (e.g., or some other visual indicator(s)) in the graph 608 corresponds to paths that were used at some point during the round, but were not in use at the end of the round, which, for example, can be useful to detect path exploration in BGP.
[00121] Path Simplification Algorithms
[00122] Currently if a node does not respond with a TTL expired, we leave the node white in the visualization. If there's loss, we might have a series of "white" nodes, typically followed by the destination, such as shown in Figure 7.
[00123] Figure 7 illustrates a graph of a visualization of routing paths including a series of nodes indicating data loss in accordance with some embodiments. As shown in graph 708, some of these white nodes 710 (e.g., or other visual indicators can be used to indicate such nodes) never reply to various agent testing, such as probes with ICMP Time Exceeded. These nodes can be referred to as real stars. Other nodes usually reply but, due to loss, occasionally the reply is not received. These nodes can be referred to as spurious stars. Distinguishing real stars from spurious stars is a challenging problem. For example, the challenge is that there is no other indication of packet loss, as both real stars and spurious stars look the same. However, a technique that can be used to address this issue by using a set of paths from multiple sources to a given destination is described below.
[00124] For example, for nodes in the middle of the path, losses will create diamond shaped elements of equal length. For example, two paths of equal length are shown below:
#1 : 3-x-x-5-6
#2: 3-x-x-x-6
where the numbers represent different nodes in path visualization and the "x" represents a node that did not reply (e.g., referred to above as a white node). In this example, node 5 replied with Time Exceeded on path #1, but not on path #2. Diamond shapes can be detected in the graph and such can be reduced (e.g., path simplification can be performed) as follows:
A-pathl-B
A-path2-B
for every diamond A-B in the graph, if B is not the final destination,
length(pathl)=length(path2), and pathl and path2 are mergeable, then the paths are merged. Mergeable paths are paths that do not have conflicting hops.
are not mergeable, because 4! =5
but, as another example:
3-4-5-6
are mergeable as there are no conflicting hops. In this case, the paths can be merged into path 3-4-5-6.
[00125] Inferring Packet Loss
[00126] In some embodiments, every time a diamond gets reduced, loss on a certain link should be accounted for as discussed below. For example, two paths are shown below:
#1 : 3-4-5
#2: 3-x-5
in which the above two paths can be merged into path 3-4-5. We know how many packets were routed through path #1 where 4 replied (e.g., N number of packets), and we also know how many packets were routed through path #2 where there was no reply after node 3 (e.g., L number of packets). Accordingly, the loss percentage on link 3-4 can be computed as L/(N+L). More generally, the loss of a node n can be determined by computing as the total number of packets lost in next hops divided by the total number of packets forwarded to next hops of n, as follows:
Loss(n) = Sum L(i) / Sum L(i) + Sum N(i)
For terminal nodes, such as nodes that do not have any next hops but are not a destination, the loss is always 100%. In some cases, it can also happen that a node is terminal for some routes, but not for other routes. In such cases, the loss can be computed as if there were a next hop on the terminal route where all N packets sent to it were lost. For these next hops, N is equal to the threshold of consecutive white nodes used in the terminal condition to stop probing a path.
[00127] In some embodiments, the graph visualization of such nodes marks loss using a visual indicator(s) (e.g., as a red circle on a node, the thickness of which is proportional to the loss percentage, or some other visual indicator(s) can be used). In some embodiments, a slider to control the amount of loss visible is provided (e.g., "Mark nodes with more than x% loss").
[00128] Diamonds Involving the Destination
[00129] In some embodiments, the destination is treated as a special case. The destination will reply to any TCP SYN packet it receives with a TCP SYNACK, which means that it will respond to any packet with TTL equal to or greater than the true number of hops to the destination. As a result, an algorithm that can be used for the destination case is similar to the path merging algorithm, but in this case paths are not required to be of the same length. For example, for the following paths:
A-pathl-C
A-path2-C
where C is the destination, but pathl and path2 are of different lengths. Then, if pathl and path2 are mergeable, the path can still be reduced, such as follows:
A-x-x-x-x-C
A-B-C
the result of the path merging is A-B-C.
[00130] Conditions to Reduce a Diamond
[00131] In some embodiments, the condition to filter cases with real stars is as follows: if we have
A-x-C
A-B-C
the diamond is reduced to A-B-C ONLY IF the following rules/conditions are satisfied:
(1) there is a dominant path inside the diamond that has a threshold percentage of routes going through it (e.g., at least 75%); in this case A-B-C needs to be a dominant path for the merging to occur; AND
(2) there is a minimum of four routes entering the diamond (e.g., this can be required to avoid false positives closer to the agents were routes are less dense); as three routes per agent are collected (e.g., in which this last condition also forces the diamond to have routes from at least two different agents).
[00132] Special Cases
[00133] In some embodiments, there are some special cases where the above discussed two rules/conditions (e.g., default rules) are altered slightly. For example, given a mergeable diamond with source node A and destination node B:
(1) If B is actually the destination and rule 1 does not apply (e.g., none of the paths in the diamond is dominant), but rule 2 still applies (e.g., at least 4 routes are entering the diamond), then paths can be simplified/reduced with only stars into a single hop notated with "?" indicating the uncertainty in the number of hops. For example:
A-x-B
A-x-x-B
A-x-C-B
can be reduced to:
A-?-B
A-x-C-B
The "?" link is used in this example, because there is not enough information is available to determine the true number of hops (e.g., or true number of distinct paths) between A and B with sufficient certainty.
(2) If there are more than two distinct paths inside a diamond, do not perform a merge unless there is a dominant path that originates from the same agent as the mergeable path. For example:
agent 1-A-B-x-F
agent2-A-B-C-F (dominant)
agent3-A-B-D-F
could not be reduced, even though A-B-C-F is dominant; neither A-B-C-F nor A-B-D-F appears in a path originating from agent 1, which of the two (if any) agent 1 is connected to cannot be determined. However, if the following path:
agent 1 -A-B-C-F
were also present in the above example, the path with the star could be merged into A-B-C-F according to the default rules as discussed above.
[00134] Information Over Time Versus Over Space
[00135] As discussed above, information collected from multiple sources can be used to infer the reality of a path from a single source. Similar techniques can be used across the time dimension. For example, by looking at the data on the preceding and subsequent intervals, the reality of the path can be inferred at the current interval. Although the space- based approach discussed above can be more convenient in some cases, such as if the necessary information is readily available at the time of visualization, in other cases, a time- based approach can also be used, such as for cases where not enough data in one time slice is available to get an accurate representation of a path.
[00136] Case Studies
[00137] Figure 8A and Figure 8B illustrate views for showing Border Gateway Protocol (BGP) and packet loss in accordance with some embodiments. Figure 8A and 8B are screen shots 800 and 810, respectively, of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
[00138] In particular, Figure 8A illustrates a data path view in which a data path from source site (e.g., Chicago node) to the lowest listed destination does not have an IP address and is shown as having packet loss (e.g., using a visual indicator for that destination node, such as a color coded indicator or other graphical user interface related visual indicator(s)). Figure 5B illustrates a BGP view, in which a window 812 shows an IP address destination for which there are no routes to reach that destination, which corresponds to the path from Chicago for which there is packet loss on the path as shown in the data path view of Figure 8A. Accordingly, Figures 5A and 5B illustrate the user of cross-layer visibility and troubleshooting into application delivery in which a root cause of a packet loss seen on the Chicago node in Figure 8A can be illustrated by navigating to a BGP view for that same time interval to determine that the Chicago node was unreachable due to BGP route information related problems. In particular, Figure 8B provides a visual representation that explains that the packet loss seen on the Chicago node in Figure 8A is a result of the unavailability to a certain BGP prefix.
[00139] DNS
[00140] Collecting and Visualizing DNS Information
[00141] Figure 9 illustrates a visualization of DNS data in accordance with some embodiments. In particular, Figure 9 is a screen shot 900 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
[00142] In some embodiments, a DNS test (e.g., that can be performed by one or more distributed agents and controlled by agent controllers, such as agent controllers 114) includes several authoritative name servers that serve a specific domain name. For example, DNS queries from agents can be sent to the specified DNS servers targeting the specified domain name (e.g., one or more domain names that an entity wants to test for, such as www.domain- name.com). For example, the following cases can be marked as errors:
• The absence of a reply - the name server does not reply within a certain timeout to the
query.
• An empty reply - the name server replied with an NXDOMAIN, or an empty resource record.
[00143] This technique can be used for identifying and resolving application level problems (e.g., application layer response time), as DNS tests can be correlated to identify which DNS name server(s) may be the root of the problem. For example, if a problem is identified with a particular DNS name server, then a user can drill down to the network layer view to determine if it is a network problem to access that DNS name server from a particular location (e.g., a location at which one or more agents are present for such testing).
[00144] As shown in Figure 9, various visualization elements are provided that are similar to previously discussed views. For example, metric selector 204 can be used to choose between Availability and Resolution Time. Timeline 206 can be used to select an instant in time to drill down to. World map 208 shows a geographical representation of the problem. Summary window 210 shows a small summary of the metrics/ number of errors. Table 212 shows the breakdown of the DNS tests per location.
[00145] For example, when no server is selected, the view can show aggregated metrics (e.g., average for the case of Availability and Minimum value for the case of
Resolution Time). A minimum can be selected as this is the most likely the value a DNS resolver would get if it had to query multiple authoritative servers for a domain.
[00146] DNSSEC
[00147] In some embodiments, a similar visualization for DNSSEC is provided, such as for the DNS extension that adds authentication and data integrity to DNS. In order to test DNSSEC delegation chains, a test from each agent is provided in which the test can start at the bottom of the chain and verify signatures all the way to the top of the chain. An error is triggered if there is any step in the resolution chain that fails, either because a resource record is missing or because a signature does not match.
[00148] Case Studies
[00149] Figure 10 illustrates Anycast troubleshooting, to identify any relevant DNS problems, in accordance with some embodiments. In particular, Figure 10 is a screen shot 1000 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
[00150] In particular, Figure 10 illustrates a DNS view, illustrating DNS resolutions to the DNS servers serverl .com (e.g., showing four errors in this example) and server2.com (e.g., showing 7 errors in this example) in table 1010, which are geographically represented by location in red on the world map view 1008, and the time interval is 6:00-6:00:47 on 3-8- 2012.
[00151] Figure 11 illustrates a path visualization that shows all problem agents routing to an Ashburn, Virginia instance in accordance with some embodiments. In particular, Figure 11 is a screen shot 1100 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
[00152] In particular, Figure 11 illustrates a path visualization to server2.com in path visualization graph 1102. In this example, DNS Anycast is used, which can announce its IP from several different locations (e.g., five different locations). This shows that a commonality among the agents that have DNS errors is that all of these agents are performing their DNS lookup with this particular Anycast server in Ashburn Virginia, which can be
inferred after looking at the location of the hops immediately preceding the server in path visualization graph 1102 as shown.
[00153] HTTP
[00154] Figure 12 illustrates a visualization of HTTP data in accordance with some embodiments. In particular, Figure 12 is a screen shot 1200 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein. The various other components as shown in Figure 12 are similar to components shown in previously discussed views, including timeline 206, world map 208, summary 210, and table 212 components of the screen shot 1200.
[00155] In some embodiments, for HTTP tests, the URL to be tested is fetched from each agent location, the time to complete each step is measured, and whether there was an error at any particular step is determined and recorded. For example, an HTTP fetch can include the following steps: DNS resolution, TCP connection establishment, SSL certificate exchange, HTTP GET request, and/or HTTP response. In some cases, the HTTP tests can capture the following metrics: Availability: an indication of errors, the aggregate availability is the fraction of agents that complete the test without any error; Response Time: or Time to First Byte (TTFB) is the time it takes for the client to start receiving HTTP data from the server; this metric is useful to understand if delays are coming from network or from backend; typically slow response times are caused by slow networks; and/or Fetch Time: this is the time it takes to actually receive all the HTTP data.
[00156] Web page
[00157] Figure 13 illustrates a visualization of a web page load performance in accordance with some embodiments. In particular, Figure 13 is a screen shot 1300 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
[00158] In some embodiments, in order to measure the performance of an entire web page, real web browsers can be used by agents (e.g., web testing agents that can be distributed and controlled by agent controllers, such as agent controllers 114) to load the web page(s) from different locations and extract the performance for each of the components of the web page(s). For example, the components can be grouped per domain and per provider
to rank Response Time and Throughput, and to better identify bottlenecks. Example metrics can include the following: DOM time: the time it takes for the document object model (DOM) to build; after the DOM is created, the user can start interacting with the page; and/or Page load time: the time it takes for the entire page to load, including style sheets and images.
[00159] In some embodiments, when a user clicks in a location in world map 1308, the waterfall at 1312 is updated. The waterfall view 1312 provides a breakdown of the individual performance of each component of the page, for example, including time measurements for each of the following: Blocked: the time the browser is waiting for a TCP connection to be available to process the request; DNS: the time it takes to do a DNS resolution for the name of the resource; Connect: the time it takes to establish a TCP connection to the server; SSL/TSL: the time it takes to do the SSL/TSL handshake; Wait: the time between sending the HTTP request and getting the first byte of the reply back from the server; and Receive: the time it takes from the first byte of the answer until the last byte is received.
[00160] Case Studies
[00161] Figure 14 illustrates a bottleneck analysis by provider for page loads in accordance with some embodiments. In particular, Figure 14 is a screen shot 1400 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
[00162] In particular, Figure 14 illustrates a page load view that indicates detailed views of page load by provider (e.g., each object can provide from different providers, such as Dell, Adobe, and Akamai for www.example.com), for a particular period of time and an average response time and throughput, using browser measurements (e.g., using web testing agents, as similarly discussed above).
[00163] Web Transactions
[00164] Figure 15 illustrates a visualization of transactions (e.g., web transactions) in accordance with some embodiments. In particular, Figure 15 is a screen shot 1500 of a GUI presented via an interface (e.g., using platform 100) that provides a visualization of a network performance using various techniques described herein.
[00165] In some embodiments, in addition to tests on individual web pages, the platform for cross-layer visibility for application delivery also supports transaction tests, such as for web-based transactions. For example, a transaction can refer to a series of scripted steps that are automatically executed in a browser. As an example, a transaction can include going to a homepage of a web site and adding an item into a shopping cart. As another example, a transaction can include going to a home page of a web site, logging into the web site with user credentials, and downloading a first home login page associated with that user's credentials.
[00166] Accordingly, in some embodiments, transactions are used to refer to extensions of single page loads, and, as shown in Figure 15, the visualization of transactions can similarly be provided as for single page loads (e.g., simple, single web page load transactions). In some embodiments, on each transaction, users are allowed to define an initial step and a final step. For example, these steps can be used to define the boundaries of interest in the transaction and to define the boundaries for computing the transaction time for the measured transaction.
[00167] As shown in Figure 15, in waterfall 1512, all the steps of the transaction are presented as vertical colored lines or using another visual indicator(s) (e.g., the x-offset corresponds to elapsed time since step 0). The individual components that are outside of the interval defined between the initial and final step can be faded out to visually emphasize the relevant steps. In the summary component 1510, a breakdown of the time spent on each step and the time each page loaded during a transaction took to load is provided. For example, metrics for transactions can include: Completion: the fraction of transaction steps that were completed during the transaction; and/or Transaction time: the time between the initial step and the final step of the transaction.
[00168] Case Studies
[00169] Figure 16 shows a summary of transaction steps and identifying bottlenecks in accordance with some embodiments. For example, Figure 16 illustrates web transaction screen shot 1600 of a summary including a time per step and time per page for an example web transaction test results. In particular, as shown, step 6 for page load #1 was measured as using a relatively significant/longer amount of time. Thus, this indicates that step 6 for page load #1 may be the cause of the bottleneck for these web transactions.
[00170] Diagnosing performance issues using Browser Extension
[00171] Performance problems can be challenging and time consuming to diagnose when they cannot be reproduced. When users experience performance problems and report to IT or to the service provider, the providers often use data from a nearby city from an external monitoring system to cross check. However, this is problematic, because it does not collect data from the end user reporting the problem.
[00172] Accordingly, in some embodiments, a new browser extension is provided
(e.g., for a web browser), which is a lightweight extension that can be installed by any users that desire to report their experience to a service provider or Information Technology (IT) department. For example, when the user clicks on a small "capture performance" icon on the browser of the user's client device (e.g., laptop, computer, smart phone, or other computing device that can execute a web browser), the add-on starts recording errors and timings for each object received by the browser, and also perform active measurements such as traceroutes at the same instant of time. The extension then exports the data collected to a collector of a service provider or IT department (e.g., cloud based network performance service provider or IT), which channels the data to the relevant customer of the service provider/IT based on the domain that was being tested. For example, data on a test to store.apple.com can be made available on the Apple users (e.g., in the service provider's data for the customer) if they exist.
[00173] Figure 17 illustrates a flow diagram for cross-layer troubleshooting of application delivery in accordance with some embodiments. In some embodiments, process 1700 is performed using platform 100 as shown in Figure 1. At 1702, collecting test results from a plurality of distributed agents for a plurality of application delivery layers is performed. At 1704, generating a graphical visualization of an application delivery state for the plurality of application delivery layers based on the test results is performed. At 1706, the graphic visualization is output.
[00174] Figure 18 illustrates another flow diagram for cross-layer troubleshooting of application delivery in accordance with some embodiments. In some embodiments, process 1800 is performed using platform 100 as shown in Figure 1. At 1802, sending tests to distributed agents (e.g., application delivery tests) to perform for different application delivery layers (e.g., to perform various active measurements across different application
delivery layers) is performed. At 1804, collecting test results from the distributed agents for the different application delivery layers is performed. At 1806, correlating the collected test results for the different application delivery layers is performed. At 1808, generating a graphical visualization based on the correlated test results for the different application delivery layers (e.g., generating a graphical visualization of application delivery results for the different layers and enabling layer correlation) is performed.
[00175] Although the foregoing embodiments have been described in some detail for purposes of clarity of understanding, the invention is not limited to the details provided. There are many alternative ways of implementing the invention. The disclosed embodiments are illustrative and not restrictive.
Claims
1. A system for cross-layer troubleshooting of application delivery, comprising:
a processor configured to:
collect test results from a plurality of distributed agents for a plurality of application delivery layers; and
generate a graphical visualization of an application delivery state for the plurality of application delivery layers based on the test results;
and
a memory coupled to the processor and configured to provide the processor with instructions.
2. The system recited in claim 1, wherein the graphical visualization of the application delivery state facilitates cross-layer troubleshooting of problems.
3. The system recited in claim 1, wherein the graphical visualization of the application delivery state facilitates cross-layer visualization and troubleshooting of application delivery.
4. The system recited in claim 1, wherein the graphical visualization of the application delivery state facilitates cross-layer troubleshooting of application delivery by providing for correlation of the test results across the plurality of application delivery layers.
5. The system recited in claim 1, wherein the graphical visualization of the application delivery state facilitates cross-layer troubleshooting of application delivery by providing for correlation of the test results across the plurality of application delivery layers using a space domain.
6. The system recited in claim 1, wherein the graphical visualization of the application delivery state facilitates cross-layer troubleshooting of application delivery by providing for correlation of the test results across the plurality of application delivery layers using a time domain.
7. The system recited in claim 1, wherein the plurality of distributed agents are controlled by an agent controller.
8. The system recited in claim 1 , wherein the graphical visualization of network performance is generated by a web tier.
9. The system recited in claim 1 , wherein a plurality of tests that can be performed by the distributed agents are configurable.
10. The system recited in claim 1, wherein the processor is further configured to:
output the graphical visualization of the application delivery state based on the test results.
11. A method of cross-layer troubleshooting of application delivery, comprising:
collecting test results from a plurality of distributed agents for a plurality of application delivery layers using a processor; and
generating a graphical visualization of an application delivery state for the plurality of application delivery layers based on the test results using a processor.
12. The method of claim 11, wherein the graphical visualization of the application delivery state facilitates cross-layer troubleshooting of problems.
13. The method of claim 11, wherein the graphical visualization of the application delivery state facilitates cross-layer visualization and troubleshooting of application delivery.
14. The method of claim 11, wherein the graphical visualization of the application delivery state facilitates cross-layer troubleshooting of application delivery by providing for correlation of the test results across the plurality of application delivery layers.
15. The method of claim 11, wherein the graphical visualization of the application delivery state facilitates cross-layer troubleshooting of application delivery by providing for correlation of the test results across the plurality of application delivery layers using a space domain.
16. The method of claim 11, wherein the graphical visualization of the application delivery state facilitates cross-layer troubleshooting of application delivery by providing for correlation of the test results across the plurality of application delivery layers using a time domain.
17. The method of claim 11, wherein the plurality of distributed agents are controlled by an agent controller.
18. The method of claim 11 , further comprising:
outputting the graphical visualization of the application delivery state based on the test results.
19. A computer program product for cross-layer troubleshooting of application delivery, the computer program product being embodied in a tangible computer readable storage medium and comprising computer instructions for:
collecting test results from a plurality of distributed agents for a plurality of application delivery layers; and
generating a graphical visualization of an application delivery state for the plurality of application delivery layers based on the test results.
20. The computer program product recited in claim 19, wherein the graphical
visualization of the application delivery state facilitates cross-layer troubleshooting of application delivery.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP13793705.8A EP2852886A4 (en) | 2012-05-21 | 2013-05-13 | Cross-layer troubleshooting of application delivery |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US201261649473P | 2012-05-21 | 2012-05-21 | |
US61/649,473 | 2012-05-21 | ||
US13/839,214 | 2013-03-15 | ||
US13/839,214 US10230603B2 (en) | 2012-05-21 | 2013-03-15 | Cross-layer troubleshooting of application delivery |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2013176918A1 true WO2013176918A1 (en) | 2013-11-28 |
Family
ID=49582326
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2013/040797 WO2013176918A1 (en) | 2012-05-21 | 2013-05-13 | Cross-layer troubleshooting of application delivery |
Country Status (3)
Country | Link |
---|---|
US (5) | US10230603B2 (en) |
EP (1) | EP2852886A4 (en) |
WO (1) | WO2013176918A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2017147210A1 (en) | 2016-02-26 | 2017-08-31 | Arista Networks, Inc. | System and method of a cloud service provider tracer |
Families Citing this family (76)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9729414B1 (en) | 2012-05-21 | 2017-08-08 | Thousandeyes, Inc. | Monitoring service availability using distributed BGP routing feeds |
US10230603B2 (en) | 2012-05-21 | 2019-03-12 | Thousandeyes, Inc. | Cross-layer troubleshooting of application delivery |
US10511497B2 (en) * | 2012-10-04 | 2019-12-17 | Fortinet, Inc. | System and method for dynamic management of network device data |
US10355930B2 (en) | 2013-03-14 | 2019-07-16 | Fortinet, Inc. | System and method of subnetting a virtual network identifier |
US9411787B1 (en) | 2013-03-15 | 2016-08-09 | Thousandeyes, Inc. | Cross-layer troubleshooting of application delivery |
US9203740B2 (en) * | 2013-07-18 | 2015-12-01 | Globalfoundries Inc. | Automated network fault location |
US20150237140A1 (en) * | 2014-02-14 | 2015-08-20 | Tenoware R&D Limited | Data storage systems and methods |
JP2017516241A (en) * | 2014-03-20 | 2017-06-15 | クイデル コーポレーション | Wireless system for monitoring disease in near real time |
CN104092730A (en) * | 2014-06-20 | 2014-10-08 | 裴兆欣 | Cloud computing system |
US20160164734A1 (en) | 2014-12-01 | 2016-06-09 | Stateless Networks, Inc. | System and method of optimizing paths in a network |
US9825834B2 (en) * | 2015-03-30 | 2017-11-21 | Airwatch Llc | Network speed detection |
US11190397B2 (en) * | 2015-05-11 | 2021-11-30 | Verisign, Inc. | Identifying trusted configuration information to perform service discovery |
CN111030929A (en) * | 2015-10-16 | 2020-04-17 | 华为技术有限公司 | Route processing method, equipment and system |
US9930029B2 (en) * | 2016-02-25 | 2018-03-27 | Nutanix, Inc. | Hypervisor agnostic bidirectional secure channel for guest agent transport |
US20180284745A1 (en) | 2016-05-09 | 2018-10-04 | StrongForce IoT Portfolio 2016, LLC | Methods and systems for self-organization of collected data using 3rd party data from a data marketplace in an industrial internet of things environment |
US11774944B2 (en) | 2016-05-09 | 2023-10-03 | Strong Force Iot Portfolio 2016, Llc | Methods and systems for the industrial internet of things |
US11327475B2 (en) | 2016-05-09 | 2022-05-10 | Strong Force Iot Portfolio 2016, Llc | Methods and systems for intelligent collection and analysis of vehicle data |
US10009336B2 (en) * | 2016-05-18 | 2018-06-26 | Cisco Technology, Inc. | Network security system to validate a server certificate |
US10659325B2 (en) * | 2016-06-15 | 2020-05-19 | Thousandeyes, Inc. | Monitoring enterprise networks with endpoint agents |
US11237546B2 (en) | 2016-06-15 | 2022-02-01 | Strong Force loT Portfolio 2016, LLC | Method and system of modifying a data collection trajectory for vehicles |
US10671520B1 (en) | 2016-06-15 | 2020-06-02 | Thousandeyes, Inc. | Scheduled tests for endpoint agents |
US11687501B2 (en) * | 2016-06-24 | 2023-06-27 | Spectra Logic Corporation | Informed data retrieval |
US10171596B2 (en) | 2016-10-06 | 2019-01-01 | Bmc Software, Inc. | Automatic server cluster discovery |
US10728113B2 (en) * | 2016-10-26 | 2020-07-28 | Zscaler, Inc. | Systems and methods for troubleshooting and performance analysis of cloud based services |
US11811623B2 (en) | 2016-10-26 | 2023-11-07 | Zscaler, Inc. | Deep tracing of user experience |
US10951462B1 (en) | 2017-04-27 | 2021-03-16 | 8X8, Inc. | Fault isolation in data communications centers |
US10908602B2 (en) | 2017-08-02 | 2021-02-02 | Strong Force Iot Portfolio 2016, Llc | Systems and methods for network-sensitive data collection |
KR20200037816A (en) | 2017-08-02 | 2020-04-09 | 스트롱 포스 아이오티 포트폴리오 2016, 엘엘씨 | Methods and systems for detection in an industrial Internet of Things data collection environment with large data sets |
US10776246B2 (en) * | 2017-08-18 | 2020-09-15 | Vmware, Inc. | Presenting a temporal topology graph of a computing environment at a graphical user interface |
US10756990B1 (en) * | 2017-09-22 | 2020-08-25 | Equinix, Inc. | Monitoring and performance improvement of enterprise applications using correlated data associated with a plurality of service layers |
US11729043B2 (en) * | 2017-10-06 | 2023-08-15 | Cisco Technology, Inc. | Traffic outage detection in the internet |
US10892976B2 (en) * | 2017-10-24 | 2021-01-12 | Keysight Technologies, Inc. | Methods, systems, and computer readable media for intelligent network topology mapping |
US10616043B2 (en) * | 2017-11-27 | 2020-04-07 | Google Llc | Real-time probabilistic root cause correlation of network failures |
CA3089761C (en) * | 2018-02-28 | 2023-10-31 | Cyber Defence Qcd Corporation | Methods and systems for cyber-monitoring and visually depicting cyber-activities |
US10909010B2 (en) | 2018-04-10 | 2021-02-02 | Nutanix, Inc. | Efficient data restoration |
US10778534B2 (en) * | 2018-06-13 | 2020-09-15 | Juniper Networks, Inc. | Virtualization infrastructure underlay network performance measurement and monitoring |
US11144185B1 (en) | 2018-09-28 | 2021-10-12 | Splunk Inc. | Generating and providing concurrent journey visualizations associated with different journey definitions |
US11762869B1 (en) | 2018-09-28 | 2023-09-19 | Splunk Inc. | Generating journey flow visualization with node placement based on shortest distance to journey start |
US11032124B1 (en) | 2018-10-24 | 2021-06-08 | Thousandeyes Llc | Application aware device monitoring |
US10848402B1 (en) | 2018-10-24 | 2020-11-24 | Thousandeyes, Inc. | Application aware device monitoring correlation and visualization |
US10887196B2 (en) * | 2018-11-28 | 2021-01-05 | Microsoft Technology Licensing, Llc | Efficient metric calculation with recursive data processing |
US10997192B2 (en) | 2019-01-31 | 2021-05-04 | Splunk Inc. | Data source correlation user interface |
US10567249B1 (en) | 2019-03-18 | 2020-02-18 | Thousandeyes, Inc. | Network path visualization using node grouping and pagination |
US10754638B1 (en) | 2019-04-29 | 2020-08-25 | Splunk Inc. | Enabling agile functionality updates using multi-component application |
US10805144B1 (en) * | 2019-06-18 | 2020-10-13 | Cisco Technology, Inc. | Monitoring interactions between entities in a network by an agent for particular types of interactions and indexing and establishing relationships of the components of each interaction |
US10983856B2 (en) * | 2019-06-28 | 2021-04-20 | Microsoft Technology Licensing, Llc | Identifying root causes of performance issues |
US11568421B1 (en) * | 2019-07-24 | 2023-01-31 | Walgreen Co. | Client side diagnostics for enhanced customer care |
WO2021021267A1 (en) * | 2019-07-31 | 2021-02-04 | Thousandeyes, Inc. | Scheduled tests for endpoint agents |
USD982021S1 (en) * | 2019-09-03 | 2023-03-28 | Silvus Technologies, Inc. | Display screen or portion thereof with a graphical user interface |
US11151125B1 (en) | 2019-10-18 | 2021-10-19 | Splunk Inc. | Efficient updating of journey instances detected within unstructured event data |
CN111106970B (en) * | 2019-12-31 | 2023-05-09 | 腾讯科技(深圳)有限公司 | Data monitoring method and device, electronic equipment and readable storage medium |
US11544410B2 (en) | 2020-01-10 | 2023-01-03 | Zscaler, Inc. | Secure access to third-party cloud-based applications |
US10972370B1 (en) * | 2020-01-10 | 2021-04-06 | Zscaler, Inc. | Monitoring and analysis of cloud-based applications |
US20220027469A1 (en) * | 2020-07-22 | 2022-01-27 | Zscaler, Inc. | Cloud access security broker systems and methods for active user identification and load balancing |
US11809447B1 (en) * | 2020-04-30 | 2023-11-07 | Splunk Inc. | Collapsing nodes within a journey model |
US11741131B1 (en) | 2020-07-31 | 2023-08-29 | Splunk Inc. | Fragmented upload and re-stitching of journey instances detected within event data |
US11374824B2 (en) * | 2020-11-27 | 2022-06-28 | At&T Intellectual Property I, L.P. | Time-based visualization for network virtualization platform |
US11604515B2 (en) | 2020-11-27 | 2023-03-14 | At&T Intellectual Property I, L.P. | Network virtualization platforms enhanced with non-visual sensory interactivity |
US11811633B2 (en) | 2021-01-14 | 2023-11-07 | Zscaler, Inc. | Traceroutes through external proxies |
US11425015B2 (en) | 2021-01-14 | 2022-08-23 | Zscaler, Inc. | Accurate differential traceroute latency calculation between hops |
US11758025B2 (en) | 2021-01-14 | 2023-09-12 | Zscaler, Inc. | Adaptive tracing using one of a plurality of protocols |
US11671438B2 (en) | 2021-01-14 | 2023-06-06 | Zscaler, Inc. | Detection of latency, packet drops, and network hops through a tunnel by tracing hops therein |
US11637766B2 (en) | 2021-01-14 | 2023-04-25 | Zscaler, Inc. | Detection of network hops and latency through an opaque tunnel and detection misconfiguration of tunnels |
US11770319B2 (en) * | 2021-01-14 | 2023-09-26 | Zscaler, Inc. | TCP traceroute using RST and SYN-ACK to determine destination reachability |
US11863415B2 (en) | 2021-01-14 | 2024-01-02 | Zscaler, Inc. | Determining endpoint and application behavior for monitoring user experience |
US11153190B1 (en) | 2021-01-21 | 2021-10-19 | Zscaler, Inc. | Metric computation for traceroute probes using cached data to prevent a surge on destination servers |
US11784904B2 (en) | 2021-01-14 | 2023-10-10 | Zscaler, Inc. | Adaptive tracing with a reduced number of probes to avoid firewall issues |
US11949578B2 (en) | 2021-01-14 | 2024-04-02 | Zscaler, Inc. | Adaptive probing to discover a protocol for network tracing |
US11546240B2 (en) * | 2021-03-01 | 2023-01-03 | Zscaler, Inc. | Proactively detecting failure points in a network |
US11647067B2 (en) | 2021-03-05 | 2023-05-09 | Zscaler, Inc. | Cached web probes for monitoring user experience |
US12010007B1 (en) * | 2021-03-16 | 2024-06-11 | Amazon Technologies, Inc. | Detecting noisy agents in network monitoring |
US11765063B2 (en) * | 2021-06-14 | 2023-09-19 | Capital One Services, Llc | System for creating randomized scaled testing |
EP4378137A1 (en) * | 2021-07-30 | 2024-06-05 | Cisco Technology, Inc. | Multi-service views for network monitoring visualization |
US20230102271A1 (en) * | 2021-09-24 | 2023-03-30 | Cisco Technology, Inc. | Detection and mitigation of domain-based anomalies |
US11855867B2 (en) | 2021-12-28 | 2023-12-26 | Palo Alto Networks, Inc. | Enhanced identification of sources of delays in packet delivery along a path |
US12028237B2 (en) | 2022-06-28 | 2024-07-02 | Zscaler, Inc. | Egress handling for networks with public internet protocol (IP) address |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20080222287A1 (en) * | 2007-03-06 | 2008-09-11 | Microsoft Corporation | Constructing an Inference Graph for a Network |
US7698396B2 (en) * | 2000-01-31 | 2010-04-13 | Hitachi Software Engineering Co., Ltd. | Method of automatically recognizing network configuration including intelligent packet relay equipment, method of displaying network configuration chart, and system thereof |
US20110286418A1 (en) * | 2009-02-03 | 2011-11-24 | Hang Liu | Method and apparatus for dynamic channel assignment and stream control in multi-hop wireless networks |
Family Cites Families (124)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5655081A (en) | 1995-03-08 | 1997-08-05 | Bmc Software, Inc. | System for monitoring and managing computer resources and applications across a distributed computing environment using an intelligent autonomous agent architecture |
US5881237A (en) | 1996-09-10 | 1999-03-09 | Ganymede Software, Inc. | Methods, systems and computer program products for test scenario based communications network performance testing |
US5940376A (en) | 1997-01-29 | 1999-08-17 | Cabletron Systems, Inc. | Method and apparatus to establish a tap-point in a switched network using self-configuring switches having distributed configuration capabilities |
US6665702B1 (en) | 1998-07-15 | 2003-12-16 | Radware Ltd. | Load balancing |
US6397359B1 (en) | 1999-01-19 | 2002-05-28 | Netiq Corporation | Methods, systems and computer program products for scheduled network performance testing |
US6611955B1 (en) * | 1999-06-03 | 2003-08-26 | Swisscom Ag | Monitoring and testing middleware based application software |
US6625648B1 (en) | 2000-01-07 | 2003-09-23 | Netiq Corporation | Methods, systems and computer program products for network performance testing through active endpoint pair based testing and passive application monitoring |
WO2001086775A1 (en) | 2000-05-05 | 2001-11-15 | Aprisma Management Technologies, Inc. | Help desk systems and methods for use with communications networks |
US7886054B1 (en) | 2000-10-11 | 2011-02-08 | Siddhartha Nag | Graphical user interface (GUI) for administering a network implementing media aggregation |
JP2002084249A (en) | 2000-09-08 | 2002-03-22 | Fujitsu Ltd | Central management apparatus |
US7937470B2 (en) | 2000-12-21 | 2011-05-03 | Oracle International Corp. | Methods of determining communications protocol latency |
US20060098586A1 (en) | 2001-03-09 | 2006-05-11 | Farrell Craig A | Method and apparatus for application route discovery |
US6990101B1 (en) | 2001-03-23 | 2006-01-24 | Advanced Micro Devices, Inc. | System and method for performing layer 3 switching in a network device |
US7149210B2 (en) | 2001-04-02 | 2006-12-12 | Demartino Kevin A | Wide area multi-service communications network based on dynamic channel switching |
US20020198985A1 (en) | 2001-05-09 | 2002-12-26 | Noam Fraenkel | Post-deployment monitoring and analysis of server performance |
US6744739B2 (en) | 2001-05-18 | 2004-06-01 | Micromuse Inc. | Method and system for determining network characteristics using routing protocols |
US7155512B2 (en) | 2001-05-23 | 2006-12-26 | Tekelec | Methods and systems for automatically configuring network monitoring system |
US20020199016A1 (en) * | 2001-06-22 | 2002-12-26 | Freedman Avraham T. | Automated control of outbound transist links in a multi-homed BGP routing environment |
US6892227B1 (en) * | 2001-12-21 | 2005-05-10 | Networks Associates Technology, Inc. | Enterprise network analyzer host controller/zone controller interface system and method |
US7299277B1 (en) | 2002-01-10 | 2007-11-20 | Network General Technology | Media module apparatus and method for use in a network monitoring environment |
US20030174650A1 (en) | 2002-03-15 | 2003-09-18 | Broadcom Corporation | Weighted fair queuing (WFQ) shaper |
US20040126840A1 (en) | 2002-12-23 | 2004-07-01 | Affymetrix, Inc. | Method, system and computer software for providing genomic ontological data |
US6993686B1 (en) | 2002-04-30 | 2006-01-31 | Cisco Technology, Inc. | System health monitoring and recovery |
US7092995B2 (en) | 2002-06-25 | 2006-08-15 | Microsoft Corporation | Testing distributed applications |
US7209963B2 (en) | 2002-07-11 | 2007-04-24 | International Business Machines Corporation | Apparatus and method for distributed monitoring of endpoints in a management region |
US7231555B2 (en) * | 2002-08-22 | 2007-06-12 | Agilent Technologies, Inc. | Method and apparatus to coordinate groups of heterogeneous measurements |
US7240325B2 (en) | 2002-09-11 | 2007-07-03 | International Business Machines Corporation | Methods and apparatus for topology discovery and representation of distributed applications and services |
CA2519378A1 (en) | 2003-03-18 | 2004-09-30 | Renesys Corporation | Methods and systems for monitoring network routing |
CA2425442A1 (en) | 2003-04-15 | 2004-10-15 | Felix Katz | Connectivity verification for internet protocol/multi-protocol label switching data communications networks |
US7529192B2 (en) | 2003-07-21 | 2009-05-05 | Arbor Networks | System and method for correlating traffic and routing information |
US7376087B2 (en) | 2003-08-13 | 2008-05-20 | Tropos Networks, Inc. | Method and apparatus for monitoring and displaying routing metrics of a network |
EP1510939A1 (en) | 2003-08-29 | 2005-03-02 | Sap Ag | A method of providing a visualisation graph on a computer and a computer for providing a visualisation graph |
US20080089347A1 (en) | 2003-08-29 | 2008-04-17 | End Ii End Communications Inc. | Systems and methods for broadband network optimization |
US8073943B2 (en) * | 2003-11-24 | 2011-12-06 | Jds Uniphase Corporation | Graphical user interface for adding measurements to existing distributed network troubleshooting system |
US20050198269A1 (en) | 2004-02-13 | 2005-09-08 | Champagne Andrew F. | Method and system for monitoring border gateway protocol (BGP) data in a distributed computer network |
US20050222815A1 (en) | 2004-03-31 | 2005-10-06 | Kevin Tolly | System and method for testing and certifying products |
JP2007533215A (en) | 2004-04-16 | 2007-11-15 | アパレント ネットワークス、インク. | Method and apparatus for automating and scaling IP network performance monitoring and analysis by active probing |
JP4530707B2 (en) | 2004-04-16 | 2010-08-25 | 株式会社クラウド・スコープ・テクノロジーズ | Network information presentation apparatus and method |
US20060023638A1 (en) | 2004-07-29 | 2006-02-02 | Solutions4Networks | Proactive network analysis system |
US7860016B1 (en) * | 2004-08-25 | 2010-12-28 | Emc Corporation | Method and apparatus for configuration and analysis of network routing protocols |
US20060074946A1 (en) | 2004-09-27 | 2006-04-06 | Performance It | Point of view distributed agent methodology for network management |
US7822837B1 (en) * | 2004-12-30 | 2010-10-26 | Packeteer, Inc. | Adaptive correlation of service level agreement and network application performance |
US7636305B1 (en) | 2005-06-17 | 2009-12-22 | Cisco Technology, Inc. | Method and apparatus for monitoring network traffic |
US8130767B2 (en) | 2005-06-17 | 2012-03-06 | Cisco Technology, Inc. | Method and apparatus for aggregating network traffic flows |
US20060291446A1 (en) | 2005-06-24 | 2006-12-28 | Donald Caldwell | Systems, methods, and devices for managing routing |
US7616579B2 (en) | 2005-07-21 | 2009-11-10 | Netcordia, Inc. | Voice over IP analysis system and method |
US8694621B2 (en) * | 2005-08-19 | 2014-04-08 | Riverbed Technology, Inc. | Capture, analysis, and visualization of concurrent system and network behavior of an application |
US8429630B2 (en) * | 2005-09-15 | 2013-04-23 | Ca, Inc. | Globally distributed utility computing cloud |
US8726243B2 (en) * | 2005-09-30 | 2014-05-13 | Telecom Italia S.P.A. | Method and system for automatically testing performance of applications run in a distributed processing structure and corresponding computer program product |
US7945658B1 (en) | 2005-12-05 | 2011-05-17 | Narus, Inc. | Method for real-time visualization of BGP analysis and trouble-shooting |
US8542612B1 (en) | 2005-12-28 | 2013-09-24 | At&T Intellectual Property Ii, L.P. | Method and apparatus for providing a wide area network infrastructure |
US8458319B2 (en) * | 2006-01-11 | 2013-06-04 | Cisco Technology, Inc. | System and method for tracking network resources |
US7633917B2 (en) | 2006-03-10 | 2009-12-15 | Cisco Technology, Inc. | Mobile network device multi-link optimizations |
US8572281B1 (en) | 2006-03-24 | 2013-10-29 | Ciena Corporation | Leaking routes among routing engines in a highly scalable router |
US8214876B2 (en) * | 2006-04-19 | 2012-07-03 | Telcordia Technologies, Inc. | System and method for statistical analysis of border gateway protocol (BGP) configurations |
US7990892B2 (en) | 2006-04-21 | 2011-08-02 | France Telecom | Method of selecting a telephony route within an IP telephony domain, and corresponding apparatus and computer program |
US7656849B1 (en) * | 2006-05-31 | 2010-02-02 | Qurio Holdings, Inc. | System and method for bypassing an access point in a local area network for P2P data transfers |
CN100388695C (en) | 2006-06-07 | 2008-05-14 | 北京邮电大学 | Monitoring and analytic system for route between domain of internet and its working method |
US8185619B1 (en) | 2006-06-28 | 2012-05-22 | Compuware Corporation | Analytics system and method |
US8194555B2 (en) | 2006-08-22 | 2012-06-05 | Embarq Holdings Company, Llc | System and method for using distributed network performance information tables to manage network communications |
US8549405B2 (en) | 2006-08-22 | 2013-10-01 | Centurylink Intellectual Property Llc | System and method for displaying a graphical representation of a network to identify nodes and node segments on the network that are not operating normally |
DK2074752T3 (en) | 2006-10-09 | 2012-08-13 | Ericsson Telefon Ab L M | Elasticity process in connection-oriented communication networks |
US7660261B2 (en) | 2006-11-14 | 2010-02-09 | The Trustees Of Columbia University In The City Of New York | Systems and methods for computing data transmission characteristics of a network path based on single-ended measurements |
US8510464B2 (en) | 2006-12-20 | 2013-08-13 | Cisco Technology, Inc. | Measuring delays from content servers to network devices on paths to a client such as for use in selecting a content server based on a common network device |
US7639625B2 (en) | 2007-03-02 | 2009-12-29 | Cisco Technology, Inc. | Tracing connection paths through transparent proxies |
US8015139B2 (en) | 2007-03-06 | 2011-09-06 | Microsoft Corporation | Inferring candidates that are potentially responsible for user-perceptible network problems |
US7921205B2 (en) * | 2007-04-13 | 2011-04-05 | Compuware Corporation | Website load testing using a plurality of remotely operating agents distributed over a wide area |
US20080263188A1 (en) | 2007-04-20 | 2008-10-23 | Verizon Business Network Services Inc. | Method and system for monitoring and analyzing of routing in ip networks |
US8751619B2 (en) | 2011-05-31 | 2014-06-10 | Cisco Technology, Inc. | Autonomous performance probing |
US7864687B2 (en) * | 2007-12-19 | 2011-01-04 | At&T Intellectual Property Ii, L.P. | Methods and apparatus for fault identification in border gateway protocol networks |
US7975045B2 (en) | 2007-12-26 | 2011-07-05 | Verizon Patent And Licensing Inc. | Method and system for monitoring and analyzing of IP networks elements |
US8325720B2 (en) | 2008-07-17 | 2012-12-04 | NETBRAIN Technologies, Inc | System and method for simulating IP network routing |
US7974219B2 (en) | 2008-09-26 | 2011-07-05 | Hewlett-Packard Development Company, L.P. | Network troubleshooting using path topology |
US7894461B2 (en) | 2008-11-20 | 2011-02-22 | At&T Intellectual Property I, L.P. | Methods and apparatus to infer the status of border gateway protocol sessions |
US7804781B2 (en) | 2008-11-20 | 2010-09-28 | At&T Intellectual Property I, L.P. | Methods and apparatus to detect border gateway protocol session failures |
US8136160B2 (en) | 2008-11-25 | 2012-03-13 | At&T Intellectual Property I, Lp | System and method to select monitors that detect prefix hijacking events |
US8521904B1 (en) | 2008-12-16 | 2013-08-27 | At&T Intellectual Property I, L.P. | Devices, systems, and/or methods for determining internet topology |
US8898280B2 (en) | 2009-02-19 | 2014-11-25 | Fluke Corporation | Methods and apparatus for determining and displaying WAN optimization attributes for individual transactions |
US8208377B2 (en) | 2009-03-26 | 2012-06-26 | Force10 Networks, Inc. | MAC-address based virtual route aggregation |
US8549124B2 (en) | 2009-05-27 | 2013-10-01 | International Business Machines Corporation | Network management discovery tool |
EP2486706B1 (en) | 2009-10-07 | 2016-12-07 | Riverbed Technology, Inc. | Network path discovery and analysis |
US8724494B2 (en) | 2009-10-07 | 2014-05-13 | Riverbed Technology, Inc. | Network multi-path discovery |
US8458301B1 (en) | 2009-10-30 | 2013-06-04 | Bradford Networks, Inc. | Automated configuration of network devices administered by policy enforcement |
US8161190B2 (en) | 2009-12-14 | 2012-04-17 | At&T Intellectual Property I, L.P. | System and method to manage static internet protocol addresses |
JP5407883B2 (en) | 2010-01-14 | 2014-02-05 | 富士通株式会社 | Topology tree creation device, program, and method |
EP2561451A4 (en) | 2010-04-19 | 2018-02-07 | Hewlett Packard Development Company, L.P. | Segmenting a web page into coherent functional blocks |
US9495473B2 (en) | 2010-07-19 | 2016-11-15 | Soasta, Inc. | Analytic dashboard with user interface for producing a single chart statistical correlation from source and target charts during a load test |
US8490055B2 (en) * | 2010-09-17 | 2013-07-16 | Ca, Inc. | Generating dependency maps from dependency data |
EP2633432A4 (en) | 2010-10-26 | 2015-10-21 | Hewlett Packard Development Co | Extraction of content from a web page |
US8527627B2 (en) | 2010-12-14 | 2013-09-03 | At&T Intellectual Property I, L.P. | Intelligent mobility application profiling with respect to identified communication bursts |
US9225554B2 (en) | 2011-01-26 | 2015-12-29 | Cisco Technology, Inc. | Device-health-based dynamic configuration of network management systems suited for network operations |
US8909762B2 (en) | 2011-03-17 | 2014-12-09 | Hewlett-Packard Development Company, L.P. | Network system management |
US8438427B2 (en) * | 2011-04-08 | 2013-05-07 | Ca, Inc. | Visualizing relationships between a transaction trace graph and a map of logical subsystems |
US8675672B1 (en) | 2011-12-30 | 2014-03-18 | Emc Corporation | Hierarchical cluster tree overlay network |
US10230603B2 (en) | 2012-05-21 | 2019-03-12 | Thousandeyes, Inc. | Cross-layer troubleshooting of application delivery |
US9729414B1 (en) * | 2012-05-21 | 2017-08-08 | Thousandeyes, Inc. | Monitoring service availability using distributed BGP routing feeds |
US8971323B2 (en) * | 2012-06-15 | 2015-03-03 | Citrix Systems, Inc. | Systems and methods for dynamic routing in a cluster |
US9143408B2 (en) | 2012-07-30 | 2015-09-22 | Hewlett-Packard Development Company, L.P. | Interprovider virtual private network path identification |
US20170134957A1 (en) | 2012-10-16 | 2017-05-11 | Mcafee, Inc. | System and method for correlating network information with subscriber information in a mobile network environment |
US9225624B2 (en) | 2012-12-20 | 2015-12-29 | Dell Products L.P. | Systems and methods for topology discovery and application in a border gateway protocol based data center |
US20140304407A1 (en) | 2012-12-21 | 2014-10-09 | Boundary, Inc. | Visualizing Ephemeral Traffic |
US9262058B2 (en) | 2013-02-26 | 2016-02-16 | Blackberry Limited | System and methods for navigating social networks |
US9413781B2 (en) | 2013-03-15 | 2016-08-09 | Fireeye, Inc. | System and method employing structured intelligence to verify and contain threats at endpoints |
US9374278B2 (en) | 2013-03-15 | 2016-06-21 | NETBRAIN Technologies, Inc | Graphic user interface based network management system to define and execute troubleshooting procedure |
US10454782B2 (en) | 2013-03-15 | 2019-10-22 | NetBrain Technologies, Inc. | System and method for automating network management tasks |
US9411787B1 (en) * | 2013-03-15 | 2016-08-09 | Thousandeyes, Inc. | Cross-layer troubleshooting of application delivery |
WO2014176752A1 (en) | 2013-04-29 | 2014-11-06 | Hewlett-Packard Development Company, L.P. | Host mobility messaging |
US9277044B2 (en) | 2013-05-09 | 2016-03-01 | Steven P. Kahn | Transportable wireless loudspeaker and system and method for managing multi-user wireless media playback over a media playback system |
US10083250B2 (en) | 2013-05-22 | 2018-09-25 | International Business Machines Corporation | Simplification of large networks and graphs |
US9288143B2 (en) | 2013-10-14 | 2016-03-15 | Hewlett Packard Enterprise Development Lp | Data flow path determination |
US9781008B1 (en) | 2014-04-11 | 2017-10-03 | Cisco Technology, Inc. | Visualization of dynamic fabric automation network topology |
US9740792B2 (en) | 2014-06-18 | 2017-08-22 | Vmware, Inc. | Connection paths for application topology |
US9886445B1 (en) | 2014-08-20 | 2018-02-06 | Vmware, Inc. | Datacenter entity information system |
US10037617B2 (en) | 2015-02-27 | 2018-07-31 | Cisco Technology, Inc. | Enhanced user interface systems including dynamic context selection for cloud-based networks |
US9539901B1 (en) | 2015-04-29 | 2017-01-10 | State Farm Mutual Automobile Insurance Company | Method and system for providing speed limit alerts |
US10110449B2 (en) | 2015-12-18 | 2018-10-23 | International Business Machines Corporation | Method and system for temporal sampling in evolving network |
US10297073B2 (en) | 2016-02-25 | 2019-05-21 | Intel Corporation | Method and apparatus for in-place construction of left-balanced and complete point K-D trees |
US10764393B2 (en) | 2016-04-21 | 2020-09-01 | Hewlett Packard Enterprise Development Lp | Determining a persistent network identity of a networked device |
US10009105B2 (en) | 2016-06-07 | 2018-06-26 | Ciena Corporation | Optical network visualization and planning |
US10466866B2 (en) | 2016-10-31 | 2019-11-05 | Cisco Technology, Inc. | Displaying a complex service topology for monitoring |
US11301500B2 (en) | 2016-12-29 | 2022-04-12 | Sap Se | Clustering for geo-enriched data |
US10388042B2 (en) | 2017-08-25 | 2019-08-20 | Microsoft Technology Licensing, Llc | Efficient display of data points in a user interface |
JP2019091257A (en) | 2017-11-15 | 2019-06-13 | 富士通株式会社 | Information processing device, information processing method, and program |
US10887192B2 (en) | 2018-08-03 | 2021-01-05 | Red Hat, Inc. | Targeted network discovery and visualizations |
-
2013
- 2013-03-15 US US13/839,214 patent/US10230603B2/en active Active
- 2013-05-13 EP EP13793705.8A patent/EP2852886A4/en not_active Ceased
- 2013-05-13 WO PCT/US2013/040797 patent/WO2013176918A1/en active Application Filing
-
2014
- 2014-03-13 US US14/210,128 patent/US9455890B2/en active Active
-
2016
- 2016-08-24 US US15/246,443 patent/US9985858B2/en not_active Expired - Fee Related
-
2019
- 2019-01-18 US US16/251,667 patent/US10986009B2/en active Active
-
2021
- 2021-04-19 US US17/234,246 patent/US20210243099A1/en active Pending
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7698396B2 (en) * | 2000-01-31 | 2010-04-13 | Hitachi Software Engineering Co., Ltd. | Method of automatically recognizing network configuration including intelligent packet relay equipment, method of displaying network configuration chart, and system thereof |
US20080222287A1 (en) * | 2007-03-06 | 2008-09-11 | Microsoft Corporation | Constructing an Inference Graph for a Network |
US20110286418A1 (en) * | 2009-02-03 | 2011-11-24 | Hang Liu | Method and apparatus for dynamic channel assignment and stream control in multi-hop wireless networks |
Non-Patent Citations (1)
Title |
---|
See also references of EP2852886A4 * |
Cited By (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2017147210A1 (en) | 2016-02-26 | 2017-08-31 | Arista Networks, Inc. | System and method of a cloud service provider tracer |
EP3420457A4 (en) * | 2016-02-26 | 2019-10-02 | Arista Networks, Inc. | System and method of a cloud service provider tracer |
US10652126B2 (en) | 2016-02-26 | 2020-05-12 | Arista Networks, Inc. | System and method of a cloud service provider tracer |
Also Published As
Publication number | Publication date |
---|---|
US20190230015A1 (en) | 2019-07-25 |
US20130311832A1 (en) | 2013-11-21 |
US20170026262A1 (en) | 2017-01-26 |
US9455890B2 (en) | 2016-09-27 |
US10986009B2 (en) | 2021-04-20 |
US10230603B2 (en) | 2019-03-12 |
US20210243099A1 (en) | 2021-08-05 |
EP2852886A1 (en) | 2015-04-01 |
US20140280917A1 (en) | 2014-09-18 |
US9985858B2 (en) | 2018-05-29 |
EP2852886A4 (en) | 2016-02-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10986009B2 (en) | Cross-layer troubleshooting of application delivery | |
US9800478B2 (en) | Cross-layer troubleshooting of application delivery | |
US9729414B1 (en) | Monitoring service availability using distributed BGP routing feeds | |
US11121947B2 (en) | Monitoring and analysis of interactions between network endpoints | |
KR102076862B1 (en) | Network performance indicator visualization method and apparatus, and system | |
Madhyastha et al. | iPlane: An information plane for distributed services | |
Javed et al. | PoiRoot: Investigating the root cause of interdomain path changes | |
KR102076861B1 (en) | Network performance diagnosis method and apparatus, and system | |
US7778165B2 (en) | Information plane for determining performance metrics of paths between arbitrary end-hosts on the internet | |
JP7096342B2 (en) | Traffic failure detection on the Internet | |
Fontugne et al. | Pinpointing delay and forwarding anomalies using large-scale traceroute measurements | |
US20060203739A1 (en) | Profiling wide-area networks using peer cooperation | |
Sundaresan et al. | Challenges in inferring internet congestion using throughput measurements | |
Yeganeh et al. | A first comparative characterization of multi-cloud connectivity in today’s internet | |
Bourgeau et al. | TopHat: supporting experiments through measurement infrastructure federation | |
US20230033521A1 (en) | Multi-service views for network monitoring visualization | |
Bermudez et al. | A distributed architecture for the monitoring of clouds and CDNs: Applications to Amazon AWS | |
Wassermann et al. | On the analysis of internet paths with DisNETPerf, a distributed paths performance analyzer | |
KR102027759B1 (en) | Network-related new device registration method and apparatus | |
Borzemski et al. | An Empirical Study of Web Quality: Measuring the Web from Wroclaw University of Technology Campus. | |
US11899568B2 (en) | Enriched application outage insights | |
Goldman et al. | Evaluating and mapping Internet connectivity in the United States | |
Yeganeh | On Measuring Internet Topology & its Applications | |
WO2023009408A1 (en) | Enriched application outage insights | |
WO2023009851A1 (en) | Multi-service views for network monitoring visualization |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 13793705 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2013793705 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |