WO2015077385A2 - Performance monitoring to provide real or near real time remediation feedback - Google Patents
Performance monitoring to provide real or near real time remediation feedback Download PDFInfo
- Publication number
- WO2015077385A2 WO2015077385A2 PCT/US2014/066480 US2014066480W WO2015077385A2 WO 2015077385 A2 WO2015077385 A2 WO 2015077385A2 US 2014066480 W US2014066480 W US 2014066480W WO 2015077385 A2 WO2015077385 A2 WO 2015077385A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- data
- client
- performance
- tenant
- isp
- Prior art date
Links
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3466—Performance evaluation by tracing or monitoring
- G06F11/3495—Performance evaluation by tracing or monitoring for systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/56—Provisioning of proxy services
- H04L67/566—Grouping or aggregating service requests, e.g. for unified processing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
- G06F11/3072—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting
- G06F11/3082—Monitoring arrangements determined by the means or processing involved in reporting the monitored data where the reporting involves data filtering, e.g. pattern matching, time or event triggered, adaptive or policy-based reporting the data filtering being achieved by aggregating or compressing the monitored data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
- H04L41/065—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis involving logical or physical relationship, e.g. grouping and hierarchies
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0654—Management of faults, events, alarms or notifications using network fault recovery
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/34—Signalling channels for network management communication
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Definitions
- Bandwidth is one factor that affects speed of a network.
- Latency is another factor that affects network speed and responsiveness. Latency may be described as delay that affects processing of network data. Network conditions, hardware and software limitations, and/or other factors may adversely affect a user's experience of some online application or service. With the emergence of cloud computing and datacenter services, it is imperative to provide timely service with minimal bottlenecks across hundreds of server computers and associated networking infrastructure serving millions of users worldwide.
- Embodiments provide for monitoring of an online user experience and/or remediating performance issues, but are not so limited.
- a computer-implemented method of an embodiment operates to receive, pre-aggregate, and aggregate client performance data as part of providing an end-to-end diagnostics monitoring and resolution service.
- a system of an embodiment is configured to aggregate performance data of a plurality of client devices or systems as part of identifying latency issues at one or more of a tenant level, geographic location level, and/or service provider level. Other embodiments are included.
- FIGURE 1 depicts an exemplary system that operates in part to provide real or near real time end user performance monitoring services.
- FIGURE 2 is a flow diagram depicting an exemplary process of pre- aggregating and aggregating performance and/or other data.
- FIGURE 3 is a block diagram depicting components of an exemplary end-to- end data processing pipeline.
- FIGURE 4 is flow diagram depicting operations of an exemplary end-to-end process used as part of providing performance diagnostic analysis and/or issue remediation services.
- FIGURE 5 is a block diagram illustrating an exemplary computing
- FIGURES 6A-6B illustrate a mobile computing device with which
- FIGURE 7 illustrates one embodiment of a system architecture for implementation of various embodiments.
- FIGURE 1 depicts an exemplary system 100 that operates in part to provide real or near real time end user performance monitoring services, but is not so limited.
- Components of the system 100 operate in part to use aggregated latency and/or other network data to mitigate and/or resolve network ecosystem issues.
- components of the system 100 can operate to provide failure zone analysis and resolution information to tenants based on aggregations of performance data.
- Components of the system 100 can be used to provide a real or near real time assessment of the usability of an online service as well as being able to identify or target failure zones to troubleshoot and/or correct any associated performance or user-experience problems.
- the system 100 includes features that provide end user performance optics to consumers of an online service including quantifying real time tenant level optics, such as by enabling one or more designated persons of a customer with an ability to view performance or other metrics of a user base across any geographical location or locations.
- components of the system 100 operate in part by collecting tenant level data to identify top latency data or other outliers for reporting or alerting within a defined location of interest. Equipped with an ability to focus at a geographic level can uncover issues specific to location, such as poor CDN performance, DNS resolution time, longer round trip times, etc.
- geographic granularity based on a service provider allows for identifying issues at an Internet Service Provider (ISP) level.
- ISP Internet Service Provider
- components of the system 100 can operate to ascertain one or more failure zones for tenants as well as identify specific users having degraded experience.
- the aggregation service 110 can use rules to generate an aggregated output 112 to generate a geographic- based latency map color coded by scale of communication latency.
- the aggregation service 110 can use configured rules to generate an aggregated output 1 12 as part of debugging and isolating issues based on geographic, ISP, and/or other parameters as described below.
- components of the system 100 operate to identify failure zones, such as by isolating an issue tied to a DNS resolver, ISP peering, network routing, non-optimal hosting locations, etc.
- components of the system 100 can be used to assess or quantify a state of a user experience for one or more locations (e.g., region, country, county, etc.), one or more tenants, a selected tenant by geographic location or ISP, and/or for selected geographic location by ISP.
- the system 100 operates in part to provide for debugging of latency or other data with additional breakdowns by: a client time, a network time, a server time, a CDN time, a connect time, etc.; identifying outlier data, such as a first number of tenants and ISPs by latency; generating historic trends on latency and other performance metrics; providing guidance data for effective edge and other server deployments; enabling pre-aggregating by configuring mailbox servers with geo-mapping capability; generating report data to gain insight into real user CDN interaction; supporting web access based and locally installed clients to reduce load times; etc.
- different types of metrics or other data can be collected and provided to the system 100 for use in quantifying user experiences.
- Components of the system 100 can operate as part of supporting use of an online service or application by proactively operating to identify specific users or user groups having a degraded experience. As described below, as part of assessing a performance state of an online service or application, quantitative comparisons can be made relative to one or more baseline experiences for a particular location or ISP. Establishing robust and up-to- date baselines allows for a more focused and confident response to performance related calls/emails and proactive aspect of identification of outliers can be used to have 360 degree loop with service consumers.
- One embodiment of the system 100 comprises a service support communication infrastructure that enables troubleshooting and remedying performance or other issues related to a server component, a client component, and/or a network condition, such as network latency issues, DNS look up issues, Content Delivery Network (CDN) issues, etc.
- data collection services comprise a decentralized architecture which partitions client data based in part on a datacenter location by processing raw client data for each server node including pre-aggregating raw data before uploading pre-aggregated data to one or more stores, such as a plurality of database servers for example, before final aggregations.
- the aggregation service 1 10 can be configured as a separate or an integrated service running on one or multiple physical machines to globally aggregate the pre-aggregated data across multiple data stores based on a set of common and/or customized metrics.
- pre-aggregating as part of collecting data at each node, processing time and use can be reduced due in part to the limited number of data points used with a final aggregation. As such, aggregated data can be generated in real or near real time.
- the aggregation service 110 of one embodiment is configured to automatically aggregate latency and/or other performance data, including navigation and/or load timing data, to identify issues at different levels or granularities, such as a tenant level, a geographical or location level, and/or an ISP level as part of efficiently remediating any realized or potential issues.
- the system 100 may include multiple server computers, including pre-aggregation servers, database servers, and/or aggregation servers, as well as client devices/systems that operate as part of an end-to-end computing architecture. It will be appreciated that servers may comprise one or more physical and/or virtual machines dependent upon the particular implementation.
- components of the system 100 are configured to collect, pre-aggregate, aggregate, and/or analyze client information as part of providing real or near real time reporting to customers regarding the state of an application or network. Additional components and/or features can be added to the system 100 as needed. For example, based on an identified latency, a customer may use the feedback to deploy an additional edge server in their network. As described below, components of the system 100 may be used to ascertain different user experiences and/or network conditions across multiple networks and network types serving a client or consumer base.
- server 102 receives information from one or more clients shown as input 104.
- input 104 includes performance data associated with a client while using an online service or application.
- raw performance data can be uploaded to server 102 for processing.
- input 104 includes information pertaining to a client experience such as loading and navigating web resources, and/or server 102 comprises a server computer that supports the use of log files to store collected data.
- a browser or other application running on a user device/system can use script code to collect information related to one or more of navigation timing parameters, resource and/or load timing parameters, and/or custom marker parameters which may be written to a server log file.
- server 102 can be configured as a MICROSOFT EXCHANGE server to use one or more fault-tolerant, transaction-based databases to store information.
- server 102 in addition to processing and memory resources, server 102 includes extensible diagnostic features that utilize a pre-aggregator 106 that operates in part on raw performance data included with input 104, but is not so limited.
- the pre-aggregator 106 of an embodiment operates to parse client data stored in log files as part of extracting and mapping the client data to one or more mapping tables.
- the pre-aggregator 106 operates to parse performance data stored in one or more log files to generate mappings, wherein the mappings are defined in part by transforming client IP address and logged client information to one or more of a geographical location (e.g., country/state), an ISP, and/or tenant global user identifier (GUID).
- the pre-aggregator 106 is configured to group performance data by one or more of IP, location, ISP, and/or tenant GUID before storing the grouped information to store 108.
- the pre-aggregator 106 can be configured to group performance data associated with client latency metrics by country/state, ISP, and/or tenant. If the logged data cannot be resolved to an ISP level, the pre-aggregator 106 can identify groups limited to country and/or tenant. It will be appreciated that country and ISP parameters can be determined according to client IP address.
- the aggregation service 110 operates on the pre-aggregated output provided by pre-aggregator 106 to generate an aggregated output 112.
- the functionality provided by the pre-aggregator 106 operates in part to increase an efficient use of processing and memory resources at the aggregation service 1 10 while also reducing power consumption since a smaller data set can be input to the aggregation service 110 to generate the aggregated output 112.
- the aggregation service 1 10 of an embodiment comprises one or more server computers and complex aggregation code that operates to provide aggregated output 1 12.
- an aggregated output 1 12 can be further processed to identify any potential failure zones and/or other issues that may be contributing to a user experience.
- the aggregation service 110 of one embodiment aggregates pre- aggregated data across all databases to quantify one or more of tenant level, country level, and/or ISP level latencies associated with a particular application, service, or other component.
- rules can be included with the aggregation service 110 to control processing of the pre-aggregated output to generate the aggregated output 112.
- the aggregated output 1 12 provides focus including correlations, trends, baseline comparisons, and/or other quantified information tied to a use experience during execution of an application or an online service.
- rules can be implemented that operate on pre-aggregated data to analyze performance based on an overall value for a region, such as by deriving the 75% percentile x and the standard deviation y for a given metric for North America. If the measurement for Mexico is greater than (x+y), it may cause escalation of a potential issue to engineering staff. Additional features are described further below.
- complex communication architectures typically employ multiple hardware and/or software components including, but not limited to, server computers, networking components, and other components that enable communication and interaction by way of wired and/or wireless networks. While some embodiments have been described, various embodiments may be used with a number of computer configurations, including hand-held devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, minicomputers, mainframe computers, etc. Various embodiments may be implemented in distributed computing environments using remote processing devices/systems that communicate over a one or more communications networks. In a distributed computing environment, program modules or code may be located in both local and remote memory. Various embodiments may be implemented as a process or method, a system, a device, article of manufacture, etc.
- FIGURE 2 is a flow diagram depicting an exemplary process 200 of pre- aggregating and aggregating performance and/or other data as part of providing performance diagnostics and/or remediation services according to an embodiment.
- the process 200 begins at 202 by receiving raw performance data.
- the process 200 at 202 can operate using a server computer to receive client-centric performance data collected by a client as part of requesting an assessment of a state of an online service or application.
- the process 200 at 202 operates to receive client performance data that includes navigation timing, page load timing, and/or other parameters to use when assessing health or user experience associated with an online service or application.
- the process 200 operates to pre-aggregate the raw performance data.
- the process 200 at 204 operates to pre-aggregate the raw performance data by parsing log files and mapping client IP addresses to one or more of tenant identifier, location identifier, and/or ISP identifier before uploading the pre-aggregated data to one or more databases for final aggregation operations.
- the process 200 operates to aggregate the pre-aggregated data.
- process 200 at 206 operates to aggregate the pre-aggregated data in part by generating an output of latency or other user experience quantifiers to identify issues at one or more of a tenant level, a location level, and/or ISP level.
- the process 200 proceeds to 210 and uses the aggregated data for latency and/or other analysis. Otherwise, the process 200 returns to 206 and continues aggregation operations.
- aggregated output can be used as part of remediating any identified issue by implementing contingency or other measures. While a certain number and order of operations are described for the exemplary flow of FIGURE 2, it will be appreciated that other numbers, combinations, and/or orders can be used according to desired implementations.
- FIGURE 3 is a block diagram depicting components of an exemplary end-to- end data processing pipeline 300 that operate in part to provide user insights into aggregated data as part of identifying infrastructure, performance, network, or other issues that may be adversely affecting use of an online application or service.
- an online service supporting cloud-based application services can include functionality to collect and quantify performance data or metrics in near real time including providing user scenario latencies and detailed breakdowns by collected metrics associated with one or more of client operational parameters, tenant parameters, IP parameters, location parameters, and/or ISP parameters.
- Components of the pipeline 300 operate in part to aggregate, pivot, and/or store data at the tenant level, IP level, geographic location level, and/or an ISP level.
- Components of the pipeline 300 operate in part to proactively monitor user experiences to reduce performance degradations while providing alerts and/or solutions to remediate end user performance issues.
- a client 302 associated with a first tenant user and client 304 associated with a second tenant user are communicating with server 306.
- log file 308 receives and stores collected data from clients 302 and 304.
- the client 302 can be implemented as part of a browser application running on a user device system, wherein script code can be used to collect information associated with use of an online application or service, such as a page load time, a time to connect, or some other parameter for example.
- the server 306 of one embodiment comprises a server computer dedicated to serving clients 302 and 304.
- server 306 includes a diagnostics service that uses an IP mapper 310 and upload component 312 for an associated node.
- the IP mapper 310 and upload component 312 operate in part to provide pre- aggregation services on the data of log file 308. As described above, a single component can be configured to perform the pre-aggregation services provided by these components.
- the IP mapper 310 of an embodiment operates in part to parse log file 308 to extract and map logged performance data or metrics based on one or more of an IP address, a location, and/or ISP for each client or tenant. According to one embodiment, the IP mapper 310 operates in part to pre-aggregate and consolidate the client data by mapping a client IP address and performance or latency data to one or more of a geographic location (e.g., country/state), an ISP, and/or a tenant global user identifier (GUID).
- a geographic location e.g., country/state
- ISP Internet Protocol Security
- GUID tenant global user identifier
- the upload component 312 operates to upload the mapped data provided by the IP mapper 310 grouped by one or more of location, ISP, and/or tenant GUID to a dedicated database 314. If the logged data cannot be resolved to an ISP level, the pre-aggregation can include groupings limited to country and/or tenant. It will be appreciated that country and ISP parameters can be determined according to a client IP address.
- components of server 306 are configured with complex programming code that operates to pre-aggregate collected client data in part by parsing the collected client data, such as by parsing performance data logs for example, and extracting user scenario, time of event, client IP, latency, tenant data and other detailed metrics based on the client information. Consequently, the server 306 is able to pre-aggregate data received from client as part of reducing the final aggregation load when quantifying latency and/or other performance issues.
- the IP mapper 310 of an embodiment operates to map client IP addresses to a geographic location depending on the mapping granularity and/or a client IP to an associated ISP based on known or to be implemented IP ranges.
- the server 306 includes analysis code that operates to parse based in part on a type of client and/or associated client data. For example, performance data of a web access client can be collected and routed to a log file of mailbox server serving the sessions, wherein the analysis code would be configured to parse the particular client information to understand a scenario, latency, and associated issues (e.g., slow navigation time, slow DNS time, etc.).
- Parsing of an embodiment operates to transform client IP address and tenant information in the log files to country/state, ISP and/or tenant GUID.
- parsing operations are performed in part using a derived mapping table generated from a generic public geo-mapping database.
- An example data entry in a geo-mapping database for parsing may include:
- the parsing operations applied by the IP mapper 310 of an embodiment result in the generation of a derived mapping table for IP to countries by scanning each data entry, sorting, and merging based on IP ranges and corresponding countries to yield:
- a mapping table can include exemplary mapping ⁇ key,value ⁇ data.
- the mapped data includes a key that is an integer value that represents a starting IP address and a value that is the country ISO code.
- IP addresses between 16777216 and 16777472 belong to AU. By sorting the keys, the table can be compressed for loading into memory for quick look-up.
- parsing operations applied by the IP mapper 310 of an embodiment result in the generation of a derived mapping table for an IP to ISP mapping as shown below (key is the same as above but the value is an ASN number of an ISP):
- server 316 processes or pre-aggregates client data of clients 318 and 320 stored in log file 321 in part by using the IP mapper 322 and upload component 324 to process and upload pre- aggregated data to another dedicated database 326.
- Dedicated databases 314 and 326 may or may not include more than one host computer.
- the pipeline can include additional components, features, and functionality.
- Server 328 processes client data of clients 330, 332, 334, and 336 stored in log file 337 in part by using the IP mapper 338 and upload component 340 to process and upload pre-aggregated data to dedicated database 326.
- databases 314 and 326 are designed to handle the performance counters and metrics collected from various machines that may be networked to provide an online application or service. Since the end user performance data brings in additional pivots, a database schema can be used to support IP, geographic location, tenant, and/or ISP metrics and parameters.
- server 306, server 316, and server 328 collect client data from a plurality of clients. For example, at the node level, server 306 can operate to pre-aggregate client data every 5 minutes using IP mapper 310 to transform the client data into predetermined pivots and the upload component 316 propagates the transformed data to database 314.
- Aggregation service 342 aggregates the pre-aggregated data across databases 314 and 326 to determine one or more of tenant level latencies, country level latencies, and/or ISP level latencies associated with an online application or service, but is not so limited.
- the aggregation service 342 operates on the pre-aggregated or transformed data to perform scope (Global and/or Site for example) level conversion on the node level data for end user metrics.
- scope Global and/or Site for example
- the aggregation service 342 has provided an aggregated output that includes quantified client performance data 346 associated with the first tenant and quantified client performance data 348 associated with the second tenant.
- a number of sample counts can be used as a weighting factor to improve statistical accuracy of the quantified client performance data.
- the aggregation service 342 can be configured to aggregate pre-aggregated data uploaded from one of more upload components at defined time intervals (e.g., run every 15 min., use for a sliding window of last 1 hour of data; run every 24 hours, use sliding window of last 24 hours of data, etc.).
- the aggregation service 342 can also be configured to pivot or group, across one or more domain controllers, by geographic location, tenant, ISP per geographic location, tenant per geographic location, and/or scope per site level.
- the aggregation service 342 operates in part to generate client scenario latency and other performance related statistics for quantifying navigation time, CDN time, authorization time, redirect time, etc.
- the aggregation service 342 can provide statistical measures/values such as average, 75% percentile, 85% percentile, 95% percentile, etc.
- the aggregation service 342 can also use dynamic bins that encompass a range of latencies with percentile values for latencies at 10th, 20th, 30th, 40th, 50th, 60th, 70th, 80th, 90th percentiles, and maximum.
- Failure zone analyzer 350 operates in part using rules that are designed to identify certain segments or characteristics of the data aggregate using statistical measures or other latency quantifications.
- the rules may be designed to identify different levels of performance (e.g., fair, poor, excellent, etc.) based on one or more quantitative measures, such as navigation time, load time, connect time, etc.
- the rules are applied to the aggregated data according to the output from the aggregation service 342.
- Exemplary rules are configurable according to each implementation. For example, rules may be based on an overall value for a region or ISP such as rules configured to prioritize consideration of certain metrics or measures over others.
- Report generator 352 operates to generate report information for reporting and/or feedback communications as to the state of an application or service along with any specific recommendations for tenants having some identified issue that may need to be addressed.
- report generator 352 can operate to dynamically generate a user insight report that lists the top number (e.g., 10) tenants for each geographic location having highest latencies or the top number of tenants having the highest latencies. While shown as integral components, it will be appreciated that failure zone analyzer 350 and report generator 352 can be configured as separate components.
- pivots can be applied solely at the aggregation service 342, or in combination with pivots applied the server 306, server 316, and/or server 328.
- the pipeline 300 of an embodiment uses performance markers as part of: reliably collecting client data; allowing segregation of successful and failed execution of scenario; allowing for filtering/segregation of monitoring data (e.g., probes); accurately marking the start and end of scenarios tied with user experience (e.g., navigation time, page load, page displayed, page interactive, etc.); and/or identifying and filling missing data to assist with detailed drill downs, such as time to complete authentication, time to download CDN resources, time to redirect to correct web-access server, etc.
- monitoring data e.g., probes
- accurately marking the start and end of scenarios tied with user experience e.g., navigation time, page load, page displayed, page interactive, etc.
- identifying and filling missing data to assist with detailed drill downs, such as time to complete authentication, time to download CDN resources, time to redirect to correct web-access server, etc.
- Navigation timing of one embodiment comprise calculated values based on each time stamp defined in the W3C Navigation Timing API.
- the W3C Navigation Timing API introduces the performance timing interface allowing JAVASCRIPT mechanisms to provide complete client-side latency measurements within applications.
- the interface can be used to measure a user's perceived page load time.
- Resource timing markers of one embodiment are the calculated values based on each time stamp defined in the W3C Resource Timing API that defines an interface allowing JAVASCRIPT mechanisms to provide complete client-side latency measurements within applications.
- the interface can be used to measure a user's perceived load time of a resource.
- exemplary markers may include:
- PLT Page load time
- ALT The PLT time without authentication time, this key only appear when "type" is ALT (boot from application cache).
- RDT The render time from web access finish retrieve session data until PLT end marker.
- client raw data includes parameters including but not limited to:
- log file 308 can include the following web-access navigation timing raw data associated with client 302 as:
- Exemplary load timing raw data associated with client 302 as:
- FIGURE 4 is flow diagram depicting operations of an exemplary end-to-end process 400 used as part of providing performance diagnostic analysis and/or issue remediation services according to an embodiment.
- the process 400 at 402 operates to collect performance data using a client executing on an end-user device/system.
- a client such as a browser or other application and scripting code (e.g., JAVASCRIPT code) collects client-centric performance data and/or requests performance diagnostic analysis services from one or more server computers associated with use an online of service or application.
- the process 400 at 402 of one embodiment operates to collect raw performance data that includes navigation timing, page load timing, and/or other parameters indicative of latencies or other performance issues as part of assessing an end- user experience associated with an online service or application.
- the process 400 at 404 operates to provide the raw performance data to a log file of a dedicated server computer.
- the process 400 at 404 includes the use of a browser executing on a user device/system to upload a client IP address and collected performance data or some portion to one or more log files.
- the process 400 operates to transform or map the logged performance data using the client IP address and mapping targets that include geographical location (e.g., country/state), ISP, and/or tenant GUID.
- the process 400 at 406 can be configured to map logged client data to a plurality of mapping tables including a first mapping table that defines IP address to geographic location mappings for the logged client data and a second mapping table that defines IP address to ISP mappings for the logged client data.
- the process 400 operates to upload the transformed data grouped by one or more of tenant GUID, geographic location, and/or ISP to one or more diagnostic service databases.
- the process 400 at 410 operates to perform aggregation operations across the one or more databases to generate latency and/or other performance related aggregations for the online service or application.
- the process 400 at 410 performs aggregation operations to determine one or more of tenant level, geographic location level, and/or ISP level latencies.
- the process 400 at 412 uses one or more rules on the aggregated data to perform a failure zone analysis to identify one or more failure or potential failure zones. For example, the process 400 at 412 can use configured rules to vet whether a user experience is poor, satisfactory, or excellent based in part on trend or baseline comparisons across all countries and/or ISPs.
- the process 400 operates to use the failure zone information as part of taking any corrective or mitigating action.
- the process 400 at 414 can use failure zone analysis information to generate online reports that identify potential network and/or communication architecture modifications as part of reducing latency or other performance related issues. While a certain number and order of operations are described for the exemplary flow of FIGURE 4, it will be appreciated that other numbers, combinations, and/or orders can be used according to desired implementations.
- the process 400 can be used in part to generate an electronic report that allows for viewing of different network metrics for an online email service to identify that users in a first location are spending longer time in CDN compared to rest of the countries in the associated region. A reviewer can then follow-up with a CDN provider in the first location to resolve the issue. Additionally, review of a geographic-ISP report for the first location reveals difference in latencies by ISP enabling ready identification of an increase in latency for one of the larger ISPs that may be contacted to inform and resolve the issue.
- the process 400 can be used to generate an electronic report that includes download times by region to identify users of a particular region having maximum download time resulting in deploying of a new edge server to reduce the impact of user networks.
- An updated report reveals a reduction in latencies for the particular region.
- the process 400 can generate an electronic report that allows a particular tenant to display a trend view and determine that a latency increase occurred in the last few days as well as TCP connecting times increased by 500ms. Based on the report, an affected tenant can be contacted to identify issues with ISP peering with another location.
- Suitable programming means include any means for directing a computer system or device to execute steps of a process or method, including for example, systems comprised of processing units and arithmetic-logic circuits coupled to computer memory, which systems have the capability of storing in computer memory, which computer memory includes electronic circuits configured to store data and program instructions or code.
- An exemplary article of manufacture includes a computer program product useable with any suitable processing system. While a certain number and types of components are described above, it will be appreciated that other numbers and/or types and/or configurations can be included according to various embodiments. Accordingly, component functionality can be further divided and/or combined with other component functionalities according to desired implementations.
- the term computer readable media as used herein can include computer storage media or computer storage. The computer storage of an embodiment stores program code or instructions that operate to perform some function. Computer storage media can include volatile and nonvolatile, removable and nonremovable media implemented in any method or technology for storage of information, such as computer readable instructions, data structures, program modules, etc.
- Computer storage media may include, but is not limited to, RAM, ROM, electrically erasable read-only memory (EEPROM), flash memory or other memory technology, CD-ROM, digital versatile disks (DVD) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium which can be used to store information and which can be accessed by a computing device. Any such computer storage media may be part of a device or system.
- communication media may include wired media such as a wired network or direct- wired connection, and wireless media such as acoustic, RF, infrared, and other wireless media.
- the embodiments and examples described herein are not intended to be limiting and other embodiments are available.
- the components described above can be implemented as part of networked, distributed, and/or other computer-implemented environment.
- the components can communicate via a wired, wireless, and/or a combination of communication networks.
- Network components and/or couplings between components of can include any of a type, number, and/or combination of networks and the corresponding network components which include, but are not limited to, wide area networks (WANs), local area networks (LANs), metropolitan area networks (MANs), proprietary networks, backend networks, cellular networks, etc.
- Client computing devices/systems and servers can be any type and/or combination of processor-based devices or systems. Additionally, server functionality can include many components and include other servers. Components of the computing environments described in the singular tense may include multiple instances of such components. While certain embodiments include software implementations, they are not so limited and encompass hardware, or mixed hardware/software solutions.
- Terms used in the description generally describe a computer-related operational environment that includes hardware, software, firmware and/or other items.
- a component can use processes using a processor, executable, and/or other code.
- Exemplary components include an application, a server running on the application, and/or an electronic communication client coupled to a server for receiving communication items.
- Computer resources can include processor and memory resources such as: digital signal processors, microprocessors, multi-core processors, etc. and memory components such as magnetic, optical, and/or other storage devices, smart memory, flash memory, etc.
- Communication components can be used to communicate computer-readable information as part of transmitting, receiving, and/or rendering electronic communication items using a communication network or networks, such as the Internet for example. Other embodiments and configurations are included.
- FIGURE 5 provides a brief, general description of a suitable computing environment in which embodiments be implemented. While described in the general context of program modules that execute in conjunction with program modules that run on an operating system on various types of computing devices/systems, those skilled in the art will recognize that the invention may also be implemented in combination with other types of computer devices/systems and program modules.
- program modules include routines, programs, components, data structures, and other types of structures that perform particular tasks or implement particular abstract data types.
- program modules may be located in both local and remote memory storage devices.
- computer 2 comprises a general purpose server, desktop, laptop, handheld, or other type of computer capable of executing one or more application programs including an email application or other application that includes email functionality.
- the computer 2 includes at least one central processing unit 8 ("CPU"), a system memory 12, including a random access memory 18 ("RAM”) and a read-only memory (“ROM”) 20, and a system bus 10 that couples the memory to the CPU 8.
- CPU central processing unit
- RAM random access memory
- ROM read-only memory
- the computer 2 further includes a mass storage device 14 for storing an operating system 24, application programs, and other program modules/resources 26.
- the mass storage device 14 is connected to the CPU 8 through a mass storage controller (not shown) connected to the bus 10.
- the mass storage device 14 and its associated computer-readable media provide non- volatile storage for the computer 2.
- computer-readable media can be any available media that can be accessed or utilized by the computer 2.
- the computer 2 may operate in a networked environment using logical connections to remote computers through a network 4, such as a local network, the Internet, etc. for example.
- the computer 2 may connect to the network 4 through a network interface unit 16 connected to the bus 10. It should be appreciated that the network interface unit 16 may also be utilized to connect to other types of networks and remote computing systems.
- the computer 2 may also include an input/output controller 22 for receiving and processing input from a number of other devices, including a keyboard, mouse, etc. (not shown). Similarly, an input/output controller 22 may provide output to a display screen, a printer, or other type of output device.
- a number of program modules and data files may be stored in the mass storage device 14 and RAM 18 of the computer 2, including an operating system 24 suitable for controlling the operation of a networked personal computer, such as the WINDOWS operating systems from MICROSOFT CORPORATION of Redmond, Washington.
- the mass storage device 14 and RAM 18 may also store one or more program modules.
- the mass storage device 14 and the RAM 18 may store application programs, such as word processing, spreadsheet, drawing, e-mail, and other applications and/or program modules, etc.
- FIGURES 6A-6B illustrate a mobile computing device 600, for example, a mobile telephone, a smart phone, a tablet personal computer, a laptop computer, and the like, with which embodiments may be practiced.
- a mobile computing device 600 for example, a mobile telephone, a smart phone, a tablet personal computer, a laptop computer, and the like, with which embodiments may be practiced.
- FIGURE 6A one embodiment of a mobile computing device 600 for implementing the embodiments is illustrated.
- the mobile computing device 600 is a handheld computer having both input elements and output elements.
- the mobile computing device 600 typically includes a display 605 and one or more input buttons 610 that allow the user to enter information into the mobile computing device 600.
- the display 605 of the mobile computing device 600 may also function as an input device (e.g., a touch screen display). If included, an optional side input element 615 allows further user input.
- the side input element 615 may be a rotary switch, a button, or any other type of manual input element.
- mobile computing device 600 may incorporate more or less input elements.
- the display 605 may not be a touch screen in some embodiments.
- the mobile computing device 600 is a portable phone system, such as a cellular phone.
- the mobile computing device 600 may also include an optional keypad 635.
- Optional keypad 635 may be a physical keypad or a "soft" keypad generated on the touch screen display.
- the output elements include the display 605 for showing a graphical user interface (GUI), a visual indicator 620 (e.g., a light emitting diode), and/or an audio transducer 625 (e.g., a speaker).
- GUI graphical user interface
- a visual indicator 620 e.g., a light emitting diode
- an audio transducer 625 e.g., a speaker
- the mobile computing device 600 incorporates a vibration transducer for providing the user with tactile feedback.
- the mobile computing device 600 incorporates input and/or output ports, such as an audio input (e.g., a microphone jack), an audio output (e.g., a headphone jack), and a video output (e.g., a HDMI port) for sending signals to or receiving signals from an external device.
- an audio input e.g., a microphone jack
- an audio output e.g., a headphone jack
- a video output e.g., a HDMI port
- FIGURE 6B is a block diagram illustrating the architecture of one embodiment of a mobile computing device. That is, the mobile computing device 600 can incorporate a system (i.e., an architecture) 602 to implement some embodiments.
- the system 602 is implemented as a "smart phone" capable of running one or more applications (e.g., browser, e-mail, calendaring, contact managers, messaging clients, games, and media clients/players).
- the system 602 is integrated as a computing device, such as an integrated personal digital assistant (PDA) and wireless phone.
- PDA personal digital assistant
- One or more application programs 666 may be loaded into the memory 662 and run on or in association with the operating system 664.
- Examples of the application programs include phone dialer programs, e-mail programs, personal information management (PIM) programs, word processing programs, spreadsheet programs, Internet browser programs, messaging programs, and so forth.
- the system 602 also includes a non-volatile storage area 668 within the memory 662. The non-volatile storage area 668 may be used to store persistent information that should not be lost if the system 602 is powered down.
- the application programs 666 may use and store information in the non-volatile storage area 668, such as e-mail or other messages used by an e-mail application, and the like.
- a synchronization application (not shown) also resides on the system 602 and is programmed to interact with a corresponding synchronization application resident on a host computer to keep the information stored in the non-volatile storage area 668 synchronized with corresponding information stored at the host computer.
- other applications may be loaded into the memory 662 and run on the mobile computing device 600.
- the system 602 has a power supply 670, which may be implemented as one or more batteries.
- the power supply 670 might further include an external power source, such as an AC adapter or a powered docking cradle that supplements or recharges the batteries.
- the system 602 may also include a radio 672 that performs the function of transmitting and receiving radio frequency communications.
- the radio 672 facilitates wireless connectivity between the system 602 and the "outside world," via a communications carrier or service provider. Transmissions to and from the radio 672 are conducted under control of the operating system 664. In other words, communications received by the radio 672 may be disseminated to the application programs 666 via the operating system 664, and vice versa.
- the visual indicator 620 may be used to provide visual notifications and/or an audio interface 674 may be used for producing audible notifications via the audio transducer 625.
- the visual indicator 620 is a light emitting diode (LED) and the audio transducer 625 is a speaker.
- LED light emitting diode
- the LED may be programmed to remain on indefinitely until the user takes action to indicate the powered-on status of the device.
- the audio interface 674 is used to provide audible signals to and receive audible signals from the user.
- the audio interface 674 may also be coupled to a microphone to receive audible input, such as to facilitate a telephone conversation.
- the microphone may also serve as an audio sensor to facilitate control of notifications, as will be described below.
- the system 602 may further include a video interface 676 that enables an operation of an on-board camera 630 to record still images, video stream, and the like.
- a mobile computing device 600 implementing the system 602 may have additional features or functionality.
- the mobile computing device 600 may also include additional data storage devices (removable and/or non-removable) such as, magnetic disks, optical disks, or tape. Such additional storage is illustrated in FIGURE 6B by the non-volatile storage area 668.
- additional data storage devices removable and/or non-removable
- FIGURE 6B Such additional storage is illustrated in FIGURE 6B by the non-volatile storage area 668.
- Data/information generated or captured by the mobile computing device 600 and stored via the system 602 may be stored locally on the mobile computing device 600, as described above, or the data may be stored on any number of storage media that may be accessed by the device via the radio 672 or via a wired connection between the mobile computing device 600 and a separate computing device associated with the mobile computing device 600, for example, a server computer in a distributed computing network, such as the Internet.
- a server computer in a distributed computing network such as the Internet.
- data/information may be accessed via the mobile computing device 600 via the radio 672 or via a distributed computing network.
- data/information may be readily transferred between computing devices for storage and use according to well-known data/information transfer and storage means, including electronic mail and collaborative data/information sharing systems.
- FIGURE 7 illustrates one embodiment of a system architecture for implementing latency identification and remediation features.
- Data processing information may be stored in different communication channels or storage types. For example, various information may be stored/accessed using a directory service 722, a web portal 724, a mailbox service 726, an instant messaging store 728, and/or a social networking site 730.
- a server 720 may provide additional latency analysis and other features. As one example, the server 720 may provide rules that are used to distribute outbound email using a number of datacenter partitions over network 715, such as the Internet or other network(s) for example.
- the client computing device may be implemented as a general computing device 702 and embodied in a personal computer, a tablet computing device 704, and/or a mobile computing device 706 (e.g., a smart phone). Any of these clients may use content from the store 716.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Computer Networks & Wireless Communication (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Computer Hardware Design (AREA)
- Environmental & Geological Engineering (AREA)
- Debugging And Monitoring (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Computer And Data Communications (AREA)
Abstract
Description
Claims
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2016533584A JP2017500791A (en) | 2013-11-22 | 2014-11-20 | Performance monitoring that provides real-time or near real-time improvement feedback |
RU2016119573A RU2016119573A (en) | 2013-11-22 | 2014-11-20 | PERFORMANCE MONITORING TO REALIZE REAL OR ALMOST REAL TIME CORRECTION |
EP14810075.3A EP3072050A2 (en) | 2013-11-22 | 2014-11-20 | Performance monitoring to provide real or near real time remediation feedback |
CN201480063665.5A CN105765907A (en) | 2013-11-22 | 2014-11-20 | Performance monitoring to provide real or near real time remediation feedback |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/087,413 US20150149609A1 (en) | 2013-11-22 | 2013-11-22 | Performance monitoring to provide real or near real time remediation feedback |
US14/087,413 | 2013-11-22 |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2015077385A2 true WO2015077385A2 (en) | 2015-05-28 |
WO2015077385A3 WO2015077385A3 (en) | 2015-08-20 |
Family
ID=52021441
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2014/066480 WO2015077385A2 (en) | 2013-11-22 | 2014-11-20 | Performance monitoring to provide real or near real time remediation feedback |
Country Status (6)
Country | Link |
---|---|
US (2) | US20150149609A1 (en) |
EP (1) | EP3072050A2 (en) |
JP (1) | JP2017500791A (en) |
CN (1) | CN105765907A (en) |
RU (1) | RU2016119573A (en) |
WO (1) | WO2015077385A2 (en) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN109906592A (en) * | 2016-11-07 | 2019-06-18 | 华为技术有限公司 | Monitor the system and method for chip property |
EP3436951A4 (en) * | 2016-03-29 | 2019-11-20 | Anritsu Company | Systems and methods for measuring effective customer impact of network problems in real-time using streaming analytics |
EP4002800A3 (en) * | 2020-11-17 | 2022-08-03 | Citrix Systems Inc. | Systems and methods for detection of degradation of a virtual desktop environment |
US11467911B2 (en) | 2020-11-17 | 2022-10-11 | Citrix Systems, Inc. | Systems and methods for detection of degradation of a virtual desktop environment |
EP3864516B1 (en) * | 2018-11-19 | 2022-12-21 | Microsoft Technology Licensing, LLC | Veto-based model for measuring product health |
Families Citing this family (33)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9800567B2 (en) * | 2014-03-31 | 2017-10-24 | Sap Se | Authentication of network nodes |
US10003492B2 (en) * | 2015-02-24 | 2018-06-19 | CENX, Inc. | Systems and methods for managing data related to network elements from multiple sources |
US10567246B2 (en) | 2015-12-15 | 2020-02-18 | At&T Intellectual Property I, L.P. | Processing performance data of a content delivery network |
US10355872B2 (en) * | 2016-02-05 | 2019-07-16 | Prysm, Inc | Techniques for a collaboration server network connection indicator |
US20170346909A1 (en) * | 2016-05-31 | 2017-11-30 | Linkedin Corporation | Client-side bottleneck analysis using real user monitoring data |
CN106656666B (en) * | 2016-12-13 | 2020-05-22 | 中国联合网络通信集团有限公司 | Method and device for acquiring first screen time of webpage |
US10666515B2 (en) * | 2017-01-17 | 2020-05-26 | International Business Machines Corporation | Control of activities executed by endpoints based on conditions involving aggregated parameters |
US10680933B2 (en) | 2017-02-02 | 2020-06-09 | Microsoft Technology Licensing, Llc | Electronic mail system routing control |
US10581954B2 (en) * | 2017-03-29 | 2020-03-03 | Palantir Technologies Inc. | Metric collection and aggregation for distributed software services |
US10482000B2 (en) | 2017-04-24 | 2019-11-19 | Microsoft Technology Licensing, Llc | Machine learned decision guidance for alerts originating from monitoring systems |
CN107122448A (en) * | 2017-04-25 | 2017-09-01 | 广州市诚毅科技软件开发有限公司 | A kind of intelligent display method and device of the estimated response time of front end page request |
US10951462B1 (en) * | 2017-04-27 | 2021-03-16 | 8X8, Inc. | Fault isolation in data communications centers |
US11645131B2 (en) * | 2017-06-16 | 2023-05-09 | Cisco Technology, Inc. | Distributed fault code aggregation across application centric dimensions |
US10476946B2 (en) * | 2017-07-27 | 2019-11-12 | Citrix Systems, Inc. | Heuristics for selecting nearest zone based on ICA RTT and network latency |
US10698756B1 (en) | 2017-12-15 | 2020-06-30 | Palantir Technologies Inc. | Linking related events for various devices and services in computer log files on a centralized server |
US10824497B2 (en) * | 2018-08-29 | 2020-11-03 | Oracle International Corporation | Enhanced identification of computer performance anomalies based on computer performance logs |
CN111475429B (en) * | 2019-01-24 | 2023-08-29 | 爱思开海力士有限公司 | memory access method |
US11068333B2 (en) | 2019-06-24 | 2021-07-20 | Bank Of America Corporation | Defect analysis and remediation tool |
CN110493075B (en) * | 2019-08-01 | 2021-06-25 | 京信通信系统(中国)有限公司 | Method, device and system for monitoring online duration of equipment |
US11558271B2 (en) * | 2019-09-04 | 2023-01-17 | Cisco Technology, Inc. | System and method of comparing time periods before and after a network temporal event |
US10924334B1 (en) * | 2019-09-12 | 2021-02-16 | Salesforce.Com, Inc. | Monitoring distributed systems with auto-remediation |
US11799741B2 (en) * | 2019-10-29 | 2023-10-24 | Fannie Mae | Systems and methods for enterprise information technology (IT) monitoring |
US11012326B1 (en) * | 2019-12-17 | 2021-05-18 | CloudFit Software, LLC | Monitoring user experience using data blocks for secure data access |
US10877867B1 (en) | 2019-12-17 | 2020-12-29 | CloudFit Software, LLC | Monitoring user experience for cloud-based services |
US11379442B2 (en) | 2020-01-07 | 2022-07-05 | Bank Of America Corporation | Self-learning database issue remediation tool |
ZA202100191B (en) | 2020-01-20 | 2023-12-20 | EXFO Solutions SAS | Method and device for estimating a number of distinct subscribers of a telecommunication network impacted by network issues |
JP7285798B2 (en) * | 2020-03-09 | 2023-06-02 | 株式会社日立製作所 | Performance analysis device, performance analysis method, and performance analysis program |
US11546408B2 (en) | 2020-11-02 | 2023-01-03 | Microsoft Technology Licensing, Llc | Client-side measurement of computer network conditions |
US20220357968A1 (en) * | 2021-05-07 | 2022-11-10 | Citrix Systems, Inc. | Heuristic Policy Recommendations in a Virtual Environment |
US12086160B2 (en) | 2021-09-23 | 2024-09-10 | Oracle International Corporation | Analyzing performance of resource systems that process requests for particular datasets |
US12038816B2 (en) * | 2021-09-24 | 2024-07-16 | Salesforce, Inc. | Determining insights related to performance bottlenecks in a multi-tenant database system preliminary class |
US12088347B2 (en) * | 2022-04-22 | 2024-09-10 | Bank Of America Corporation | Intelligent monitoring and repair of network services using log feeds provided over Li-Fi networks |
US20230385173A1 (en) * | 2022-05-27 | 2023-11-30 | Microsoft Technology Licensing, Llc | Real-time report generation |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6738933B2 (en) * | 2001-05-09 | 2004-05-18 | Mercury Interactive Corporation | Root cause analysis of server system performance degradations |
JP2007158623A (en) * | 2005-12-02 | 2007-06-21 | Matsushita Electric Ind Co Ltd | Method of monitoring quality of video distribution service and terminal |
CN101548268B (en) * | 2006-10-05 | 2014-05-21 | 瓦拉泰克有限公司 | Advanced contention detection |
CN101192227B (en) * | 2006-11-30 | 2011-05-25 | 阿里巴巴集团控股有限公司 | Log file analytical method and system based on distributed type computing network |
DE602006015827D1 (en) * | 2006-12-08 | 2010-09-09 | Ubs Ag | Method and apparatus for detecting the IP address of a computer and related location information |
JP5158189B2 (en) * | 2008-03-07 | 2013-03-06 | 日本電気株式会社 | Mail receiving device |
US20090245114A1 (en) * | 2008-04-01 | 2009-10-01 | Jayanth Vijayaraghavan | Methods for collecting and analyzing network performance data |
KR101940815B1 (en) * | 2009-01-28 | 2019-01-21 | 헤드워터 리서치 엘엘씨 | Quality of service for device assisted services |
US9203913B1 (en) * | 2009-07-20 | 2015-12-01 | Conviva Inc. | Monitoring the performance of a content player |
US9021362B2 (en) * | 2010-07-19 | 2015-04-28 | Soasta, Inc. | Real-time analytics of web performance using actual user measurements |
CN102291594B (en) * | 2011-08-25 | 2015-05-20 | 中国电信股份有限公司上海信息网络部 | IP network video quality detecting and evaluating system and method |
US8452871B2 (en) * | 2011-08-27 | 2013-05-28 | At&T Intellectual Property I, L.P. | Passive and comprehensive hierarchical anomaly detection system and method |
-
2013
- 2013-11-22 US US14/087,413 patent/US20150149609A1/en not_active Abandoned
-
2014
- 2014-11-20 JP JP2016533584A patent/JP2017500791A/en active Pending
- 2014-11-20 WO PCT/US2014/066480 patent/WO2015077385A2/en active Application Filing
- 2014-11-20 EP EP14810075.3A patent/EP3072050A2/en not_active Withdrawn
- 2014-11-20 RU RU2016119573A patent/RU2016119573A/en not_active Application Discontinuation
- 2014-11-20 CN CN201480063665.5A patent/CN105765907A/en active Pending
-
2017
- 2017-09-29 US US15/720,983 patent/US20180027088A1/en not_active Abandoned
Non-Patent Citations (1)
Title |
---|
None |
Cited By (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP3436951A4 (en) * | 2016-03-29 | 2019-11-20 | Anritsu Company | Systems and methods for measuring effective customer impact of network problems in real-time using streaming analytics |
US10686681B2 (en) | 2016-03-29 | 2020-06-16 | Anritsu Company | Systems and methods for measuring effective customer impact of network problems in real-time using streaming analytics |
CN109906592A (en) * | 2016-11-07 | 2019-06-18 | 华为技术有限公司 | Monitor the system and method for chip property |
EP3526933A4 (en) * | 2016-11-07 | 2019-08-21 | Huawei Technologies Co., Ltd. | System and methods for monitoring performance of slices |
CN109906592B (en) * | 2016-11-07 | 2020-10-09 | 华为技术有限公司 | System and method for monitoring slicing performance |
US10827366B2 (en) | 2016-11-07 | 2020-11-03 | Huawei Technologies Co., Ltd. | System and methods for monitoring performance of slices |
EP3864516B1 (en) * | 2018-11-19 | 2022-12-21 | Microsoft Technology Licensing, LLC | Veto-based model for measuring product health |
EP4002800A3 (en) * | 2020-11-17 | 2022-08-03 | Citrix Systems Inc. | Systems and methods for detection of degradation of a virtual desktop environment |
US11467911B2 (en) | 2020-11-17 | 2022-10-11 | Citrix Systems, Inc. | Systems and methods for detection of degradation of a virtual desktop environment |
US12001287B2 (en) | 2020-11-17 | 2024-06-04 | Citrix Systems, Inc. | Systems and methods for detection of degradation of a virtual desktop environment |
Also Published As
Publication number | Publication date |
---|---|
US20150149609A1 (en) | 2015-05-28 |
EP3072050A2 (en) | 2016-09-28 |
JP2017500791A (en) | 2017-01-05 |
US20180027088A1 (en) | 2018-01-25 |
RU2016119573A (en) | 2017-11-23 |
RU2016119573A3 (en) | 2018-08-10 |
WO2015077385A3 (en) | 2015-08-20 |
CN105765907A (en) | 2016-07-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US20180027088A1 (en) | Performance monitoring to provide real or near real time remediation feedback | |
US11042474B2 (en) | Scheduled tests for endpoint agents | |
US10644962B2 (en) | Continuous monitoring for performance evaluation of service interfaces | |
US10951489B2 (en) | SLA compliance determination with real user monitoring | |
US20160070691A1 (en) | Method and system for auto-populating electronic forms | |
EP3864516B1 (en) | Veto-based model for measuring product health | |
US20150067147A1 (en) | Group server performance correction via actions to server subset | |
JP7105356B2 (en) | Mapping Entity to Account | |
US20100198649A1 (en) | Role tailored dashboards and scorecards in a portal solution that integrates retrieved metrics across an enterprise | |
US10275338B2 (en) | Automated system for fixing and debugging software deployed to customers | |
US11522765B2 (en) | Auto discovery of network proxies | |
US20160321906A1 (en) | Alert management within a network based virtual collaborative space | |
CN114208125A (en) | Network problem node identification using traceroute aggregation | |
US8819704B1 (en) | Personalized availability characterization of online application services | |
US12047839B2 (en) | Out of box user performance journey monitoring | |
WO2015148238A1 (en) | End user performance analysis | |
US20140344418A1 (en) | Dynamic configuration analysis | |
WO2021021267A1 (en) | Scheduled tests for endpoint agents | |
US20160226719A1 (en) | Network based virtual collaborative problem solving space | |
US10425452B2 (en) | Identifying changes in multiple resources related to a problem | |
US20170223136A1 (en) | Any Web Page Reporting and Capture | |
US8689058B2 (en) | Centralized service outage communication | |
US11328369B2 (en) | Network liquidity to engagement mapping | |
US10505894B2 (en) | Active and passive method to perform IP to name resolution in organizational environments | |
US10838950B2 (en) | Dynamic review cadence for intellectual capital |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 14810075 Country of ref document: EP Kind code of ref document: A2 |
|
DPE1 | Request for preliminary examination filed after expiration of 19th month from priority date (pct application filed from 20040101) | ||
REEP | Request for entry into the european phase |
Ref document number: 2014810075 Country of ref document: EP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2014810075 Country of ref document: EP |
|
ENP | Entry into the national phase |
Ref document number: 2016119573 Country of ref document: RU Kind code of ref document: A |
|
ENP | Entry into the national phase |
Ref document number: 2016533584 Country of ref document: JP Kind code of ref document: A |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
REG | Reference to national code |
Ref country code: BR Ref legal event code: B01A Ref document number: 112016011022 Country of ref document: BR |
|
ENP | Entry into the national phase |
Ref document number: 112016011022 Country of ref document: BR Kind code of ref document: A2 Effective date: 20160516 |