EP3510747B1 - Automated performance debugging of production applications - Google Patents

Automated performance debugging of production applications Download PDF

Info

Publication number
EP3510747B1
EP3510747B1 EP16915480.4A EP16915480A EP3510747B1 EP 3510747 B1 EP3510747 B1 EP 3510747B1 EP 16915480 A EP16915480 A EP 16915480A EP 3510747 B1 EP3510747 B1 EP 3510747B1
Authority
EP
European Patent Office
Prior art keywords
thread
call
processor
data
critical
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP16915480.4A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP3510747A1 (en
EP3510747A4 (en
Inventor
Yawei Wang
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Microsoft Technology Licensing LLC
Original Assignee
Microsoft Technology Licensing LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Microsoft Technology Licensing LLC filed Critical Microsoft Technology Licensing LLC
Publication of EP3510747A1 publication Critical patent/EP3510747A1/en
Publication of EP3510747A4 publication Critical patent/EP3510747A4/en
Application granted granted Critical
Publication of EP3510747B1 publication Critical patent/EP3510747B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/02—Capturing of monitoring data
    • H04L43/022—Capturing of monitoring data by sampling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
    • G06F11/0715—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a system implementing multitasking
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
    • G06F11/3017—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is implementing multitasking
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
    • G06F11/3419—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment by assessing time
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
    • G06F11/3428—Benchmarking
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3447—Performance evaluation by modeling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466—Performance evaluation by tracing or monitoring
    • G06F11/3476—Data logging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3604—Software analysis for verifying properties of programs
    • G06F11/3612—Software analysis for verifying properties of programs by runtime analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/362—Software debugging
    • G06F11/3636—Software debugging by tracing the execution of the program
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/362—Software debugging
    • G06F11/3644—Software debugging by instrumenting at runtime
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3692—Test management for test results analysis
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/04—Processing captured monitoring data, e.g. for logfile generation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00—Arrangements for monitoring or testing data switching networks
    • H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
    • H04L43/0805—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability
    • H04L43/0817—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters by checking availability by checking functioning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0766—Error or fault reporting or storing
    • G06F11/0778—Dumping, i.e. gathering error/state information after a fault for later diagnosis

Definitions

  • Cloud computing is an internet-based on-demand type of computing in which a pool of configurable processing resources including networks, servers, storage, applications, services and so on are shared among many customers. Cloud computing and storage solutions provide users and enterprises with a multitude of capabilities to store and process data in third-party data centers.
  • an interceptor software is provided and code injection in e.g. a framework module of a thread is also considered. Thread states and thread performance information can be gathered and analyzed in order to determine which modules are slowing the operation of the computer down.
  • US 2011/0016357 A1 discloses a technique for call-stacks representation for easier analysis of thread dumps. Execution data for a number of process threads are accessed by a parser in a thread dump wherein execution data includes a number of call-stacks. Process threads have corresponding call-stacks comprising information about active program subroutines of the thread.
  • Threads are grouped in a number of sets that include threads with one or more actions in common.
  • the sets correspond to a same action or subroutine in the call-stacks of the grouped threads.
  • a tree representation of the execution data is generated based on the active actions in the call-stacks.
  • automated performance debugging of customer source code or non-framework code in a production environment can be performed by collecting performance data using a lightweight sampling technique.
  • Performance data is the data that shows how the production application is performing. For example, it can refer to but is not limited to referring to, information about CPU utilization, threads and their call stack traces in a performance debugging context.
  • Framework code is the executable code that serves as an abstraction layer for an operating system to provide generic functionality. Often, framework code can be selectively changed by additional user-written code (code that is not framework code and not operating system code), thus providing application-specific functionality.
  • Framework code generally has libraries, tool sets, and application programming interface (API) support to facilitate development of software applications.
  • Non-framework code e.g., user-written code
  • Non-framework code can comprise user-implemented extensions to the framework code.
  • Performance data can be collected. Performance data can be serialized into storage. The data can be analyzed off-line. The data can be used to identify critical threads. The data can be fed into a decision-tree based model to identify critical threads associated with processing slowdowns. A call graph prefix binary tree can be built to analyze call stack patterns of critical threads to highlight hot spots. A view of wall clock time elapsed on the hot spots can be provided. Through observation of analysis results including the call stack pattern view, the hotspot (code that is causing a performance slowdown) can be identified. In addition to debugging, the information can be used for performance tuning. Performance tuning can improve application performance.
  • the call frame of an identified hotspot can include the module name and method name as well as elapsed clock time of the code associated with the slowdown.
  • static code review can be easily scoped to just reviewing the piece of method code indicated.
  • the subject matter disclosed herein can be leveraged to tune production application performance.
  • a hybrid cloud system may begin with requests form thin or thick clients that flow through a cloud service front-end to an on-premises web service to a third party line of business application built on diverse technologies which have data access components interacting with a database layer. Additional services (such as social identity providers, request routing, authentication, authorization and so on) are likely to come into play. Should a performance anomaly (e.g., response slowdown) occur, it is not at all clear which layer is causing the anomaly.
  • a performance anomaly e.g., response slowdown
  • a hang mode dump is a full user-mode dump that includes the entire memory space of a process, the program's executable image itself, the handle table, and other information that will be useful to the debugger.
  • a user-mode debugger is a debugger that is used to debug a process running executable code, which is separated from other processes by the OS (operating system). The debugger and the debuggee (the executable being debugged) typically run on the same user mode system.
  • An application is said to be in a low CPU hang state when the application becomes unresponsive to any request while the process hosting the application is consuming little or no CPU (central processing unit) time.
  • High CPU hang is a type of unresponsiveness in which the process hosting the executable is consuming a great deal of system processor(s) CPU time, (usually near 100%) and the process is slow performing, freezing or even shutting down.
  • the subject matter described herein deals with both low and high CPU hang.
  • overall performance anomalies for production applications can be identified.
  • the dynamic behavior over time of hosting processes on the same or different computers can be analyzed.
  • Call sites that may be performance bottlenecks or may be causing hangs can be identified.
  • a lightweight sampling strategy can collect predicates representing key performance data in production scenarios.
  • Performance predicates can provide information about the subject (e.g., what the performance issue is, what caused the performance issue, etc.).
  • the data can be fed into a model (e.g., such as but not limited to a model based on a decision tree) to identify critical threads running the problematic call sites.
  • the results along with the key performance data can be used to build a call graph prefix binary tree for analyzing call stack patterns. Data collection, analysis and visualizations of results can be performed.
  • automated performance debugging and tuning can be applied to production applications.
  • One or more of the following features can be included: guarantee limited runtime overhead (as used herein, limited runtime overhead is overhead comprising less than 5% of total overhead for real-life large-scale applications including enterprise server applications).
  • On-the-fly analysis and offline reporting can be performed. Someone who has only basic application and system knowledge can understand what the process is doing during the monitoring period.
  • automated performance debugging can be applied to all processes in a large-scale distributed system simultaneously to identify the causes of overall performance anomaly.
  • automated performance debugging can be applied to applications that run with black boxes. A black box is a component about which users have very little knowledge.
  • T ⁇ n ⁇ ⁇ CPUi n , CallSitei n > n ⁇ 0 ... N
  • CPUi n is the user time of thread i in sample n
  • CallSitei n are call stack patterns of thread i in sample n.
  • Tl n ⁇ and P t ⁇ are the Key Performance Data.
  • a lightweight sampling strategy as described herein can be used against a target process to generate Key Performance Data. Choosing an appropriate sampling strategy can be pivotal in achieving the low overhead that production environments typically demand. Traditionally, sampling occurs by invasively attaching a debugger to the process and capturing a hang mode dump or instrumenting the whole application beforehand to emit runtime traces.
  • the lightweight sampling strategy described herein can inject a thread into a target process to collect predicates.
  • a remote process handle for the target process can be created. The handle can be provided to one or more processes to create an argument buffer and invoke the appropriate code. When the code is invoked, a new thread can be created to collect predicates.
  • the sampling duration and interval (the sampling period) can be specified. A snapshot of all the running threads can be taken.
  • FIG. 2c illustrates an example 250 of a call graph and its corresponding call sites.
  • an application has a call stack running in thread T where the call stack starts at a process MAIN() 251 and finishes at RET() 253.
  • the call site N() 252 is the hypothetical performance bottleneck taking 5 seconds to return while the whole execution from MAIN() 251 to RET() 253 takes 6 seconds.
  • O 1 254 can be the collection of call paths that do not include N() 252.
  • dashed lines in the call graph demonstrate there are multiple calls or hoops between the caller and callee.
  • dashed line 255 indicates that indicates that Main() 251 calls F() 259 more than once.
  • the solid lines (such as solid line 257 and solid line 258) refer to direct caller/callee relationships.
  • line 257 indicates that caller B() 260 calls callee N() 252 once.
  • Each rectangular box (e.g., box 251a) refers to one call site and the rounded rectangular box 254a refers to the collection of branches in the graph that do not call call site N() 252.
  • FIG. 2d illustrates an example of a call stack 270 (e.g., the call stack for CallSitei n n ⁇ ⁇ 0... 11 ⁇ ) .
  • the call stack patterns of interest start (e.g., the calls from A() 262 to B() 260 to N() 252 collectively 276a) to appear on the call stack.
  • Each frame in the call stack column reflects a call stack trace captured in a particular sampling of a particular thread.
  • the call stack trace can include multiple frames.
  • the bottom frame, frame 273 is the first call site in the call flow while the top frame 274 is the last.
  • System and Framework Call Sites 275 refers to a call to framework code and the ellipses such as ellipsis 276 indicates that there are multiple hoops between two call sites. Hence in frame 272a lines 277, 278 and 279 indicate that A() 262 calls C() 261 and C() 261 calls N() 252.
  • a decision tree or classification tree can be created from an existing set of training cases.
  • the decision tree can be used to classify new cases.
  • Various decision tree construction algorithms can be used including but not limited to the algorithms ID3 and C4.5 by Quinlan. While both these algorithms use formulas based on entropy and information gain, other formulas are contemplated.
  • the decision tree based model can be used to classify a thread as Normal or Critical.
  • a normal thread is unlikely to have call stack information that helps with debugging or problem isolation (i.e., system idle threads).
  • a system idle thread is a thread that is idle (doing nothing).
  • an idle thread can be a thread on which no non-framework executable is running.
  • Such a thread is classified as normal.
  • a critical thread is likely to have information that helps to identify a thread that is causing a performance anomaly (e.g., busy GC (garbage collection thread), busy worker thread, consistently busy threads with the same series of customer call sites).
  • busy GC garbage collection thread
  • busy worker thread consistently busy threads with the same series of customer call sites.
  • a ranking of busy threads can be computed.
  • Busy threads can be ranked by CPU usage.
  • Critical threads are classified by a model based on a decision tree.
  • the decision tree model can be constructed based on busy threads ranking, call stack length, the presence of non-framework call sites and the presence of non-framework call sites in consecutive samples.
  • a critical thread can be busy because the thread has a heavy workload because of a problematic call site.
  • a critical thread can be a thread that is not busy because it is waiting for a resource.
  • a critical thread can be a thread that is not busy because it is blocking a resource.
  • the call stack of a critical thread usually starts and ends with the OS or framework and has a non-framework section in the middle, the non-framework section including the possible problematic call sites (e.g., the call sites referenced by reference numerals 276a, 277, 278 and 279).
  • Busy Thread Ranking along with three other attributes, the length of the call stack, "Call Stack Length", the presence of non-framework call sites in the call stack, "Has non-framework call sites” and the presence of non-framework call sites in consecutive samples, "Has non-framework call sites in consecutive samples" can be used in a training set for building the decision tree based model.
  • Busy Thread Ranking reflects the CPU utilization of each busy thread in any time interval(s).
  • a call graph prefix binary tree can be created that clusters over-time call stacks to highlight hot spots and hot paths.
  • the call graph prefix binary tree can provide a rough view of wall clock time elapsed on the hot spots and/or hot paths.
  • FIG. 2e illustrates an example of building a call graph prefix binary tree 291 for call stack 270 of FIG. 2d .
  • the notation + X (Y) CallSite Z means the CallSite has been hit X times in pattern Z with maximum consecutive hits Y.
  • +3(5)N1 means that callsite N has been sampled 3 times and appeared in 5 consecutive samples in call stack pattern 1.
  • There are 3 call stack patterns in FIG. 2d pattern 1 (ADCBN) 291b, pattern 2 (FN) 291c and pattern 3 (EN) 291d.
  • FIG. 1a illustrates a system 100 comprising an example of an automated performance debugging system for production applications in accordance with aspects of the subject matter described herein. All or portions of system 100 may reside on one or more computers or computing devices such as the computers described below with respect to FIG. 3 . System 100 or portions thereof may be provided as a stand-alone system or as a plug-in or add-in.
  • System 100 or portions thereof may include information obtained from a service (e.g., in the cloud) or may operate in a cloud computing environment.
  • a cloud computing environment can be an environment in which computing services are not owned but are provided on demand.
  • information may reside on multiple devices in a networked cloud and/or data can be stored on multiple devices within the cloud.
  • System 100 can be an on-premises automated performance debugging system.
  • System 100 can include one or more computing devices such as, for example, computing device 102.
  • Contemplated computing devices include but are not limited to desktop computers, tablet computers, laptop computers, notebook computers, personal digital assistants, smart phones, cellular telephones, mobile telephones, sensors, server computers and so on.
  • a computing device such as computing device 102 can include one or more processors such as processor 142, etc., and a memory such as memory 144 that communicates with the one or more processors.
  • System 100 may include one or more program modules that when loaded into the memory 144 and accessed by the one or more processors such as processor 142, etc., cause the processor to perform the action or actions attributed to the one or more program modules.
  • the processor(s) may be configured to perform the action or actions attributed to the one or more program modules.
  • System 100 may include any one of or any combination of any number of the following: a worker thread pool such as worker thread pool 109 from which a worker thread such as worker thread 111 can be assigned to an application process such as application process 105, a lightweight sampling program module or modules such as lightweight sampling module 107, a data collection program module or modules such as data collection module 113, an injected thread such as injected thread 115 injected into application process 105, a storage device such as storage device or storage 117, a computing process such as computing process 119, a data analysis program module such as data analysis module 121 and/or analysis results such as analysis results 123 generated by data analysis process 1 22.
  • a worker thread pool such as worker thread pool 109 from which a worker thread such as worker thread 111 can be assigned to an application process such as application process 105
  • a lightweight sampling program module or modules such as lightweight sampling module 107
  • a data collection program module or modules such as data collection module 113
  • an injected thread such as injected thread 115 injected into application process
  • a worker thread 111 from a worker thread pool 109 can be injected into a application process 105 executing code of a data collection module 113.
  • a lightweight sampling module 107 can periodically collect data (e.g., key performance data 118) which can be serialized and stored in storage 117. The lightweight sampling module 107 can initiate a thread injection into the application process 105.
  • a data analysis process 122 of a computing process 119 can analyze the data and provide analysis results 123.
  • FIG. 1b illustrates a system 101 comprising another example of an automated performance debugging system for production applications in accordance with aspects of the subject matter described herein. All or portions of system 101 may reside on one or more computers or computing devices such as the computers described below with respect to FIG. 3 . System 101 or portions thereof may be provided as a stand-alone system or as a plug-in or add-in.
  • System 101 or portions thereof may include information obtained from a service (e.g., in the cloud) or may operate in a cloud computing environment.
  • a cloud computing environment can be an environment in which computing services are not owned but are provided on demand.
  • information may reside on multiple devices in a networked cloud and/or data can be stored on multiple devices within the cloud.
  • Contemplated computing devices include but are not limited to desktop computers, tablet computers, laptop computers, notebook computers, personal digital assistants, smart phones, cellular telephones, mobile telephones, sensors, server computers and so on.
  • a computing device such as computing device can include one or more processors and a memory that communicates with the one or more processors.
  • System 101 can include one or more computing devices such as, for example, computing devices such as one or more application servers 125, one or more web servers 127, one or more database servers 129.
  • the computing devices may comprise a data center.
  • a data collection agent 135 may be downloaded from the cloud 133 via a network such the internet 139.
  • Key performance data such as key performance data 137 can be uploaded to the cloud 133, analyzed by a data analysis component 138 and analysis results 143 can be rendered on a client device 141.
  • a data collection agent 135 can be resident on an on-premises computer.
  • Key performance data 137 may be analyzed by a resident data analysis component and analysis results 143 can be generated on an on-premises device. Any combination of cloud and on-premises components is contemplated.
  • FIG. 1c illustrates a system 103 comprising an example of a cloud automated performance debugging system for production applications in accordance with aspects of the subject matter described herein. All or portions of system 103 may reside on one or more computers or computing devices such as the computers described below with respect to FIG. 3 . System 103 or portions thereof may be provided as a stand-alone system or as a plug-in or add-in.
  • System 103 or portions thereof may include information obtained from a service (e.g., in the cloud) or may operate in a cloud computing environment.
  • a cloud computing environment can be an environment in which computing services are not owned but are provided on demand.
  • information may reside on multiple devices in a networked cloud and/or data can be stored on multiple devices within the cloud.
  • Contemplated computing devices include but are not limited to desktop computers, tablet computers, laptop computers, notebook computers, personal digital assistants, smart phones, cellular telephones, mobile telephones, sensors, server computers and so on.
  • a computing device such as computing device can include one or more processors and a memory that communicates with the one or more processors.
  • System 103 can include one or more cloud computing devices such as, for example, computing devices in a data center 160.
  • a cloud data collection agent 150 can collect data from a production environment 156 in the cloud 158 and can serialize the data (e.g., key performance data) to cloud storage 154.
  • a cloud automatic diagnostic service 152 can provide analysis results 153 to a client 162.
  • FIG. 2a illustrates an example of a method 200 for performance debugging of production applications in accordance with aspects of the subject matter disclosed herein.
  • the method described in FIG. 2a can be practiced by a system such as but not limited to the one described with respect to FIG. 1a and/or FIG. 1b and/or FIG. 1c .
  • method 200 describes a series of operations that are performed in a sequence, it is to be understood that method 200 is not limited by the order of the sequence depicted. For instance, some operations may occur in a different order than that described. In addition, one operation may occur concurrently with another operation. In some instances, not all operations described are performed. In some instances, not all operations performed are illustrated.
  • data can be collected.
  • the data can be key performance data.
  • the data can be collected by a data collection agent.
  • the data can be collected by a thread injected into an application process.
  • the data can be collected using a lightweight sampling technique described more fully above.
  • data can be streamed to storage.
  • the data can be streamed to storage so that the analysis of the data can be performed off line.
  • the data can be serialized before it is stored.
  • the stored data can be analyzed.
  • the analysis can be performed off line.
  • the key performance data can be analyzed to identify critical threads. Critical threads can be identified using a decision tree based model.
  • the call stack patterns of the critical threads can be analyzed using a call graph prefix binary tree.
  • a analysis results can be provided.
  • the analysis results can be results comprising one or more of: a CPU utilization view, a listing of critical thread data, and/or a call stack pattern view. The analysis results can be provided visually.
  • FIG. 2b is a block diagram illustrating an automated performance debugging system in accordance with aspects of the subject matter described herein. All or portions of system 210 may reside on one or more computers or computing devices such as the computers described below with respect to FIG. 3 . System 210 or portions thereof may be provided as a stand-alone system or as a plug-in or add-in.
  • System 210 or portions thereof may include information obtained from a service (e.g., in the cloud) or may operate in a cloud computing environment.
  • a cloud computing environment can be an environment in which computing services are not owned but are provided on demand.
  • information may reside on multiple devices in a networked cloud and/or data can be stored on multiple devices within the cloud.
  • System 210 can include one or more computing devices.
  • Contemplated computing devices include but are not limited to desktop computers, tablet computers, laptop computers, notebook computers, personal digital assistants, smart phones, cellular telephones, mobile telephones, sensors, server computers and so on.
  • a computing device can include one or more processors and a memory that communicates with the one or more processors.
  • System 210 may include one or more program modules that when loaded into the memory and accessed by the one or more processors cause the processor to perform the action or actions attributed to the one or more program modules.
  • the processor(s) may be configured to perform the action or actions attributed to the one or more program modules.
  • System 210 may operate as follows.
  • One or more worker threads such as worker thread 211 from worker thread pool 209 can be assigned to application process 205.
  • a thread such as injected thread 215 can be injected into a data collection process 213.
  • a data collection agent can collect data such as key performance data 214a from the application process 205 being debugged or tuned.
  • a lightweight sampling process such as lightweight sampling process 207 can be used to collect samples of data.
  • the collected data (e.g., key performance data 214a) can be stored in a storage device such as storage 217.
  • An analysis agent such as analysis agent 221 within a computing process such as computing process 219 can retrieve and/or analyze data such as key performance data 214b in storage 217 to generate analysis results such as analysis results 223.
  • a worker thread from the worker thread pool can be injected into the data collection process.
  • the lightweight sampling process can periodically collect data which can be serialized and stored in storage.
  • a data analysis process can analyze the data and provide analysis results on a display or other device.
  • FIG. 2f illustrates an example of elements that can be included in analysis results for performance debugging in accordance with aspects of the subject matter described herein.
  • Chart 280 is an example of a CPU view.
  • the CPU view shows the overall user mode CPU utilization of the computing device.
  • the y axis 280a represents the percentage of user mode CPU utilization.
  • the x axis 280b represents sampling time intervals.
  • Each vertical box in chart 280 refers to user mode CPU utilization from sample T x - 1 to T x .
  • the CPU view can show the amount of CPU utilization each thread accounts for. For example, "35" 281 in the first vertical box 282 indicates that the total user mode CPU utilization of all threads in the time interval from Sample 0 to Sample 1 is 35%.
  • the chart of CPU utilization breakdown 288 is a list that shows which thread accounts for how much user mode CPU utilization for that sampling interval.
  • vertical box 289 has a total user mode CPU utilization of 76% and there are 9 contributors (threads), e.g.. Thread identifier (TID) X1, Thread X2, Thread X3 ... to Thread X9.
  • Threads e.g.. Thread identifier (TID) X1, Thread X2, Thread X3 ... to Thread X9.
  • TID Thread identifier
  • Thread X1 has CPU utilization of Y1%
  • Thread X2 accounts for Y2% of CPU user mode utilization etc., to Y9% for Thread X9, respectively.
  • the sum of Y1, Y2...to Y9 is the total user mode CPU utilization, 76 percent.
  • CPU view (chart 280) can depict busy threads at each time interval during the sampling period. Busy threads ranking can be computed. Busy threads ranking along with the other three attributes "Call Stack Length", "Has non-framework call sites” and “Has non-framework call sites in consecutive samples” can be fed into a decision tree model to create Critical Threads View chart 220.
  • Chart 220 is an example of a view of critical threads.
  • the Critical Threads View, chart 220 depicts the identified critical threads and information about the critical threads including thread ID, column 283, CPU utilization, column 284, e.g. 45% 284a, if it's blocking, column 285, when the thread is sampled, column 286, (Entry 286a 1,7,9,12 means that the thread X1 was sampled from the first to seventh sampling time interval and from the 9th to 12th sampling time interval. In all for entry 286a there are 6 (1 to 7) plus 3 (9 to 12) or 9 sampling intervals. Column 287 is the total sampled time.
  • each interval is assumed to be a half second, thus column 287 entry 287a indicates a total sampling time of 4.5 seconds, (9 sampling intervals of 1 ⁇ 2 second each equals 4.5 seconds). Given a sampling interval of one second and a sampling duration of 10 seconds, 11 samples will be taken at Time 0, Time 1 ... to Time 10.
  • Chart 230 is an example of a portion of a Call Stack Pattern View for a selected critical thread from a critical thread list such as the critical thread list, chart 220.
  • the complete Call Stack Pattern view depicts all the call stack patterns constructed for the critical thread.
  • Each critical thread has a corresponding Call Stack Pattern View which includes multiple call stack patterns.
  • a busy threads ranking and decision tree classification can result in a critical threads list.
  • Each thread in the critical threads list can have a call stack pattern view.
  • a critical thread of interest from the CPU utilization breakdown chart e.g., CPU utilization breakdown chart 288 of chart 280
  • the view can include one or more clustered call stack patterns.
  • Each call stack pattern can be clustered from multiple call stacks captured during monitoring duration. They can be clustered using Call Graph Prefix Binary Tree data structure and algorithm. This enables a user to focus on the highlighted call sites because the highlighted call sites are the causes of the performance anomaly.
  • Each call stack pattern can have multiple call stack frames, each of which can include module and method name and the maximum consecutive samples in which it is found.
  • mscorlib 400 in the top call stack frame 401 is a module having a method, method A 402.
  • the "20" in call stack frame 401 represents the maximum consecutive samples in which mscorlib! methodA is found. If each sampling has a duration of 1 second, mscorlib! methodA would have executed for 20 seconds.
  • All of the code represented in the chart 230 except for call stack frames 404 and 405 are framework or operating system code (discernible from the module names).
  • call stack frames 404 and 405 The code represented by call stack frames 404 and 405 is user code, discernible because the module name is "UserModule".
  • the call stack frame 405 is a user module 405a with method X 405b and 20 maximum consecutive samples (MCS) 405c.
  • MCS maximum consecutive samples
  • This call stack frame may be distinguished in some way (e.g., highlighting, coloring of text, etc.) to draw attention to it as a possible candidate for causing a performance problem e.g., (being a hotspot).
  • Call stack frame 404 is also relevant because it is user code and has a relatively large MCS. Thus a user can review the code of methodX (and potentially method Y) to determine the reasons for the large MCS.
  • the implementation of a Call Graph Binary Tree and the algorithm can include a field that records the maximum consecutive samples (e.g., maximum consecutive samples 403) where the call stack frame appears. A coarse calculation of execution time of each call stack frame can be conducted based on maximum consecutive samples.
  • the Call Graph Binary Tree can include a field that records the maximum consecutive samples where the call stack frame appears. In call stack frame 405 it can be seen that in UserModule 406, method X 407 is present in 20 consecutive samples. From the three types of views, the root cause of the performance related issue can be identified.
  • a computing device comprising a processor and a memory connected to the processor.
  • the processor can be configured to provide a program module that collects key performance data for an application executing in a production environment using a lightweight sampling strategy. It can analyze the key performance data to identify a critical thread. It can analyze a call stack pattern of the critical thread. It can provide analysis results comprising debug information for the application executing in the production environment.
  • the key performance data can include data from execution of the application.
  • the lightweight sampling strategy can inject a data collection agent into a process executing the application, employing use of a thread injected into the process.
  • the key performance data can be fed into a decision tree model to identify critical threads, the critical threads running call sites causing a performance slowdown of the application. Busy threads between any two points in time within a sampling duration can be determined by ranking busy threads.
  • Described herein is a method comprising configuring a processor of a computing device to receive key performance data for automated debugging of a production application, analyze the key performance data to identify a critical thread, the critical thread comprising a thread running a call site, the call site causing a performance slowdown of the production application, analyze a call stack pattern of the critical thread using a call graph prefix binary tree; and provide analysis results.
  • Busy threads between any two points in time within a sampling duration can be identified by ranking busy threads.
  • a critical thread can be identified using a model based on a decision tree.
  • a call stack pattern of the critical thread can be identified using a call graph prefix binary tree.
  • Key performance data can be collected using a lightweight sampling comprising creating a handle to a process in which the production application executes, injecting a thread into the process, injecting a data collection agent into the process to collect the key performance data.
  • a computing device comprising a processor; a memory; the memory connected to the processor; the processor configured to receive key performance data for automated debugging of a production application, analyze the key performance data to identify a critical thread; analyze a call stack pattern of the critical thread using a call graph binary tree; and provide analysis results identifying a call site associated with a performance slowdown.
  • the processor can be further configured to identify the critical thread using a decision tree based model, the critical thread comprising a thread running a call site, the call site causing a performance slowdown of the production application.
  • a CPU utilization view displaying a busy threads ranking can be provided.
  • a call stack pattern view that identifies hot spots can be provided.
  • the processor can be further configured to provide critical thread data.
  • FIG. 3 and the following discussion are intended to provide a brief general description of a suitable computing environment 510 in which various embodiments of the subject matter disclosed herein may be implemented. While the subject matter disclosed herein is described in the general context of computer-executable instructions, such as program modules, executed by one or more computers or other computing devices, those skilled in the art will recognize that portions of the subject matter disclosed herein can also be implemented in combination with other program modules and/or a combination of hardware and software. Generally, program modules include routines, programs, objects, physical artifacts, data structures, etc. that perform particular tasks or implement particular data types. Typically, the functionality of the program modules may be combined or distributed as desired in various embodiments.
  • the computing environment 510 is only one example of a suitable operating environment and is not intended to limit the scope of use or functionality of the subject matter disclosed herein.
  • Computer 512 may include at least one processing unit 514, a system memory 516, and a system bus 518.
  • the at least one processing unit 514 can execute instructions that are stored in a memory such as but not limited to system memory 516.
  • the processing unit 514 can be any of various available processors.
  • the processing unit 514 can be a graphics processing unit (GPU).
  • the instructions can be instructions for implementing functionality carried out by one or more components or modules discussed above or instructions for implementing one or more of the methods described above. Dual microprocessors and other multiprocessor architectures also can be employed as the processing unit 514.
  • the computer 512 may be used in a system that supports rendering graphics on a display screen.
  • the system memory 516 may include volatile memory 520 and nonvolatile memory 522.
  • Nonvolatile memory 522 can include read only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM) or flash memory.
  • Volatile memory 520 may include random access memory (RAM) which may act as external cache memory.
  • the system bus 518 couples system physical artifacts including the system memory 516 to the processing unit 514.
  • the system bus 518 can be any of several types including a memory bus, memory controller, peripheral bus, external bus, or local bus and may use any variety of available bus architectures.
  • Computer 512 may include a data store accessible by the processing unit 514 by way of the system bus 518.
  • the data store may include executable instructions, 3D models, materials, textures and so on for graphics rendering.
  • Computer 512 typically includes a variety of computer readable media such as volatile and nonvolatile media, removable and non-removable media.
  • Computer readable media may be implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.
  • Computer readable media include computer-readable storage media (also referred to as computer storage media) and communications media.
  • Computer storage media includes physical (tangible) media, such as but not limited to, RAM, ROM, EEPROM, flash memory or other memory technology, CDROM, digital versatile disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices that can store the desired data and which can be accessed by computer 512.
  • Communications media include media such as, but not limited to, communications signals, modulated carrier waves or any other intangible media which can be used to communicate the desired information and which can be accessed by computer 512.
  • FIG. 3 describes software that can act as an intermediary between users and computer resources.
  • This software may include an operating system 528 which can be stored on disk storage 524, and which can allocate resources of the computer 512.
  • Disk storage 524 may be a hard disk drive connected to the system bus 518 through a non-removable memory interface such as interface 526.
  • System applications 530 take advantage of the management of resources by operating system 528 through program modules 532 and program data 534 stored either in system memory 516 or on disk storage 524. It will be appreciated that computers can be implemented with various operating systems or combinations of operating systems.
  • a user can enter commands or information into the computer 512 through an input device(s) 536.
  • Input devices 536 include but are not limited to a pointing device such as a mouse, trackball, stylus, touch pad, keyboard, microphone, voice recognition and gesture recognition systems and the like. These and other input devices connect to the processing unit 514 through the system bus 518 via interface port(s) 538.
  • An interface port(s) 538 may represent a serial port, parallel port, universal serial bus (USB) and the like.
  • Output devices(s) 540 may use the same type of ports as do the input devices.
  • Output adapter 542 is provided to illustrate that there are some output devices 540 like monitors, speakers and printers that require particular adapters.
  • Output adapters 542 include but are not limited to video and sound cards that provide a connection between the output device 540 and the system bus 518.
  • Other devices and/or systems or devices such as remote computer(s) 544 may provide both input and output capabilities.
  • Computer 512 can operate in a networked environment using logical connections to one or more remote computers, such as a remote computer(s) 544.
  • the remote computer 544 can be a personal computer, a server, a router, a network PC, a peer device or other common network node, and typically includes many or all of the elements described above relative to the computer 512, although only a memory storage device 546 has been illustrated in FIG. 3 .
  • Remote computer(s) 544 can be logically connected via communication connection(s) 550.
  • Network interface 548 encompasses communication networks such as local area networks (LANs) and wide area networks (WANs) but may also include other networks.
  • Communication connection(s) 550 refers to the hardware/software employed to connect the network interface 548 to the bus 518.
  • Communication connection(s) 550 may be internal to or external to computer 512 and include internal and external technologies such as modems (telephone, cable, DSL and wireless) and ISDN adapters, Ethernet cards and so on.
  • a computer 512 or other client device can be deployed as part of a computer network.
  • the subject matter disclosed herein may pertain to any computer system having any number of memory or storage units, and any number of applications and processes occurring across any number of storage units or volumes.
  • aspects of the subject matter disclosed herein may apply to an environment with server computers and client computers deployed in a network environment, having remote or local storage.
  • aspects of the subject matter disclosed herein may also apply to a standalone computing device, having programming language functionality, interpretation and execution capabilities.
  • the various techniques described herein may be implemented in connection with hardware or software or, where appropriate, with a combination of both.
  • the methods and apparatus described herein, or certain aspects or portions thereof may take the form of program code (i.e., instructions) embodied in tangible media, such as floppy diskettes, CD-ROMs, hard drives, or any other machine-readable storage medium, wherein, when the program code is loaded into and executed by a machine, such as a computer, the machine becomes an apparatus for practicing aspects of the subject matter disclosed herein.
  • the term "machine-readable storage medium” shall be taken to exclude any mechanism that provides (i.e., stores and/or transmits) any form of propagated signals.
  • the computing device will generally include a processor, a storage medium readable by the processor (including volatile and non-volatile memory and/or storage elements), at least one input device, and at least one output device.
  • One or more programs that may utilize the creation and/or implementation of domain-specific programming models aspects, e.g., through the use of a data processing API or the like, may be implemented in a high level procedural or object oriented programming language to communicate with a computer system.
  • the program(s) can be implemented in assembly or machine language, if desired. In any case, the language may be a compiled or interpreted language, and combined with hardware implementations.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Quality & Reliability (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Hardware Design (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Signal Processing (AREA)
  • Computing Systems (AREA)
  • Software Systems (AREA)
  • Environmental & Geological Engineering (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Mathematical Physics (AREA)
  • Debugging And Monitoring (AREA)
EP16915480.4A 2016-09-09 2016-09-09 Automated performance debugging of production applications Active EP3510747B1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2016/098516 WO2018045545A1 (en) 2016-09-09 2016-09-09 Automated performance debugging of production applications

Publications (3)

Publication Number Publication Date
EP3510747A1 EP3510747A1 (en) 2019-07-17
EP3510747A4 EP3510747A4 (en) 2020-04-01
EP3510747B1 true EP3510747B1 (en) 2021-09-08

Family

ID=61562537

Family Applications (1)

Application Number Title Priority Date Filing Date
EP16915480.4A Active EP3510747B1 (en) 2016-09-09 2016-09-09 Automated performance debugging of production applications

Country Status (4)

Country Link
US (1) US10915425B2 (zh)
EP (1) EP3510747B1 (zh)
CN (1) CN109716730B (zh)
WO (1) WO2018045545A1 (zh)

Families Citing this family (17)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10514890B2 (en) * 2017-11-15 2019-12-24 Accenture Global Solutions Limited Test case and data selection using a sampling methodology
US11709939B2 (en) * 2018-05-04 2023-07-25 New York University Anomaly detection in real-time multi-threaded processes on embedded systems and devices using hardware performance counters and/or stack traces
CN110609759B (zh) * 2018-06-15 2021-09-14 华为技术有限公司 一种故障根因分析的方法及装置
US10965663B2 (en) * 2018-07-26 2021-03-30 Microsoft Technology Licensing, Llc Troubleshooting single sign on failure
US10970055B2 (en) * 2018-08-21 2021-04-06 International Business Machines Corporation Identifying software and hardware bottlenecks
CN109176601B (zh) * 2018-09-21 2022-04-29 北京云迹科技股份有限公司 用于机器人的开门测试方法及装置
CN109697456B (zh) * 2018-11-21 2021-02-09 华为技术有限公司 业务分析方法、装置、设备及存储介质
CN110806931B (zh) * 2019-10-31 2022-11-22 上海依图网络科技有限公司 分布式服务的调用链路跟踪实现方法、装置及电子设备
CN113297052B (zh) * 2020-02-21 2024-04-26 腾讯科技（深圳）有限公司 应用程序的卡顿事件定位方法、装置、存储介质和设备
CN113810341B (zh) * 2020-06-12 2023-08-22 武汉斗鱼鱼乐网络科技有限公司 一种识别目标网络团体的方法及系统、存储介质、设备
US11599342B2 (en) * 2020-09-28 2023-03-07 Red Hat, Inc. Pathname independent probing of binaries
JP7011696B1 (ja) 2020-10-08 2022-01-27 レノボ・シンガポール・プライベート・リミテッド 電子機器、制御方法、及び学習済みモデル
US11928518B2 (en) 2021-08-10 2024-03-12 Kyndryl, Inc. Noisy-neighbor detection and remediation
CN113934475B (zh) * 2021-08-10 2022-09-06 荣耀终端有限公司 应用调用的分析方法及电子设备
US12093389B2 (en) 2022-03-14 2024-09-17 Microsoft Technology Licensing, Llc Data traffic characterization prioritization
CN114880159B (zh) * 2022-07-13 2022-09-13 腾讯科技（深圳）有限公司 数据处理方法、装置、设备及存储介质
CN118227431A (zh) * 2022-12-19 2024-06-21 华为技术有限公司 性能分析方法、可读存储介质、程序产品及电子设备

Family Cites Families (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
AU2001249451A1 (en) 2000-03-27 2001-12-11 Accenture Llp System, method, and article of manufacture for an automated scripting solution for enterprise testing
US8473922B2 (en) 2001-09-19 2013-06-25 Hewlett-Packard Development Company, L.P. Runtime monitoring in component-based systems
US7827535B2 (en) 2003-12-10 2010-11-02 Oracle International Corporation Application performance tuning server-side component
US7526759B2 (en) 2005-04-19 2009-04-28 International Business Machines Corporation Debugging prototyped system solutions in solution builder wizard environment
US7987462B2 (en) * 2006-11-16 2011-07-26 International Business Machines Corporation Method for automatic throttling of work producers
US8336033B2 (en) * 2007-03-30 2012-12-18 Sap Ag Method and system for generating a hierarchical tree representing stack traces
US8286134B2 (en) * 2008-07-15 2012-10-09 International Business Machines Corporation Call stack sampling for a multi-processor system
CN101364098B (zh) * 2008-09-12 2010-07-28 南京航空航天大学 一种将梯形图转换为指令表程序及解释执行的方法与系统
US8307246B2 (en) * 2008-10-29 2012-11-06 Aternity Information Systems Ltd. Real time monitoring of computer for determining speed of various processes
US9632909B2 (en) * 2008-12-16 2017-04-25 Microsoft Technology Licensing, Llc Transforming user script code for debugging
US7996722B2 (en) * 2009-01-02 2011-08-09 International Business Machines Corporation Method for debugging a hang condition in a process without affecting the process state
US8694519B2 (en) 2009-03-03 2014-04-08 Empire Technology Development, Llc Real estate analysis system
CN101615136B (zh) * 2009-06-26 2012-07-18 中兴通讯股份有限公司 linux环境下快速转换调用堆栈地址的系统及方法
US8099631B2 (en) 2009-07-17 2012-01-17 Sap Ag Call-stacks representation for easier analysis of thread dump
CN101630285A (zh) * 2009-08-07 2010-01-20 华南理工大学 一种应用于嵌入式系统的软件性能测试方法
US8938533B1 (en) * 2009-09-10 2015-01-20 AppDynamics Inc. Automatic capture of diagnostic data based on transaction behavior learning
US8584098B2 (en) * 2009-12-04 2013-11-12 Sap Ag Component statistics for application profiling
US8756586B2 (en) 2009-12-10 2014-06-17 Tata Consultancy Services Limited System and method for automated performance testing in a dynamic production environment
US8627317B2 (en) * 2010-03-25 2014-01-07 International Business Machines Corporation Automatic identification of bottlenecks using rule-based expert knowledge
US8839209B2 (en) 2010-05-12 2014-09-16 Salesforce.Com, Inc. Software performance profiling in a multi-tenant environment
US8086910B1 (en) * 2010-06-29 2011-12-27 Alcatel Lucent Monitoring software thread execution
US9274919B2 (en) 2011-04-29 2016-03-01 Dynatrace Software Gmbh Transaction tracing mechanism of distributed heterogenous transactions having instrumented byte code with constant memory consumption and independent of instrumented method call depth
CN102411534B (zh) * 2011-07-25 2014-12-31 中国科学院声学研究所 一种断点调试方法和断点调试装置
CN102685180B (zh) * 2011-10-18 2015-07-08 国网电力科学研究院 一种面向云计算的网络安全预警方法
CN102789702B (zh) 2011-12-08 2015-03-25 南京奥派信息技术有限责任公司 基于云计算的网络实践教学平台
CN102624879A (zh) 2012-02-28 2012-08-01 浪潮电子信息产业股份有限公司 一种云计算数据应用终端
CN102857560B (zh) 2012-08-15 2016-04-20 华数传媒网络有限公司 一种面向多业务应用的云存储数据分布方法
US9330119B2 (en) * 2013-04-11 2016-05-03 Oracle International Corporation Knowledge intensive data management system for business process and case management
US9569260B2 (en) * 2013-05-31 2017-02-14 Microsoft Technology Licensing, Llc Efficient priority-aware thread scheduling
US10289411B2 (en) * 2013-11-18 2019-05-14 Microsoft Technology Licensing, Llc Diagnosing production applications
CN105242962B (zh) * 2015-11-24 2018-07-03 无锡江南计算技术研究所 基于异构众核的轻量级线程快速触发方法
CN106936793B (zh) * 2015-12-30 2020-03-17 腾讯科技（深圳）有限公司 一种信息拦截处理方法及终端

Also Published As

Publication number Publication date
CN109716730B (zh) 2021-10-22
US10915425B2 (en) 2021-02-09
CN109716730A (zh) 2019-05-03
US20190196937A1 (en) 2019-06-27
EP3510747A1 (en) 2019-07-17
WO2018045545A1 (en) 2018-03-15
EP3510747A4 (en) 2020-04-01

Similar Documents

Publication Publication Date Title
EP3510747B1 (en) Automated performance debugging of production applications
CN112910945B (zh) 请求链路跟踪方法和业务请求处理方法
US8627317B2 (en) Automatic identification of bottlenecks using rule-based expert knowledge
US9753826B2 (en) Providing fault injection to cloud-provisioned machines
US8694967B2 (en) User interface inventory
US8850396B2 (en) Performing software testing based on grouping of tests using test list entity
US20200379892A1 (en) Automated determination of operating parameter configurations for applications
US20080163015A1 (en) Framework for automated testing of enterprise computer systems
US20140115565A1 (en) Test similarity detection with method call sequence analysis
US20090222697A1 (en) Dynamic functional testing coverage based on failure dependency graph
US9772682B1 (en) Method and system for dynamic selection of application dialog layout design
US20220303361A1 (en) Resiliency Control Engine for Network Service Mesh Systems
US11704185B2 (en) Machine learning-based techniques for providing focus to problematic compute resources represented via a dependency graph
Shu et al. Massive distributed and parallel log analysis for organizational security
US11526413B2 (en) Distributed tracing of huge spans for application and dependent application performance monitoring
US20190065346A1 (en) Performance data collection for a distributed computing system
US20180143897A1 (en) Determining idle testing periods
US20170177272A1 (en) Methods and systems for memory suspect detection
US10169194B2 (en) Multi-thread sequencing
Merenstein et al. {CNSBench}: A cloud native storage benchmark
US12061539B2 (en) System and method for anomaly detection and root cause automation using shrunk dynamic call graphs
US9258374B2 (en) Method and system for capturing expertise of a knowledge worker in an integrated breadcrumb trail of data transactions and user interactions
US11720810B2 (en) Reducing mean time to find problems in enterprise information technology systems using bots
US11392375B1 (en) Optimizing software codebases using advanced code complexity metrics
Angelopoulos et al. GcLite: an expert tool for analyzing garbage collection behavior

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20190304

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
REG Reference to a national code

Ref country code: DE

Ref legal event code: R079

Ref document number: 602016063613

Country of ref document: DE

Free format text: PREVIOUS MAIN CLASS: H04L0029080000

Ipc: G06F0011360000

A4 Supplementary search report drawn up and despatched

Effective date: 20200302

RIC1 Information provided on ipc code assigned before grant

Ipc: G06F 11/30 20060101ALI20200225BHEP

Ipc: G06F 11/07 20060101ALI20200225BHEP

Ipc: G06F 11/36 20060101AFI20200225BHEP

Ipc: G06F 11/34 20060101ALI20200225BHEP

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20201221

GRAJ Information related to disapproval of communication of intention to grant by the applicant or resumption of examination proceedings by the epo deleted

Free format text: ORIGINAL CODE: EPIDOSDIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

INTC Intention to grant announced (deleted)
GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20210414

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

Ref country code: AT

Ref legal event code: REF

Ref document number: 1429210

Country of ref document: AT

Kind code of ref document: T

Effective date: 20210915

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602016063613

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

RAP4 Party data changed (patent owner data changed or rights of a patent transferred)

Owner name: MICROSOFT TECHNOLOGY LICENSING, LLC

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20210908

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211208

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211208

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1429210

Country of ref document: AT

Kind code of ref document: T

Effective date: 20210908

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20211209

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20210930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220108

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20220110

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602016063613

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210909

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210909

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210930

26N No opposition filed

Effective date: 20220609

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210930

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20210930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20230512

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO

Effective date: 20160909

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20230822

Year of fee payment: 8

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20210908

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20240820

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20240820

Year of fee payment: 9