WO2018227548A1 - Analyzing performance impact of system update - Google Patents

Analyzing performance impact of system update Download PDF

Info

Publication number
WO2018227548A1
WO2018227548A1 PCT/CN2017/088617 CN2017088617W WO2018227548A1 WO 2018227548 A1 WO2018227548 A1 WO 2018227548A1 CN 2017088617 W CN2017088617 W CN 2017088617W WO 2018227548 A1 WO2018227548 A1 WO 2018227548A1
Authority
WO
WIPO (PCT)
Prior art keywords
measurement
domain
cost
performance
system performance
Prior art date
Application number
PCT/CN2017/088617
Other languages
French (fr)
Inventor
Chow Kingsum
Wanyi ZHU
Original Assignee
Alibaba Group Holding Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Alibaba Group Holding Limited filed Critical Alibaba Group Holding Limited
Priority to CN201780091959.2A priority Critical patent/CN110998539B/en
Priority to PCT/CN2017/088617 priority patent/WO2018227548A1/en
Publication of WO2018227548A1 publication Critical patent/WO2018227548A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
    • G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/3051—Monitoring arrangements for monitoring the configuration of the computing system or of the computing system component, e.g. monitoring the presence of processing resources, peripherals, I/O links, software programs
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/30—Monitoring
    • G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3452—Performance evaluation by statistical analysis

Definitions

  • Performance variation due to different software/hardware versions can affect capacity planning of machines launching the services in sophisticated applications like data centers. Furthermore, performance variations of software applications due to added performance tools will affect capacity planning as well. Solutions are needed to determine realistic performance impacts in such scenarios to manage capacity planning in, e.g., major cloud service and data center operators to serve users efficiently and satisfactorily.
  • FIG. 1 illustrates an example system for analyzing system impact of an additional element.
  • FIG. 2 illustrates an example operation environment of the system of FIG. 1.
  • FIG. 3 illustrates an example operation process of the system of FIG. 1.
  • FIG. 4 illustrates another example operation process of the system of FIG. 1.
  • FIG. 5 illustrates another example operation process of the system of FIG. 1.
  • a software developer releases a new version of a software application.
  • a throughput of, e.g., 1000 transactions per second may be obtained.
  • a throughput of 1050 transactions per second is obtained.
  • a software developer adds a performance monitoring capability, e.g., a performance monitoring element, to an existing system.
  • a performance monitoring capability e.g., a performance monitoring element
  • a throughput of, e.g., 1000 transactions per second is obtained.
  • the same software application is operated with the monitoring element, the same throughput of 1000 transactions per second is obtained.
  • the performance monitoring element With the assumption that the performance monitoring element introduces no performance overhead, the performance monitoring element is deployed in a data center. Later on, it may be found that the performance monitoring element brings about adverse performance impact on the software application.
  • Performance overhead in computer science is generally considered any individual and/or combination of excess or indirect computation time, memory usage, bandwidth, or other resources that are required to attain a particular performance goal.
  • Performance overhead include various domains, e.g., computation time and memory usage.
  • Performance goals may be in different domains, like system throughput and response speed.
  • a performance goal may be specific to a software application, but normally involves the performance of the whole system.
  • domains of performance goals and performance overheads may overlap and/or may be inter-correlated. For example, system load (domain) may be evaluated as a performance goal in some situations and may be evaluated as system overhead in other situations.
  • a system performance domain refers to a domain that is evaluated, in a specific impact assessment scenario, as a performance goal of a computing system
  • a system cost domain refers to a domain that may be evaluated with a system performance domain but not as a performance goal.
  • a system cost domain in one analysis scenario may be a system performance domain in another analysis scenario and vice versa.
  • a target system cost domain is a system cost domain that is selected to be evaluated together with a system performance domain in determining a system impact of a system update, e.g., an additional element configured to operate with a computing system (hereinafter referred to as a target element) .
  • a target element may include a software element, a hardware element, and/or a combination of a hardware and a software element.
  • a software application outputs a key performance goal metric ( “P” ) , e.g., a throughput.
  • P may be used as a system performance domain for the software application.
  • multiple performance overheads may be involved, e.g., CPU usage ( “U” ) , memory allocation ( “V” ) , networking bandwidth ( “W” ) , I/O loads ( “X” ) .
  • U, V, W, X may be used as target system cost domains depending on the specific analysis scenarios.
  • An example target element may be a monitoring plug-in to the software application.
  • System operation data in system cost domains CPU U, memory V, networking W and IO X operations
  • corresponding data in system performance domain (s) throughput “P”
  • the measurement data obtained with the computing system operating without the target element is referred to the baseline data.
  • the software application may also be launched with the target element (here the monitoring plug-in application) under controlled environment, e.g., other factors all being the same.
  • System operation data in system cost domains (CPU U’ , memory V’ , networking W’ and IO X’ operations) along with corresponding data in system performance domain (s) (throughput P’ ) may be measured and recorded as datasets, referred to herein as comparison data.
  • a data analysis scheme may be used to analyze the baseline data and the comparison data to obtain a baseline analytical result and a comparison analytical result, respectively.
  • the data analysis scheme analyzes measurements of system performance domain together/in relation to the corresponding measurements in target system cost domain (s) .
  • Various data analysis scheme may be used and all are included in the disclosure.
  • a data analysis scheme may analyze measurements in the system performance domain on a basis of per unit of measurements in a target system cost domain.
  • a data analysis scheme may analyze system performance throughput data P (P’ ) in relation to CPU usage data U (U’ ) .
  • the system performance value generated by a unit of system cost may be determined as:
  • the comparison analytical result may be analyzed in relation to the baseline analytical result to evaluate a system impact of operating the target element with the computing system including the software application.
  • the (P/U) / (P’ /U’ ) may be used to determine the impact of the target element to the performance of computer system. For example, if (P/U) / (P’ /U’ ) > 1, it may be determined that the target element has a negative impact on the system performance because less system performance metric is generated per unit of the target system cost. If (P/U) / (P’ /U’ ) ⁇ 1, it may be determined that the target element has a positive impact on the system performance because more system performance metric is generated per unit of the target system cost.
  • the measurements on the system performance domain (s) and/or the system cost domain (s) may be pre-processed before the analysis.
  • the measurements on P and the measurements on U may be pre-processed to obtain the average P measurement and average U measurement for the P/U (P’ /U’ ) analysis.
  • the P/U (P’ /U’ ) analysis may be conducted at multiple intervals and/or ranges of U and/or P measurements to refine the analysis.
  • a data analysis scheme may include a regression analysis between system performance measurements on a system performance domain and system cost measurements on one or more system cost domains.
  • the regression analysis may be between a single system cost domain and the system performance domain.
  • a regression equation may be:
  • regression analysis may also be conducted between multiple system cost domains and the system performance domain.
  • a regression equation may be:
  • a same test value in U (or P) may be applied to both the baseline regression equation and the comparison regression equation to obtain a result value P r in P (or U) and a result value P’ r in P’ (or U’ ) .
  • the result values may be used as the baseline analytical result and the comparison analytical result, respectively.
  • a highest possible value in at least one of the system performance domain (P or P’ ) measurement or the system cost domain (U or U’ ) may be used as the test value.
  • the comparison analytical result P’ r may be analyzed in relation to the baseline analytical result P r to determine an impact of the target element on the performance of computing system including the software application.
  • P’ r /P r may be evaluated. If P’ r /P r > 1, it may be determined that the target element has a positive impact on the computing system because more system performance metric, e.g., throughput P, is generated when the computing system operates with the target element as compared to without the target element.
  • system performance metric e.g., throughput P
  • the regression analysis scheme may also be implemented separately for system performance measurements and system cost measures at multiple different ranges of values.
  • a technique to select a target system cost domain to analyze with a system performance domain may also be included.
  • a targeting system cost domain may be selected based on whether a system performance metric is limited/restricted by the system cost domain.
  • a limiting cost domain i.e., a cost domain that limits/restricts the system performance metric, may be selected as the target system cost domain. Any approach to determine a limiting cost domain may be possible and all are included in the disclosure.
  • measurements on multiple system cost domains may be obtained with measurements in the system performance domain P for, e.g., the baseline data (and/or comparison data) .
  • Each system cost domain has a maximum utilization that cannot be exceeded. For example, CPU utilization may not exceed 100%, network bandwidth may not exceed 10 Mbps, etc.
  • Three separate regression equations may be generated using system cost measures in each of the system cost domains U, V, W and system performance measurements in the system performance domain P.
  • a corresponding maximum (highest possible) performance metric P is obtained using the respective regression equation, e.g., by applying the highest possible value (i.e., highest utilization) of the system cost domain into the respective regression equation.
  • the system cost domain associated with the lowest maximum metric P is determined as the limiting cost domain.
  • system cost domain V may be used as the target system cost domain to be analyzed with system performance domain P in determining the impact of the target element.
  • the evaluation of system performance impact of a target element may include multiple system performance domains and multiple system cost domains.
  • a system cost domain may be analyzed with multiple system performance domains, and/or the analysis of a system performance domain may include multiple system cost domains.
  • system 10 may include one or more memory 100 storing computer executable instructions, which when executed by one or more processing units, configure the processing units and the related computing device (s) to implement a system identification unit 110, a data receiving unit 120, an analysis unit 130, and a control unit 140.
  • Data receiving unit 120 may further include an original system data receiving unit 122 and an updated system data receiving unit 124.
  • Analysis unit 130 may further include an original system analyzing unit 132, an updated system analyzing unit 134, a comparison unit 136, and a target system cost domain determination unit 138.
  • System 10 may also include one or more processing units (PU) 150, interfacing units 160, communication units 170 and/or other components 180.
  • PU processing units
  • system 10 and the units thereof e.g., system identification unit 110, data receiving unit 120, analysis unit 130, and control unit 140, are illustrated as a single system, it does not necessarily mean that all components of system 10 are physically or functionally located within a single computing system. All the units as illustrated in system 10 of FIG. 1 may be located on separate computing systems communicating and functioning together in a distributed computing environment.
  • FIG. 2 illustrate an example operation environment 200 of system 10. Note that FIG. 2 illustrates operation environment 200 in a distributed computing scheme, which is not necessary for the implementation. Some or all of the communication shown in FIG. 2 may be implemented through data bus (i.e., local communication) within a single computing system. For example, system 10 may be included in one of the computing system 240 as a built-in analysis unit.
  • one or more units of system 10 may communicate with data source (s) 230 (230-1 to 230-N) and/or computing system (s) 240 (240-1 to 240-M) through a network 220.
  • Each data source 230 may acquire/collect performance data/measurements of one or more computing systems 240 in one or more system performance domains and one or more system cost domains.
  • System 10 may receive system performance data and/or system cost data from data source 230, or may acquire/collect such data directly with a computing system 240.
  • System performance data/measurements and/or system cost data/measurements may be acquired/collected using any approaches and all are included in the disclosure.
  • different data source 230 may engage different mechanisms to acquire data and/or may process acquired data of computing system (s) 240 differently such that the data of different data sources 230 may include different formats.
  • the sampling frequency of data sources 230 may also vary, which may be reflected in the data entry points of the dataset generated and received by system 10.
  • System 10 may adopt techniques to resolve such issues of inconsistent data formats.
  • Target elements 250 may include software elements, hardware elements and/or combinations of hardware/software elements.
  • a target element 250 may be configured to operate with a computing system 240 and/or a unit thereof.
  • Data source 230 and/or system 10 may be configured to acquire/collect measurements in system performance domain (s) and/or system cost domain (s) for computing system 240 operating with and without the relevant target element 250.
  • the acquired data may be initially processed locally in different data sources 230 in a distributed computing scheme.
  • data source 230 may initially process the acquired system performance data and system cost data to generate datasets with system performance data entries and corresponding system cost data entries associated to one another to facilitate further analysis.
  • Data source 230 may further process the generated data set to improve the data quality, e.g., by eliminating unusual data entries.
  • the locally and initially processed system performance data and/or system cost data may be received by system 10, through data communication via network 220, for further processing.
  • Such a tiered processing of the acquired system performance/cost data in the distributed computing environment may save communication bandwidth, improve capacity allocation, and enable architecture flexibility and adaptability, which are all technical advantages.
  • system identification unit 110 may be configured to identify a computing system 240 and a corresponding target element 250 configured to operate with the computing system 240.
  • the identification may be achieved through user inputs, communication with data source 230, and/or other approaches.
  • Data receiving unit 120 may be generally configured to receive system performance data/measurements and/or system cost data/measurements of identified computing system 240 operating with/without corresponding target element 250.
  • the measurement data may be acquired and/or received from data sources 230 or from other inside/outside channels.
  • Data receiving unit 120 may also be configured to merge measurements in the system performance domain and measurements in the system cost domain into a dataset using a same time scale.
  • data receiving unit 120 and/or data source 230 may acquire measurements in system cost domains separately than the measurements in system performance domains. Such separately acquired system cost data and system performance data may need to be merged and associated together to enable the further analysis. For example, a system cost measurement may need to be linked, e.g., on sampling point, to a system performance measurement to be analyzed together.
  • data receiving unit 120 may receive system performance measurements and system cost measurements from multiple data sources with different formats and time scales. Such data from different data sources need to be processed and merged into a dataset for further analysis.
  • original system data receiving unit 122 may be configured to receive baseline system performance measurements of computing system 240 in a system performance domain (s) and baseline system cost measurements of computing system 240 in a system cost domain (s) including a target system cost domain (s) .
  • the baseline system performance measurements and the baseline system cost measurements may be obtained/acquired with respect to the computing system 240 operating without the corresponding target element 250.
  • Updated system data receiving unit 124 may be configured to receive comparison system performance measurements of computing system 240 in the same system performance domain and comparison system cost measurements of computing system 240 in the same system cost domain (s) including the target system cost domain (s) . As described herein, the comparison system performance measurements and the comparison system cost measurements may be obtained/acquired with respect to the computing system 240 operating with the corresponding target element 250.
  • the acquiring of the baseline data and the comparison data is under a well controlled environment such that other than the inclusion/exclusion of target element 250, the rest of the computing system 240 and the operation thereof is controlled as identical.
  • Analysis unit 130 may be generally configured to analyze the received baseline data (baseline system performance and baseline system cost data) and comparison data (comparison system performance data and comparison system cost data) to evaluate a system impact of operating the identified target element 250 with the identified computing system 240.
  • original system analyzing unit 132 may be configured to generate a first analytical result using the baseline system performance measurements and the baseline system cost measurements under a data analysis scheme. Any data analysis scheme that analyzes the system performance measurements together with system cost measurements may be used and all are included in the disclosure.
  • the data analysis scheme may include analyzing the (baseline) system performance measurements on a basis of per unit of the (baseline) system cost measurement, e.g., P/U as described herein as an illustrative example.
  • the data analysis scheme may include a regression analysis between the (baseline) system performance measurements and the (baseline) system cost measurements.
  • Updated system analyzing unit 134 may be configured to generate a second analytical result using the comparison system performance measurements and the comparison system cost measurements under the same data analysis scheme as used by the original system analyzing unit 134.
  • Comparison unit 136 may be configured to analyze the second analytical result in relation to the first analytical result to evaluate a system impact of operating the target element 250 with the computing system 240. Any analyzing approaches may be used and all are included in the disclosure. For example, in the case that a regression analysis scheme is used to obtain the first and second analytical results, P’ r /P r may be used to evaluate the system impact of target element 250 on computing system 240. If P’ r /P r > 1, it may be determined that the target element 250 has a positive impact on the performance of computing system 240 because more system performance metrics, e.g., throughput P, is generated when the computing system 240 operates with the target element 250 as compared to without the target element 250.
  • system performance metrics e.g., throughput P
  • Target system cost domain determination unit 138 may be configured to select, among multiple system cost domains, a system cost domain to be a target system cost domain to analyze with the system performance domain. In an example, target system cost domain determination unit 138 may determine a limiting cost domain among the multiple system cost domains as the target system cost domain.
  • Control unit 140 may be configured to control the configuration and/or operation of computing system 240 based on the analysis result generated by analysis unit 130. For example, control unit 140 may control whether the identified computing system 240 operates with the identified target element 250 or not. Control unit 140 may also control computing system 240 operates with target element 250 in identified ranges of system cost metrics and/or system performance metrics, e.g., in the case that the system impact analysis is conducted on multiple ranges of system cost measurements/system performance measurements.
  • system identification unit 110 may identify as an analysis object a computing system 240 and a corresponding target element 250 configured to operate with the computing system 240.
  • original system data receiving unit 122 may receive baseline system performance measurements of identified computing system 240 in a system performance domain and baseline system cost measurements of identified computing system 240 in a system cost domain (s) including a target system cost domain (s) .
  • the baseline system performance measurements and the baseline system cost measurements may be obtained/acquired with the identified computing system 240 operating without the corresponding (identified) target element 250.
  • updated system data receiving unit 124 may receive comparison system performance measurements of identified computing system 240 in the same system performance domain and comparison system cost measurements of identified computing system 240 in the same system cost domain (s) including the same target system cost domain (s) .
  • the comparison system performance measurements and the comparison system cost measurements may be obtained/acquired with the identified computing system 240 operating with the corresponding identified target element 250.
  • original system analyzing unit 132 may generate a first analytical result using the baseline system performance measurements and the baseline system cost measurements of the identified computing system 240 under a data analysis scheme. Any data analysis scheme that analyzes the system performance measurements together with system cost measurements may be used and all are included in the disclosure.
  • the data analysis scheme may include analyzing the (baseline) system performance measurements on a basis of per unit of the (baseline) system cost measurement, e.g., P/U as described herein as an illustrative example.
  • the data analysis scheme may include a regression analysis between the (baseline) system performance measurements and the (baseline) system cost measurements.
  • updated system analyzing unit 134 may generate a second analytical result using the comparison system performance measurements and the comparison system cost measurements of identified computing system 240 under the same data analysis scheme as used by the original system analyzing unit 134 in example operation 340. For example, in the case that the original system analyzing unit 134 uses the scheme of system performance metric per unit of system cost, e.g., P/U, updated system analyzing unit 134 uses the same scheme and obtains P’ /U’ .
  • the scheme of system performance metric per unit of system cost e.g., P/U
  • example operation 340 the same regression analysis is used in example operation 350.
  • FIG. 4 shows an example operation flow of the regression analysis in obtaining the first analytical result and the second analytical result.
  • a baseline regression equation is generated using the baseline system performance measurements and the baseline system cost measurements.
  • the baseline regression equation may be:
  • a test value of one of system performance domain or system cost domain may be applied to the baseline regression equation to obtain a first result value of the other one of the system performance domain or the system cost domain as the first analytical result.
  • 50%CPU usage (U) is applied to the baseline equation and a resulted throughput value (P r ) 600 is obtained as the first analytical result.
  • a comparison regression equation may be generated using comparison system performance measurements and comparison system cost measurements.
  • the comparison regression equation may be:
  • example operation 440 the same test value (as in example operation 420) is applied to the comparison regression equation to obtain a second result value as the second analytical value.
  • the same 50%CPU usage (U’ ) is applied to the comparison regression equation and a resulted throughput metric (P r ’ ) 550 is obtained as the second analytical result.
  • comparison unit 136 may analyze the second analytical result in relation to the first analytical result to evaluate a performance impact of the system update, here, i.e., operating the identified target element 250 with the identified computing system 240.
  • Any analyzing approaches may be used and all are included in the disclosure.
  • P’ r /P r may be used to evaluate the performance impact of target element 250 on computing system 240.
  • control unit 140 may control the computing system 240 in system configuration and/or operation based on the analysis result generated by analysis unit 130. For example, control unit 140 may control whether the identified computing system 240 operates with the identified target element 250 or not. Control unit 140 may also control the identified computing system 240 operates with the identified target element 250 in specified ranges of system cost metrics and/or system performance metrics.
  • FIG. 5 shows an example process flow for determining a target system cost domain to be analyzed with the system performance domain.
  • the selection of the target cost domain may be based on the baseline data (system performance measurements and system cost measurements) and/or may be based on the comparison data (system performance measurements and system cost measurements) .
  • baseline data is used in the selection of target system cost domain as an illustrative example.
  • multiple regression equations are generated using baseline system cost measurements in each of multiple system cost domains (e.g., U, V, W) and the baseline system performance measurements in a same system performance domain (e.g., P) .
  • a highest possible system performance value is obtained through each of the multiple regression equations (e.g., for each of U, V, W) .
  • the highest system performance value may be obtained based on the range of the corresponding system cost value, e.g., CPU usage U cannot be higher than 100%.
  • the highest system performance value may also be obtained based on the specific regression equation, e.g., if the regression equation is not linear.
  • the lowest one of the multiple highest possible system performance values obtained through the multiple regression equations is identified. For illustrative example, if the highest system performance value obtained through the regression equations for system cost domains U, V, W are P u , P v , P w , respectively, the lowest one among P u , P v , P w may be identified, e.g., P u .
  • the system cost domain corresponding to the lowest one of the multiple highest possible system performance values obtained through the multiple regression equations may be identified as the target system cost domain.
  • CPU usage (U) may be identified as the target system cost domain because P u is the lowest one among P u , P v , P w obtained through the regression equations each between one of system cost domains U, V, W and system performance domain P.
  • the identified target system cost domain may be used to analyze, together with the system performance domain, the performance impact of identified target element 250 on identified computing system 240. It should be appreciated that the analysis of the system impact may be conducted for multiple system cost domains with the system performance domain, either separately or in combination. The system impact analysis may also be conducted for multiple system performance domains in relation to the same multiple system cost domain and/or different system cost domains.
  • the examples provided in the disclosure are only for illustrative purposes and do not limit the scope of the disclosure.
  • FIGS. 3-5 can be implemented in hardware, software, or a combination thereof.
  • the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations.
  • computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types.
  • hardware components perform one or more of the operations.
  • Such hardware components may include or be incorporated into processors, application-specific integrated circuits (ASICs) , programmable circuits such as field programmable gate arrays (FPGAs) , or in other ways.
  • ASICs application-specific integrated circuits
  • FPGAs field programmable gate arrays
  • the memory may include computer readable media such as a volatile memory, a Random Access Memory (RAM) , and/or non-volatile memory, e.g., Read-Only Memory (ROM) or flash RAM, and so on.
  • RAM Random Access Memory
  • ROM Read-Only Memory
  • the memory is an example of a computer readable medium.
  • Computer readable media include non-volatile, volatile, mobile and non-mobile media, and can implement information storage through any method or technology.
  • the information may be computer readable instructions, data structures, program modules or other data.
  • Examples of storage media of a computer include, but not limited to, Phase-change RAMs (PRAMs) , Static RAMs (SRAMs) , Dynamic RAMs (DRAMs) , other types of RAMs, ROMs, Electrically Erasable Programmable Read-Only Memories (EEPROMs) , flash memories or other memory technologies, Compact Disk Read-Only Memories (CD-ROMs) , Digital Versatile Discs (DVDs) or other optical memories, cassettes, cassette and disk memories or other magnetic memory devices or any other non-transmission media, and can be used for storing information accessible to the computation device.
  • the computer readable media exclude transitory media, such as modulated data signals and carriers.
  • the terms "include” , “comprise” , or any variants thereof are intended to cover a non-exclusive inclusion, such that a process, a method, a product, or a device that includes a series of elements not only includes such elements but also includes other elements not specified expressly, or may further include inherent elements of the process, method, product, or device. In the absence of more restrictions, an element limited by “include a/an" does not exclude other same elements existing in the process, method, product, or device that includes the element.
  • a computing system comprising: a processing unit; a memory containing computer executable instructions, which when executed by the processing unit, configured the processing unit to implement a system configuration controller, the system configuration controller being operable to: identifying an application operable in the computing system, and a target element configured to operate with the application; acquire a first system performance measurement of the computing system in a system performance domain and a first system cost measurement of the computing system in a target system cost domain, the first system performance measurement and the first system cost measurement being obtained with the application being operated in the computing system without the target element; acquire a second system performance measurement of the computing system in the system performance domain and a second system cost measurement of the computing system in the target system cost domain, the second system performance measurement and the second system cost measurement being obtained with the application operated in the computing system with the target element; generate a first analytical result using the first system performance measurement and the first system cost measurement under a data analysis scheme; generate a second analytical result using the second system performance measurement and the second system cost measurement under the same data analysis scheme; and analyze the
  • Clause 2 the computing system of clause 1, wherein the data analysis scheme includes analyzing a system performance measurement on a basis of per unit of a system cost measurement.
  • Clause 3 the computing system of clause 1, wherein the data analysis scheme includes a regression analysis between a system performance measurement and a system cost measurement.
  • Clause 4 the computing system of clause 3, wherein the generating the first analytical result includes: generating a first regression equation using the first system performance measurement and the first system cost measurement, and applying a test value of one of the system performance domain or the system cost domain into the first regression equation to obtain a first result value of other one of the system performance domain or the system cost domain as the first analytical result, and the generating the second analytical result includes: generating a second regression equation using the second system performance measurement and the second system cost measurement, and applying the test value into the second regression equation to obtain a second result value as the second analytical result.
  • Clause 5 the computing system of clause 1, wherein the first analytical result and the second analytical result are generated using a highest possible value in at least one of the first system cost measurement or the second system cost measurement.
  • Clause 6 the computing system of clause 1, wherein system configuration controller is further operable to merge a measurement in the system performance domain and a measurement in the system cost domain into a dataset using a same time scale.
  • Clause 7 the computing system of clause 1, wherein system configuration controller is further operable to control at least one of a configuration or an operation of the application with respect to the target element.
  • Clause 8 a method, comprising: identifying a computing system and a target element configured to operate with the computing system; receiving a first system performance measurement of the computing system in a system performance domain and a first system cost measurement of the computing system in a target system cost domain, the first system performance measurement and the first system cost measurement being obtained with the computing system operating without the target element; receiving a second system performance measurement of the computing system in the system performance domain and a second system cost measurement of the computing system in the target system cost domain, the second system performance measurement and the second system cost measurement being obtained with the computing system operating with the target element; generating a first analytical result using the first system performance measurement and the first system cost measurement under a data analysis scheme; generating a second analytical result using the second system performance measurement and the second system cost measurement under the same data analysis scheme; and analyzing the second analytical result in relation to the first analytical result to evaluate an impact of operating the target element with the computing system.
  • Clause 9 the method of clause 8, wherein the data analysis scheme includes analyzing a system performance measurement on a basis of per unit of a system cost measurement.
  • Clause 10 the method of clause 9, wherein an average value of the system performance measurement and an average value of the system cost measurement are used in calculating the analyzing.
  • Clause 11 the method of clause 8, wherein the data analysis scheme includes a regression analysis between a system performance measurement and a system cost measurement.
  • Clause 12 the method of clause 11, wherein the generating the first analytical result includes: generating a first regression equation using the first system performance measurement and the first system cost measurement, and applying a test value of one of the system performance domain or the system cost domain into the first regression equation to obtain a first result value of other one of the system performance domain or the system cost domain as the first analytical result, and the generating the second analytical result includes: generating a second regression equation using the second system performance measurement and the second system cost measurement, and applying the test value into the second regression equation to obtain a second result value as the second analytical result.
  • Clause 13 the method of clause 12, wherein, the generating the first regression equation includes generating multiple regression equations, each by using a first system cost measurement in one of multiple system cost domains and the first system performance measurement, determining a limiting cost domain among the multiple system cost domains by analyzing maximum system performance values obtained through each of the multiple regressions, and using the limiting cost domain as the target system cost domain.
  • Clause 14 the method of clause 8, wherein the first analytical result and the second analytical result are generated using a highest possible value in at least one of the system performance domain or the target system cost domain.
  • Clause 15 the method of clause 8, further comprising merging a measurement in the system performance domain and a measurement in the system cost domain into a dataset using a same time scale.
  • Clause 16 a method, comprising: receiving system performance measurements of a computing system in a system performance domain and system cost measurement in multiple system cost domains; generating multiple regression equations each between the system performance domain and one of the multiple system cost domains and using the system performance measurements and system cost measurements in the one of the multiple system cost domains; determining a highest possible system performance value for each of the multiple regression equations to obtain a pool of highest possible system performance values; identifying a lowest system performance value in the pool; and identifying a system cost domain associated with the lowest system performance value in the pool as a target system cost domain; and analyzing system cost measurements in the target system cost domain with the system performance measurements to determine a performance of the computing system.
  • Clause 17 the method of clause 16, wherein the analyzing includes analyzing a system performance measurement on a basis of per unit of a system cost measurement in the target system cost domain.
  • Clause 18 the method of clause 16, wherein the analyzing includes generating the first analytical result includes: receiving a first system performance measurement of the computing system in the system performance domain and a first system cost measurement of the computing system in the target system cost domain, the first system performance measurement and the first system cost measurement being obtained with the computing system operating in an original configuration; receiving a second system performance measurement of the computing system in the system performance domain and a second system cost measurement of the computing system in the target system cost domain, the second system performance measurement and the second system cost measurement being obtained with the computing system operating in an updated configuration; generating a first analytical result using the first system performance measurement and the first system cost measurement under a data analysis scheme; generating a second analytical result using the second system performance measurement and the second system cost measurement under the same data analysis scheme; and analyzing the second analytical result in relation to the first analytical result to evaluate a system impact of the updated configuration of the computing system.
  • Clause 19 the method of clause 18, further comprising controlling a configuration of the computing system based on the evaluated system impact of the updated system.
  • Clause 20 the method of clause 18, wherein further comprising controlling an operation of the computing system in the updated configuration.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Quality & Reliability (AREA)
  • General Physics & Mathematics (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Evolutionary Biology (AREA)
  • Probability & Statistics with Applications (AREA)
  • Computer Hardware Design (AREA)
  • Debugging And Monitoring (AREA)

Abstract

Computing system with a built-in performance impact evaluation mechanism. Baseline system performance and cost data are acquired with an original system configuration; comparison system performance and cost data are acquired with an updated system configuration; baseline data and comparison data are analyzed separately using the same analysis scheme to obtain baseline analytical result and comparison analytical result, which are analyzed together to determine a performance impact of the updated system configuration.

Description

ANALYZING PERFORMANCE IMPACT OF SYSTEM UPDATE BACKGROUND
Performance variation due to different software/hardware versions can affect capacity planning of machines launching the services in sophisticated applications like data centers. Furthermore, performance variations of software applications due to added performance tools will affect capacity planning as well. Solutions are needed to determine realistic performance impacts in such scenarios to manage capacity planning in, e.g., major cloud service and data center operators to serve users efficiently and satisfactorily.
BRIEF DESCRIPTION OF THE DRAWINGS
The detailed description is set forth with reference to the accompanying figures. In the figures, the left-most digit (s) of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different figures indicates similar or identical items or features.
FIG. 1 illustrates an example system for analyzing system impact of an additional element.
FIG. 2 illustrates an example operation environment of the system of FIG. 1.
FIG. 3 illustrates an example operation process of the system of FIG. 1.
FIG. 4 illustrates another example operation process of the system of FIG. 1.
FIG. 5 illustrates another example operation process of the system of FIG. 1.
DETAILED DESCRIPTION
The disclosure provides a solution to analyze performance impact of an update to a computing system. In the following detailed description, references are made to the accompanying drawings that form a part hereof, and in which are shown by way of illustration specific configurations or examples, in which like numerals represent like elements throughout the several figures.
1. Overview
When evaluating a software application release, it is important to determine the performance change that comes with the new version. When evaluating the overhead of a performance tool being deployed in a data center, it is also required to determine the resulted performance impact. In the disclosure herein, techniques in determining such performance impact are described from multiple perspectives.
To facilitate the understanding of the disclosure, two example scenarios may be used to illustrate the implementations of the solutions.
In a first example scenario, a software developer releases a new version of a software application. When the old version was operated, a  throughput of, e.g., 1000 transactions per second may be obtained. Regarding the new version, a throughput of 1050 transactions per second is obtained. With the assumption that the new version is better due to the higher throughput, the software developer releases the new version and later observes that the new version is actually slower than the old version.
In a second example scenario, a software developer adds a performance monitoring capability, e.g., a performance monitoring element, to an existing system. When a software application is operated without the performance monitoring element, a throughput of, e.g., 1000 transactions per second, is obtained. When the same software application is operated with the monitoring element, the same throughput of 1000 transactions per second is obtained. With the assumption that the performance monitoring element introduces no performance overhead, the performance monitoring element is deployed in a data center. Later on, it may be found that the performance monitoring element brings about adverse performance impact on the software application.
Observations of both example scenarios illustrate that throughput measurement itself may not reliably indicate a performance impact of a system update. The techniques disclosed herein resolve, among others, these disadvantages.
Performance overhead in computer science is generally considered any individual and/or combination of excess or indirect computation time, memory usage, bandwidth, or other resources that are required to attain a particular performance goal. Performance overhead include various domains, e.g., computation time and memory usage. Performance goals may be in different domains, like system throughput and response speed. A performance goal may be specific to a software application, but normally involves the performance of the whole system. Many times, domains of performance goals and performance overheads may overlap and/or may be inter-correlated. For example, system load (domain) may be evaluated as a performance goal in some situations and may be evaluated as system overhead in other situations.
In the disclosure herein, a system performance domain refers to a domain that is evaluated, in a specific impact assessment scenario, as a performance goal of a computing system, and a system cost domain refers to a domain that may be evaluated with a system performance domain but not as a performance goal. A system cost domain in one analysis scenario may be a system performance domain in another analysis scenario and vice versa. A target system cost domain is a system cost domain that is selected to be evaluated together with a system performance domain in determining a system impact of a system update, e.g., an additional element configured to operate with a computing system (hereinafter referred to as a target element) . A target element may include a software element, a hardware element, and/or a combination of a hardware and a software element.
For an illustrative example, it is assumed that a software application outputs a key performance goal metric ( “P” ) , e.g., a throughput. P may be used as a system performance domain for the software application. In the analysis of impacts of a target element on throughput P of the software application, multiple performance overheads may be involved, e.g., CPU usage ( “U” ) , memory allocation ( “V” ) , networking bandwidth ( “W” ) , I/O loads  ( “X” ) . One or more of U, V, W, X may be used as target system cost domains depending on the specific analysis scenarios. An example target element may be a monitoring plug-in to the software application.
In the operation of the performance impact analysis, the software application is first launched without the target element. System operation data in system cost domains (CPU U, memory V, networking W and IO X operations) along with corresponding data in system performance domain (s) (throughput “P” ) may be measured and recorded as datasets. For descriptive purpose, the measurement data obtained with the computing system operating without the target element is referred to the baseline data.
The software application may also be launched with the target element (here the monitoring plug-in application) under controlled environment, e.g., other factors all being the same. System operation data in system cost domains (CPU U’ , memory V’ , networking W’ and IO X’ operations) along with corresponding data in system performance domain (s) (throughput P’ ) may be measured and recorded as datasets, referred to herein as comparison data.
A data analysis scheme may be used to analyze the baseline data and the comparison data to obtain a baseline analytical result and a comparison analytical result, respectively. The data analysis scheme analyzes measurements of system performance domain together/in relation to the corresponding measurements in target system cost domain (s) . Various data analysis scheme may be used and all are included in the disclosure.
In an example, a data analysis scheme may analyze measurements in the system performance domain on a basis of per unit of  measurements in a target system cost domain. For illustrative example, a data analysis scheme may analyze system performance throughput data P (P’ ) in relation to CPU usage data U (U’ ) . The system performance value generated by a unit of system cost may be determined as:
P/U (baseline)
P’/U’ (comparison)
After the baseline analytical result and the comparison analytical result are generated/obtained using the analysis scheme, the comparison analytical result may be analyzed in relation to the baseline analytical result to evaluate a system impact of operating the target element with the computing system including the software application.
Any approaches to analyze the comparison analytical result in relation to the baseline analytical result may be used and all are included in the disclosure. In an example, in the case that P/U (P’ /U’ ) is used to generate the analytical results, the (P/U) / (P’ /U’ ) may be used to determine the impact of the target element to the performance of computer system. For example, if (P/U) / (P’ /U’ ) > 1, it may be determined that the target element has a negative impact on the system performance because less system performance metric is generated per unit of the target system cost. If (P/U) / (P’ /U’ ) < 1, it may be determined that the target element has a positive impact on the system performance because more system performance metric is generated per unit of the target system cost.
It should be appreciated that the measurements on the system performance domain (s) and/or the system cost domain (s) may be pre-processed before the analysis. For example, in the case that P/U (P’ /U’ ) is  used to generate the analytical results, the measurements on P and the measurements on U may be pre-processed to obtain the average P measurement and average U measurement for the P/U (P’ /U’ ) analysis. Further, the P/U (P’ /U’ ) analysis may be conducted at multiple intervals and/or ranges of U and/or P measurements to refine the analysis.
In another example, a data analysis scheme may include a regression analysis between system performance measurements on a system performance domain and system cost measurements on one or more system cost domains. The regression analysis may be between a single system cost domain and the system performance domain. For an illustrative example, a regression equation may be:
P ＝ a*U + b, where a, b are constants, or
P ＝ a*U2 + b*U + c, where a, b and c are constants.
The regression analysis may also be conducted between multiple system cost domains and the system performance domain. For an illustrative example, a regression equation may be:
P ＝ a*U + b*W + c*X + d, where a, b, c, d are constants.
It should be appreciated that the above example regression equations are provided for illustrative purpose only and do not limit the scope of the disclosure. Any regression scheme may be possible and all are included in the disclosure. For illustrative purposes only, a regression analysis scheme between system performance metric throughput P and CPU usage U, P ＝ a*U + b, is used as an illustrative example to describe the regression analysis herein.
A baseline regression equation P ＝ a*U + b may be generated using the baseline data. A comparison regression equation P’ ＝ a’ *U’ + b’ may be generated using the comparison data. A same test value in U (or P) may be applied to both the baseline regression equation and the comparison regression equation to obtain a result value Pr in P (or U) and a result value P’ r in P’ (or U’ ) . The result values may be used as the baseline analytical result and the comparison analytical result, respectively.
In an example, a highest possible value in at least one of the system performance domain (P or P’ ) measurement or the system cost domain (U or U’ ) may be used as the test value.
The comparison analytical result P’ r may be analyzed in relation to the baseline analytical result Pr to determine an impact of the target element on the performance of computing system including the software application.
For example, P’ r /Pr may be evaluated. If P’ r /Pr > 1, it may be determined that the target element has a positive impact on the computing system because more system performance metric, e.g., throughput P, is generated when the computing system operates with the target element as compared to without the target element.
Similarly to the per unit of system cost measurement (P/U) analysis scheme, the regression analysis scheme may also be implemented separately for system performance measurements and system cost measures at multiple different ranges of values.
A technique to select a target system cost domain to analyze with a system performance domain may also be included. In an example, a targeting system cost domain may be selected based on whether a system performance metric is limited/restricted by the system cost domain. A limiting cost domain, i.e., a cost domain that limits/restricts the system performance metric, may be selected as the target system cost domain. Any approach to determine a limiting cost domain may be possible and all are included in the disclosure.
In an example, measurements on multiple system cost domains, e.g., U, V, W, may be obtained with measurements in the system performance domain P for, e.g., the baseline data (and/or comparison data) .
Each system cost domain has a maximum utilization that cannot be exceeded. For example, CPU utilization may not exceed 100％, network bandwidth may not exceed 10 Mbps, etc.
Three separate regression equations may be generated using system cost measures in each of the system cost domains U, V, W and system performance measurements in the system performance domain P.
For each resources U, V and W, a corresponding maximum (highest possible) performance metric P is obtained using the respective regression equation, e.g., by applying the highest possible value (i.e., highest utilization) of the system cost domain into the respective regression equation. The system cost domain associated with the lowest maximum metric P is determined as the limiting cost domain.
For illustrative example, assuming that the regression analysis of U and P generates a maximum throughput P of 1000, the regression analysis of V and P generates a maximum throughput P of 900, and the regression analysis of W and P generates a maximum throughput P of 950. As the regression analysis of V and P generates the lowest maximum throughput P, the corresponding system cost domain of V may be determined as the limiting cost domain. Therefore, system cost domain V may be used as the target system cost domain to be analyzed with system performance domain P in determining the impact of the target element.
It should be appreciated that the evaluation of system performance impact of a target element may include multiple system performance domains and multiple system cost domains. A system cost domain may be analyzed with multiple system performance domains, and/or the analysis of a system performance domain may include multiple system cost domains.
2. Example Devices
Referring to FIG. 1, an example system 10 for analyzing performance impact of a system update is illustrated. As shown in FIG. 1, system 10 may include one or more memory 100 storing computer executable instructions, which when executed by one or more processing units, configure the processing units and the related computing device (s) to implement a system identification unit 110, a data receiving unit 120, an analysis unit 130, and a control unit 140.
Data receiving unit 120 may further include an original system data receiving unit 122 and an updated system data receiving unit 124. Analysis unit 130 may further include an original system analyzing unit 132, an updated system analyzing unit 134, a comparison unit 136, and a target system cost domain determination unit 138.
System 10 may also include one or more processing units (PU) 150, interfacing units 160, communication units 170 and/or other components 180.
It should be appreciated that although system 10 and the units thereof, e.g., system identification unit 110, data receiving unit 120, analysis unit 130, and control unit 140, are illustrated as a single system, it does not necessarily mean that all components of system 10 are physically or functionally located within a single computing system. All the units as illustrated in system 10 of FIG. 1 may be located on separate computing systems communicating and functioning together in a distributed computing environment.
FIG. 2 illustrate an example operation environment 200 of system 10. Note that FIG. 2 illustrates operation environment 200 in a distributed computing scheme, which is not necessary for the implementation. Some or all of the communication shown in FIG. 2 may be implemented through data bus (i.e., local communication) within a single computing system. For example, system 10 may be included in one of the computing system 240 as a built-in analysis unit.
As shown in FIG. 2, one or more units of system 10 may communicate with data source (s) 230 (230-1 to 230-N) and/or computing  system (s) 240 (240-1 to 240-M) through a network 220. Each data source 230 may acquire/collect performance data/measurements of one or more computing systems 240 in one or more system performance domains and one or more system cost domains. System 10 may receive system performance data and/or system cost data from data source 230, or may acquire/collect such data directly with a computing system 240. System performance data/measurements and/or system cost data/measurements may be acquired/collected using any approaches and all are included in the disclosure.
It is understood that different data source 230 may engage different mechanisms to acquire data and/or may process acquired data of computing system (s) 240 differently such that the data of different data sources 230 may include different formats. The sampling frequency of data sources 230 may also vary, which may be reflected in the data entry points of the dataset generated and received by system 10. System 10 may adopt techniques to resolve such issues of inconsistent data formats.
Target elements 250 may include software elements, hardware elements and/or combinations of hardware/software elements. A target element 250 may be configured to operate with a computing system 240 and/or a unit thereof. Data source 230 and/or system 10 may be configured to acquire/collect measurements in system performance domain (s) and/or system cost domain (s) for computing system 240 operating with and without the relevant target element 250.
In an example, as illustrated in FIG. 2, the acquired data may be initially processed locally in different data sources 230 in a distributed computing scheme. For example, data source 230 may initially process the  acquired system performance data and system cost data to generate datasets with system performance data entries and corresponding system cost data entries associated to one another to facilitate further analysis. Data source 230 may further process the generated data set to improve the data quality, e.g., by eliminating unusual data entries. The locally and initially processed system performance data and/or system cost data may be received by system 10, through data communication via network 220, for further processing. Such a tiered processing of the acquired system performance/cost data in the distributed computing environment may save communication bandwidth, improve capacity allocation, and enable architecture flexibility and adaptability, which are all technical advantages.
In operation, system identification unit 110 may be configured to identify a computing system 240 and a corresponding target element 250 configured to operate with the computing system 240. The identification may be achieved through user inputs, communication with data source 230, and/or other approaches.
Data receiving unit 120 may be generally configured to receive system performance data/measurements and/or system cost data/measurements of identified computing system 240 operating with/without corresponding target element 250. The measurement data may be acquired and/or received from data sources 230 or from other inside/outside channels.
Data receiving unit 120 may also be configured to merge measurements in the system performance domain and measurements in the system cost domain into a dataset using a same time scale. In some scenarios, data receiving unit 120 and/or data source 230 may acquire measurements in  system cost domains separately than the measurements in system performance domains. Such separately acquired system cost data and system performance data may need to be merged and associated together to enable the further analysis. For example, a system cost measurement may need to be linked, e.g., on sampling point, to a system performance measurement to be analyzed together. In other scenarios, data receiving unit 120 may receive system performance measurements and system cost measurements from multiple data sources with different formats and time scales. Such data from different data sources need to be processed and merged into a dataset for further analysis.
Specifically, original system data receiving unit 122 may be configured to receive baseline system performance measurements of computing system 240 in a system performance domain (s) and baseline system cost measurements of computing system 240 in a system cost domain (s) including a target system cost domain (s) . As described herein, the baseline system performance measurements and the baseline system cost measurements may be obtained/acquired with respect to the computing system 240 operating without the corresponding target element 250.
Updated system data receiving unit 124 may be configured to receive comparison system performance measurements of computing system 240 in the same system performance domain and comparison system cost measurements of computing system 240 in the same system cost domain (s) including the target system cost domain (s) . As described herein, the comparison system performance measurements and the comparison system  cost measurements may be obtained/acquired with respect to the computing system 240 operating with the corresponding target element 250.
In an example, the acquiring of the baseline data and the comparison data is under a well controlled environment such that other than the inclusion/exclusion of target element 250, the rest of the computing system 240 and the operation thereof is controlled as identical.
Analysis unit 130 may be generally configured to analyze the received baseline data (baseline system performance and baseline system cost data) and comparison data (comparison system performance data and comparison system cost data) to evaluate a system impact of operating the identified target element 250 with the identified computing system 240. Specifically, original system analyzing unit 132 may be configured to generate a first analytical result using the baseline system performance measurements and the baseline system cost measurements under a data analysis scheme. Any data analysis scheme that analyzes the system performance measurements together with system cost measurements may be used and all are included in the disclosure.
For example, the data analysis scheme may include analyzing the (baseline) system performance measurements on a basis of per unit of the (baseline) system cost measurement, e.g., P/U as described herein as an illustrative example. For another example, the data analysis scheme may include a regression analysis between the (baseline) system performance measurements and the (baseline) system cost measurements.
Updated system analyzing unit 134 may be configured to generate a second analytical result using the comparison system performance  measurements and the comparison system cost measurements under the same data analysis scheme as used by the original system analyzing unit 134.
Comparison unit 136 may be configured to analyze the second analytical result in relation to the first analytical result to evaluate a system impact of operating the target element 250 with the computing system 240. Any analyzing approaches may be used and all are included in the disclosure. For example, in the case that a regression analysis scheme is used to obtain the first and second analytical results, P’ r /Pr may be used to evaluate the system impact of target element 250 on computing system 240. If P’ r /Pr > 1, it may be determined that the target element 250 has a positive impact on the performance of computing system 240 because more system performance metrics, e.g., throughput P, is generated when the computing system 240 operates with the target element 250 as compared to without the target element 250.
Target system cost domain determination unit 138 may be configured to select, among multiple system cost domains, a system cost domain to be a target system cost domain to analyze with the system performance domain. In an example, target system cost domain determination unit 138 may determine a limiting cost domain among the multiple system cost domains as the target system cost domain.
Control unit 140 may be configured to control the configuration and/or operation of computing system 240 based on the analysis result generated by analysis unit 130. For example, control unit 140 may control whether the identified computing system 240 operates with the identified target element 250 or not. Control unit 140 may also control computing system 240  operates with target element 250 in identified ranges of system cost metrics and/or system performance metrics, e.g., in the case that the system impact analysis is conducted on multiple ranges of system cost measurements/system performance measurements.
3. Example Processes
Referring to FIG. 3, an example operation process 300 of system 10 of FIG. 1 is illustrated. In example operation 310, system identification unit 110 may identify as an analysis object a computing system 240 and a corresponding target element 250 configured to operate with the computing system 240.
In example operation 320, original system data receiving unit 122 may receive baseline system performance measurements of identified computing system 240 in a system performance domain and baseline system cost measurements of identified computing system 240 in a system cost domain (s) including a target system cost domain (s) . The baseline system performance measurements and the baseline system cost measurements may be obtained/acquired with the identified computing system 240 operating without the corresponding (identified) target element 250.
In example operation 330, updated system data receiving unit 124 may receive comparison system performance measurements of identified computing system 240 in the same system performance domain and comparison system cost measurements of identified computing system 240 in the same system cost domain (s) including the same target system cost domain (s) . As described herein, the comparison system performance  measurements and the comparison system cost measurements may be obtained/acquired with the identified computing system 240 operating with the corresponding identified target element 250.
In example operation 340, original system analyzing unit 132 may generate a first analytical result using the baseline system performance measurements and the baseline system cost measurements of the identified computing system 240 under a data analysis scheme. Any data analysis scheme that analyzes the system performance measurements together with system cost measurements may be used and all are included in the disclosure.
For example, the data analysis scheme may include analyzing the (baseline) system performance measurements on a basis of per unit of the (baseline) system cost measurement, e.g., P/U as described herein as an illustrative example. For another example, the data analysis scheme may include a regression analysis between the (baseline) system performance measurements and the (baseline) system cost measurements.
In example operation 350, updated system analyzing unit 134 may generate a second analytical result using the comparison system performance measurements and the comparison system cost measurements of identified computing system 240 under the same data analysis scheme as used by the original system analyzing unit 134 in example operation 340. For example, in the case that the original system analyzing unit 134 uses the scheme of system performance metric per unit of system cost, e.g., P/U, updated system analyzing unit 134 uses the same scheme and obtains P’ /U’ .
Similarly, in a case that regression analysis is used in example operation 340, the same regression analysis is used in example operation 350.
FIG. 4 shows an example operation flow of the regression analysis in obtaining the first analytical result and the second analytical result. In example operation 410, a baseline regression equation is generated using the baseline system performance measurements and the baseline system cost measurements. For illustrative example, the baseline regression equation may be:
P＝ 1000 *U + 100
In example operation 420, a test value of one of system performance domain or system cost domain may be applied to the baseline regression equation to obtain a first result value of the other one of the system performance domain or the system cost domain as the first analytical result. For illustrative example, 50％CPU usage (U) is applied to the baseline equation and a resulted throughput value (Pr) 600 is obtained as the first analytical result.
In example operation 430, a comparison regression equation may be generated using comparison system performance measurements and comparison system cost measurements. For illustrative example, the comparison regression equation may be:
P’＝ 800 *U’ + 150
In example operation 440, the same test value (as in example operation 420) is applied to the comparison regression equation to obtain a second result value as the second analytical value. Following the illustrative example, the same 50％CPU usage (U’ ) is applied to the comparison regression equation and a resulted throughput metric (Pr’ ) 550 is obtained as the second analytical result.
Referring back to FIG. 3, in example operation 360, comparison unit 136 may analyze the second analytical result in relation to the first analytical result to evaluate a performance impact of the system update, here, i.e., operating the identified target element 250 with the identified computing system 240. Any analyzing approaches may be used and all are included in the disclosure. For example, in the case that regression analysis scheme is used to obtain the first and second analytical results, P’ r /Pr may be used to evaluate the performance impact of target element 250 on computing system 240. Here following the above illustrative example, Pr’ ＝ 550 and Pr ＝ 600, P r’ /Pr < 1. It may be determined that the target element 250 has a negative impact on the computing system 240 because less system performance metric, e.g., throughput P, is generated when computing system 240 operates with the target element 250 as compared to without the target element 250.
In example operation 370, control unit 140 may control the computing system 240 in system configuration and/or operation based on the analysis result generated by analysis unit 130. For example, control unit 140 may control whether the identified computing system 240 operates with the identified target element 250 or not. Control unit 140 may also control the identified computing system 240 operates with the identified target element 250 in specified ranges of system cost metrics and/or system performance metrics.
FIG. 5 shows an example process flow for determining a target system cost domain to be analyzed with the system performance domain. The selection of the target cost domain may be based on the baseline data (system performance measurements and system cost measurements) and/or may be based on the comparison data (system performance measurements and  system cost measurements) . In the description herein, baseline data is used in the selection of target system cost domain as an illustrative example.
In example operation 510, multiple regression equations are generated using baseline system cost measurements in each of multiple system cost domains (e.g., U, V, W) and the baseline system performance measurements in a same system performance domain (e.g., P) .
In example operation 520, a highest possible system performance value (P) is obtained through each of the multiple regression equations (e.g., for each of U, V, W) . The highest system performance value may be obtained based on the range of the corresponding system cost value, e.g., CPU usage U cannot be higher than 100％. The highest system performance value may also be obtained based on the specific regression equation, e.g., if the regression equation is not linear.
In example operation 530, the lowest one of the multiple highest possible system performance values obtained through the multiple regression equations is identified. For illustrative example, if the highest system performance value obtained through the regression equations for system cost domains U, V, W are Pu, Pv, Pw, respectively, the lowest one among Pu, Pv, Pw may be identified, e.g., Pu.
In example operation 540, the system cost domain corresponding to the lowest one of the multiple highest possible system performance values obtained through the multiple regression equations may be identified as the target system cost domain. Following the above illustrative example, CPU usage (U) , may be identified as the target system cost domain because Pu is the lowest one among Pu, Pv, Pw obtained through  the regression equations each between one of system cost domains U, V, W and system performance domain P.
The identified target system cost domain may be used to analyze, together with the system performance domain, the performance impact of identified target element 250 on identified computing system 240. It should be appreciated that the analysis of the system impact may be conducted for multiple system cost domains with the system performance domain, either separately or in combination. The system impact analysis may also be conducted for multiple system performance domains in relation to the same multiple system cost domain and/or different system cost domains. The examples provided in the disclosure are only for illustrative purposes and do not limit the scope of the disclosure.
The processes described above in association with FIGS. 3-5 can be implemented in hardware, software, or a combination thereof. In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types. In other embodiments, hardware components perform one or more of the operations. Such hardware components may include or be incorporated into processors, application-specific integrated circuits (ASICs) , programmable circuits such as field programmable gate arrays (FPGAs) , or in other ways. The order in which the operations are described is not intended to  be construed as a limitation, and any number of the described operations can be combined in any order and/or in parallel to implement the processes.
The memory may include computer readable media such as a volatile memory, a Random Access Memory (RAM) , and/or non-volatile memory, e.g.， Read-Only Memory (ROM) or flash RAM, and so on. The memory is an example of a computer readable medium.
Computer readable media include non-volatile, volatile, mobile and non-mobile media, and can implement information storage through any method or technology. The information may be computer readable instructions, data structures, program modules or other data. Examples of storage media of a computer include, but not limited to, Phase-change RAMs (PRAMs) , Static RAMs (SRAMs) , Dynamic RAMs (DRAMs) , other types of RAMs, ROMs, Electrically Erasable Programmable Read-Only Memories (EEPROMs) , flash memories or other memory technologies, Compact Disk Read-Only Memories (CD-ROMs) , Digital Versatile Discs (DVDs) or other optical memories, cassettes, cassette and disk memories or other magnetic memory devices or any other non-transmission media, and can be used for storing information accessible to the computation device. According to the definitions herein, the computer readable media exclude transitory media, such as modulated data signals and carriers.
It should be further noted that, the terms "include" , "comprise" , or any variants thereof are intended to cover a non-exclusive inclusion, such that a process, a method, a product, or a device that includes a series of elements not only includes such elements but also includes other elements not specified expressly, or may further include inherent elements of the process, method,  product, or device. In the absence of more restrictions, an element limited by "include a/an…" does not exclude other same elements existing in the process, method, product, or device that includes the element.
Described above are merely the examples of the present application, which are not used to limit the present application. For those skilled in the art, the present application may have various alterations and changes. Any modification, equivalent replacement, improvement, and the like made within the spirit and principle of the present application shall be included in the scope of the claims of the present application.
The disclosure may be appreciated with the following clauses:
Clause 1: a computing system, comprising: a processing unit； a memory containing computer executable instructions, which when executed by the processing unit, configured the processing unit to implement a system configuration controller, the system configuration controller being operable to: identifying an application operable in the computing system, and a target element configured to operate with the application； acquire a first system performance measurement of the computing system in a system performance domain and a first system cost measurement of the computing system in a target system cost domain, the first system performance measurement and the first system cost measurement being obtained with the application being operated in the computing system without the target element； acquire a second system performance measurement of the computing system in the system performance domain and a second system cost measurement of the computing system in the target system cost domain, the second system performance measurement and the second system cost measurement being obtained with  the application operated in the computing system with the target element； generate a first analytical result using the first system performance measurement and the first system cost measurement under a data analysis scheme； generate a second analytical result using the second system performance measurement and the second system cost measurement under the same data analysis scheme； and analyze the second analytical result in relation to the first analytical result to evaluate an impact of operating the target element with the application in the computing system.
Clause 2: the computing system of clause 1, wherein the data analysis scheme includes analyzing a system performance measurement on a basis of per unit of a system cost measurement.
Clause 3: the computing system of clause 1, wherein the data analysis scheme includes a regression analysis between a system performance measurement and a system cost measurement.
Clause 4: the computing system of clause 3, wherein the generating the first analytical result includes: generating a first regression equation using the first system performance measurement and the first system cost measurement, and applying a test value of one of the system performance domain or the system cost domain into the first regression equation to obtain a first result value of other one of the system performance domain or the system cost domain as the first analytical result, and the generating the second analytical result includes: generating a second regression equation using the second system performance measurement and the second system cost measurement, and applying the test value into the second regression equation to obtain a second result value as the second analytical result.
Clause 5: the computing system of clause 1, wherein the first analytical result and the second analytical result are generated using a highest possible value in at least one of the first system cost measurement or the second system cost measurement.
Clause 6: the computing system of clause 1, wherein system configuration controller is further operable to merge a measurement in the system performance domain and a measurement in the system cost domain into a dataset using a same time scale.
Clause 7: the computing system of clause 1, wherein system configuration controller is further operable to control at least one of a configuration or an operation of the application with respect to the target element.
Clause 8: a method, comprising: identifying a computing system and a target element configured to operate with the computing system； receiving a first system performance measurement of the computing system in a system performance domain and a first system cost measurement of the computing system in a target system cost domain, the first system performance measurement and the first system cost measurement being obtained with the computing system operating without the target element； receiving a second system performance measurement of the computing system in the system performance domain and a second system cost measurement of the computing system in the target system cost domain, the second system performance measurement and the second system cost measurement being obtained with the computing system operating with the target element； generating a first analytical result using the first system performance measurement and the first  system cost measurement under a data analysis scheme； generating a second analytical result using the second system performance measurement and the second system cost measurement under the same data analysis scheme； and analyzing the second analytical result in relation to the first analytical result to evaluate an impact of operating the target element with the computing system.
Clause 9: the method of clause 8, wherein the data analysis scheme includes analyzing a system performance measurement on a basis of per unit of a system cost measurement.
Clause 10: the method of clause 9, wherein an average value of the system performance measurement and an average value of the system cost measurement are used in calculating the analyzing.
Clause 11: the method of clause 8, wherein the data analysis scheme includes a regression analysis between a system performance measurement and a system cost measurement.
Clause 12: the method of clause 11, wherein the generating the first analytical result includes: generating a first regression equation using the first system performance measurement and the first system cost measurement, and applying a test value of one of the system performance domain or the system cost domain into the first regression equation to obtain a first result value of other one of the system performance domain or the system cost domain as the first analytical result, and the generating the second analytical result includes: generating a second regression equation using the second system performance measurement and the second system cost measurement, and applying the test value into the second regression equation to obtain a second result value as the second analytical result.
Clause 13: the method of clause 12, wherein, the generating the first regression equation includes generating multiple regression equations, each by using a first system cost measurement in one of multiple system cost domains and the first system performance measurement, determining a limiting cost domain among the multiple system cost domains by analyzing maximum system performance values obtained through each of the multiple regressions, and using the limiting cost domain as the target system cost domain.
Clause 14: the method of clause 8, wherein the first analytical result and the second analytical result are generated using a highest possible value in at least one of the system performance domain or the target system cost domain.
Clause 15: the method of clause 8, further comprising merging a measurement in the system performance domain and a measurement in the system cost domain into a dataset using a same time scale.
Clause 16: a method, comprising: receiving system performance measurements of a computing system in a system performance domain and system cost measurement in multiple system cost domains； generating multiple regression equations each between the system performance domain and one of the multiple system cost domains and using the system performance measurements and system cost measurements in the one of the multiple system cost domains； determining a highest possible system performance value for each of the multiple regression equations to obtain a pool of highest possible system performance values； identifying a lowest system performance value in the pool； and identifying a system cost domain associated with the lowest system performance value in the pool as a target system cost domain；  and analyzing system cost measurements in the target system cost domain with the system performance measurements to determine a performance of the computing system.
Clause 17: the method of clause 16, wherein the analyzing includes analyzing a system performance measurement on a basis of per unit of a system cost measurement in the target system cost domain.
Clause 18: the method of clause 16, wherein the analyzing includes generating the first analytical result includes: receiving a first system performance measurement of the computing system in the system performance domain and a first system cost measurement of the computing system in the target system cost domain, the first system performance measurement and the first system cost measurement being obtained with the computing system operating in an original configuration； receiving a second system performance measurement of the computing system in the system performance domain and a second system cost measurement of the computing system in the target system cost domain, the second system performance measurement and the second system cost measurement being obtained with the computing system operating in an updated configuration； generating a first analytical result using the first system performance measurement and the first system cost measurement under a data analysis scheme； generating a second analytical result using the second system performance measurement and the second system cost measurement under the same data analysis scheme； and analyzing the second analytical result in relation to the first analytical result to evaluate a system impact of the updated configuration of the computing system.
Clause 19: the method of clause 18, further comprising controlling a configuration of the computing system based on the evaluated system impact of the updated system.
Clause 20: the method of clause 18, wherein further comprising controlling an operation of the computing system in the updated configuration.

Claims (21)

  1. A computing system, comprising:
    a processing unit,
    a memory containing computer executable instructions, which when executed by the processing unit, configured the processing unit to implement a system configuration controller, the system configuration controller being operable to:
    identifying an application operable in the computing system, and a target element configured to operate with the application；
    acquire a first system performance measurement of the computing system in a system performance domain and a first system cost measurement of the computing system in a target system cost domain, the first system performance measurement and the first system cost measurement being obtained with the application being operated in the computing system without the target element；
    acquire a second system performance measurement of the computing system in the system performance domain and a second system cost measurement of the computing system in the target system cost domain, the second system performance measurement and the second system cost measurement being obtained with the application operated in the computing system with the target element；
    generate a first analytical result using the first system performance measurement and the first system cost measurement under a data analysis scheme；
    generate a second analytical result using the second system performance measurement and the second system cost measurement under the same data analysis scheme； and
    analyze the second analytical result in relation to the first analytical result to evaluate an impact of operating the target element with the application in the computing system.
  2. The computing system of claim 1, wherein the data analysis scheme includes analyzing a system performance measurement on a basis of per unit of a system cost measurement.
  3. The computing system of claim 1, wherein the data analysis scheme includes a regression analysis between a system performance measurement and a system cost measurement.
  4. The computing system of claim 3, wherein the generating the first analytical result includes:
    generating a first regression equation using the first system performance measurement and the first system cost measurement, and
    applying a test value of one of the system performance domain or the system cost domain into the first regression equation to obtain a first result value of other one of the system performance domain or the system cost domain as the first analytical result,
    and the generating the second analytical result includes:
    generating a second regression equation using the second system performance measurement and the second system cost measurement, and
    applying the test value into the second regression equation to obtain a second result value as the second analytical result.
  5. The computing system of claim 1, wherein the first analytical result and the second analytical result are generated using a highest possible value in at least one of the first system cost measurement or the second system cost measurement.
  6. The computing system of claim 1, wherein system configuration controller is further operable to merge a measurement in the system performance domain and a measurement in the system cost domain into a dataset using a same time scale.
  7. The computing system of claim 1, wherein system configuration controller is further operable to control at least one of a configuration or an operation of the application with respect to the target element.
  8. A method, comprising:
    identifying a computing system and a target element configured to operate with the computing system；
    receiving a first system performance measurement of the computing system in a system performance domain and a first system cost measurement of the computing system in a target system cost domain, the first system  performance measurement and the first system cost measurement being obtained with the computing system operating without the target element；
    receiving a second system performance measurement of the computing system in the system performance domain and a second system cost measurement of the computing system in the target system cost domain, the second system performance measurement and the second system cost measurement being obtained with the computing system operating with the target element；
    generating a first analytical result using the first system performance measurement and the first system cost measurement under a data analysis scheme；
    generating a second analytical result using the second system performance measurement and the second system cost measurement under the same data analysis scheme； and
    analyzing the second analytical result in relation to the first analytical result to evaluate an impact of operating the target element with the computing system.
  9. The method of claim 8, wherein the data analysis scheme includes analyzing a system performance measurement on a basis of per unit of a system cost measurement.
  10. The method of claim 9, wherein an average value of the system performance measurement and an average value of the system cost measurement are used in calculating the analyzing.
  11. The method of claim 8, wherein the data analysis scheme includes a regression analysis between a system performance measurement and a system cost measurement.
  12. The method of claim 11, wherein the generating the first analytical result includes:
    generating a first regression equation using the first system performance measurement and the first system cost measurement, and
    applying a test value of one of the system performance domain or the system cost domain into the first regression equation to obtain a first result value of other one of the system performance domain or the system cost domain as the first analytical result,
    and the generating the second analytical result includes:
    generating a second regression equation using the second system performance measurement and the second system cost measurement, and
    applying the test value into the second regression equation to obtain a second result value as the second analytical result.
  13. The method of claim 12, wherein,
    the generating the first regression equation includes generating multiple regression equations, each by using a first system cost measurement in one of multiple system cost domains and the first system performance measurement,
    determining a limiting cost domain among the multiple system cost domains by analyzing maximum system performance values obtained through each of the multiple regressions, and
    using the limiting cost domain as the target system cost domain.
  14. The method of claim 8, wherein the first analytical result and the second analytical result are generated using a highest possible value in at least one of the system performance domain or the target system cost domain.
  15. The method of claim 8, further comprising merging a measurement in the system performance domain and a measurement in the system cost domain into a dataset using a same time scale.
  16. A method, comprising:
    receiving system performance measurements of a computing system in a system performance domain and system cost measurement in multiple system cost domains；
    generating multiple regression equations each between the system performance domain and one of the multiple system cost domains and using the system performance measurements and system cost measurements in the one of the multiple system cost domains；
    determining a highest possible system performance value for each of the multiple regression equations to obtain a pool of highest possible system performance values；
    identifying a lowest system performance value in the pool； and
    identifying a system cost domain associated with the lowest system performance value in the pool as a target system cost domain； and
    analyzing system cost measurements in the target system cost domain with the system performance measurements to determine a performance of the computing system.
  17. The method of claim 16, wherein the analyzing includes analyzing a system performance measurement on a basis of per unit of a system cost measurement in the target system cost domain.
  18. The method of claim 16, wherein the analyzing includes generating the first analytical result includes:
    receiving a first system performance measurement of the computing system in the system performance domain and a first system cost measurement of the computing system in the target system cost domain, the first system performance measurement and the first system cost measurement being obtained with the computing system operating in an original configuration；
    receiving a second system performance measurement of the computing system in the system performance domain and a second system cost measurement of the computing system in the target system cost domain, the second system performance measurement and the second system cost measurement being obtained with the computing system operating in an updated configuration；
    generating a first analytical result using the first system performance measurement and the first system cost measurement under a data analysis scheme；
    generating a second analytical result using the second system performance measurement and the second system cost measurement under the same data analysis scheme； and
    analyzing the second analytical result in relation to the first analytical result to evaluate a system impact of the updated configuration of the computing system.
  19. The method of claim 18, further comprising controlling a configuration of the computing system based on the evaluated system impact of the updated system.
  20. The method of claim 18, wherein further comprising controlling an operation of the computing system in the updated configuration.
  21. A method, comprising:
    identifying a computing system and a target element configured to operate with the computing system；
    receiving a first system performance measurement of the computing system in a system performance domain and a first system cost measurement of the computing system in a system cost domain, the first system performance measurement and the first system cost measurement being obtained with the computing system operating without the target element；
    receiving a second system performance measurement of the computing system in the system performance domain and a second system cost measurement of the computing system in the system cost domain, the second system performance measurement and the second system cost measurement being obtained with the computing system operating with the target element；
    generating a first analytical result using the first system performance measurement and the first system cost measurement under a data analysis scheme；
    generating a second analytical result using the second system performance measurement and the second system cost measurement under the same data analysis scheme； and
    analyzing the second analytical result in relation to the first analytical result to evaluate an impact of operating the target element with the computing system.
PCT/CN2017/088617 2017-06-16 2017-06-16 Analyzing performance impact of system update WO2018227548A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
CN201780091959.2A CN110998539B (en) 2017-06-16 2017-06-16 Performance impact analysis of system updates
PCT/CN2017/088617 WO2018227548A1 (en) 2017-06-16 2017-06-16 Analyzing performance impact of system update

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2017/088617 WO2018227548A1 (en) 2017-06-16 2017-06-16 Analyzing performance impact of system update

Publications (1)

Publication Number Publication Date
WO2018227548A1 true WO2018227548A1 (en) 2018-12-20

Family

ID=64659476

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/088617 WO2018227548A1 (en) 2017-06-16 2017-06-16 Analyzing performance impact of system update

Country Status (2)

Country Link
CN (1) CN110998539B (en)
WO (1) WO2018227548A1 (en)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140282422A1 (en) * 2013-03-12 2014-09-18 Netflix, Inc. Using canary instances for software analysis
CN105468510A (en) * 2014-09-05 2016-04-06 北京畅游天下网络技术有限公司 Method and system for evaluating and tracking software quality
CN105653303A (en) * 2014-11-10 2016-06-08 阿里巴巴集团控股有限公司 Application installation control method and device
US20160224453A1 (en) * 2015-01-30 2016-08-04 Lindedln Corporation Monitoring the quality of software systems

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7793269B2 (en) * 2005-02-15 2010-09-07 Ebay Inc. Parallel software testing based on a normalized configuration
CN102025548B (en) * 2010-12-20 2014-12-10 中兴通讯股份有限公司 Multi-version smooth upgrading method and device for user-defined performance indexes
US9268663B1 (en) * 2012-04-12 2016-02-23 Amazon Technologies, Inc. Software testing analysis and control
CN104102668A (en) * 2013-04-12 2014-10-15 袁晓东 Automatic database performance analysis and early warning system
CN104182333B (en) * 2013-05-23 2018-03-09 阿里巴巴集团控股有限公司 A kind of performance test methods and equipment
US9225621B2 (en) * 2013-06-25 2015-12-29 Netflix, Inc. Progressive deployment and termination of canary instances for software analysis
US9594665B2 (en) * 2014-03-05 2017-03-14 Microsoft Technology Licensing, Llc Regression evaluation using behavior models of software applications
US9886262B2 (en) * 2015-03-16 2018-02-06 Microsoft Technology Licensing, Llc Adaptive upgrade to computing systems
CN106453549B (en) * 2016-10-08 2019-09-13 中国工商银行股份有限公司 A kind of distributed software version dissemination method and system

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20140282422A1 (en) * 2013-03-12 2014-09-18 Netflix, Inc. Using canary instances for software analysis
CN105468510A (en) * 2014-09-05 2016-04-06 北京畅游天下网络技术有限公司 Method and system for evaluating and tracking software quality
CN105653303A (en) * 2014-11-10 2016-06-08 阿里巴巴集团控股有限公司 Application installation control method and device
US20160224453A1 (en) * 2015-01-30 2016-08-04 Lindedln Corporation Monitoring the quality of software systems

Also Published As

Publication number Publication date
CN110998539A (en) 2020-04-10
CN110998539B (en) 2023-06-13

Similar Documents

Publication Publication Date Title
Pein et al. Heterogeneous change point inference
US11003793B2 (en) Identification of optimal data utility-preserving anonymization techniques by evaluation of a plurality of anonymization techniques on sample data sets that correspond to different anonymization categories
Mehmani et al. Predictive quantification of surrogate model fidelity based on modal variations with sample density
US9967715B2 (en) Determining location of a user device
Yin et al. Optimal linear combinations of multiple diagnostic biomarkers based on Youden index
US20080306903A1 (en) Cardinality estimation in database systems using sample views
US10067746B1 (en) Approximate random number generator by empirical cumulative distribution function
US20210049482A1 (en) State dependent calibration of qubit measurements
US10496514B2 (en) System and method for parallel processing prediction
Wang et al. Bayesian service demand estimation using gibbs sampling
Cremonesi et al. Indirect estimation of service demands in the presence of structural changes
Rana et al. Machine learning approach for quality assessment and prediction in large software organizations
US11636386B2 (en) Determining data representative of bias within a model
Goldreich The uniform distribution is complete with respect to testing identity to a fixed distribution
De Iaco et al. Using simultaneous diagonalization to identify a space–time linear coregionalization model
US10248462B2 (en) Management server which constructs a request load model for an object system, load estimation method thereof and storage medium for storing program
Wang et al. A multilevel simulation optimization approach for quantile functions
Hunold et al. Automatic verification of self-consistent MPI performance guidelines
WO2018227548A1 (en) Analyzing performance impact of system update
US9986450B2 (en) Resource consumption estimation with out of norm determination in mobile devices
Tandon et al. Kernel ridge regression via partitioning
Moosbauer et al. Towards explaining hyperparameter optimization via partial dependence plots
Das et al. Analysis of collinear data: multicollinearity
Lopez-Benitez et al. Accurate estimation of the minimum primary channel activity time in cognitive radio based on periodic spectrum sensing observations
US10521274B2 (en) Systems and methods relating to host configuration optimization in a data center environment

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17913798

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 17913798

Country of ref document: EP

Kind code of ref document: A1