JP6530462B2 - Determining and monitoring the performance capabilities of computer resource services - Google Patents

Determining and monitoring the performance capabilities of computer resource services Download PDF

Info

Publication number
JP6530462B2
JP6530462B2 JP2017194858A JP2017194858A JP6530462B2 JP 6530462 B2 JP6530462 B2 JP 6530462B2 JP 2017194858 A JP2017194858 A JP 2017194858A JP 2017194858 A JP2017194858 A JP 2017194858A JP 6530462 B2 JP6530462 B2 JP 6530462B2
Authority
JP
Japan
Prior art keywords
computer
performance
computer system
system
resource service
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
JP2017194858A
Other languages
Japanese (ja)
Other versions
JP2018022520A (en
Inventor
エリシャ,サイモン,ジェレミー
Original Assignee
アマゾン・テクノロジーズ・インコーポレーテッド
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority to US13/899,407 priority Critical patent/US9384115B2/en
Priority to US13/899,407 priority
Priority to US13/899,449 priority patent/US9584364B2/en
Priority to US13/899,449 priority
Application filed by アマゾン・テクノロジーズ・インコーポレーテッド filed Critical アマゾン・テクノロジーズ・インコーポレーテッド
Publication of JP2018022520A publication Critical patent/JP2018022520A/en
Application granted granted Critical
Publication of JP6530462B2 publication Critical patent/JP6530462B2/en
Application status is Active legal-status Critical
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance or administration or management of packet switching networks
    • H04L41/08Configuration management of network or network elements
    • H04L41/0803Configuration setting of network or network elements
    • H04L41/0806Configuration setting of network or network elements for initial configuration or provisioning
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3409Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3409Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
    • G06F11/3428Benchmarking
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3452Performance evaluation by statistical analysis
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3466Performance evaluation by tracing or monitoring
    • G06F11/3495Performance evaluation by tracing or monitoring for systems
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L41/00Arrangements for maintenance or administration or management of packet switching networks
    • H04L41/50Network service management, i.e. ensuring proper service fulfillment according to an agreement or contract between two parties, e.g. between an IT-provider and a customer
    • H04L41/5038Testing of service level quality, e.g. simulating service usage
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing packet switching networks
    • H04L43/08Monitoring based on specific metrics
    • H04L43/0805Availability
    • H04L43/0817Availability functioning
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing packet switching networks
    • H04L43/50Testing arrangements
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/30Monitoring
    • G06F11/34Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
    • G06F11/3409Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
    • G06F11/3433Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment for load management
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F9/00Arrangements for program control, e.g. control units
    • G06F9/06Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
    • G06F9/44Arrangements for executing specific programs
    • G06F9/455Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
    • G06F9/45533Hypervisors; Virtual machine monitors
    • G06F9/45558Hypervisor-specific management and integration aspects
    • G06F2009/45591Monitoring or debugging support
    • GPHYSICS
    • G06COMPUTING; CALCULATING; COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F2201/00Indexing scheme relating to error detection, to error correction, and to monitoring
    • G06F2201/815Virtual
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04LTRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
    • H04L43/00Arrangements for monitoring or testing packet switching networks
    • H04L43/08Monitoring based on specific metrics
    • H04L43/0876Network utilization

Description

Cross-reference to related applications

  This application is related to US Patent Application No. 13 / 899,449, filed May 21, 2013, entitled "REPORTING PERFORMANCE CAPABILITIES OF A COMPUTER RESOURCE SERVICE" to Simon Jeremy Elisha, the disclosure of which is incorporated by reference in its entirety. Is incorporated herein by reference.

  Network accessible computing services, sometimes referred to now as cloud computing services or remote computing services, offer customers flexibility in their choice in computing. When a customer creates a new virtual machine instance within a computer resource service or migrates an instance to a computer resource service, the customer may assess the resource characteristics of the instance running in the computer resource service. This may include, among other possibilities, processor type and performance, memory size and performance, input / output capabilities, temporary storage size, and network capabilities.

  Each instance type can have different performance profiles, providing the customer with a degree of choice. However, due to various inconsistencies, selecting between various instance types for performance at a given time and place (eg, a particular data center) can be difficult. For example, an instance may be hosted on various underlying physical hardware, which may alter the actual performance of the instance (e.g. older hardware may be replaced by newer hardware) May indicate various performances). Also, instances implemented in "crowded" locations (e.g. noisy neighbors) within computer resource services may provide reduced performance over areas with fewer users. Thus, understanding the performance profile within an instance of a computer resource service can be difficult.

  The various features of the implementation will be more fully appreciated and better understood with reference to the following detailed description of the implementation, when considered in conjunction with the accompanying drawings.

8 illustrates an example environment in which a resource monitoring tool can monitor computer resource services in accordance with various implementations. 7 illustrates an example process for monitoring the performance of computer resource services in accordance with various implementations. 8 illustrates another example of an environment in which a resource monitoring tool can monitor computer resource services in accordance with various implementations. 8 illustrates another example of an environment in which a resource monitoring tool can monitor computer resource services in accordance with various implementations. 7 illustrates another example of a process for monitoring the performance of computer resource services in accordance with various implementations. 8 illustrates an example data structure for storing performance metrics of computer resource services in accordance with various implementations. 5 illustrates an example of a hardware configuration of a computing device in accordance with various implementations.

  Implementations of the present disclosure relate to systems and methods for determining and monitoring the performance of computer resource services. A resource monitoring tool can start a test virtual machine on a computer system of the computer resource service to determine and monitor the performance of the computer resource service in real time. The resource monitoring tool can then monitor various metrics that show the performance of the test virtual machine over time, such as processor performance, memory performance, input / output (I / O) performance, and network performance. The resource monitoring tool may store the monitored performance metrics, provide the selected performance metrics to the user, and use the performance metrics to manage computer resource services.

  According to an implementation, to accurately measure performance, the resource monitoring tool can select a computer system representative of the computer resource service and the computer resource of the computer resource service for testing. For example, if the computer system is distributed at different locations, the resource monitoring tool can select computer systems at different locations so that performance at each location can be monitored and compared. Similarly, where the computer system includes, for example, various computing resources (processors of various types and generations, memories, storage devices, etc.), the resource monitoring tool represents a computer system that represents various computing resources. It can be selected. Thus, the resource monitoring tool can determine and compare the performance of differently located and configured computer systems.

  In implementations, performance metrics may be utilized by computer resource services to manage resources of computer resource services. For example, when allocating computing resources to an instance, the computer resource service utilizes, in real time, performance metrics to select computing resources that are optimal for the user's demand, and to balance the current utilization of computing resources. It is possible to select resources that are currently available or not fully utilized. Similarly, computer resource services can utilize performance metrics to determine when a computer system should be added to a service or when an existing computer system should be updated. Thus, the performance metrics provided by the resource monitoring tool can enable computer resource services to manage the provided computer resources efficiently and effectively. Similarly, performance metrics may be provided to the user to enable the user to make an accurate determination of when to select the configuration of computing resources and instances in the computer resource service.

  FIG. 1 illustrates an example of a resource monitoring tool 100 capable of determining and monitoring a performance computer resource service 102 in accordance with various implementations. While FIG. 1 illustrates the various components contained within the resource monitoring tool 100, FIG. 1 illustrates one example of the monitoring tool, additional components may be added and existing components may be added. It can be removed. Similarly, while FIG. 1 illustrates various components included within computer resource service 102, FIG. 1 illustrates an example of computer resource service, and additional components may be added and existing Components of can be removed.

  In implementations, resource monitoring tool 100 may be configured to determine and monitor the performance of computer resource service 102. Computer resource service 102 may be configured to deliver computing resources as a service to users over one or more networks. For example, the user may be able to set up computing resources, for example on an hourly basis, by the number of uses. As illustrated in FIG. 1, computer resource service 102 may include a number of computer systems, eg, set of computer systems 104, set of computer systems 106, and set of computer systems. In order to provide computer resources as a service, computer resource service 102 may be configured to execute one or more machine instances (MIs) on any of computer system 104, computer system 106, and computer system 108.

  The MI may correspond to one or more of computer system 104, computer system 106, or computer system 108. Similarly, the MI may correspond to a virtual machine (VM) hosted by one or more of computer system 104, computer system 106, and computer system 108. The VM may be a virtualized computer system, or a software implementation of the computer system layered on any of computer system 104, computer system 106, and computer system 108. The VM's access to the underlying computer system can be controlled through a hypervisor or virtual machine monitor. The VM can provide multiple and / or different operating system environments for launching simultaneously on a single computer system.

  In the computer resource service 102, each MI can be controlled by different users, they can have administrator access only to their own MI, and can not have access to instances of other users . Multiple MIs can be run simultaneously on computer systems that include parallel processors, although multiple instances appear to run simultaneously on multithreaded computer systems with fewer processors than instances. In some cases, the various MIs executing on the computer system are controlled by two or more different users, and in all other cases, all of the MIs are controlled by a single user.

  Computer resource service 102 may also include one or more management systems 111. One or more management systems 111 may be configured to control the operation and configuration of computer resource service 102. One or more management systems 111 initiate execution of the MI on computer system 104, computer system 106, and computer system 108, configure the MI in computer resource service 102, computer system 104, computer system 106, and The MI running on computer system 108 may be terminated and configured to migrate existing MI between computer system 104, computer system 106, and computer system 108. One or more management systems 111 may also be configured to provide an interface to users requesting computer services from computer resource service 102. One or more management systems 111 may be implemented as a physical computer system, such as one or more of computer system 104, computer system 106, and computer system 108. Similarly, one or more management systems 111 may be implemented as one or more MIs executing within computer resource service 102.

  In an implementation, resource monitoring tool 100 is configured to determine and monitor the performance of computer resources provided by computer resource service 102 by testing the operation of computer system 104, computer system 106, and computer system 108. can do. In operation, the performance of computer system 104, computer system 106, and computer system 108 is the number of MIs running on the computer system, the amount of MIs running on the computer system, the physics supporting the computer system. Hardware and software, as well as other similar factors. To test the operation of computer system 104, computer system 106, and computer system 108, resource monitoring tool 100 may be configured to utilize one or more test VMs 110. Resource monitoring tool 100 may be configured to start test VM 110 on computer system 104, computer system 106, and computer system 108 to test the performance of test VM 110.

  To accurately represent the computer resource service 102, the resource monitoring tool 100 can be configured to select the number and placement of test VMs 110 based on the configuration of the computer resource service 102. In computer resource service 102, computer system 104, computer system 106, and computer system 108 may be located in various places. The location of computer system 104, computer system 106, and computer system 108 may include any granularity of isolation within computer resource service 102. For example, the location may include a particular computer system in a data center, various servers in a server rack, various data centers, various geographic locations or regions, etc. Thus, to test various locations, resource monitoring tool 100 may be configured to select and test several computer systems from each location of computer system 104, computer system 106, and computer system 108. it can.

  Similarly, computer systems at various locations can present various computer resources. For example, computer system 104, computer system 106, and computer system 108 may include various configurations of hardware and / or software resources. Also, computer system 104, computer system 106, and computer system 108 may include various generations of hardware and / or software resources. For example, one or more of computer system 104, computer system 106, and computer system 108 may include older computer systems, which typically use older hardware and software resources, Provide reduced computing resources. Thus, the resource monitoring tool 100 can be configured to select from each location a number of computer systems that are representative of different configurations and generations.

  When selecting the number of computer systems, the resource monitoring tool 100 selects a sufficient number of computer systems from each of various locations, different configurations, different generations, etc. to accurately represent the computer system. can do. That is, the resource monitoring tool 100 can be configured to select a statistically insignificant computer system from each of various places, various configurations, various generations, and the like. For example, with respect to a particular location, configuration, or generation, resource monitoring tool 100 may select a number of computer systems that statistically represent all computer systems of that particular location, configuration, or generation. it can. The number of computer systems can be a percentage (e.g., 10%, 5%, 1%, etc.) of the total number of computer systems in a particular location, configuration, or generation. When selecting a computer system from a particular location, configuration, or generation, the resource monitoring tool 100 can randomly select a computer system from that particular location, configuration, or generation.

  In implementations, when testing computer resources, the resource monitoring tool 100 can be configured to utilize various configurations of the test VM 110. Specifically, the resource monitoring tool 100 can select a test VM 110 that utilizes various amounts of computer resources provided by the computer resource service 102. The resource monitoring tool 100 can be configured to select a pre-configured test VM 110 presented by the computer resource service 102. Similarly, the resource monitoring tool 100 can be configured to determine various configurations of the test VM 110 that represent various levels of computer resources.

  In an implementation, resource monitoring tool 100 is a software program that may be stored on or executed by a computer system as part of computer resource service 102 or external to computer resource service 102. It can be configured. Similarly, resource monitoring tool 100 can be configured as a software module that is part of another application program. In any of the examples, the resource monitoring tool 100 should be written in various programming languages to correspond to various operating systems such as JAVA, C ++, Python code, Visual Basic, HTML, XML, etc., computer system architecture, etc. Can. As illustrated in FIG. 1, the resource monitoring tool 100 can include a monitoring module 112, performance data storage 114, and an interface 116. Although illustrated as part of a single application program, any of the components of the resource monitoring tool 100 can be implemented as separate software programs or modules.

  In an implementation, the monitoring module 112 may be configured to test the performance of the test VM 110 and monitor the performance metrics of the test VM 110. When test VM 110 executes within computer resource service 102, monitoring module 112 may be configured to test the performance of test VM 110. In particular, monitoring module 112 may be configured to perform one or more benchmark tests that measure the performance of test VM 110. Benchmark tests include processor performance (eg, software execution, score based on instruction execution, etc.), memory performance (eg, latency, read speed, write speed, etc.), input / output (I / O) performance (eg, List any kind of conventional tests that measure I / O per second, I / O bandwidth etc), network performance (eg network bandwidth, network packets per second etc) etc it can.

  From benchmark testing, the monitoring module 112 can be configured to collect performance metrics. Performance metrics include the results of benchmark tests. Once collected, resource monitoring tool 100 may be configured to store performance metrics in performance data store 114. Performance data store 114 can be implemented as one or more data structures that can store performance metrics. Performance data storage 114 may also store details of computer resource service 102, details of computer system 104, computer system 106, and computer system 108, details of test VM 110, details of performance testing, and the like.

  Performance data store 114 may be configured to store performance metrics and other data in a manner that links and correlates relevant information. For example, with respect to a particular set of performance metrics, performance data store 114 may identify the particular test VM 110 details that generated the particular performance metric set, the particular computer system details that the particular test VM 110 performed, the particular computer system Location, details of performance tests performed on a particular test VM 110, etc. can be associated.

  In an implementation, the resource monitoring tool 100 can be configured to use the monitoring agent to assist in the testing and monitoring of the test VM 110. The monitoring agent may be implemented as a software program that executes on the test VM 110 and is configured to perform one or more of the performance tests. To collect performance metrics, the monitoring agent can be configured to send performance to the resource monitoring tool 100. The monitoring agent may send performance metrics in real time, periodically, on demand, and combinations thereof. Similarly, the resource monitoring tool 100 can be configured to communicate with the monitoring agent, eg, via the monitoring module 112, to retrieve performance metrics from the monitoring agent. Resource monitoring tool 100 may retrieve performance metrics in real time, periodically, on demand, and combinations thereof.

  In addition to collecting performance metrics, resource monitoring tool 100 can be configured to perform statistical analysis on performance metrics. The resource monitoring tool 100 can perform statistical analysis to analyze and compare computer system performance, performance of various configurations of test VMs, and combinations thereof. Statistical analysis may include, for example, the performance of the computer system, the performance of the various configurations of the test VM, such as median, mean, standard deviation, statistical ranking based on relationships, statistical modeling, and combinations thereof. It may be any kind of procedure that generates statistics useful for analysis and comparison. Once determined, the resource monitoring tool 100 can store statistics in the performance data store 114.

  For example, resource monitoring tool 100 can compare over time performance metrics of particular configurations of test VMs running on a computer system. For example, the resource monitoring tool 100 can determine the median, mean, and standard deviation of performance metrics collected over time to assess performance over time. In addition, for example, the resource monitoring tool 100 can determine a particular time in which the performance metrics fall within a certain percentage of the performance metrics at other times.

  Similarly, for example, whether the resource monitoring tool 100 is at the same location, at various locations, or both, of the various computing systems hosting the particular configuration of the test VM Statistics on performance metrics can be determined. In this example, the resource monitoring tool 100 can compare statistics to determine the performance of the computing system relative to one another. In addition, the resource monitoring tool 100 determines statistics on performance metrics of different configurations of the test virtual machine, whether running on the same computer system, on different computer systems, or both. can do. The resource monitoring tool 100 can compare statistics to determine the performance of the various configurations of the test virtual machine relative to one another.

  The resource monitoring tool 100 may also include an interface 116 that allows the user device 120 to communicate with the resource monitoring tool 100 via one or more networks 118 and to access performance metrics. The user device 120 may include a desktop, a laptop, a computer device such as a server, or a mobile device such as a smartphone, a tablet computer, a mobile phone, or a personal digital assistant. The user device 120 includes hardware resources (eg, processor, memory, storage, network interface, I / O device, etc.) and software resources (eg, operating system, application program, mobile application) found in a conventional computer device. Etc.).

  In implementations, interface 116 may be any type of interface that allows the user to communicate locally and remotely with resource monitoring tool 100. For example, the interface 116 includes a graphical user interface ("GUI") or command line interface that allows the user to interact with the resource monitoring tool through the computer system on which the resource monitoring tool 100 is running. be able to. Also, for example, interface 116 may include a web-based interface that allows user device 120 to remotely communicate with resource monitoring tool 100 using a web-based application such as a web browser. Similarly, for example, interface 116 may include an application programming interface (“API”) that allows applications on user device 120 to communicate with resource monitoring tool 100. In some implementations, the interface may be a web service API.

  In an implementation, the user of user device 120 may desire to assess the performance of computer resource service 102. For example, the user may wish to determine the best place to start within the computer resource service, computer system, and configuration of the MI. To achieve this, the user device 120 can send a request to the resource monitoring tool 100 to provide a set of performance metrics. In response, resource monitoring tool 100 may be configured to search performance data store 114 and retrieve a set of performance metrics that match the requirements of user device 120. The resource monitoring tool 100 can be configured to send a set of performance metrics to the user device 120.

  In order to provide meaningful information to user device 120, resource monitoring tool 100 may be configured to utilize filters when determining and retrieving a set of performance metrics for user device 120. In particular, the resource monitoring tool 100 can utilize filters to determine specific performance metrics to retrieve for user devices. The filter may include one or more parameters that specify which performance metrics are desired by the user device 120.

  For example, the filter may include one or more parameters for a computer system in computer resource service 102. These can include identification of the particular location or region where the computer system is located, identification of the particular configuration of the computer system, etc. In addition, for example, the filter may include one or more parameters regarding the test VM 110 or a test performed on the test VM 110. These parameters may include identification of a particular type or configuration of test VM 110, identification of where or where test VM 110 was launched, identification of a benchmark test, date or time range on which the test was performed, etc. Also, for example, the filter may include one or more parameters around the performance metric itself. These parameters may include specific types of performance metrics (eg, processor performance, memory performance, I / O performance, network performance, etc.), specific values for performance metrics, specific statistics for performance metrics (eg, average values, etc.) For example, the median, a certain percentile, etc. can be mentioned.

  A complete description of interface 116 and the processes associated with interface 116 may be found in US patent application Ser. No. 13/899, filed May 21, 2013, entitled “REPORTING PERFORMANCE CAPABILITIES OF A COMPUTER RESOURCE SERVICE” to Simon Jeremy Elisha. , 449, the disclosure of which is incorporated herein by reference in its entirety.

  FIG. 2 illustrates a process 200 for determining and monitoring the performance of computer resource services in accordance with various implementations. In the process 200, the illustrated steps are examples, any of the illustrated steps can be removed, additional steps can be added, and the order of the illustrated steps can be changed. Additionally, although process 200 is described as implemented by resource monitoring tool 100, one or more of the stages of process 200 can be implemented by any hardware or software component.

  At 204, the resource monitoring tool 100 can optionally determine the computer system within the computer resource service 102. The resource monitoring tool 100 may determine identification information regarding the computer system 104, computer system 106, and computer system 108, such as the location of the computer system, the name of the computer system, the network address of the computer system, access information regarding the computer system, etc. it can. In addition, resource monitoring tool 100 can determine details of computer system 104, computer system 106, and computer system 108, such as computer system hardware and software components. To determine this information, resource monitoring tool 100 can scan computer system 104, computer system 106, and computer system 108. Similarly, the resource monitoring tool 100 can communicate with one or more management systems 111 of the computer resource service 102 to obtain information. Additionally, the location and configuration of the computer system can also be preprogrammed into the resource monitoring tool 100.

  At 206, the resource monitoring tool 100 can determine the location distribution of the computer system and the set of computer systems that are representative of the computing resources of the computer system. That is, the resource monitoring tool 100 can select a computer system to represent various locations of the computer system and various computer resource configurations of the computer system. The number of computer systems selected may be statistically significant to be able to determine an accurate measure of performance.

  For example, if computer system 104 is located at a first location and computer system 106 is located at a second location, resource monitoring tool 100 may include a number of computer systems 104 and a number of computer systems representative of each location. Computer system 106 may be selected to provide reliable performance metrics, such as, for example, 5% of computer system 104 and 5% of computer system 106. Similarly, in this example, computer system 104 may have two computer resource configurations, eg, a first group of computer systems 104 having the same processor, memory, and network bandwidth, The second group of systems 104 has the same processor, memory, and network bandwidth but with various things from the first group. When selecting a computer system from computer system 104, resource monitoring tool 100 can select a statistically significant number from both the first group and the second group.

  At 208, the resource monitoring tool 100 can start a test VM on a set of computer systems. The resource monitoring tool 100 can select the same configuration for each of the test VMs so that performance can be compared and correlated between computer systems. For example, the resource monitoring tool 100 can select a test VM 100 configured to present particular computer resources (eg, processing power, memory, I / O resources, network bandwidth, etc.). The resource monitoring tool 100 can configure the test VM 110. Similarly, the resource monitoring tool 100 can select a pre-configured virtual machine to start on the computer system.

  At 210, the resource monitoring tool 100 can monitor performance metrics of test VMs on a set of computer systems. The resource monitoring tool 100 can remotely monitor performance metrics via a network connection. Similarly, resource monitoring tool 100 can install a performance monitoring agent on one or more of the test virtual machines and receive or retrieve performance metrics from the monitoring agent. The resource monitoring tool 100 can monitor performance metrics over time to determine performance metrics at various times.

  The resource monitoring tool 100 and / or the monitoring agent may perform various benchmark tests on the test VM 110. Benchmark tests include processor performance (eg, software execution, score based on instruction execution, etc.), memory performance (eg, latency, read speed, write speed, etc.), input / output (I / O) performance (eg, List any kind of conventional tests that measure I / O per second, I / O bandwidth etc), network performance (eg network bandwidth, network packets per second etc) etc it can.

  At 212, the resource monitoring tool 100 can store performance metrics to be monitored for the test VM. Resource monitoring tool 100 may store performance metrics in one or more data structures, such as performance data store 114. The resource monitoring tool 100 can store performance metrics and other information. Other information may include, for example, the information collected at 204.

  At 214, the resource monitoring tool 100 can repeat the process for various configurations of test VMs. In order to obtain an overview of computer resources of the computer resource service 102, the resource monitoring tool 100 may, for example, provide various computer resources (eg, processing power, memory, I / O resources, network bandwidth, etc.). The testing and monitoring of performance metrics for various configurations of the test VM 110 can be performed.

  At 216, the resource monitoring tool 100 can perform resource management and monitoring based on the performance metrics. The resource monitoring tool 100 can utilize performance metrics to assist in any process that requires the determination of the performance of computer resources of the computer resource service 102. Management and monitoring of resources can be implemented by the resource monitoring tool 100. Similarly, resource monitoring tool 100 can provide performance metrics to another system, such as one or more resource management systems 111, to assist in managing resources.

  For example, performance metrics can be used to determine the specific machine on which to launch a new MI. For example, if a particular computer system exhibits good performance metrics for a test VM, that particular computer system can be selected (or given higher priority) to host the new MI. Similarly, if a particular computer system shows poor performance metrics for a test VM, that particular computer system may pause (or give lower priority) for the new MI host it can.

  In addition, for example, performance metrics can be utilized to balance the currently hosted MI within computer resource service 102. For example, if a particular computer system shows poor performance metrics for a test VM, migrating one or more MIs hosted by that particular computer system to various computer systems that show better performance metrics it can.

  Also, for example, performance metrics can be used to price the provision of computer resources by computer resource service 102. For example, if a particular computer system exhibits poor performance metrics for a test VM and is associated with older hardware of that particular system, a lower price may be set by the computer resources provided by that particular computer system. Similarly, if, for example, a particular computer system exhibits good performance metrics for a test VM and is associated with the newer hardware of that particular system, the computer resources provided by that particular computer system will set a higher price. It can be done.

  Further, for example, performance metrics can be used to determine whether to upgrade one or more of the computer systems. For example, if a particular computer system exhibits poor performance metrics for a test VM and is associated with older hardware of that particular system, then a decision can be made to update the hardware of that particular computer system.

  At 218, the resource monitoring tool 100 can provide performance metrics to the requesting user. Resource monitoring tool 100 may receive a request from a user, such as user device 120, for a set of performance metrics. For example, resource monitoring tool 100 can receive a request via interface 116. The request may include one or more filters that specify a particular set of performance metrics required by the user. In response, the resource monitoring tool 100 can search the performance data store 114 based on the filter and retrieve a set of performance metrics. Resource monitoring tool 100 can provide a set of performance metrics to the user via interface 116.

  Once received, the user can perform various operations using the set of performance metrics. For example, user device 120 may perform operations associated with computer resource services. The user device 120 can utilize the performance metrics to assist in any process that requires the determination of the performance of computer resources of the computer resource service 102.

  For example, performance metrics can be utilized to determine the location of a particular computer system or computer system on which to launch a new MI. For example, if a particular computer system, or a particular location of a computer system, indicates a good performance metric for the test VM, the user device 120 initiates a new MI on a particular computer system or at a particular location (or Can request that start). Similarly, if, for example, the user device receives a set of performance metrics for various configurations of the test VM, the user device 120 sets that performance metric to select the configuration of the MI to start within the computer resource service Can be used. In addition, for example, performance metrics can be utilized to balance the currently hosted MI within computer resource service 102. For example, if a particular computer system exhibits poor performance metrics for a test VM, one or more MIs hosted by that particular computer system may migrate to various computer systems that exhibit better performance metrics. it can.

  After 218, the process can end, repeat, or return to any point.

  FIGS. 3A and 3B illustrate another example of a resource monitoring tool 100 capable of determining and monitoring a performance computer resource service 302 according to various implementations. While FIGS. 3A and 3B illustrate various components included within the resource monitoring tool 100, FIGS. 3A and 3B illustrate an example of the monitoring tool, and additional components may be added, existing Components of can be removed. Similarly, while FIGS. 3A and 3B illustrate various components contained within computer resource service 302, FIGS. 3A and 3B illustrate an example of computer resource service with additional components added. And existing components may be removed.

  In implementations, computer resource service 302 may provide computing resources as a service to users located at regional and geographically distinct locations. In providing services to geographically and geographically distinct locations, computer systems of computer resource service 302 can be distributed among various geographical and regional locations. As illustrated in FIG. 3A, computer resource service 302 can be divided into several geographic locations or areas, such as, for example, geographic location 1 304 and geographic location 2 306. That is, different computer systems supporting computer resource service 302 can be located at different geographic locations to support computer resources presented at different geographic locations.

  For example, computer resource service 302 may include several data centers, such as, for example, data center A 308, data center B 310, data center C 312, data center D 314, and data center E 316. As illustrated in FIG. 3A, data center A 308 and data center B 310 may be located at geographic location 1 304. Data center C 312, data center D 314, and data center E 316 may be located at geographic location 2 306. Data centers within a particular geographic location may also be distributed within the subregions of that particular geographic location.

  As illustrated in FIG. 3B, data center A may include several computer systems, such as computer system 318, computer system 320, and computer system 322. Similarly, data center B 310, data center C 312, data center D 314, and data center E 316 can include several computer systems. In order to determine and monitor the performance of the data, the resource monitoring tool 100 may be configured to start test VMs 324, 326 and 328 on computer system 318, computer system 320 and computer system 322, respectively. it can. As described below, the resource monitoring tool 100 can be configured to select computer systems to accurately represent the computer resources of the data center A 308 of the computer resource service 302.

  When deciding where to host the MI within computer resource service 302, computer resource service 302 may wish to determine the performance of the computer system across all geographic and regional locations. Similarly, when utilizing computer resource service 302, a user may wish to determine a computer system that will provide computing resources that are optimal for the user's needs. For example, users may wish to determine the performance of computer systems within their geographic location, and computer systems within other geographic locations. Thus, the resource monitoring tool 100 can monitor and determine performance metrics representative of the location of the computer system and the configuration of the computer system.

  FIG. 4 illustrates a process 400 for monitoring the performance of computer resource services, such as computer resource services 302, in accordance with various implementations. In the process 400, the illustrated steps are examples, any of the illustrated steps may be removed, additional steps may be added, and the order of illustrated steps may be changed. . Additionally, although the process 400 is described as implemented by the resource monitoring tool 100, one or more of the stages of the process 400 can be implemented by any hardware or software component.

  At 404, the resource monitoring tool 100 can optionally determine the geographical location of the data center within the computer resource service. The resource monitoring tool 100 can scan computer resource services to determine the geographical location of the data center within the computer resource service. Similarly, the resource monitoring tool 100 can communicate with one or more management systems 111 of computer resource services to determine the geographical location of the data center within the computer resource service. The geographical location of the data center can also be preprogrammed into the resource monitoring tool 100.

  For example, referring to FIG. 3A, the resource monitoring tool 100 may use the geographic resource 1 304 having the data center A 308 and the data center B 310, the data resource C 302, the data center D 314, and the data center E 316. It can be decided to include the geographical location 2 306 that it has.

  At 406, the resource monitoring tool 100 can select a data center within the geographic location to test. For example, the resource monitoring tool 100 can select any one of the data center A 308, the data center B 310, the data center C 312, the data center D 314, and the data center E 316. The resource monitoring tool 100 may randomly select a particular data center, or may select a particular data center based on a predetermined order.

  At 408, the resource monitoring tool 100 can optionally determine the computer system in the data center. The resource monitoring tool 100 can determine the identity and details of the computer system in the data center currently being tested. The identification information may include the name of the computer system, the network address of the computer system, and access information on the computer system. The details of the computer system can include the hardware and software components of the computer system.

  For example, with respect to data center A 308, resource monitoring tool 100 determines identification information for computer system 318, computer system 320, and computer system 322, such as computer system name, computer system network address, access information for computer system, etc. can do. In addition, resource monitoring tool 100 can determine details of computer system 318, computer system 320, and computer system 322, such as hardware and software components of the computer system. To determine this information, resource monitoring tool 100 can scan computer system 318, computer system 320, and computer system 322. Similarly, resource monitoring tool 100 can communicate with one or more management systems 111 of computer resource service 302 to obtain information.

  At 410, the resource monitoring tool 100 can determine the set of computer systems in the data center that represents the various configurations of the computer systems. For each of the various configurations of the computer system, the resource monitoring tool 100 is sufficiently large to provide several computer systems that are statistically significant, ie, statistically meaningful performance metrics, for each configuration. The sample of the computer system can be selected.

  For example, computer system 318 and computer system 320 may include computer systems having various configurations (eg, various generations or types of hardware resources). The resource monitoring tool 100 represents each location, for example 5% of the computer system 318 and 5% of the computer system 320, and provides multiple computer systems 318 and multiple computer systems 320 that provide reliable performance metrics. It can be selected.

  At 412, the resource monitoring tool 100 can start a test VM on a set of computer systems. The resource monitoring tool 100 can select the same configuration for each of the test VMs so that performance can be compared and correlated between computer systems. For example, with respect to data center A 308, resource monitoring tool 100 may be configured with test VMs 324, 326, and 328 configured to present specific computer resources (eg, processing power, memory, I / O resources, network bandwidth, etc.). You can choose the configuration for The resource monitoring tool 100 can configure test VMs 324, 326 and 328. Similarly, the resource monitoring tool 100 can select a test VM preconfigured for the test VMs 324, 326, and 328 and start it on the computer system.

  At 414, the resource monitoring tool 100 can monitor the performance of the test virtual machine. The resource monitoring tool 100 can remotely monitor performance metrics via a network connection. Similarly, resource monitoring tool 100 can install a performance monitoring agent on one or more of the test virtual machines and receive or retrieve performance metrics from the monitoring agent. The resource monitoring tool 100 can monitor performance metrics over time to determine performance metrics at various times.

  The resource monitoring tool 100 and / or the monitoring agent may launch various benchmark tests on the test VMs 324, 326 and 328. Benchmark tests include processor performance (eg, software execution, score based on instruction execution, etc.), memory performance (eg, latency, read speed, write speed, etc.), input / output (I / O) performance (eg, List any kind of conventional tests that measure I / O per second, I / O bandwidth etc), network performance (eg network bandwidth, network packets per second etc) etc it can.

  At 416, the resource monitoring tool 100 can store the monitored performance metrics in the test virtual machine. Resource monitoring tool 100 may store performance metrics in one or more data structures, such as performance data store 114. The resource monitoring tool 100 can store performance metrics and other information. Other information may include, for example, the information collected at 204.

  For example, FIG. 5 illustrates an example data structure that can store performance metrics monitored from a test VM. Although FIG. 5 illustrates an example of a data structure, the illustrated structure is an example, and any type of data structure can be utilized to store performance metrics. Similarly, although FIG. 5 illustrates an example of a performance metric that may be stored, the illustrated performance metric is an example, and any type of performance metric or other information is monitored and stored by the resource monitoring tool 100. be able to.

  As illustrated, the resource monitoring tool 100 can store performance metrics in the table 500. The table 500 can include several rows and columns. Each row can store performance metrics and other data for a particular test VM. Each row can store a specific type of information about the test VM. The table 500 can include a column 502 that stores an identifier for each test VM. The identifier may be any value that uniquely identifies a test VM. For example, the identifier may be a numerical value such as "00001".

  The table 500 may also include a column 504 that stores configuration information describing computing resources assigned to the test VM. The configuration information is assigned to the test VM such as a memory assigned to the test VM, a processing capability assigned to the test VM, a storage device assigned to the test VM, a network resource assigned to the test VM, and a data processing architecture of the test VM Any information describing computing resources can be mentioned. For example, as illustrated, column 504 can include the following information: "allocated memory (number of bytes); processing capacity (computing unit); storage (number of bytes); architecture (number of bits)". For example, for test VM "00001", column 504 may include "1.7 GB" of allocated memory, "5 CU" of processing power, "320 GB" of storage, and "32 bit" architecture.

  The table 500 may also include columns 506 and 508 that store location information of the computer system hosting the test VM. Column 506 may include regional or geographical information of the test VM, such as regional or geographical identification and description. For example, for test VM "00001", column 506 may include the location of "Geographical location 1" (as referenced in FIG. 3A). Column 508 may include additional location information regarding a particular location, such as a subregion or data center that includes a computer system hosting the test VM. For example, for test VM "00001", column 508 may include "Data Center A", which indicates a computer system hosting test VM "00001".

  The table 500 may also include a column 510 storing an identifier of the computer system hosting the test VM. The identifier may be any value that uniquely identifies the computing system. For example, the identifier may be a media access control (MAC) address of a computer system. For example, for test VM "00001", column 510 may include the MAC address in hexadecimal, the value "00: A0: C9: 14: C8: 29".

  The table 500 may also include a column 512 that stores time and date information of tests performed by the test VM. Time and date information may include any information identifying when a test was performed, when a performance metric was collected, etc. For example, as illustrated, column 512 may include the date as "day, month, year" and time as "hour: minute: second". For example, for test VM “00001”, column 512 may include the date “May 1, 2013” and the time “12:15:06”.

  The table 500 may also include columns 514, 516, 518, and 520 that store performance metrics collected by the test VM. Column 514 may store performance metrics associated with processor performance. Processor performance may be any value generated by a particular benchmark test. For example, for test VM "00001", column 514 may include the value "1025", which corresponds to the score of the particular benchmark test. Column 516 may store performance metrics associated with memory performance testing. For example, as illustrated, column 516 can include the performance metric: “Page Update Rate (seconds); Latency (seconds)”. For example, for test VM "00001", column 516 may include a page update rate of "5" seconds and a latency of "3" seconds.

  Column 518 may store performance metrics associated with I / O performance testing. For example, as illustrated, column 518 can include the performance metric: "input / output per second: latency (seconds)." For example, for test VM "00001", column 518 may include "100,000" IOPS and a "10" second latency. Column 520 may store performance metrics associated with network performance testing. For example, as illustrated, column 520 may include the performance metric: "network bandwidth (M bits per second); number of dropped packets; latency (seconds)". For example, for test VM “00001”, column 520 may include “9.8” Mbit / s network bandwidth, “10” number of dropped packets, and “3” second latency.

  Table 500 may also include columns 522 that may store any additional information. For example, the columns can include information such as test intervals, identification of benchmark tests to be performed, identification of systems running the resource monitoring tool 100, and the like.

  At 418, the resource monitoring tool 100 can determine whether to test additional virtual machine configurations. In order to obtain an overview of the computer resources of the computer resource service 302, the resource monitoring tool 100 may, for example, provide various computer resources (eg, processing power, memory, I / O resources, network bandwidth, etc.). Testing and monitoring of performance metrics for various configurations of test VMs 324, 326, and 328, such as, may be performed.

  At 420, the resource monitoring tool 100 can determine if there are additional data centers in the geographic location. For example, resource monitoring tool 100 may determine that data center B, 310 also needs to be tested. The resource monitoring tool 100 can repeat the above process for the data center B, 310. Thus, resource monitoring tool 100 can determine and monitor performance metrics for all data centers in geographic location 1, 304.

  At 422, the resource monitoring tool 100 can determine if additional geographic locations exist. For example, the resource monitoring tool 100 can determine that the geographic location 2 306 needs to be tested. The resource monitoring tool 100 may repeat the above process for data center C 312, data center D 314, and data center E 316 at geographic location 306. Thus, resource monitoring tool 100 can determine and monitor performance metrics for all data centers in both geographic location 1 304 and geographic location 2 306.

  The resource monitoring tool 100 can keep on monitoring performance metrics over time. Thus, the resource monitoring tool 100 can determine performance metrics over time and determine how the performance metrics change over time.

  After 422, the process can end, repeat, or return to any point.

  In the implementations described above, the performance metrics collected by the resource monitoring tool 100 can be utilized for various processes associated with computer resource services. In an implementation, as described above, performance metrics may determine which computer system within computer resource service 302 should support MI, computer resource pricing of computer resource service 302, computer resource service 302 It can be used to make decisions regarding computer system updates.

  In the implementations described above, resource monitoring tool 100 can continue to monitor performance metrics over time. Thus, the resource monitoring tool 100 can determine performance metrics over time and determine how the performance metrics change over time. In addition, after a certain period of time, the resource monitoring tool 100 may terminate the test VM to enable computing resources to be utilized by users of computer resource services.

  FIG. 6 illustrates an example of a hardware configuration for a computing device 600 implementing the resource monitoring tool 100 that can be used to implement one or more of the processes described above. While FIG. 6 illustrates various components included in computing device 600, FIG. 6 illustrates one example of a computing device where additional components may be added and existing components are removed. It can be done.

  As illustrated in FIG. 6, computing device 600 may include one or more processors 602 of various core configurations and clock frequencies. Computing device 600 may also include one or more memory devices 604 that act as main memory during operation of computing device 600. For example, in operation, a copy of resource monitoring tool 100 may be stored on one or more memory devices 604. The computing device 600 also includes one or more peripheral interfaces 606, such as a keyboard, mouse, touch pad, computer screen, touch screen, etc., to enable human interaction with the computing device 600 and its operation. be able to.

  The computing device 600 is also one to communicate via one or more networks, such as an Ethernet adapter, a wireless transceiver, or a serial network component, to communicate with a wired or wireless medium using a protocol. The above network interface 608 can also be included. Computing device 600 may also include various physical, such as flash drives, hard drives, random access memories, etc., for storing data such as images, files, and program instructions for execution by one or more processors 602. One or more storage devices 610 of dimensions and storage capacity may also be included.

  Additionally, computing device 600 may include one or more software programs 612, such as resource monitoring tool 100. One or more software programs 612 may include instructions that cause one or more processors 602 to perform the processes described herein. Copies of one or more software programs 612 may be stored on one or more memory devices 604 and / or one or more storage devices 610. Similarly, data utilized by one or more software programs 612, such as, for example, performance data storage 114, may be stored in one or more memory devices 604 and / or one or more storage devices 610.

  In an implementation, computing device 600 may communicate with one or more remote user devices 614 and computer resource services 618 via network 616. One or more remote user devices 614 may be any type of conventional computing device. For example, one or more user devices 614 may be desktops, laptops, servers, etc., or portable devices such as smart phones, tablet computers, cell phones, personal digital assistants, and the like. Network 616 may be any type of network, such as a local area network, wide area network, virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, an infrared network, a wireless network, and any combination thereof. . Network 616 may support communication using any of a variety of commercially available protocols such as TCP / IP, OSI, FTP, UPnP, NFS, CIFS, and AppleTalk. Network 616 can be, for example, a local area network, a wide area network, a virtual private network, the Internet, an intranet, an extranet, a public switched telephone network, an infrared network, a wireless network, and any combination thereof.

  In an implementation, computing device 600 may exchange data over network 616 with one or more user devices 614 and computer resource services 618. For example, computing device 600 can receive requests for one or more of the performance metrics from one or more user devices 614.

  The computing device 600 and the resource monitoring tool 100 may be implemented as part of at least one service or web service, and may, for example, be part of a service oriented architecture. As described above, the resource monitoring tool 100 can implement an interface, for example, a web service API. The computing device 600 may exchange data with one or more user devices 614 via the web services API during operation of at least one service or web service. Services such as web services can communicate using any suitable type of messaging, for example using Extensible Markup Language (XML) formatted messages, and SOAP ("Simple Object Access Protocol Can be exchanged using an appropriate protocol such as The processes provided or performed by such services can be described in any suitable language, such as Web Services Description Language (WSDL). The use of languages such as WSDL enables functionality such as automatic generation of client-side code in various SOAP frameworks.

  In an implementation, computing device 600 may be utilized as part of a web server architecture. In a web server architecture, computing device 600 may launch any of a variety of server or mid-tier applications, including HTTP servers, FTP servers, CGI servers, data servers, Java servers, and business application servers. The computing device 600 may also respond to requests from one or more remote user devices 614, for example, Java, C, C #, or C ++, or any scripting language such as Perl, Python Executing the program or script by executing one or more web applications that may be implemented as one or more scripts or programs written in any programming language such as TCL, etc., and combinations thereof It is possible. The computing device 600 may also include a database server, including but not limited to, commercially available from Oracle®, Microsoft®, Sybase®, and IBM®. Are listed.

  Computing device 600 may include the various data storage described above as well as other memory and storage media. They may reside on storage media that is local to (and / or reside in) one or more of the various locations, eg, computers, or remotely from any or all of the computers on the network. Can. In some implementations, the information can reside in a storage area network ("SAN"), which is well known to those skilled in the art. Similarly, any necessary files for performing functions originating from a computer, server or other network device may be stored locally and / or remotely as required.

  In implementations, the components of computing device 600 described above need not be enclosed within a single housing, or even need to be located in close proximity to one another. Those skilled in the art will appreciate that the computing device 600 may include any type of hardware components, including any necessary accompanying firmware or software for implementing the disclosed implementations. It will be appreciated that the components that are provided are merely examples. The computing device 600 may also be partially or wholly implemented by an electronic circuit element or processor, such as an application specific integrated (ASIC) or field programmable gate array (FPGA).

Embodiments of the present disclosure can be described in view of the following clauses.
1.1 A non-transitory computer readable storage medium storing instructions for causing one or more processors to implement a method for determining the performance of a computer resource service, the method comprising:
Determining a set of computer systems on which to test the performance of the computer resource service, the set of computer systems comprising at least one of a first computer system and a second computer system Determining at least one of the first computer system is located at a first location and the second computer system is located at a second location;
Starting at least one virtual machine on each computer system in the set of computer systems, wherein the at least one virtual machine on each computer system is configured according to the first configuration And
Determining performance metrics of the at least one virtual machine running on each computer system in the set of computer systems;
Storing a record including performance metrics, a first configuration of the at least one virtual machine, and location information of each computer system running on the at least one virtual machine, with respect to the at least one virtual machine; , Non-transitory computer readable storage medium.
2. Determining the set of computer systems
Determining that the first computer system comprises at least one computer system having a first hardware configuration and at least one computer system having a second hardware configuration;
Selecting the at least one computer system having the first hardware configuration and the at least one computer system having the second hardware configuration to be included in the set of computer systems; The non-transitory computer readable storage medium of clause 1 including.
3. The method is
Starting at least one additional virtual machine on each computer system in the set of computer systems, the at least one additional virtual machine having a configuration different from the first configuration; Starting up,
A clause further comprising: monitoring additional performance metrics of the at least one additional virtual machine running on each computer system in the set of computer systems; and storing the additional performance metrics. One non-transitory computer readable storage medium.
4. The method is
Installing a monitoring agent on the at least one virtual machine on each computer system in the set of computer systems, wherein the monitoring agent collects, installs, a portion of the performance metrics The non-transitory computer readable storage medium of clause 1 further comprising.
5. A computer-implemented method for determining the performance of a computer resource service, comprising:
Determining a set of computer systems on which to test the performance of the computer resource service, wherein each computer system in the set of computer systems comprises a distribution of locations of the computer system and a number of computer systems. Representing and representing at least one of the differences in computer resources;
Starting at least one virtual machine on each computer system in the set of computer systems;
Monitoring by the processor performance metrics of the at least one virtual machine running on each computer system in the set of computer systems;
Storing the performance metric in a computer readable storage medium.
6. Installing a monitoring agent on the at least one virtual machine on each computer system in the set of computer systems, wherein the monitoring agent collects, installs a portion of the performance metrics The computer implemented method of clause 5 further comprising:
7. Monitoring the performance metric is:
The computer-implemented method of clause 6, comprising receiving the portion of a performance metric from the monitoring agent.
8. The computer-implemented method of clause 5, wherein the performance metric comprises at least one of processor performance, memory performance, input / output performance, and network communication performance.
9. Determining one of the computer systems having available computer resources based at least in part on the performance metric;
The computer implemented method of clause 5, further comprising: migrating an existing virtual machine to the one of the computer systems.
10. Determining, based at least in part on the performance metric, one of the computer systems requiring an upgrade;
The computer implemented method of clause 5, further comprising: providing notification of the required upgrade.
11. Receiving a request to provide the set of performance metrics via a web service protocol;
Retrieving the set of performance metrics from the performance metrics stored in the computer readable storage medium based at least in part on the request;
The computer-implemented method of clause 5, further comprising: providing the set of performance metrics to a source of the request.
12. Receiving a request to start a virtual machine within the computer resource service;
The computer-implemented method of clause 5, further comprising: determining from the computer system at least one computer system on which to start the virtual machine based at least in part on the performance metric.
13. The first portion of the computer system is located at a first geographical location and the second portion of the computer system is located at a second geographical location
The computer-implemented method of clause 5, wherein the set of computer systems comprises at least one computer system from a first part and at least one computer system from a second part.
14. Starting at least one additional virtual machine on each computer system in the set of computer systems, the at least one additional virtual machine configured differently than the at least one virtual machine Having, starting, and
Monitoring by the processor additional performance metrics of the at least one additional virtual machine running on each computer system in the set of computer systems;
The computer-implemented method of clause 5, further comprising storing the additional performance metric in the computer readable storage medium.
15. A system for determining the performance of a computer resource service, comprising:
At least one memory device containing instructions;
A network interface configured to communicate with the computer resource service, the computer resource service comprising a computer system;
Coupled to at least one memory device and a network interface,
Determining a set of computer systems on which to test the performance of the computer resource service, wherein each computer system in the set of computer systems comprises a location distribution of the computer system and a computer of the computer system Representing and representing at least one of the differences in resources;
Starting at least one virtual machine on each computer system in the set of computer systems;
Monitoring performance metrics of the at least one virtual machine running on each computer system in the set of computer systems via the network interface;
Storing the performance metric in a computer readable storage medium; and at least one processor configured to execute instructions implementing the method.
16. The at least one processor is
Installing a monitoring agent on the at least one virtual machine on each computer system in the set of computer systems;
The system of clause 15, further configured to execute the instructions to implement the method, further comprising receiving a portion of the performance metric from the monitoring agent via the network interface.
17. The monitoring agent comprises instructions executing on the at least one virtual machine to collect the performance metric including at least one of processor performance, memory performance, input / output performance, and network communication performance. Article 16 system.
18. The at least one processor is
Determining one of the computer systems having available computer resources based at least in part on the performance metric;
The system of clause 15, further comprising: migrating an existing virtual machine to the one of the computer systems; and further configured to execute the instructions implementing the method.
19. The at least one processor is
Determining, based at least in part on the performance metric, one of the computer systems requiring an upgrade;
The system of clause 15, further comprising: providing notification of the required upgrade; and performing the instructions to implement the method.
20. The at least one processor is
Receiving a request to provide the set of performance metrics via the network interface;
Retrieving the set of performance metrics from the performance metrics stored in the computer readable storage medium based at least in part on the request;
The system of clause 15, further comprising: providing the set of performance metrics to a source of the request via the network interface; further configured to execute the instructions implementing the method.
21. The at least one processor is
Receiving a request to start a virtual machine within the computer resource service;
Determining from the computer system at least one computer system on which to start the virtual machine based at least in part on the performance metric to execute the instructions to implement the method The system of clause 15 further configured.
22. The first portion of the computer system is located at a first geographical location and the second portion of the computer system is located at a second geographical location
15. The system of clause 15, wherein the set of computer systems comprises at least one computer system from a first part and at least one computer system from a second part.
23. The at least one processor is
Starting at least one additional virtual machine on each computer system in the set of computer systems, the at least one additional virtual machine configured differently than the at least one virtual machine Having, starting, and
Monitoring additional performance metrics of the at least one additional virtual machine running on each computer system in the set of computer systems;
The system of clause 15, further comprising: storing a record of the additional performance metric; and further configured to execute the instructions implementing the method.
Embodiments of the present disclosure can also be described in view of the following clauses.
1.1 A non-transitory computer readable storage medium storing instructions for causing one or more processors to perform a method for evaluating the performance of a computer resource service, the method comprising:
Receiving one or more filters to be included in the request for performance metrics associated with the computer resource service, the one or more filters responsive to the request for any of the performance metrics Monitoring the performance metrics of a test virtual machine running on a computer system of the computer resource service via receiving and a network, including one or more parameters used to determine what to provide Sending the request to a system that is
Receiving a response comprising the set of performance metrics determined by the one or more filters via the network, the set of performance metrics being performance metrics associated with the configuration of the test virtual machine Including, receiving, and
Determining a configuration from the configuration of the test virtual machine based at least in part on the set of performance metrics;
And, according to the configuration, starting a virtual machine within the computer resource service.
2. The one or more parameters may include the location of the computer system of the computer resource service, the configuration of the computer system of the computer resource service, the configuration of the test virtual machine, benchmark tests performed by the test virtual machine, performance The non-transitory computer readable storage medium of clause 1 comprising at least one of metric type, date, time, statistics associated with the performance metric.
3. Determining the configuration from the configuration of the test virtual machine is:
Determining the desired performance of the virtual machine;
Comparing the desired performance to the set of performance metrics;
The non-transitory computer readable storage medium of clause 1 comprising determining the configuration of the test virtual machine based at least in part on the comparison.
4. The method is
The non-transitory computer readable storage medium of clause 1 further comprising determining from the computer system at least one computer system on which to start the virtual machine based at least in part on the set of performance metrics.
5. The method is
The non-transitory computer readable storage medium of clause 1 further comprising determining at least one location of the computer system for starting the virtual machine based at least in part on the set of performance metrics.
6. A computer implemented method for performance evaluation of a computer resource service, comprising:
Receiving a request for performance of the computer resource service via a network interface, the request being used to determine a performance metric to provide in response to the request. Receiving, including parameters
Determining the set of performance metrics based at least in part on the one or more parameters, the set of performance metrics being a distribution of computer systems within the computer resource service and the computer system Representing, representatively at least one of the computer resource differences of
Sending a response comprising the set of performance metrics via the network interface.
7. The computer-implemented method of clause 6, further comprising determining the performance metric from a test virtual machine running on the computer system of the computer resource service.
8. The computer-implemented method of clause 6, further comprising receiving a request to start at least one virtual machine on at least one computer system associated with the set of performance metrics in response to sending the response.
9. The computer implemented method of clause 6, wherein the network interface is a web service interface.
10. The one or more parameters may include the location of the computer system of the computer resource service, the configuration of the computer system of the computer resource service, the configuration of the test virtual machine, benchmark tests performed by the test virtual machine, performance 6. The computer implemented method of clause 6, comprising at least one of metric type, date, time, statistics associated with the performance metric.
11. A computer-implemented method for evaluating the performance of a computer resource service, comprising:
Providing a request for the performance of the computer resource service via a network interface, the request being used to determine a performance metric to provide in response to the request. Receiving, including parameters
Receiving the set of performance metrics, the set of performance metrics being representative of at least one of a location distribution of a computer system within the computer resource service and a difference of the computer resources of the computer system Representing, receiving, and
Performing an operation associated with the computer resource service based at least in part on the set of performance metrics.
12. To carry out the operation is
Determining a first computer system and a second computer system from the computer system of the computer resource service, the first computer system being located at a first location, the second computer system Is located at the second place, determining,
Determining that the first computer system is operating at a desired performance based at least in part on the set of performance metrics.
Launching at least one virtual machine on at least one of the first computer systems.
13. To carry out the operation is
Determining a first computer system and a second computer system from the computer system of the computer resource service, the first computer system comprising a first configuration, the second computer system comprising Determining, including a second configuration;
Determining that the first computer system is operating at a desired performance based at least in part on the set of performance metrics.
Launching at least one virtual machine on at least one of the first computer systems.
14. To carry out the operation is
Determining a first configuration of the test virtual machine and a second configuration of the test virtual machine;
Determining that the first configuration is operating at a desired performance based at least in part on the set of performance metrics;
The computer-implemented method of clause 11, comprising starting at least one virtual machine on the computer system according to the first configuration.
15. To carry out the operation is
Determining, based at least in part on the set of performance metrics, one of the computer systems having available computer resources;
Moving the existing virtual machine to the one of the computer systems.
16. A system for evaluating the performance of a computer resource service, comprising
At least one memory device containing instructions;
A display device,
Coupled to at least one memory device and the display device;
Sending a request for the performance of the computer resource service, the request comprising one or more parameters used to determine which of the performance metrics to provide in response to the request. Including, sending, and
Receiving a response including the set of performance metrics determined by the one or more parameters;
At least one processor configured to execute the instructions to perform a method comprising outputting a portion of the set of performance metrics via the display device.
17. The system of clause 16, further comprising an input device coupled to the at least one processor, wherein the input device receives the one or more parameters.
18. The system of clause 16, further comprising a network device configured to communicate with the system monitoring the performance metric, wherein the request is sent to the system via a web service protocol.
19. The system of clause 16, further comprising a network device configured to communicate with a computer system of the computer resource service, wherein the at least one processor is configured to monitor the performance metric.
20. The network resource is further configured to communicate with the computer system of the computer resource service, the at least one processor in communication with a monitoring agent, and via the network device one of the performance metrics from the monitoring agent. The system of clause 16 configured to collect parts.
21. The at least one processor is
Determining a first configuration of the test virtual machine and a second configuration of the test virtual machine;
Determining that the first configuration is operating at a desired performance based at least in part on the set of performance metrics;
The system of clause 16, configured to execute instructions implementing the method, further comprising: starting at least one virtual machine on the computer system according to the first configuration.

  The particular implementations described above can be implemented as computer applications or programs. Computer programs can exist in various forms, both active and inactive. For example, a computer program may consist of source code, object code, executable code or other formatted program instructions, one or more software programs, software modules, or both, firmware program (s), or It can exist as a hardware description language (HDL) file. Any of the above may be embodied in compressed or uncompressed form on computer readable media and signals, including computer readable storage devices and media. Examples of computer readable storage devices and media include conventional computer system RAM (random access memory), ROM (read only memory), EPROM (erasable programmable ROM), EEPROM (electrically erasable programmable ROM) And magnetic or optical disks or tapes. An example of a computer readable signal is a signal that may be configured to access a computer system hosting or activating the present teachings, whether modulated or not using a carrier, such as the Internet or other network. Contains the signal to be downloaded through Specific examples of the foregoing include the distribution of executable software program (s) of a computer program on a CD-ROM or via download on the Internet. In a sense, the Internet is itself a computer readable medium as an abstract entity. The same is generally true for computer networks.

  Although the present teachings have been described with reference to examples of those implementations, one skilled in the art can make various modifications to the described implementations without departing from its true spirit and scope. Will. The terms and descriptions used herein are set forth by way of illustration only and are not meant to be limiting. In particular, although the method is described by way of example, the steps of the method can be performed in an order different from that illustrated or simultaneously. Furthermore, the terms "include", "include", "have", "have", "with", or variations thereof are used either in the form for carrying out the invention or in the claims. To the extent that such terms are intended to be inclusive, in a manner similar to the term "comprising." As used herein, for example, with respect to the listing of items such as A and B, the terms "one or more of" and "at least one of" may be A alone, B alone, or A And B. Furthermore, unless otherwise specified, the term "set" should be interpreted as "one or more". Also, the terms "couple" or "couples" are intended to mean either an indirect or direct connection. Thus, if the first device couples to the second device, the connection may be through a direct connection, or through an indirect connection through other devices, components, and connections.

  For the purpose of simplicity and illustration, the principles of the present teachings are described above primarily by reference to examples of its various implementations. However, one skilled in the art will appreciate that the same principles are equally applicable to many different types of information and systems, and may be implemented in many different types of information and systems, and any such variations are true to the present teachings. It will be readily appreciated that it does not depart from the spirit and scope of the invention. Furthermore, in the foregoing Detailed Description, reference is made to the accompanying drawings, which illustrate specific examples of various implementations. Electrical, mechanical, logical and structural changes may be made to the example of various implementations without departing from the spirit and scope of the present teachings. Accordingly, the above-described modes for carrying out the invention should not be interpreted in a limiting sense, and the scope of the present teachings is defined by the appended claims and their equivalents.

Claims (13)

  1. A non-transitory computer readable storage medium storing instructions for causing one or more processors of a user device to perform a method of evaluating the performance of a computer resource service, the method comprising the following steps performed by the user device: :
    Via a network, the system for monitoring the performance of the computer resource service, and transmitting a request for configuration of the performance metric for a particular computer system from among the arrangement of a plurality of computer systems of the computer resource services , a system for monitoring the performance, to collect performance metrics of the plurality of configurations, utilizes a test virtual machine on different ones of said computer system, wherein the request, the particular configuration of the computer system Specifying a filter including one or more parameters identifying the location of the computer system of the computer resource service, the configuration of the computer system of the computer resource service, the test Configuration of virtual machines, including at least one of the benchmark test performed by the test virtual machine, statistics relating to the type of performance metric or the performance metric, the steps of the transmission,
    Via the network, the performance of the system to monitor, comprising: receiving a response including a set of performance metrics that have been specified by the one or more filters in the request, the set of performance metrics, Said receiving, comprising performance metrics of different ones of said configurations of said computer system;
    Selecting at least partially based, one consists in the configuration of the computer system of the computer resource services to a set of the received performance metric,
    According to the configuration wherein the selected, by the computer resource services, sending a request to start the new virtual machine,
    And a non-transitory computer readable storage medium.
  2. The step of selecting the configuration from the configuration of the test virtual machine comprises:
    Determining the desired performance of the virtual machine;
    Comparing the desired performance to the set of performance metrics;
    Determining the configuration of the test virtual machine based at least in part on the comparison;
    A non-transitory computer readable storage medium according to claim 1, comprising
  3. The method is
    Based at least in part on the set of performance metrics before Symbol further comprising determining the at least one computer system from the computer system for starting the virtual machine, non-transitory computer-readable storage according to claim 1 Medium.
  4. The method is
    The non-transitory computer readable storage medium of claim 1, further comprising determining at least one location of the computer system for starting the virtual machine based at least in part on the set of performance metrics.
  5. A computer implemented method for evaluating the performance of a computer resource service, comprising:
    Providing a plurality of different configurations of computer systems by the computer resource service;
    Receiving a request for a particular performance metric of the plurality of different configurations of the computer system via a network interface, the request identifying the particular one of the configurations Specify one or more parameters, and the one or more parameters indicate the location of the computer system of the computer resource service, the configuration of the computer system of the computer resource service, the type of performance metric, or the performance metric. Receiving, including at least one of associated statistics ;
    Determining the set of performance metrics for a particular one of the computer systems based at least in part on the one or more parameters specified in the request, the set of performance metrics being: and location distribution of a computer system in the computer resource services, represents at least one of differences in computer resources of the computer system, the set of performance metrics, and evaluate the performance of the computer resource services, the computer It is at least one by the system for selecting at least one computer system to request so as to start a new virtual machine available on the system, and wherein the determining,
    Sending a response including the set of the determined performance metrics via the network interface;
    Computer-implemented methods, including:
  6. The method is
    The computer-implemented method of claim 5 , further comprising determining the performance metric from a test virtual machine running on the computer system of the computer resource service.
  7. The method is
    6. The method of claim 5 , further comprising receiving a request to start at least one virtual machine on at least one computer system associated with the set of performance metrics in response to sending the response. Computer implemented method.
  8. Wherein the one or more parameters in the configuration of the pre-Symbol test virtual machine further includes at least one of the benchmark test performed by the test virtual machine, the computer implemented method of claim 6.
  9. A computer implemented method for evaluating the performance of a computer resource service, comprising:
    Via the network interface, the system for monitoring the performance of the computer resource service, by sending a request for performance metric for a particular configuration of the computer system from the plurality of configurations of the computer resource service computer systems Said system for monitoring said performance utilizes test virtual machines on different ones of said computer systems to collect performance metrics of said plurality of configurations, said request being for said particulars of said computer systems specify one or more parameters identifying a configuration, the one or more parameters, the location of the computer system of the computer resource services, configuration of the computer system of the computer resource services, configuration of the test virtual machine And said benchmark tests performed by the test virtual machine, including at least one of statistics relating to the type of performance metric or said performance metric, and the transmission,
    Receiving from the system for monitoring the performance the set of performance metrics specified by the one or more parameters in the request, the set of performance metrics being locations of computer systems in the computer resource service distribution and represents at least one of differences in computer resources of the computer system, the set of performance metrics, and evaluate the performance of the computer resource services, at least one new virtual on the computer system I Ri is utilized in the system for selecting at least one computer system to request so as to start the machine, the method comprising the reception,
    Performing an operation associated with the computer resource service based at least in part on the set of performance metrics;
    Only including,
    To perform the operation is:
    Determining a first computer system and a second computer system from among the computer systems of the computer resource service, the first computer system being located at a first location, the second computer The system being located at a second location;
    Determining that the first computer system is operating at a desired performance based at least in part on the set of performance metrics.
    Starting at least one virtual machine on at least one of the first computer systems;
    Including, computer-implemented method.
  10. A computer implemented method for evaluating the performance of a computer resource service, comprising:
    Transmitting a request of performance metrics of a particular configuration of the computer system from among a plurality of configurations of the computer system of the computer resource service to a system that monitors the performance of the computer resource service via a network interface The system for monitoring the performance utilizes test virtual machines on different ones of the computer systems to collect performance metrics of the plurality of configurations, and the request is for the particular configuration of the computer system Specify one or more parameters identifying the location of the computer system of the computer resource service, the configuration of the computer system of the computer resource service, the configuration of the test virtual machine And said benchmark tests performed by the test virtual machine, including at least one of statistics relating to the type of performance metric or said performance metric, and the transmission,
    Receiving from the system for monitoring the performance the set of performance metrics specified by the one or more parameters in the request, the set of performance metrics being locations of computer systems in the computer resource service At least one new virtual machine on the computer system, representing at least one of a distribution and differences in computer resources of the computer system, the set of performance metrics evaluating the performance of the computer resource service Receiving by the system selecting at least one computer system for requesting to start
    Performing an operation associated with the computer resource service based at least in part on the set of performance metrics;
    Including
    To perform the operation is:
    Determining a first computer system and a second computer system among the computer systems of the computer resource service, the first computer system including a first configuration, the second computer system Includes the second configuration, and
    Determining that the first computer system is operating at a desired performance based at least in part on the set of performance metrics.
    Starting at least one virtual machine on at least one of the first computer systems;
    Including, computer-implemented method.
  11. A computer implemented method for evaluating the performance of a computer resource service, comprising:
    Transmitting a request of performance metrics of a particular configuration of the computer system from among a plurality of configurations of the computer system of the computer resource service to a system that monitors the performance of the computer resource service via a network interface The system for monitoring the performance utilizes test virtual machines on different ones of the computer systems to collect performance metrics of the plurality of configurations, and the request is for the particular configuration of the computer system Specify one or more parameters identifying the location of the computer system of the computer resource service, the configuration of the computer system of the computer resource service, the configuration of the test virtual machine And said benchmark tests performed by the test virtual machine, including at least one of statistics relating to the type of performance metric or said performance metric, and the transmission,
    Receiving from the system for monitoring the performance the set of performance metrics specified by the one or more parameters in the request, the set of performance metrics being locations of computer systems in the computer resource service At least one new virtual machine on the computer system, representing at least one of a distribution and differences in computer resources of the computer system, the set of performance metrics evaluating the performance of the computer resource service Receiving by the system selecting at least one computer system for requesting to start
    Performing an operation associated with the computer resource service based at least in part on the set of performance metrics;
    Including
    To perform the operation is:
    Determining a first configuration of the test virtual machine and a second configuration of the test virtual machine;
    Determining that the first configuration is operating at a desired performance based at least in part on the set of performance metrics.
    Starting at least one virtual machine on the computer system according to the first configuration;
    Including, computer-implemented method.
  12. A computer implemented method for evaluating the performance of a computer resource service, comprising:
    Transmitting a request of performance metrics of a particular configuration of the computer system from among a plurality of configurations of the computer system of the computer resource service to a system that monitors the performance of the computer resource service via a network interface The system for monitoring the performance utilizes test virtual machines on different ones of the computer systems to collect performance metrics of the plurality of configurations, and the request is for the particular configuration of the computer system Specify one or more parameters identifying the location of the computer system of the computer resource service, the configuration of the computer system of the computer resource service, the configuration of the test virtual machine And said benchmark tests performed by the test virtual machine, including at least one of statistics relating to the type of performance metric or said performance metric, and the transmission,
    Receiving from the system for monitoring the performance the set of performance metrics specified by the one or more parameters in the request, the set of performance metrics being locations of computer systems in the computer resource service At least one new virtual machine on the computer system, representing at least one of a distribution and differences in computer resources of the computer system, the set of performance metrics evaluating the performance of the computer resource service Receiving by the system selecting at least one computer system for requesting to start
    Performing an operation associated with the computer resource service based at least in part on the set of performance metrics;
    Including
    To perform the operation is:
    Determining one of the computer systems having available computer resources based at least in part on the set of performance metrics;
    Migrating an existing virtual machine to one of the computer systems;
    Including, computer-implemented method.
  13. A system for evaluating the performance of a computer resource service, comprising
    At least one storage device containing instructions;
    A display device,
    A system coupled to the at least one storage device and the display device, and monitoring the performance of the computer resource service via a network interface, a particular computer from among a plurality of computer system configurations of the computer resource service Sending a request for a performance metric of a configuration of the system, wherein the system for monitoring the performance utilizes a test virtual machine on a different one of the computer systems, and the performance metric of the plurality of configurations The request specifies one or more parameters identifying the particular configuration of the computer system, the one or more parameters indicating the location of the computer system of the computer resource service, the computer resource The computer system configuration of the source service, the configuration of the test virtual machine, the benchmark test performed by the test virtual machine, the type of performance metric, or at least one of the statistics associated with the performance metric, Sending and
    Receiving from the system monitoring the performance a response comprising the set of performance metrics specified by the one or more parameters in the request, the set of performance metrics being on the computer resource service Receiving, used by the system for selecting at least one computer system from among the computer systems of the computer resource service for requesting to start at least one new virtual machine. ,
    Outputting a portion of the set of performance metrics via the display device;
    At least one processor configured to execute the instructions causing the method to perform.
    A system for evaluating the performance of computer resource services, comprising:
JP2017194858A 2013-05-21 2017-10-05 Determining and monitoring the performance capabilities of computer resource services Active JP6530462B2 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
US13/899,407 US9384115B2 (en) 2013-05-21 2013-05-21 Determining and monitoring performance capabilities of a computer resource service
US13/899,407 2013-05-21
US13/899,449 US9584364B2 (en) 2013-05-21 2013-05-21 Reporting performance capabilities of a computer resource service
US13/899,449 2013-05-21

Related Parent Applications (1)

Application Number Title Priority Date Filing Date
JP2016515005 Division 2014-05-20

Publications (2)

Publication Number Publication Date
JP2018022520A JP2018022520A (en) 2018-02-08
JP6530462B2 true JP6530462B2 (en) 2019-06-12

Family

ID=56560626

Family Applications (2)

Application Number Title Priority Date Filing Date
JP2016515005A Active JP6224824B2 (en) 2013-05-21 2014-05-20 Determine and monitor the performance capabilities of computer resource services
JP2017194858A Active JP6530462B2 (en) 2013-05-21 2017-10-05 Determining and monitoring the performance capabilities of computer resource services

Family Applications Before (1)

Application Number Title Priority Date Filing Date
JP2016515005A Active JP6224824B2 (en) 2013-05-21 2014-05-20 Determine and monitor the performance capabilities of computer resource services

Country Status (5)

Country Link
US (2) US9584364B2 (en)
EP (1) EP3000040B1 (en)
JP (2) JP6224824B2 (en)
CN (2) CN108182140A (en)
WO (1) WO2014189899A1 (en)

Families Citing this family (29)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9817864B1 (en) * 2013-06-13 2017-11-14 Amazon Technologies, Inc. Flexible pivot querying of monitoring data with zero setup
US9542294B2 (en) * 2013-07-09 2017-01-10 International Business Machines Corporation Method to apply perturbation for resource bottleneck detection and capacity planning
US9811435B2 (en) * 2013-09-03 2017-11-07 Cisco Technology, Inc. System for virtual machine risk monitoring
US9509759B2 (en) * 2013-09-09 2016-11-29 International Business Machines Corporation Service agreement performance validation in a cloud hosted environment
GB2518151A (en) * 2013-09-11 2015-03-18 Ibm Network anomaly detection
US9246840B2 (en) 2013-12-13 2016-01-26 International Business Machines Corporation Dynamically move heterogeneous cloud resources based on workload analysis
US9495238B2 (en) 2013-12-13 2016-11-15 International Business Machines Corporation Fractional reserve high availability using cloud command interception
US10069753B2 (en) * 2014-03-17 2018-09-04 Cloud Physics, Inc. Relationship-based resource-contention analysis system and method
US20150324724A1 (en) * 2014-05-06 2015-11-12 International Business Machines Corporation Benchmarking performance of a service organization
US10515000B2 (en) * 2014-08-26 2019-12-24 Cloudy Days, Inc. Systems and methods for performance testing cloud applications from multiple different geographic locations
US10075352B2 (en) * 2014-12-09 2018-09-11 Ca, Inc. Monitoring user terminal applications using performance statistics for combinations of different reported characteristic dimensions and values
US9928151B1 (en) * 2014-12-12 2018-03-27 Amazon Technologies, Inc. Remote device interface for testing computing devices
US9876703B1 (en) * 2014-12-17 2018-01-23 Amazon Technologies, Inc. Computing resource testing
US9824001B2 (en) * 2015-02-09 2017-11-21 Wipro Limited System and method for steady state performance testing of a multiple output software system
US20160283566A1 (en) * 2015-03-27 2016-09-29 Ca, Inc. Analyzing sorted mobile application operational state sequences based on sequence metrics
US9772920B2 (en) * 2015-04-29 2017-09-26 Apollo Education Group, Inc. Dynamic service fault detection and recovery using peer services
US10200261B2 (en) * 2015-04-30 2019-02-05 Microsoft Technology Licensing, Llc Multiple-computing-node system job node selection
US20160344657A1 (en) * 2015-05-20 2016-11-24 International Business Machines Corporation PROVIDING PERFORMANCE ALTERNATIVES BASED ON COMPARATIVE PRICE AND PERFORMANCE DATA OF A RUNNING SaaS INSTANCE
CN107735771A (en) * 2015-06-09 2018-02-23 机械地带有限公司 Distributed expandable workload is tested
US10514994B2 (en) 2015-06-09 2019-12-24 Satori Worldwide, Llc Distributed scalable workload testing
US10362099B2 (en) 2015-06-23 2019-07-23 Amazon Technologies, Inc. Multiple instance types serving a single workload or application
US10374933B2 (en) * 2015-07-31 2019-08-06 Verizon Patent And Licensing Inc. Systems and methods for monitoring operational statuses of network services
US20170060720A1 (en) * 2015-08-27 2017-03-02 Ixia Methods, systems, and computer readable media for vendor-neutral testing and scoring of systems under test
WO2017144432A1 (en) * 2016-02-26 2017-08-31 Nokia Solutions And Networks Oy Cloud verification and test automation
US20170289061A1 (en) * 2016-03-31 2017-10-05 Lenovo Enterprise Solutions (Singapore) Pte. Ltd. Soft reservation techniques and systems for virtualized environments
US20170322827A1 (en) * 2016-05-06 2017-11-09 Nutanix, Inc. Testing and benchmarking for enterprise data centers
CN105847088A (en) * 2016-05-18 2016-08-10 浪潮电子信息产业股份有限公司 Virtual machine performance testing system based on cloud service
US20180234504A1 (en) * 2017-02-13 2018-08-16 Citrix Systems, Inc. Computer system providing cloud-based health monitoring features and related methods
WO2019070269A1 (en) * 2017-10-05 2019-04-11 Hewlett-Packard Development Company, L.P. Selecting computer configurations based on application usage-based clustering

Family Cites Families (31)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5668944A (en) 1994-09-06 1997-09-16 International Business Machines Corporation Method and system for providing performance diagnosis of a computer system
AU2003278862A1 (en) 2002-06-25 2004-01-06 International Business Machines Corporation Method and system for monitoring performance of application in a distributed environment
JP4119239B2 (en) 2002-12-20 2008-07-16 株式会社日立製作所 Computer resource allocation method, resource management server and computer system for executing the method
JP4313336B2 (en) * 2005-06-03 2009-08-12 株式会社日立製作所 Monitoring system and monitoring method
US7694189B2 (en) * 2007-02-28 2010-04-06 Red Hat, Inc. Method and system for remote monitoring subscription service
JP4980792B2 (en) 2007-05-22 2012-07-18 株式会社日立製作所 Virtual machine performance monitoring method and apparatus using the method
US8365184B2 (en) * 2007-08-31 2013-01-29 Apple Inc. Multi-core resource utilization planning
CA2697965C (en) * 2007-08-31 2018-06-12 Cirba Inc. Method and system for evaluating virtualized environments
US9294296B2 (en) 2007-10-31 2016-03-22 Hewlett Packard Enterprise Development Lp Automated test execution in a shared virtualized resource pool
JP4906686B2 (en) * 2007-11-19 2012-03-28 三菱電機株式会社 Virtual machine server sizing apparatus, virtual machine server sizing method, and virtual machine server sizing program
US8175863B1 (en) 2008-02-13 2012-05-08 Quest Software, Inc. Systems and methods for analyzing performance of virtual environments
US8161479B2 (en) 2008-06-13 2012-04-17 Microsoft Corporation Synchronizing virtual machine and application life cycles
US8046468B2 (en) * 2009-01-26 2011-10-25 Vmware, Inc. Process demand prediction for distributed power and resource management
US9600332B2 (en) * 2009-04-28 2017-03-21 Cisco Technology, Inc. Server load balancing based on virtual utilization, physical utilization, and feedback
US20120198063A1 (en) * 2009-10-09 2012-08-02 Nec Corporation Virtual server system, autonomous control server thereof, and data processing method and computer program thereof
US8705513B2 (en) 2009-12-15 2014-04-22 At&T Intellectual Property I, L.P. Methods and apparatus to communicatively couple virtual private networks to virtual machines within distributive computing networks
US8645529B2 (en) * 2010-10-06 2014-02-04 Infosys Limited Automated service level management of applications in cloud computing environment
JP5870927B2 (en) * 2010-10-07 2016-03-01 日本電気株式会社 Server system, management apparatus, server management method, and program
US8856783B2 (en) 2010-10-12 2014-10-07 Citrix Systems, Inc. Allocating virtual machines according to user-specific virtual machine metrics
US9736252B2 (en) * 2010-11-23 2017-08-15 Red Hat, Inc. Migrating subscribed services in a cloud deployment
US8909784B2 (en) 2010-11-23 2014-12-09 Red Hat, Inc. Migrating subscribed services from a set of clouds to a second set of clouds
US8738972B1 (en) * 2011-02-04 2014-05-27 Dell Software Inc. Systems and methods for real-time monitoring of virtualized environments
US9021473B2 (en) * 2011-03-14 2015-04-28 International Business Machines Corporation Hardware characterization in virtual environments
JP5501276B2 (en) * 2011-03-18 2014-05-21 株式会社エヌ・ティ・ティ・データ Virtual machine placement apparatus, virtual machine placement method, virtual machine placement program
US8806484B2 (en) * 2011-04-18 2014-08-12 Vmware, Inc. Host selection for virtual machine placement
TWI476586B (en) 2011-07-13 2015-03-11 Inst Information Industry Cloud-based test system, method and computer readable storage medium storing thereof
US8732302B2 (en) * 2011-07-15 2014-05-20 Inetco Systems Limited Method and system for monitoring performance of an application system
US8793535B2 (en) * 2011-07-21 2014-07-29 Microsoft Corporation Optimizing system usage when running quality tests in a virtual machine environment
JPWO2013027332A1 (en) * 2011-08-25 2015-03-05 日本電気株式会社 information processing apparatus, information processing method, and program
WO2013062545A1 (en) 2011-10-27 2013-05-02 Hewlett-Packard Development Company, L.P. Generating a visual representation
US9323628B2 (en) * 2012-10-09 2016-04-26 Dh2I Company Instance level server application monitoring, load balancing, and resource allocation

Also Published As

Publication number Publication date
WO2014189899A1 (en) 2014-11-27
US9584364B2 (en) 2017-02-28
EP3000040A4 (en) 2017-01-18
EP3000040A1 (en) 2016-03-30
JP6224824B2 (en) 2017-11-01
EP3000040B1 (en) 2018-07-18
US20140351394A1 (en) 2014-11-27
JP2016522947A (en) 2016-08-04
JP2018022520A (en) 2018-02-08
CN108182140A (en) 2018-06-19
US9384115B2 (en) 2016-07-05
CN105308576A (en) 2016-02-03
US20140351412A1 (en) 2014-11-27
CN105308576B (en) 2018-03-13

Similar Documents

Publication Publication Date Title
US8260840B1 (en) Dynamic scaling of a cluster of computing nodes used for distributed execution of a program
US9207984B2 (en) Monitoring and automatic scaling of data volumes
JP5926864B2 (en) System and method for configuring a cloud computing system
US8607200B2 (en) Executing a web application at different stages in the application life cycle
JP5458308B2 (en) Virtual computer system, virtual computer system monitoring method, and network device
JP3861087B2 (en) Virtual machine management apparatus and program
US8412810B1 (en) Provisioning and managing a cluster deployed on a cloud
US9942105B2 (en) Animated globe showing real-time web user performance measurements
US9542232B2 (en) System and method for transforming legacy desktop environments to a virtualized desktop model
US20130091285A1 (en) Discovery-based identification and migration of easily cloudifiable applications
US9229842B2 (en) Active waterfall charts for continuous, real-time visualization of website performance data
JP6144346B2 (en) Scaling virtual machine instances
US9294557B2 (en) Hardware level generated interrupts indicating load balancing status for a node in a virtualized computing environment
US20120084443A1 (en) Virtual provisioning with implementation resource boundary awareness
US9152532B2 (en) System and method for configuring a cloud computing system with a synthetic test workload
US9720989B2 (en) Dynamic partitioning techniques for data streams
JP2006011860A (en) System configuration management program and system configuration management device
US20140047342A1 (en) System and method for allocating a cluster of nodes for a cloud computing system based on hardware characteristics
US9491248B2 (en) Real-time analytics of web performance using actual user measurements
EP3069228B1 (en) Partition-based data stream processing framework
US9262231B2 (en) System and method for modifying a hardware configuration of a cloud computing system
US9720569B2 (en) Cloud-based custom metric/timer definitions and real-time analytics of mobile applications
CN103002005A (en) Cloud service monitoring system
US8769340B2 (en) Automatically allocating clients for software program testing
US20140173618A1 (en) System and method for management of big data sets

Legal Events

Date Code Title Description
A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20181019

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20181106

A601 Written request for extension of time

Free format text: JAPANESE INTERMEDIATE CODE: A601

Effective date: 20190206

A521 Written amendment

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20190322

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20190416

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20190516

R150 Certificate of patent or registration of utility model

Ref document number: 6530462

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150