WO2014088398A1

WO2014088398A1 - Automated test environment deployment with metric recommender for performance testing on iaas cloud

Info

Publication number: WO2014088398A1
Application number: PCT/MY2013/000218
Authority: WO
Inventors: Zakaria Alli Mohamad; Ikhwan Ismail Bukhary
Original assignee: Mimos Berhad
Priority date: 2012-12-06
Filing date: 2013-11-29
Publication date: 2014-06-12
Also published as: MY176486A

Abstract

Automated test environment deployment for performance testing on laaS Cloud is provided by utilizing metric recommender. The system and method of the present invention comprising a user interface (104) for setting up test performance; a Metric Agent Module (106, 206) for introducing list of suitable metrics to be captured according to performance goals; an Interface Agent Module (212) for interfacing with API (Application Programming Interface) of Performance Test Tool and Cloud Controller; a Deployment Agent Module (108, 208) for provisioning test environment based on test scripts; a Reporting Agent Module (110, 210) for collecting and aggregating test result from Data Store (220) and displaying in portal; a Metric Store (102, 202) for storing list of metrics; a Metric Logger (216) for monitoring and collecting required metrics during test execution; a Data Store (118, 220) for storing test results from Performance Test Tool (114, 218) and Metric Logger (216). The present invention utilizes a metric recommender wherein the performance test metric is automated using physical resources of virtual machine in which the complexity of setting up test environment will be overcome by providing cloud computing to set up test virtual environment. Metrics are aggregated using metric agents to achieve performance goals by iteration to validate and verify all quantitative and qualitative attributes as reliability, stability or resource utilization of the applications.

Description

AUTOMATED TEST ENVIRONMENT DEPLOYMENT WITH METRIC

RECOMMENDER FOR PERFORMANCE TESTING ON IAAS CLOUD

FIELD OF INVENTION

The present invention relates to a system and method for automating test environment deployment utilizing metric recommender for performance testing on laaS (Infrastructure-as-a-Service) Cloud. BACKGROUND ART

Performance testing is an assessment which determines the performance of a system under a particular workload. Simulated load will be introduced at stages and key point metrics are captured for analysis. Improvements are incorporated by developers at each iteration. The assessment of the test cycle continues until performance goals are met to validate and verify quantitative and qualitative attributes such as reliability, stability or resource utilization of the applications.

The process of the construction of performance test involves various activities from setting up a test environment, identifying performance goals, designing test cases, performing and monitoring test execution and identifying suitable metrics to be collected before analyzing the same for improvement. A set of performance test tools are required in order to design the test cases as well as the right test environment to simulate the load. The complexity of setting a system under test (SUT), identification of suitable test tool and identification of suitable test metrics will lead to the approach of the present invention. The system and method of the present invention handles performance test life cycle in a more simplified manner which will simplify the test cycle.

The present invention proposes a system and method to automate the SUT environment setup using laaS cloud infrastructure. The present invention also assist tester in achieving performance goals based on threshold set forth and simplify and automate selection of performance metrics based on Application Type; and Operating System (OS) flavours & distributions. The present invention provides metric recommender wherein performance test metric is automated using physical resources of virtual machine in which the complexity of setting up test environment will be overcome by providing cloud computing to set up test virtual environment and aggregating the correct metrics using metric agents to achieve performance goals.

The subject matter claimed herein is not limited to embodiments that solve any disadvantages or that operate only in environments such as those described above. Rather, this background is only provided to illustrate one exemplary technology area where some embodiments described herein may be practice.

SUMMARY OF INVENTION

The present invention provides a system and method to automate test environment deployment utilizing metric recommender for performance testing on laaS (Infrastructure-as-a-Service) Cloud. The system comprising at least one user interface (104) for setting up test performance; at least one Metric Agent Module (106, 206) for introducing list of suitable metrics to be captured according to performance goals; at least one Interface Agent Module (212) for interfacing with API (Application Programming Interface) of Performance Test Tool and Cloud Controller; at least one Deployment Agent Module (108, 208) for provisioning test environment based on test scripts; at least one Reporting Agent Module (110, 210) for collecting and aggregating test result from Data Store (220) and displaying in portal; at least one Metric Store (102, 202) for storing list of metrics; and at least one Data Store (118, 220) for storing test results from Performance Test Tool (114, 218) and Metric Logger (216).

Another aspect of the present invention provides for at least one Metric Agent Module (106, 206) and at least one Metric Store (102, 202) which is used for Metric Suggestion Functionality wherein Metric Suggestion Functionality is based on application used by developer and data stored in Data Store (118, 220).

A further aspect of the present invention provides for at least one Deployment Agent Module (108, 208) for provisioning test environment based on test scripts. The said Deployment Agent (108, 208) having means for communicating with Cloud Controller to provision virtual test environment based on requirements stated in test scripts; communicating with Performance Test Tool to configure test plan; and installing and configuring System under Test (SUT) in virtual test environment.

Another aspect of the present invention provides a method (300, 400) to automate test environment deployment utilizing metric recommender for performance testing on laaS (Infrastructure-as-a-Service) Cloud. The method comprising steps of initializing test by setting performance goals (302); provisioning test environment based on test scripts (304); performing test execution (306, 412); monitoring, capturing and storing test results based on pre-defined metrics (414);analyzing, reporting and retesting to ensure test completed as expected (308); and provisioning test tool by extracting test results and displaying in portal (416).

A further aspect of the present invention provides for the further steps for initializing test by setting performance goals. The said method comprising further steps of introducing list of suitable metrics to be captured according to performance goals (402); obtaining list of suggested metrics to create test specifications (404); and preparing test scripts for test deployment (404). Another aspect of the present invention provides for a method for provisioning test environment based on test scripts. The method comprising steps of communicating with Cloud Controller to provision virtual test environment based on requirements stated in test scripts (406); communicating with Performance Test Tool to configure test plan (408); and installing and configuring System under Test (SUT) in virtual test environment (410). Communication with Cloud Controller to provision virtual test environment based on requirements stated in test scripts further comprising launching SUT environment which consist of a plurality of System under Test Virtual Machines (SUTVMs). Another aspect of the present invention provides a method for provisioning test tool by extracting test results and displaying in portal. The method comprising steps of communicating with API (Application Programming Interface) of Performance Test Tool (502); and retrieving configuration setting from test scripts which contains load test design (504).

A further aspect of the present invention provides a method for communicating with Performance Test Tool to configure test plan. The method comprising steps of launching test tool at SaaS (Software as a Service) application and aligning test according to test goals (602); inserting test scripts to Test Tools (604); forwarding test case on median goals (606); and retrieving test results (608).

Another aspect of the present invention provides a method for communicating with Performance Test Tool to configure test plan. The method comprising steps of launching test tool at SaaS (Software as a Service) application and aligning test according to test goals (602); inserting test scripts to Test Tools (604); forwarding test case on median goals (606); and retrieving test results (608).

Another aspect of the present invention provides a method for installing and configuring System under Test in virtual test environment. The method comprising steps of realigning SUTVMs to ensure that each VMs will not compete with other resources (702); and realigning SUTVMs to reflect stability of test results (704).

The present invention consists of features and a combination of parts hereinafter fully described and illustrated in the accompanying drawings, it being understood that various changes in the details may be made without departing from the scope of the invention or sacrificing any of the advantages of the present invention.

BRIEF DESCRIPTION OF ACCOMPANYING DRAWINGS

To further clarify various aspects of some embodiments of the present invention, a more particular description of the invention will be rendered by references to specific embodiments thereof, which are illustrated in the appended drawings. It is appreciated that these drawings depict only typical embodiments of the invention and are therefore not to be considered limiting of its scope. The invention will be described and explained with additional specificity and detail through the accompanying drawings in which: FIG. 1.0 illustrates the system architecture of the present invention.

FIG. 2.0 illustrates the component of the system of the present invention.

FIG. 3.0 illustrates a simplified method of the present invention to automate test environment deployment utilizing metric recommender for performance testing on laaS (Infrastructure-as-a-Service) Cloud.

FIG. 4.0 is a flowchart illustrating a detailed method of the present invention to automate test environment deployment utilizing metric recommender for performance testing on laaS (Infrastructure-as-a-Service) Cloud.

FIG. 5.0 is a flowchart illustrating the method for provisioning test tool by extracting test results and displaying in a portal. FIG. 6.0 is a flowchart illustrating the method for communicating with Performance Test Tool to configure test plan.

FIG. 7.0 is a flowchart illustrating the method for installing and configuring System under Test in virtual test environment.

TABLE 1.0 is a table detailing the metrics to be stored in Metrics Store. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present invention provides a system and method for automating test environment deployment utilizing metric recommender for performance testing on laaS (Infrastructure-as-a-Service) Cloud. Hereinafter, this specification will describe the present invention according to the preferred embodiments. It is to be understood that limiting the description to the preferred embodiments of the invention is merely to facilitate discussion of the present invention and it is envisioned without departing from the scope of the appended claims.

Reference is first being made to FIG. 1.0 and FIG. 2.0 respectively. FIG. 1.0 illustrates the system architecture of the present invention while FIG. 2.0 illustrates the component of the system of the present invention to automate test environment deployment utilizing metric recommender for performance testing on laaS (Infrastructure-as-a-Service) Cloud. The present invention is accessed via a web portal which is hosted either on physical or virtual machine. Configuration of test plan and default metrics as well as aggregation of test results into a dashboard is conducted on portals which act as a test platform. All parameters for initialization will be compiled by portals and thereafter forwarded to Metrics Agent. Initialization is conducted on parameters such as performance criteria (e.g. 5 seconds for 100 concurrent users), load criteria (e.g. 100 concurrent users transaction with 50Kb per user), application architecture (e.g. using Tomcat JBoss, MySQL with load balance) and etc.

As illustrated in FIG. 1.0 and FIG. 2.0, the system architecture of the present invention comprising a user interface (104) for tester to set up test performance; a Metric Agent Module (106, 206) for introducing list of suitable metrics to be captured according to performance goals; an Interface Agent Module (212) for interfacing with API (Application Programming Interface) of Performance Test Tool and Cloud Controller; a Deployment Agent Module (108, 208) for provisioning test environment based on test scripts; a Reporting Agent Module (110, 210) for collecting and aggregating test result from Data Store (220) and displaying in portal; a Metric Store (102, 202) for storing list of metrics; and a Data Store (118, 220) for storing test results from Performance Test Tool (114, 218) and a Metric Logger (216) for monitoring and collecting required metrics during test execution. The Metric Agent Module (106, 206) and the Metric Store (102, 202) is used for Metric Suggestion Functionality wherein Metric Suggestion Functionality is based on application used by developer and data stored in Data Store (118, 220). The list of suggested metrics to be stored in Data Store (118,220) is detailed in Table 1.0. Reference is now being made to FIG. 3.0 and FIG. 4.0 respectively. FIG. 3.0 illustrates a simplified method of the present invention to automate test environment deployment utilizing metric recommender for performance testing on laaS (Infrastructure-as-a- Service) Cloud while FIG. 4.0 is a flowchart illustrating a detailed method of the present invention to automate test environment deployment utilizing metric recommender for performance testing on laaS (Infrastructure-as-a-Service) Cloud. As illustrated in FIG. 3.0, automated test environment deployment with metric recommender for performance testing on laaS Cloud is initialized by first setting performance goals (302). The methodology for setting performance goals further comprising steps of introducing list of suitable metrics to be captured according to performance goals (402); obtaining list of suggested metrics to create test specifications (404); and preparing test scripts for test deployment (404).

To perform the test, the tester must first select test tool such as JMeter, IBM RPT and designing the test cases. Test tool must adhere to the SUT interaction whether it is SOAP, REST or other API. Thereafter, the tester needs to design the test cases. Each test case must reflect the objective of the performance goal to ensure that the system is able to sustain at least 1000 concurrent connections or request and each request needs to be replied within a 5 second time frame. Suitable metrics are to be indentified and identifying suitable metrics involves a further step of collecting and aggregating system metrics to ensure problems or bottleneck are tracked at the right spots.

Thereafter, test environment is provisioned based on test scripts (304) as tester needs to configure test environment to mimic the production environment. In a complex multi-tier system, it may consist of database, application server, or web server. Provisioning on physical server for testing is expensive. Cloud Computing environment provides for a faster setup and provisioning in a control environment.

Upon successful creation of test specification by the Metric Agent Module (106, 206), the said Metric Agent Module (106, 206) communicates with the Interface Agent Module (212) for deployment process. Subsequently, the Interface Agent Module (212) uses test specifications which comprise of test environment setup and performance tool configurations to deploy and perform test execution (306, 412). The Interface Agent Module (212) communicates with the Cloud Controller for launching required virtual resources as stated in the specifications. The Cloud controller which runs on a specific platform is used to manage, launch, terminate and configure virtual test resources by exposing the web-service Application Programming Interfaces (APIs) for communication purpose with the Interface Agent Module (212). The default setup of the present invention comprises of web server (e.g. Apache Tomcat, Apache PHP and etc.), application server (e.g. JBoss, PHP, .NET and etc.) and database server (e.g. MySQL, Postgress and etc.).

Test cases are configured based on the specification through the Interface Agent Module (212) wherein the Interface Agent Module (212) communicates with performance test tool (114, 218) through APIs. For example, test specifications consist of the number of virtual users, the type of test plan, load size, test period and etc. Performance test tool (114, 218) includes the software for measuring performance, reliability, compatibility and characteristics such as JMeter, LoadRunner, Rational and etc. The said performance tool can be hosted in either physical or virtual resources. Test results obtained from the test execution is stored in Data Store (118, 210) for analysis and reporting while the collection of metrics are in accordance with the specifications provided by the Metrics Agent Module (106, 206).

The Deployment Agent Module (108, 208) provisions test environment in accordance to test scripts to ensure accuracy of test. Upon the availability of test beds (222), the

Deployment Agent (108, 208) begins the installation and configuration of the SUT.

Thereafter, the required application and database scheme will be uploaded by the tester via a portal which will invoke the Deployment Agent (108, 208) to communicate with deployed virtual test beds (222). The said Deployment Agent (108, 208) provisions test environment based on test scripts by communicating with the Cloud Controller to provision virtual test environment based on requirements stated in test scripts (406); communicating with Performance Test Tool to configure test plan (408) and thereafter install and configure SUT in virtual test environment (410). Uploaded files will be transferred to virtual machines using file transfer protocol (FTP). Installation and configuration is done by the script which is bundled together with the virtual machine. The Deployment agent will trigger the script once the files are successfully transferred. Communicating with the Cloud Controller to provision virtual test environment based on requirements stated in test scripts further comprising launching SUT environment which consist of a plurality of System under Test Virtual Machines (SUTVMs).

Reference is now being made to FIG. 6.0 and FIG. 7.0 respectively. FIG. 6.0 is a flowchart illustrating the method for communicating with Performance Test Tool to configure test plan and FIG. 7.0 is a flowchart is a flowchart illustrating the method for installing and configuring System under Test in virtual test environment. As illustrated in FIG. 6.0, communicating with Performance Test Tool to configure test plan further comprising steps of launching test tool at SaaS (Software as a Service) application and aligning test according to test goals (602); inserting test scripts to Test Tools (604); forwarding test case on median goals (606); and retrieving test results (608). Thereafter, as illustrated in FIG. 7.0, the methodology for installing and configuring the SUT in virtual test environment comprising steps of realigning SUTVMs to ensure that each VMs will not compete with other resources (702); and realigning SUTVMs to reflect stability of test results (704). Once the SUT is successfully installed and configured, the script will notify the tester that the system is ready to be tested.

Test execution process will be initiated by the tester through the portal which triggers the performance test tool to initiate the test plan. During initiation and execution of the test (306, 412), required metrics will be monitored and captured by the Metric Logger (216). These metrics are provided by Metrics Agent (106,206) in the test specifications (e.g. a Java-based web application running on Apache Tomcat, JBoss and MySQL server). By default, the number of response time, throughput, latency and bandwidth will be monitored by the performance test tool (114, 218). At the server side, the Metric Logger (216) monitors the performance of CPU, memory, network RX TX, disk read-write, JBoss service and etc which is not covered by the said tools. At least one Metric Logger (216) is embedded in all servers to monitor and collect the required metrics during test execution. Data Store (118, 220) stores test results in XML format for analysis and reporting process. The system monitors, captures and stores the test results accordingly in accordance to the pre-defined metrics (414).Test results collected will be stored in Data Store (118, 220) in XML format for analysis and reporting process. Thereafter, the system monitors, captures and store the test results based on pre-defined metrics (414). Reporting Agent (110, 210) is responsible in formatting the test results stored in Data Store in readable and graphical representation. Results obtained enable tester to dictate the areas that produced higher bottleneck to resolve the issue. The tester has the option to analyze, report and retest the test using the same resources with different test specifications to ensure test is completed as expected (308).

Reference is now being made to FIG. 5.0. FIG. 5.0 is a flowchart illustrating the method for provisioning test tool by extracting test results and displaying in a portal. Test tools are provisioned by extracting test results and displaying the formatted results into the portal dashboard (416) by communicating with API (Application Programming Interface) of Performance Test Tool (502); and retrieving configuration setting from test scripts which contains load test design (504). Upon completion of the test cycle, formatted results will be extracted and displayed on the portal dashboard.

The system and method of the present invention provides an automated test environment deployment utilizing metric recommender for performance testing on laaS (Infrastructure-as-a-Service) Cloud. The metric recommender of the present invention enables automated test environment deployment wherein performance test metric is automated using physical resources of virtual machine in which the complexity of setting up test environment will be overcome by providing cloud computing to set up test virtual environment and aggregating the correct metrics using metric agents to achieve performance goals.

The present invention may be embodied in other specific forms without departing from its essential characteristics. The described embodiments are to be considered in all respects only as illustrative and not restrictive. The scope of the invention is, therefore indicated by the appended claims rather than by the foregoing description. All changes, which come within the meaning and range of equivalency of the claims, are to be embraced within their scope.

Claims

CLAI S

1. A system (100, 200) to automate test environment deployment utilizing metric recommender for performance testing on laaS (Infrastructure-as-a-Service) Cloud, the system comprising:

at least one user interface (104) for setting up test performance; at least one Metric Agent Module (106, 206) for introducing list of suitable metrics to be captured according to performance goals;

at least one Interface Agent Module (212) for interfacing with API

(Application Programming Interface) of Performance Test Tool and at least one Cloud Controller;

at least one Deployment Agent Module (108, 208) for provisioning test environment based on test scripts;

at least one Reporting Agent Module (110, 210) for collecting and aggregating test result from Data Store (220) and displaying in portal; at least one Metric Store (102, 202) for storing list of metrics; at least one Metric Logger (216) for monitoring and collecting required metrics during test execution;

at least one Data Store (118, 220) for storing test results from Performance Test Tool (114, 218) and Metric Logger (216).

2. A system (100, 200) according to Claim 1 , wherein the at least one Metric Agent Module (106, 206) and the at least one Metric Store (102, 202) is used for Metric Suggestion Functionality wherein Metric Suggestion Functionality is based on application used by developer and data stored in Data Store (118, 220).

3. A system (100, 200) according to Claim 1 , wherein the at least one Deployment Agent Module (108, 208) for provisioning test environment based on test scripts further having means for:

communicating with Cloud Controller to provision virtual test environment based on requirements stated in test scripts;

communicating with Performance Test Tool to configure test plan; and installing and configuring System under Test (SUT) in virtual test environment.

A method (300, 400) to automate test environment deployment utilizing metric recommender for performance testing on laaS (Infrastructure-as-a-Service) Cloud, the method comprising steps of:

initializing test by setting performance goals (302);

provisioning test environment based on test scripts (304);

performing test execution (306, 412);

monitoring, capturing and storing test results based on pre-defined metrics (414);

analyzing, reporting and retesting to ensure test completed as expected (308); and

provisioning test tool by extracting test results and displaying in portal (416).

A method (400) according to Claim 4, wherein initializing test by setting performance goals further comprising steps of:

introducing list of suitable metrics to be captured according to performance goals (402);

obtaining list of suggested metrics to create test specifications (404); and preparing test scripts for test deployment (404).

A method (400) according to Claim 4, wherein provisioning test environment based on test scripts further comprising steps of:

communicating with Cloud Controller to provision virtual test environment based on requirements stated in test scripts (406);

communicating with Performance Test Tool to configure test plan (408); and

installing and configuring System under Test (SUT) in virtual test environment (410).

7. A method (500) according to Claim 4, wherein provisioning test tool by extracting test results and displaying in portal further comprising steps of:

communicating with API (Application Programming Interface) of Performance Test Tool (502); and retrieving configuration setting from test scripts which contains load test design (504).

8. A method according to Claim 6, wherein communicating with Cloud Controller to provision virtual test environment based on requirements stated in test scripts further comprising launching SUT environment which consist of a plurality of System under Test Virtual Machines (SUTVMs).

9. A method (600) according to Claim 6, wherein communicating with Performance Test Tool to configure test plan further comprising steps of:

launching test tool at SaaS (Software as a Service) application and aligning test according to test goals (602);

inserting test scripts to Test Tools (604);

forwarding test case on median goals (606); and

retrieving test results (608).

10. A method (700) according to Claim 6, wherein installing and configuring System under Test in virtual test environment further comprising steps of:

realigning SUTVMs to ensure that each VMs will not compete with other resources (702); and

realigning SUTVMs to reflect stability of test results (704).