JP2003271418A - Method for testing software object of web base - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system for performing a remote test for middleware of an application in an N-hierarchy model via a network. <P>SOLUTION: The test system includes a test code generator, a test engine for executing a plurality of copies of a test code, and a data analyzer for analyzing a result and presenting the result to a human user. The system automatically generates the test code, and is capable of exercising components of the middleware remotely developed by using information on the components usable for an application under test. A plurality of copies of the test code are executed by a synchronizing system. Execution time for a plurality of events is recorded and is presented in one of a plurality of formats. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

BACKGROUND OF THE INVENTION Distributed computing has been in use for many years. Distributed computing is very often used in "enterprise-wide" applications. Enterprise-wide applications are applications that allow many groups of people to work together on a common task. Typically, enterprise-wide applications perform functions that are important to a company's business. In banks, for example, people at all bank branches must have access to a database of accounts for all bank customers.
Similarly, an insurance company must have access to a database containing information about all policyholders for all people in the company. Software that performs these functions is commonly known as enterprise-wide applications.

The architecture of enterprise-wide applications is changing as the available hardware and software are developed. The currently popular architecture is called the N-tier enterprise model. The most popular N-tier enterprise is the three-tier model. The three layers are the front end, middleware, and back end. The back end is a database. The front end is sometimes referred to as the "client" or graphical user interface (GUI). Middleware is software that manages the interaction with the database and acquires "business logic." Business logic tells the system how to inspect, process, and report data in ways that are useful to people in the enterprise.

The middleware resides in a computer called a server. The databases can be on the same computer or different computers. The "client" is always on an individual's personal computer.
All computers are connected by a network. Since many people use enterprise-wide applications, such systems will be configured for simultaneous user availability and there will be many clients connected to a single server. Often, many clients will be connected to the server at the same time.

Those familiar with Internet commerce are N
It will be appreciated that the hierarchical model also describes many Internet websites selling goods and services. For example, a car auctioning website would fit the N-tier model. In such applications, a database is provided to keep track of buyers, sellers, and auctioned properties. In addition, databases must be provided to keep track of bids as they are entered. Middleware
It provides access to these databases and hides the business logic of transactions such as when to accept bids and when to declare items to be sold. In the field of distributed computing,
It makes no difference whether the "client" using the application is an employee of one company or many Internet users around the world. Here, an example of application undertest will be given,
They are not intended to limit the use of the invention. The invention described herein can be used by developers of enterprise-wide or web-based applications.

One advantage of the N-tier model is that the middleware will be very componentizable and can be described in component standards so that it will be easily integrated with software in other tiers. Sun Microsystems Enterprise Java ™
Beans (EJB), Microsoft COM, D
COM, COM + and SOAP (Simple Object Access Protocol) are examples of specified standards for commercially available components. Where EJB
Is used as an example of a component standard used to implement middleware in an N-tier model, but it should be understood that the concepts described herein can be used with other component standards. .

The EJB is written in the JAVA (registered trademark) language and is intended to be "platform independent". Platform-independent means that an application behaves the same regardless of the hardware or operating system it is running on. Platform independence is achieved through the use of "containers". A container is software designed for a particular platform. It provides a standardized environment that ensures that applications used in the platform independent language work correctly. Containers are always commercially available software, and application developers will develop rather than create them.

Componentized software is
Different types of applications, or software in which "objects" are created separately, but are designed to allow objects to work together. In order for this to happen, the object must have a standard interface that is understood and accessible by other objects. Software languages enforce some parts of these interfaces. The software interface is used by the discovery mechanism to discover interface information when it is not directly available as part of the system. If the interface is not used, software objects will not be able to work with other objects. Other practices are enforced by convention. These programming practices are known to all, so the company that creates the container can rely on them when creating the container. As a result, if these practices are not followed, the container will not be able to work properly. As such, there are indirect mechanisms for implementing these practices.

Applications are typically tested in one of two ways. Objects are tested as they are described. Each is tested to ensure that it performs the intended function. When an object is assembled into a completed application, the entire application is always tested. Until now, application testing has typically been done by applying test input at the client end and observing the application response.
This process has various drawbacks. One is that a relatively large workforce is required, especially to develop load or scalability tests. There is no easy way to create a test program, instantiate it with test data, run a test, and collect the results.

Several tools called "profilers" are available. However, these tools keep track of disk usage, memory usage, application undertest thread usage, and so on. They do not provide data about application performance based on load.

Other tools can be used to automate the execution of tests on the application. For example, M
A, Waltham's RSW Software Company is e-Loa
Providing a product called d. This tool simulates load on application undertest,
Provides information about application performance. However, this tool does not provide information about the components in the application. We understand that software developers find such information very useful.

Teradyne So, NH, Nashua
Further available are automatic test generation tools such as TestMaster, available from ftware and System Test. This type of tool provides a means to reduce the manual effort to generate tests. TestMaster works from the application undertest state model. Such applications are very useful for generating functional tests during application development. When the application model is specified, Tes
The tMaster may be instructed to generate a suite of tests that may be suitable for a particular task, such as fully exercising some parts of the changed application. Model-based testing is particularly useful for functional testing of particularly large applications, but is not fully automated as it requires the creation of a state model for the application being tested.

[0012] We note that the second drawback of enterprise-wide application testing is that a critical performance metric to measure is often related to how the application behaves as the number of concurrent users increases. I understand that. If too many users log at the same time, there is an example of website crashing or behavior that is too slow to annoy normal users. In the past, the load was informally simulated, for example by letting some people try to use the application at the same time. MA,
There are several tools to provide the load with a test application such as e-Load available from Waltham's RSW.

However, the performance of application underloading is generally known only after the application is deployed in the desired operating environment.
Therefore, the biggest problem faced by application developers is that it cannot be a test to see if each object behaves as designed or even works together as a system. How many concurrent users the middleware application can accommodate in the transaction response time specified so far,
The bottleneck is caused by the lack of useful tools that help application developers to determine which objects in an application to identify under real world load conditions. Response time is one major load test measurement. The other is throughput. Response time measures the time it takes for an individual transaction to complete. Throughput is a measure of how many transactions are being processed per unit of time and measures the amount of work the system is doing. In addition, developers who wish to test or identify components want to do so without writing their own tests, and without providing their software components to others.

SUMMARY OF THE INVENTION With the above background in mind, it is an object of the present invention to provide a web-based test harness to facilitate remote testing and / or verification of software components deployed in a remote system. It is to be. Software components can be load tested remotely. The tests used to test and verify the remote software component can be automatically generated from or for the remote software component. Tests can be performed on a single software component, multiple software components, or an application containing one or more software components. In the presently preferred embodiment, the test system analyzes response time measurements from multiple software objects in the application and predicts which software objects in the application are likely to be performance bottlenecks.

DETAILED DESCRIPTION OF THE INVENTION FIG. 1 depicts a test system 110 in accordance with the present invention. The system is an application under test 114 for remote testing. Here, the application under test 114 is an N-layer model application. More specifically, it is a three-tier database application. The application under test 114 can represent a database for a bank or insurance company, or it can represent an internet application. application·
The particular function of undertest 114 is not critical to the invention.

Moreover, the particular hardware on which test system 110 and application undertest 114 reside is not critical to the invention. It suffices to have some connections between the two deployed apart from each other. In the figure, the connection is provided by the Internet 122. In this scenario, test system 110 may be deployed on a server owned by a test company or application service provider (ASP) 100. Many applications are written using platform-independent technology so that splications work the same on many different platforms.
Platform independent technology is intended to facilitate the execution of applications on any platform.

Application under test 114
Is a software application known in the art. It contains middleware 116 that hides some business logic. The user accesses the application via the client device. Many types of client devices are possible as networks become more prevalent and lists grow. Personal computers, telephone systems, and even household appliances that have a microcontroller can be client devices. In use, it is contemplated that there may be multiple users connected to the application under test 114. The number of users who are simultaneously accessing the application undertest 114 is one indicator of application "load."

In the exemplary embodiment, access to the application undertest is provided by a graphical user interface (GUI) 124 of the type known in the art. Software is known that manages interactions between multiple users and applications. Such software is sometimes called a web server. A web server works in conjunction with a browser and is software commonly found on most PCs.

Web server and browser are HTML
Exchange information in a standard format known as. The HTML file contains tags and specific information associated with each tag. The tag signals the type associated with the information to the browser, allowing the browser to display the information in the proper format. For example, a tag signals whether the information is the title of a page or the information is a link to another web page. The browser creates a screen display in a particular window based on one or more HTML pages sent by the web server.

When the user enters a command or data into the browser window, the browser formats this information using information about the HTML page and sends it to the web server. Thus, the web server knows how to process the commands and data coming from the user.

The GUI 124 passes the received information and command to the middleware 116. In the example of FIG. 1, the middleware 116 is depicted as a middleware application created with the EJB. Container 130
Is a commercially available container in the preferred embodiment. Inside the container, there are many enterprise Java (registered trademark) beans 132. Each Java (registered trademark) beans 132
Can be more generally thought of as a software component. The GUI 124 sends the information to the appropriate EJB.
Pass to 132. Application Under Test 1
The output from 14 is G for display to the user.
Returned to the UI 124.

EJB 132 collectively executes database applications in the illustrative example. EJ
B132 manages the interaction with the data from databases 126 and 128 and their processing. They will perform such a database function as a set value in a particular record, or a value obtained from a particular record. Another function is to create columns in the database and discover columns in the database. The EJB that accesses the database
Often called "Entity Beans".

Another type of EJB performs a computational or control function. These are called "session beans". Session beans perform functions such as inspecting data entries or reporting to the user that an entry is incorrect. Session beans call entity beans to perform database access.

Each type of database transaction is controlled by a single bean that executes only that function, while programming the application so that some entity beans execute a function that is not strictly tied to database access. Separation is generally preferred. Similarly, some session beans will perform database access functions without calling entity beans. Thus, although different test techniques are described for test session beans and entity beans, some EJBs can have attributes of both entity beans and session beans. After all, a full test of any bean could utilize the techniques of testing entity beans and testing session beans.

The test system 110 is the Internet 1
On 22, the application under test 114 EJB 132 can be accessed. This way, each bean can be exercised remotely for testing on the developer's site. In this way, the developer simply allows the ASP 100 to access the software components over the Internet and perform the tests. In the preferred embodiment, the test is primarily directed at determining the response time of the beans,
More generally it is directed to determining the response time of components or objects used to create application undertests. Knowing the response time of the bean allows conclusions about the performance of the application. As mentioned above, response time is one major load test measurement. Another is throughput. Response time measures the time to complete an individual transaction. Throughput is a measure of how many transactions are being processed per unit of time and measures the amount of work the system is doing. Details of test system 110 are described below.

In the exemplary embodiment, test system 110 is software installed on one or more servers at an ASP location. In the preferred embodiment, the test system 110 is a JAVA application. Similar to the application under test 114, the test system 110 is controlled by the graphical user interface 150. GUI 150 may be a web server known in the art.

Returning now to FIG. 2, the test system 11
0 details are shown. The test system 110 performs several functions. One function is the generation of test code. The second function is to execute test code to remotely exercise one or more EJBs in the application under test. Another function is to record and analyze the results of test code execution. These functions are performed by software executing on one or more computers coupled to internal network (Intranet) 102.
The software uses commercially available languages to perform the functions described below.

FIG. 2 illustrates that test system 110 has a distributed architecture. The software components are installed on several different computers in the test system. Computers are used to provide capabilities to multiple users, perform multiple tasks, and perform a large number of tests. The particular number of computers and the distribution of software components of the test system on these computers is not critical to the invention.

The coordinator 210 is a software application that interfaces with the GUI 150. The main purpose of the coordinator 210 is to route the user's requests to the appropriate server in a user transparent manner. Turning to FIG. 3, the coordinator 210 is shown in more detail. However, it should be understood that FIG. 3 shows a conceptual structure of the coordinator 210. Coordinator 210 is not a single, but separately identified piece of software.
For example, it may be implemented as coordination software within various other components of test system 110. Further, it should be appreciated that the web server used to implement the GUI 150 also provides coordination functions such as querying multiple requests from multiple users, individually or coordinated.

The coordinator 210 is the distribution unit 3
Contains twelve. The distribution unit 312 is preferably a software program running on the server.
The user's request is received by the distribution unit 312.
When a request is received, distribution unit 312 determines the type of resources needed to process the request.
For example, a request to generate code must be sent to the server running the code generator.

The coordinator 210 contains several queues for holding pending requests. Each queue is implemented in the memory of the server that implements the coordinator 210. In FIG. 3, cue 3
18A. ． ． 318C is described. Each queue 31
8A. ． ． 318C corresponds to a particular type of resource. For example, queue 318A may include code generator requests, queue 318B may include test engine requests, and queue 318C may include data analysis requests. The distribution unit queues each request based on the type of resource required to process the request, queue 318A. ． ．
318C to any one.

Each queue 318A. ． ． 318C, the queue manager 320A. ． ． 320C is present. Each queue manager is preferably implemented as software running on a server implementing coordinator 210 or a server implementing an associated coordinator 210. Each queue manager maintains a list of servers in test system 110 that can respond to requests in its associated queue. The queue manager sends the request at the top of the queue to the server equipped to handle the request. The connection between the queue manager and the server equipped to handle the requests is on the Internet 102. If there are other servers available and there are more requests in the queue, the queue manager will send the next request in the queue to the available servers. If no servers are available, each queue manager waits for one of the servers to finish processing its assigned request.

When the request is processed, a server such as a code generator or test engine returns a report to the queue manager. In response, the queue manager sends another request from the queue and provides the result to distribution unit 312. The distribution unit 312 can be returned to the user who made the request, indicating that the request was completed, giving the result or the location of the result. For example, after the test code is generated, the user may receive an indication of where the test code is stored. After the test is run, the user can receive a report of the average run time of the test or the location of the file storing each measurement made during the test.

Those skilled in the art will appreciate that software systems for processing user commands, including commands from multiple users, are well known. Such systems must have an interface for receiving commands from users, processing those commands, and presenting the results to the user. Such an interface allows those results to be used by the user to implement further commands. Such an interface is often utilized here and is generally depicted as GUI 150. For example, GUI 150 may allow a user to enter a command that directs the code to be generated to test a particular application. Once the code is generated, the GUI 150 allows the user to specify that the test should be run utilizing that test code.

It is possible that some requirements may require coordination of multiple hardware elements. As described below, one of the functions of the test engine is to apply a load that simulates multiple users. In some examples,
A single computer can simulate multiple users by executing multiple client threads. However, there is a limit to the number of client threads that can be executed on the server.

To operate the test system 110,
Test code must be present. The user can provide the test code. The test code is NH, Nashua's Teradyne Softw
It may be provided by an automatic code generation system, such as TESTMASTER sold by are and System Test. Nevertheless, FIG. 2 illustrates that code generators 212A and 212B are used in the preferred embodiment for creating code. Returning to FIG. 4, further details of code generator 212 are shown.

The code generator 212 includes a number of scripts 512. Each script is a series of steps that the code generator 212 must perform to create the code that performs a particular type of test. The script can be prepared in any convenient programming language. Test system 1
A script is provided for each type of test that 10 will perform. User input regarding the type of test desired specifies which script 512 should be used to generate code at any given time.

The selected script 512 collects the test code 516. The information needed to gather test code 516 comes from several sources. One source of information is the test template 514. There are some steps required in most types of tests. For example, the object being tested must be expanded (deplo
y), some initialization sequences are required. If the test is repeated, there must be code to start the test at the specified start time, and the end time of the test must be recorded. further,
There must be code that allows the required data to be logged during the test. After the test there are also some stopping steps required. For example, if initialization starts with a request for a reference to a particular EJB, the test code may stop at that reference being released. The test code that these steps execute is captured in a set of templates 514.

In addition, there may be different templates that ensure that the test code 516 properly reflects the input provided by the user. For example, different containers may require different command formats to achieve the same result. One way these different formats can be reflected in test code 516 is by having different templates for each container. Alternatively, the user could specify the type of information to be recorded during the test. In this example, the data logging preference can be achieved by having a set of templates with different command lines that cause the data to be recorded during the test. An exemplary template is shown in Appendix 2.

Templates are written to fill a particular space to customize the code for the particular object being tested. In the preferred embodiment, code generator 212 generates code for testing a particular EJB in an application under test. Information that would need to be filled for many templates is being tested
This is the description of B. Another information that may be included is user code to put the application undertest in a state suitable for testing. For example, in testing a component of an application that maintains a database of account information for banks, you need to create a specific account in the database to use for testing purposes, or else before testing. The application needs to be initialized. The code required to trigger these events is unique to the application and will therefore be best inserted in the test code by the tester testing the application. In this exemplary embodiment, this code is inserted into the template and then brought to the final test code.

The template may further include space for a human tester to fill other information such as a particular set of data to use for testing. However, in the presently preferred embodiment, the dataset is provided by a human user in the form of a data table.

Code generator 212 may generate functional tests. A function test is a test aimed at determining whether the bean will perform exactly the required function. In functional testing, software undertesting is exercised with many test cases to ensure that it works correctly in all situations. A data table showing the expected output for various inputs is used to create the functional test software. However, in the presently preferred embodiment, the code generator 212 first generates test code that executes the load test. In a load test, it is not necessary to stimulate the software to exercise all the possible functions and function sets that the software will perform.
Rather, it is usually sufficient to provide one test condition. The purpose of load testing is to measure how the behavior of the software decreases as the number of concurrent users of the application increases.

In the preferred embodiment, the test system 110 includes scripts 512 for implementing various types of load tests. One type of load test determines the EJB's response time. This allows the test system to vary the load on the EJB to determine a reduction in response time corresponding to the increased load. Another type of load test is a regression type load test. In the recursive load test, the script performs operations to determine if the EJB is responding in the same way that the EJB responded to some baseline stimuli. In general, the response to the baseline stimulus represents the correct behavior of the EJB. Having a recursive test allows the test system 110 to increase the load on the bean and determine the error rate as a function of load.

Script 512 is used to generate test code 516 for these types of load tests.
Must create test code specific to the bean under test. The user provides the GUI 150 with information about which bean to test. In the preferred embodiment, this information is provided by the human tester who provides the filename in the application undertest that contains the "deployment descriptor" for the particular bean undertest. This information specifies where in the network the bean under test should be found. The script 512 uses this information to see what test code must be generated to test the bean.

The script 512 can generate code using the attributes of the platform independent language in which the bean is described. For example, in the Sun JAVA language used herein, each bean has an application program interface called "reflection." More specifically, each bean has a "home" interface and a "remote" interface. The "home" interface reveals information about how to create or discover a remote interface in a bean. The remote interface reveals how this code can be accessed by the client software. Of particular interest in the preferred embodiment, the home and remote interfaces provide the information needed to create a test program for accessing the bean.

The use of reflection allows any program to determine what is known as the "property" and "method" of a bean. Bean properties describe the data type and attributes for the variables used in the bean. All variables used in the bean must have properties associated with them. Thus, the script 512 automatically determines what method needs to be exercised to test the bean and the variables that need to be generated to stimulate the method. The variables that will depend on the method when tested can be further determined. In the preferred embodiment, this information is stored in the symbol table 515.

The symbol table 515 is a file in any conventional file format. Once the information about methods and properties is captured in the table,
Script 515 uses this information to create test code that exercises the methods and properties of the particular component undertest. Especially script 51
5 automatically creates a variable of the correct data type and assigns it a type-matched value for any variable used in the bean.

The data generator 518 uses the information from the reflection interface to generate values for the variables used during the testing of the bean. There are many ways in which an appropriate value can be generated for each variable used in testing a particular bean. However, in the commercial embodiment of the present invention, a selection of three different algorithms that the data generator 518 will use to generate the data values is provided. The user can specify "maximum", "minimum", or "random". When the maximum selection is specified, the data generator 518 analyzes the characterization obtained by the reflection interface to determine the maximum allowed value. If the user specifies "minimum", the data generator 518 will generate the smallest possible value. If the user specifies random, the data generator 518 selects a random value between the maximum and minimum.

In many instances where a load test is desired,
The exact value of a particular value is not important. For example, when testing whether a bean stores and retrieves values from a database properly, it doesn't always matter which value is stored and retrieved. It only matters if the value read from the database is the same as the stored value. Also, when timing the behavior of a particular bean, what value is input to the method is less important. In these scenarios, the data generator 518 can automatically generate values for variables used in the test code.

If the particular value of the variable used in the test is important, code generator 212 provides the user with another option. Data generator 51
Rather than the derived value of the variable from 8, the script 512 may be instructed to obtain the data value from the user-provided data table 520. The user may want to provide a data table for further load testing, for example, when the execution time of a particular function will depend on the value of the input data.

The data table is simply implemented as a file in one of the computers of the network 122. Entries in the table that specify values for particular variables to use as inputs and outputs to particular methods are separated by delimiters in the file. The standard format for such tables is the CSV format. In the preferred embodiment, test system 110 includes a file editor of the type using conventional techniques for creating and editing such files. In addition, test system 110 may include the ability to import files having the required formats using known techniques.

The bean method describes the function that the bean can perform. Part of the method description is the characteristics of variables that are inputs or outputs to the method. The second part of each method description that can be determined by the reflection interface is the command needed to invoke this method. Script 512
The script 512 is optional in the bean as it can determine the code needed to trigger any method as described above and generate a data value suitable for providing as input to that method. It is possible to generate code that calls the method of. The record technology occupies the user's data table (popula).
can be used to

In the preferred embodiment directed to load testing, the order in which the bean's methods are called is not critical to efficient testing. in this way,
The script 512 can automatically generate useful test code by invoking each method of the bean. There are currently three methods for ordering reflected scripts, mainly recording, code filtering, and UML metadata.

More mature tests can be built automatically depending on the patterns defined for the language. S
In un JAVA ™, the entity beans for controlling access to the database should have methods with the prefix “set” or “get”. These prefixes signal whether the method writes data to or reads data from the database. The method name affix indicates which value is written to or read from the database. For example, a method named setSSN should perform a function that writes a value to the database for the parameter identified as the SSN. getSSN should read the value from the parameter named SSN.

By utilizing these defined patterns, the script 512 can generate code for exercising and confirming the operation of both methods. The test code generated to test these methods will first exercise the method setSSN by providing it with the arguments created by the data generator 518. Then the method getSSN will be exercised. If the get method returns the same value as the argument provided to the set method, then it can be verified that the database access was performed in a scheduled manner.

For many types of enterprise wide applications, the beans that are most likely to be load sensitive are those that access the database. Thus, tests of set and get methods only provide very useful load test information.

However, the amount of testing done can be expanded if necessary. Some beans also include methods to create or find rows in the database. Traditionally, methods to create or find columns in databases have been named to begin with "create" or "find". By reflecting the bean interface in this manner, the script 512 can further determine how to test these methods. These methods are as executable as the set and get methods. The property exposed by the application interface will be the described format of each column in the database.
When the create method is used in this way, the data is automatically generated to fill the column, thus performing the create method satisfactorily.

In the preferred embodiment, the find method is exercised using data from the data table 520 provided by the user. Databases often have test columns inserted into them, especially for testing. Such test columns could be written to the data table 520. However, to create a column, fill it with data, and then find fin
It would be possible to exercise the d method.

Once the command to exercise the method of the EJB has been created, the script 512 can insert the necessary commands into the client test code 516 to record the output of the test. If the test does a lot of error checking, test code 5
16 must include an instruction to record an error in log 216. Similarly, if the test is measuring response time, the test code 516 must include instructions to write information to the log 216 that may determine the response time.
In the described embodiment, all major database functions can be performed without user-supplied test code. In some examples,
It is possible to exercise all functions with all automatically generated data. All the required information could be generated from the application undertest object code only. An important feature of the preferred embodiment is "minimum invasive",
That is, no user is required to run the test, and the test does not impact the customer's environment.
There is no harm from the Invasive Test. The client code executes code exactly as the user would write it.

There are several possible ways in which the “response time” of remote software components can be measured. As mentioned above, response time is one of the main load test measurements. Another is throughput. Response time measures the time required for an individual transaction to complete. Throughput is a measure of how many transactions are being processed per unit of time,
Measure the work done by the system. One way is that the total time to execute all the methods in the bean is measurable. Another way is that the start-up time of the bean can be measured. The start-up time is the time it takes for the first method to be executed after the bean is first accessed. Another method to measure response time is to measure the time it takes to execute a writing method. As another variation, the response time may be measured based on the time taken to execute only the "get" method or the time taken to execute only the "set" method.

Depending on which measurement of response time is used, different measurements must be recorded. For example, if only the total response time is needed, it is sufficient to simply record the time at which the portion of the test code exercising the entire method performs the start and end. If a startup response time is required, the client test code must first record the time to access the bean under test, the time the first method in the test sequence prepares to call.
On the other hand, if the response time is calculated for each method, the client test code must record the time before and after each method is called,
Some indication of the method being called must be further logged. In these cases the information needs to be recorded for only a subset of the methods, but similar information must be recorded if the response of only the "get" or "set" function has to be measured. .

Furthermore, if there are multiple users to be simulated, there will be multiple values for each data point. For example, if the test system 110 simulates remote users, the time it takes the bean to respond to each simulated remote user can be different, reaching up to 100 different response time measurements. The response time for 100 users may be presented as the maximum response time, ie the time it takes for all 100 simulated users to complete the bean under test. Alternatively, the average time to complete can be reported as response time. As another variant, a range of values can be reported.

In the preferred embodiment, client test code 516 includes instructions to record all the information and all possible display formats that would be needed for any possible measurement of response time. The start and end times of execution of all methods are recorded. In addition, instructions are recorded that allow the method to be identified. To support factor-based analysis rather than delay, the actual and expected results of each method execution are recorded so that errors can be detected. In addition, exception occurrences are logged. The data analyzer then reviews the log and displays the response time according to any format and using any definition of response time desired. That is, the data analyzer can count the number of exceptions or errors.

Once the data is stored, the user can specify the desired format in which the data will be presented. In the preferred embodiment, the test system 11
0 has the ability to present the test results graphically and help the tester understand the performance bottlenecks of the operation, and in particular the application undertest 114.

One important output is the response time vs. load graph. The log file contains the start and end times of the test run for a particular test case. The test case contains measurements at several different load conditions (ie, test engine 214A ... 214).
C simulates a different number of concurrent users). In this way the data analyzer can read the data in the log and identify the results obtained under different load conditions. This data can be graphed.

Another useful analysis is the number of errors per second generated as a function of the number of concurrent users. To perform this analysis, test code 516 may include instructions to write an error message to the log whenever a test statement produces an incorrect result. In the database context, incorrect results are identifiable if the "get" function does not return the same one passed as an argument to the "set" function. That is, when accessed, it can be identified if the bean does not respond or responds with an exception condition. As mentioned above, the data analyzer 218
Reads the number of errors in different states, passing through the log file and simulated. If desired, the error can be expressed as an error count, that is, the error count divided by the time it takes to perform the test.

Some examples of the types of output that can be provided are transactions per user per second, response time per user, exceptions per user, errors per user, method response time, response time vs execution time. , Is a graph showing transaction versus execution time per second. Various methods of measuring response time have been described above. In the preferred embodiment, a transaction is defined as the execution of one method, but other definitions are possible.

Execution time is defined as the total elapsed time to execute a test case and will include the time to set up the execution of the EJB. Viewing response time as a function of elapsed time is useful, for example, in revealing problems such as "memory leaks." A memory leak is a condition in which some of the memory on the server running the application undertest is used unproductively. When more memory is used unproductively, less memory is available to perform application undertests and execution is delayed. Alternatively, viewing the results in this format may reveal that application undertesting makes efficient use of caching. When caching is used efficiently, execution time can decrease as elapsed time increases.

By describing the structure of the test system 110 and giving examples of its applications, some important features of the test system 110 can be seen. One of the features is that the performance of application under test is easily obtained and a lot of data is obtained in an automatic manner. Software developers can use test systems to find specific beans that tend to be performance bottlenecks in applications. The developer
Rewrite these beans or change their deployment descriptors. For example, one aspect of the deployment descriptor is the instantiation (i
The number of bean copies that can be nstantiated) is shown. The developer may increase the number of instantiations of the bean if the bean is the bottleneck.

The test system described herein provides an easy and accurate tool for remotely testing EJBs for scalability. Create a number of virtual users specified by the user who calls the EJB while deployed on the application server. The tool searches for EJB undertests, automatically generates client test programs, uses rule-based data or provided data, multithreads client test programs, and drives EJB undertests to Do things. The result is a series of graphs that report performance versus number of users, providing useful information that makes the format easy to use.

Another feature of the invention is that the test does not require any changes in the application under test,
That is, the tests are run without the need to additionally install a special test agent on the server, including the software undertest. The generated test code 516 exercises the bean in the application under test using remote procedure calls.

Another feature of the described embodiment of test system 110 is that it is scalable. More test engines can be added to increase the number of tests that can be run at the same time, that is, to increase the size of tests that can be run. Similarly, more code generators can be added to support the simulation of a large number of simultaneous users. The designated number of copies of each component is not critical to the invention. The actual number of components in a given embodiment tends to vary from installation to installation. As applications support more users, more test engines tend to be needed.

Another feature of the described embodiment is that the test is done in the simplest configuration in the application under test, the bean in the described example. This method has two advantages. First, the test is very simply generated with minimal human intervention. Second, it allows software developers to focus on the points of the software that need to be changed or adjusted to improve performance.

It should be appreciated that various other analyzes can be performed. When the load increases, there are often some points at which the performance of the system changes dramatically.
In some examples, the time to complete a transaction increases dramatically. The dramatic change in transaction process time indicates that the system could not handle the load efficiently. However, the decrease in processing time is
It further indicates that the load limit has been reached. In some cases, the system under test will react to the error message more quickly than it would take to produce an accurate response. Thus, if the only parameter being tracked is response time, a decrease in processing time as a function of load can signal that the maximum load has been exceeded. Of course, an error or an increase in error rate may signal that the maximum load has been exceeded. Furthermore, by executing multiple test cases and concentrating each test case on a different bean, the test system 110 may automatically determine the performance bottleneck bean and assign a load rate to the application under test 114. .

While one embodiment has been described, many alternative embodiments or variations can be made. For example, it has been described that the test system 110 automatically generates test code and automatically generates test code that exercises the bean tracking database access patterns. These beans are sometimes referred to as "entity beans". Generally, there are other beans in the application that perform calculations on the data and control the execution time of the entity beans. These beans are sometimes referred to as "session beans". Session beans do not tend to track defined program patterns that simplify the generation of test code for entity beans. As a result, test code for automatically generated session beans will not be able to fully test these beans. In the described embodiment, the human tester is expected to provide test code to test the session bean if the automatically generated test is incorrect.

One possible variation on the described embodiment is that the completeness of the test for session beans can be increased. One way to increase the accuracy of testing session beans is to capture data about the execution of these beans during the actual operation of the application under test 114. This data allows the automated system to determine things such as the appropriate data values that can be used to build the data table. That is, the acquired data will allow the automated system to determine the order in which the session beans will access other session beans or entity beans to create a genuine test.

Further, as described, test code is generated to test a particular bean, which is a simple construct or "component" of application undertest. The test may focus on different configurations, such as a particular method on the bean. Test code can be generated to test a particular method in a bean. It was also described that the system records the start and end times of test code execution. The time of other events may be recorded instead or in addition. For example, the start and end times of individual methods can be recorded and the performance of individual methods can be determined.

Alternatively, the complexity of the configuration being tested can be increased. Multiple beans can be tested simultaneously to determine the interactions between the beans. For example, multiple test cases may be run simultaneously such that one test case exercises a specified instance of one bean and another test case exercises a specified number of instances of a second bean.

As another example of possible changes, the number of templates used to construct the test code can be changed. One possibility is that each template contains all the steps needed to start, run and stop the test. Therefore, the test code would be created by filling a single template. Alternatively, each template may only include the steps necessary to perform one function, such as start, test, or stop. In this realization,
The test code will be created by stringing multiple templates together.

It was further described that many concurrent users were "synchronized" in running tests. Simultaneous users are simulated by synchronizing copies of test code on different servers and the same server.
The term "synchronized" should not be construed as limited to methods suggesting that the multiple copies each perform the exact same function at the exact same time. Therefore, if the executions are described here as being synchronized, then it is only necessary that each copy of the code requires an application under test during the window of time that the test is executed. It is said that. Some copies of the code may start executing sooner or end sooner than the other. But as long as there is overlap in execution time, it can be said that the test programs are synchronized, that is, they are executed simultaneously.

As a further modification, the test system 11
It was stated that 0 provides an output that indicates the performance of the application undertest as a function of load. These outputs, in graphical or tabular form, can be used by application developers to identify the number of concurrent users who are likely to encounter problems in the application. Potential problems are revealed in various ways, such as by sudden changes in response time or error rate as a function of load. The test system 110 is easily programmable to automatically identify patterns in the output that point to these problem points.

Another useful modification is the test system 110.
Can help identify settings for various parameters in the deployment descriptor. As mentioned above, the deployment identifier for a bean identifies a parameter such as memory usage and a "pool number" that represents the number of instances of the bean created during application initialization. These and other settings in the deployment identifier can impact the performance time and maximum load an application can handle. One use of the test system described above is to allow test cases to be repeated for different settings in the deployment descriptor. Human testers can analyze performance changes for different settings of deployment descriptors. However, the test system 110
Is programmable to automatically edit the bean's deployment descriptors by varying parameters that affect pooling or memory usage. The test system 110 can automatically collect and present data indicating the impact of deployment descriptors on application performance.

Higher levels of automation can be achieved by the test system 110. For example, the test system 110 can test the beans in the application and analyze the test results for each bean. The test system 110 can identify the bean (s) that reflects the performance bottleneck (ie, display an unacceptable response time for the minimum number of concurrent users). The test system 110 then runs tests on those beans to find the setting in the deployment descriptor that balances the performance of the beans in the application.
Computer technology (that is, adjusting deployment descriptor settings appropriately so that a bottleneck bean does not perform worse than other beans) is evolving at a rapid rate, and the hardware and hardware that make up application undertesting and test systems Improved or enhanced versions of software components are available. The descriptors for one device in a class are exemplary and not intended to be limiting, and other devices of the same class may be substituted by one of ordinary skill in the art.

Further, the object being tested is an EJB, which is written in the Java (registered trademark) language. The same technique is equally applicable to applications with components implemented in other languages.
For example, applications written according to the COM standard can be Visual Basic, and applications written for the CORBA standard can be written in C ++.

Regardless of the particular language used, these standards are intended to allow separately developed components to work together. Therefore, each must provide a mechanism for other applications, such as test system 110,
Determine how to access the methods and properties of those components. However, there may be differences in the particular commands used to access the component.

In one embodiment, the code generator 212 is implemented in such a way that it can be easily modified to generate test code for applications written in another language. In particular, the code generator 212 stores the intermediate results as a symbol table independent of the particular language used to program the application undertest. The symbol table lists methods and properties for the component being tested. Information can be determined from the symbol table and input from the user when these methods are accessed, what data is used for a particular test, and what kind of data is recorded. Thus, many functions of code generator 212 are independent of the particular language used to implement application undertesting.

Thus, the language-specific aspects of code generator 212 are easily separated, representing a relatively small portion of code generator 212. In particular, the language-specific information is needed to access the application undertest and obtain the information for the symbol table. Language-specific information is also needed to format the generated client test code. However, these portions of code generator 212 can be replaced to allow test system 110 to test applications written in other languages. In addition, test system 110 includes multiple versions of the language-specific portion, allowing the user to specify the language of the application undertest as input.

Although described with respect to the preferred embodiments of the present invention, it will be apparent to those skilled in the art that other embodiments containing these concepts can be used. Further, the software included as part of the present invention can be implemented in a computer program product that includes a computer usable medium. For example, such computer usable media may be a hard drive device, a CD-ROM, a DVD-RO, storing a computer readable program code segment.
M, a computer diskette, or other readable memory device. The computer-readable medium may further include a communication link having program code segments carried as digital or analog signals, or optical, wired, or wireless. It is therefore proposed that the present invention should not be limited to the described embodiments, but only by the spirit and scope of the appended claims.

[Brief description of drawings]

FIG. 1 describes a remote application under test by the test system of the present invention.

FIG. 2 shows the test system of the invention in more detail.

FIG. 3 is a diagram showing the coordinator of FIG. 2 in more detail.

FIG. 4 shows the code generator of FIG. 2 in more detail.

[Explanation of symbols]

100 Application Service Provider 110 test system 114 Application Under Test 210 Coordinator 212 Code Generator

─────────────────────────────────────────────────── ───

[Procedure amendment]

[Submission date] May 21, 2002 (2002.5.2)
1)

[Procedure Amendment 1]

[Document name to be corrected] Drawing

[Correction target item name] All drawings

[Correction method] Change

[Correction content]

[Figure 1]

[Fig. 2]

[Figure 3]

[Figure 4]

Claims (23)

[Claims] 1. A method of remotely testing a computer application undertest on a computer network, the method comprising: providing test code to exercise a component of the application undertest; and a network for the remote application undertest. Running a first instance of the test code that spans the remote application undertest, recording performance data for the component of the remote application undertest, analyzing the recorded performance data, Indicating performance characteristics of the component. 2. The method of claim 1, further comprising executing at least one further instance of the test code across a network for the remote application undertest. 3. The method of claim 2, further comprising synchronizing the execution of one instance of the test code with another instance of the test code. 4. The method of claim 1, wherein providing a test code comprises automatically generating the test code. 5. The application undertest is written in an object-oriented language, and the step of providing a test code includes the step of providing a test code exercising one object in the application. The method of claim 1, wherein 6. The method of claim 3, wherein the synchronizing step includes the step of starting each instance of the test code at about the same time. 7. The method of claim 1, wherein the analyzing step includes providing a graphical display having the number of instances of the test code as an independent variable, the dependent variable being performance data. Method. 8. The step of analyzing comprises the step of providing a graphical display having the number of instances of the test code as an independent variable, the dependent variable being obtained from the performance data. The method described in. 9. The application undertest resides on a first server on the network, the application has a remote interface, and the test code resides on at least a second computer on the network. Method according to claim 1, characterized in that the application undertest is exercised using the remote interface of the undertest. 10. The method of claim 1, wherein the analyzing step includes the step of displaying the analyzed data to a human user using a graphical user interface. 11. A method for remotely testing a computer application under test, comprising the steps of: a) assigning a test status to a test system via a user interface; -Initiating the collection of test data regarding the performance of at least one component of the undertest, c) specifying the output format of the test data by the user interface to the test system, and d) the remote application. Displaying the response of at least one component of an undertest in a specified format. 12. The method of claim 11, wherein the specified format is a graphical format that indicates response time as a function of load conditions. 13. The method of claim 11, wherein the specified graphical format is a Hi-Lo plot. 14. The step of collecting test data comprises:
A method comprising the step of initiating execution of multiple copies of a test program, the number of copies being executed simultaneously being related to the load condition.
The method according to 1. 15. The method of claim 11, wherein the step of specifying an output format includes the step of specifying a method for which the response is measured. 16. The step of collecting test data comprises:
Recording the execution time between selected points in the test program for each of the concurrent copies of the test program, and analyzing the recorded execution time for all copies of the test program. The method of claim 11, wherein 17. The analyzing step comprises the step of determining an average and maximum execution time for each of the load conditions.
The method described in. 18. A) computer application
The undertest includes software residing on a server that controls access to the computer database; b) the server is connected to a network, the application undertest is accessed simultaneously by multiple clients on the network, c ) The test system resides on at least a second server connected to a network and is deployed remotely from the application under test,
The method according to claim 11, wherein: 19. The application undertest includes a plurality of components,
The method according to claim 11. 20. The method of claim 19, wherein the component comprises Enterprise Java Beans. 21. The method of claim 19, wherein each component includes a plurality of functions and the test code exercises the functions of the component. 22. The event of claim 16, wherein the time recorded event comprises a time when a command is issued to an access function of the component and a time when execution of the command is completed. Method. 23. A system for determining the performance of a remotely deployed application undertest in response to a load comprises: a) coordination software; and b) receiving commands from cooperating software as input and client code as output. At least one code generator having: c) a computer server having as input a command from the coordination software, each having a plurality of threads executing an instance of the client's test code; One test engine, and d) at least one data log having computer memory holding timing data created by the instance of the client test code in the plurality of threads. , Including, system.