JP2012088883A - Test method, test management server, and test management program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a test method for a system that lightens the burden on a user associated with a test.SOLUTION: There is provided the test method for the computer system including a test environment in which one or more test servers are installed and a test management server which manages an operation test of a system to be tested in the tests environment. The test management server has an analytic item table. The method includes: a procedure in which the test management server constructs the system to be tested on the test server based upon written definitions in which operation settings of the system to be tested are defined; a procedure in which the test management server places the system to be tested in operation based upon test conditions in which test contents of the system to be tested are defined; a procedure in which the test environment transmits operation information on the system to be tested to the test management server; a procedure in which the test management server analyzes the received operation information by an analyzing method set in the analytic item table; and a procedure in which the management server outputs an analytic result.

Description

  The present invention relates to a test method, a test management server, and a test management program.

  When developing a system such as a business system, it is important to discover system problems by testing the performance and reliability of the built system and formulate solutions for the discovered problems (Patent Literature) 1).

  In Patent Document 1, regarding the development of an application program, a failure factor and a solution are identified from a failure location pointed out by the user in an interactive format and the execution history information of the program, and the identified failure factor and solution are identified by the user. The technology presented in is disclosed.

JP 2001-249828 A

  However, in the technique disclosed in Patent Document 1, the user has to manually point out the defective portion, and thus there is a problem that the burden on the user related to the test is heavy. In addition, this technology is a mechanism for developing application programs, and cannot be directly applied to the development of systems such as business systems.

  The present invention has been made in consideration of the above-described problems, and an object of the present invention is to provide a system test method that reduces a burden on a user related to a test.

  In order to solve the above problems, for example, the configuration described in the claims is adopted.

  The present application includes a plurality of means for solving the above problems. To give an example, a test environment including one or more test servers, and a test environment connected to the test environment via a network, A test method in a computer system comprising a test management server for managing an operation test of an operating test target system, wherein each of the one or more test servers and the test management server includes a processor for executing a program, A memory for storing a program executed by the processor, and the test management server includes an analysis object to be analyzed in the operation test, an analysis method for the analysis object, and reference information relating to the analysis of the analysis object And an analysis item table associated with the test management server. And a procedure for constructing the test target system on the test server based on a definition document in which operation settings of the test target system are defined, and the test management server defines the test contents of the test target system. A procedure for operating the test target system based on the tested conditions, a procedure for the test environment to transmit operating information indicating an operating status of the test target system to the test management server, and the test management server Analyzing the received operation information by an analysis method set in the analysis item table, and determining whether an analysis result satisfies the reference information associated with the analysis method, and the test A management server including a procedure for outputting a determination result based on the determination procedure.

  According to the present invention, it is possible to reduce a burden on a user related to a test.

  Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

It is a figure which shows the structural example of the computer system of embodiment of this invention. It is a flowchart explaining the whole operation | movement of the computer system of embodiment of this invention. It is a figure which shows an example of the definition document of embodiment of this invention. It is a figure which shows an example of the test conditions of embodiment of this invention. It is a figure which shows an example of the definition file of embodiment of this invention. It is a figure which shows an example of the construction script of embodiment of this invention. It is a figure which shows an example of the operation information of embodiment of this invention. It is a figure which shows an example of the analysis item table of embodiment of this invention. It is a figure which shows an example of the problem / cause / solution table of the embodiment of the present invention. It is a figure which shows an example of the test history pool of embodiment of this invention. It is a figure which shows an example of the screen displayed on the user terminal of embodiment of this invention. It is a figure which shows the other example of the screen displayed on the user terminal of embodiment of this invention. It is a flowchart which shows the control logic of the problem analysis of step S13 of FIG. It is a figure which shows an example of the analysis result setting table of embodiment of this invention. It is a flowchart which shows the control logic of display information creation of the problem, the cause, and the solution of step S15 of FIG. It is a figure which shows an example of the table for a problem, a cause, and a solution method of embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating a configuration example of a computer system 1 according to the embodiment of this invention. A computer system 1 shown in FIG. 1 includes an operation test environment 10, an operation test management server 20, and a plurality of user terminals 3 connected to each other via a network 4 such as a LAN (Local Area Network).

  The operation test environment 10 is a test environment for performing a test in the computer system 1. The operation test environment 10 includes a plurality of construction target servers 11-1, ..., 11-N. Each of these construction target servers 11-1,..., 11 -N is a virtual server constructed in the operation test environment 10. Each construction target server 11-i (i = 1 to N) is a memory (virtual memory) 12-i, a CPU (Central Processing Unit) (virtual CPU) 13-i, an interface (virtual interface) 14-i, The computer includes a disk drive (virtual drive) 15-i. Hereinafter, the above-described components are simply referred to as a construction target server 11, a memory 12, a CPU 13, an interface 14, and a disk drive 15. Moreover, although the case where the construction target server 11 is a virtual server will be described, it may not be a virtual server.

  The memory 12 is a storage device such as a RAM (Random Access Memory) that reads and stores a program stored in the disk drive 15 when the construction target server 11 is activated. The memory 12 also stores files, data, and the like necessary for program execution. The CPU 13 is an arithmetic processing device that executes a program stored in the memory 12. The interface 14 is an interface device for connecting to an external network or the like. The disk drive 15 is a storage device such as an HDD that stores programs, files, data, and the like. In this disk drive 15, a definition file 50, a construction script 60, and operation information 70 are stored. Details of each information will be described later.

  The operation test management server 20 is a computer device for managing the operation test environment 10. The operation test management server 20 includes a memory 21, a CPU 22, an interface 23, and a disk drive 24.

  The memory 21 is a storage device such as a RAM that reads and stores a program stored in the disk drive 24 when the operation test management server 20 is activated. The memory 21 also stores files, data, and the like necessary for program execution.

  The CPU 22 includes a program stored in the memory 21 (a system construction processing unit 210 that constructs a test environment in the operational test environment 10, a test execution / analysis processing unit 220 that executes a test in the operational test environment 10 and analyzes the test result). (Including). The interface 23 is an interface device for connecting to an external network or the like. The disk drive 24 is a storage device such as an HDD that stores programs, files, data, and the like. The disk drive 24 stores an analysis item table 80, a problem / cause / solution table 90, and a test history pool 100. Details of each information will be described later.

  User terminals 3-1,..., 3-M are computer devices used by users. The user terminal 3-j (an integer from j = 1 to M) includes a main storage device, a CPU, an interface device, a disk drive, an input device (such as a keyboard and a mouse), and an output device (such as a display). Not shown). In the following description, the user terminals 3-1, ..., 3-M are simply referred to as user terminals 3.

  FIG. 2 is a flowchart for explaining the overall operation of the computer system 1 according to the embodiment of this invention. Here, the operation of the entire system of the computer system 1 will be described.

  First, in step S1, the user inputs the definition document 30, the test condition 40, and the business application on the user terminal 3 (S1). The business application here is application software for operating a business system to be tested on the operation test environment 10. Then, the user terminal 3 transmits the input definition document 30, test condition 40, and business application to the operation test management server 20 (S2).

  Thereafter, in step S3, the system construction processing unit 210 of the operational test management server 20 starts construction of the operational test environment 10 based on the received definition document 30 (S3). Specifically, the definition file 50 is created based on the received definition document 30.

  Thereafter, in step S4, the created definition file 50 and construction script 60 are transmitted to the operation test environment 10 (S4). Then, the operation test environment 10 constructs a test environment based on the received definition file 50 and construction script 60 (S5).

  Thereafter, the system construction processing unit 210 incorporates the business application received in step S2 into the operation test environment 10 (S6). That is, the business application is transmitted to the operation test environment 10 (S7). Then, the operation test environment 10 incorporates (installs) the received business application (S8).

  Thereafter, in step S9, the test execution / analysis processing unit 220 of the operation test management server 20 operates the operation test environment 10 based on the test condition 40 received in step S2 (S9). Then, the operation test environment 10 is activated (S10) and is in an operating state (S11).

  The test target system can be automatically constructed on the operation test environment 10 by the processing in steps S1 to S11 described above. This can reduce costs and risks associated with testing in a production environment.

  Thereafter, in step S12, the operation test environment 10 transmits the operation information 70 to the operation test management server 20 (S12). Then, the test execution / analysis processing unit 220 analyzes the problem based on the received operation information 70 and the test history pool 100 stored in the disk drive 24 (S13). The process of step S13 will be described later in detail.

  Thereafter, in step S14, the test execution / analysis processing unit 220 stores the received operation information 70 and the analysis result of step S13 in the test history pool 100 (S14). Thereafter, display information of the problem / cause / solution to be presented to the user is created based on the analysis result of step S13 and the problem / cause / solution table 90 stored in the disk drive 24 (S15). The process of step S15 will be described later in detail.

  Thereafter, in step S16, the test execution / analysis processing unit 220 stores the display information of the problem / cause / solution generated in step S15 in the test history pool 100 (S16). Thereafter, the display information of the problem / cause / solution created in step S15 and the information stored in the test history pool 100 are transmitted to the user terminal 3 as display information (S17). Then, the user terminal 3 displays the received display information (S18).

  Through the processing in steps S12 to S18 described above, the operation test management server 20 performs analysis of the problem in the operation test environment 10, discovery of the problem, and presentation of the solution to the discovered problem to the user terminal 3. Can be implemented automatically.

  Thereafter, in step S19, the user selects one or more solutions on the display screen of the user terminal 3 (S19). Then, the user terminal 3 transmits information on the solution selected by the user to the operation test management server 20 (S20).

  Thereafter, in step S21, the test execution / analysis processing unit 220 creates a definition document 30 (modified definition document 30) that satisfies the received solution (S21). Further, the received solution information and the created definition document 30 are stored in the test history pool 100 (S22).

  Thereafter, in step S23, the test execution / analysis processing unit 220 determines whether reconstruction (manual), reconstruction (automatic), or no reconstruction is performed (S23). In the case of reconstruction (manual) (YES in S23 (manual)), the user terminal 3 downloads the definition document 30 created in step S21 (S24), returns to step S1, and repeats the processing. In the case of reconstruction (automatic) (YES in S23 (automatic)), the operation test management server 20 returns to S3 and repeats the process. On the other hand, when the reconstruction is not performed (NO in S23), the operation test management server 20 ends the process.

  Through the processing in steps S19 to S23 described above, the operation test management server 20 can prompt the user to select a solution and automatically create a definition document 30 that satisfies the solution selected by the user. As a result, the retest can be performed based on the created definition document 30.

  In addition, when repeating a series of processes after step S23 or S24, when there are a plurality of definition documents 30 created in step S21, each of the same number of construction target servers (virtual servers) 11 as the definition documents 30 is set to each You may build based on the definition document 30.

  FIG. 3 is a diagram illustrating an example of the definition document 30 according to the embodiment of this invention. The definition document 30 is a file in which performance (requirements) required for the system and system operation settings are defined. The definition document 30 shown in FIG. 3 includes a plurality of items 31 and a setting value 32 for each item 31.

  The column 301 defines the required performance of the system that the number of requests per second is at least 10. The column 302 defines the required performance of the system that the CPU usage rate is 70% at maximum. A column 303 defines the required performance of the system that the used memory size per session is 1 MB at the maximum. The column 304 defines the required performance of the system that the number of simultaneous logins is at least 10. The column 305 defines the required performance of the system such that the cumulative number of full GCs is 5 at the maximum. On the other hand, the column 306 defines a system operation setting that the size of the New area of the Java (registered trademark, the same applies hereinafter) heap area is 256 MB.

  As described above, the definition document 30 defines performance required for the system and system operation settings. The defined information is not limited to the information shown in FIG.

  FIG. 4 is a diagram illustrating an example of the test condition 40 according to the embodiment of this invention. The test condition 40 is a file in which test conditions for the operational test environment 10 are defined. The test condition 40 shown in FIG. 4 includes a plurality of items 41 and a set value 42 for each item 41.

  In column 401, a load condition is defined. In the example shown in FIG. 4, the number of executions of processing that gives a load in one iteration is five, the time in one iteration is 10 seconds, and the number of iterations is 100, that is, 10 seconds. A load condition is defined such that a process of applying a load five times is repeated 100 times.

  On the other hand, the load content is defined in the column 402. In the example shown in FIG. 4, the load is that the various parameters shown in the column 402 are transmitted as the load 1 to the 80th port of the server indicated by the server name (or IP address) of 192.168.102.1. The contents are defined.

  As described above, the test condition 40 defines the test condition of the operation test environment 10. The defined information is not limited to the information shown in FIG.

  FIG. 5 is a diagram illustrating an example of the definition file 50 according to the embodiment of this invention. The definition file 50 is a file obtained by changing the definition document 30 into a data format that can be processed by the operation test environment 10. This definition file 50 is generated in step S3 of FIG.

  The property file format as shown in FIG. 5A and the XML file format as shown in FIG. 5B are given as examples, but other data formats can be used as long as the data can be processed by the operation test environment 10. There may be.

  FIG. 6 is a diagram illustrating an example of the construction script 60 according to the embodiment of this invention. The construction script 60 is a file for automatically executing commands necessary for construction of the construction target servers 11-1 to 11-N in the operation test environment 10. In the example shown in FIG. 6, the construction script 60 is shown as a batch file (script).

  FIG. 7 is a diagram illustrating an example of the operation information 70 according to the embodiment of this invention. The operation information 70 is a file in which information indicating an operation status when the operation test environment 10 is operated according to the test condition 40 is recorded. As illustrated in FIG. 7, the operation information 70 includes an access log 71, a CPU usage rate log 72, and a JavaVM log 73.

  In the access log 71, the start time of each request generated in the operation test environment 10, the contents of each request, and the processing time (unit: second) of each request are recorded. The CPU usage rate log 72 records the CPU usage rate at each time. The Java VM log 73 records the time when GC (Garbage Collection), Full GC, etc. occur, and the amount of change in the memory size used at that time.

  As described above, the operation information 70 records information indicating an operation state when the operation test environment 10 is operated according to the test condition 40. The information to be recorded is not limited to the information shown in FIG.

  FIG. 8 is a diagram illustrating an example of the analysis item table 80 according to the embodiment of this invention. The analysis item table 80 is data used for problem analysis in step S13 of FIG. The analysis item table 80 shown in FIG. 8 includes a plurality of analysis objects 81, an analysis method 82 corresponding to each analysis object 81, and an OK / NG condition 83 corresponding to each analysis object 81.

  The column 801 defines that the state of the J2EE server is confirmed as an analysis method 82 when the analysis target 81 is a J2EE server. As the OK / NG condition 83, when the J2EE server is in the start state, OK is returned, that is, it is determined that there is no problem in the J2EE server, and when the J2EE server is not in the start state, NG is returned, that is, J2EE. It is defined that the server determines that there is a problem.

  The column 802 defines that the Java VM log 73 is analyzed as the analysis method 82 when the analysis target 81 is the full GC accumulation count, the accumulation count of the full GC is confirmed, and compared with the setting value of the definition document 30. Yes. Further, as the OK / NG condition 83, when the Full GC cumulative number is less than or equal to the set value of the definition document 30, the system satisfies the required performance of the system, so OK is returned. It is defined that NG is returned because the required performance is not satisfied.

  In a column 803, as an analysis method 82 when the analysis target 81 is the CPU usage rate, the CPU usage rate log 72 is analyzed, the peak value of the CPU usage rate is obtained, and compared with the setting value of the definition document 30. Is defined. Further, as an OK / NG condition 83, when the peak value of the CPU usage rate is equal to or less than the setting value of the definition document 30, the system satisfies the required performance of the system, so that OK is returned and is larger than the setting value of the definition document 30. Is defined to return NG because it does not satisfy the required performance of the system.

  As described above, the analysis item table 80 lists analysis items when the operation test management server 20 analyzes a problem. Note that the information to be listed is not limited to the information shown in FIG.

  FIG. 9 is a diagram illustrating an example of the problem / cause / solution table 90 according to the embodiment of this invention. The problem / cause / solution table 90 is data used in step S15 in FIG. The problem / cause / solution table 90 shown in FIG. 9 includes a plurality of problems 91, cause items 92 corresponding to each problem 91, and solution items 93 corresponding to each cause item 92.

  In the problem 91, problems corresponding to the analysis items listed in the analysis item table 80 (columns 801 to 803 in FIG. 8) are defined. In the cause item 92, one or more causes corresponding to each problem 91 are defined with priority. In the solution item 93, one or more solutions for each cause item 92 are defined with priority.

  In column 901, the cause of the problem that the CPU usage rate is larger than the set value (the problem corresponding to column 803 in FIG. 8) is that the occurrence interval of FullGC is short (priority 1) and the maximum number of simultaneous executions is large. (Priority level 2) is defined. Further, as a solution for each of these causes, # 2 (link to each solution in column 902), it is defined to reduce the execution queue size. The column “definition document creation” of the solution item 93 is information indicating whether or not the definition document 30 can be automatically generated based on the solution method shown in the column “solution”.

  The column 902 defines a cause for the problem that the number of accumulated Full GCs is large (a problem corresponding to the column 802 in FIG. 8) and a solution for each cause.

  As described above, the problem / cause / solution method table 90 sets the problem / cause / solution of the operation test environment 10 in association with the analysis item table 80. The operation test management server 20 can create display information of the problem / cause / solution according to the analysis result by using this problem / cause / solution table 90 in step S15 of FIG.

  FIG. 10 is a diagram illustrating an example of the test history pool 100 according to the embodiment of this invention. The test history pool 100 is a table for storing various information such as test results of the operation test environment 10. The test history pool 100 illustrated in FIG. 10 includes a test name 101, a previously selected solution 102, operation information 103, an analysis result 104, a problem / cause / solution 105, and a currently selected solution 106. And the created definition document 107.

  The test name 101 stores an identifier for uniquely identifying each test. The column 1001 stores “initial test” as the initial test. On the other hand, in columns 802 to 807, “(test target server ID) _ (ID of the solution selected last time) _ (n-th test)” is stored.

  For example, in the test shown in the column 1002, the solution “1_ # 3-1-1” selected by the user after executing the initial test shown in the column 1001 (see the column of the solution 106 selected this time in the column 1001), This is the second test executed after being reflected in the operational test environment 10. Therefore, “1_ # 3-1-1_2” is stored in the test name 101 of the column 1002. Similarly, the test shown in the column 1003 includes the solution “1_ # 2-1-1” selected by the user after executing the second test shown in the column 1002 (the column of the solution 106 selected this time in the column 1002). Is a third test to be executed after being reflected in the operation test environment 10. Therefore, “1_ # 2-1-1_3” is stored in the test name 101 of the column 1003.

  The solution 102 selected last time stores the solution selected in the previous test of the test indicated by the test name 101 (for example, the test in the column 1001 if the test is in the column 1002).

  The operation information 103 stores a file path for the operation information 70 received by the operation test management server 20 in step S12 of FIG. 2 when the test indicated by the test name 101 is executed.

  In the analysis result 104, an analysis result setting table 140 (described later in detail with reference to FIG. 14) created in step S13 of FIG. 2 when the test indicated by the test name 101 is executed is stored.

  The problem / cause / solution 105 is a problem / cause / solution storage table 160 created in step S15 of FIG. 2 when a test indicated by a test name is executed (detailed later with reference to FIG. 16). Is stored.

  The solution 106 selected this time stores one or more solutions selected in step S19 of FIG. 2 when the test indicated by the test name is executed.

  The created definition document 107 stores a file path for the definition document 30 created in step S21 of FIG. 2 when the test indicated by the test name is executed.

  As described above, the test history pool 100 stores various information such as test results of the operation test environment 10 in association with each other.

  FIG. 11 is a diagram illustrating an example of the screen 110 displayed on the user terminal 3 according to the embodiment of this invention. This screen 110 is displayed on the user terminal 3 in step S18 of FIG. The screen 110 includes a selection unit 111, display contents 112, a check box 113, and buttons 114 and 115.

  The selection unit 111 is provided for the user to select contents to be displayed on the user terminal 3. In the example shown in FIG. 11, the “solution” of the operation test environment A is selected. The case where “test history” is selected will be described later with reference to FIG.

  In the display content 112, content corresponding to the selection in the selection unit 111 (here, the problem / cause / solution storage table 160) is displayed.

  The check box 113 is a check box that is provided for the user to select one or more solution methods on the user terminal 3 and can be checked for each solution method. Note that a solution cannot be selected (one for which the definition document 30 cannot be automatically generated) cannot be selected. The user can select one or more solutions and check using an input device such as a mouse.

  The button 114 is a button for downloading the definition document 30 generated based on the solution selected by the user. When this button 114 is pressed, the process proceeds to YES (manual) in step S23 of FIG. On the other hand, the button 115 is a button for executing a retest. When this button 115 is pressed, the process proceeds to YES (automatic) in step S23 of FIG.

  By displaying the screen 110 as described above on the user terminal 3, the operation test management server 20 prompts the user to select a solution. In addition, a definition document 30 that satisfies the solution selected by the user is automatically created, and the created definition document 30 is downloaded to the user terminal 3, or a retest is performed based on the created definition document 30. It becomes possible to do.

  FIG. 12 is a diagram illustrating another example of the screen 120 displayed on the user terminal 3 according to the embodiment of this invention. This screen 120 is displayed on the user terminal 3 in step S18 of FIG. In the following description, the same components as those in FIG.

  A screen 120 shown in FIG. 12 includes selection units 111 and 121, display contents 112 and 122, a check box 113, and buttons 114 and 115.

  The selection unit 111 is provided for the user to select contents to be displayed on the user terminal 3. In the example shown in FIG. 12, “test history” of the operation test environment A is selected. A selection unit 121 is provided to select a test history that the user wants to display on the user terminal 3. In the example shown in FIG. 12, “1_ # 3-1-1_2 (second test)” is selected.

  The display content 112 displays the content corresponding to the selection in the selection unit 111 (here, the problem / cause / solution storage table 160 relating to the second test). The display content 122 displays information related to the solution selected in the previous test (here, the first test).

  By displaying the screen 110 as described above on the user terminal 3, the operation test management server 20 prompts the user to select a solution. Unlike the screen 110 shown in FIG. 11 described above, by displaying the past test history, the user can select an optimal solution method with reference to the past test history.

  FIG. 13 is a flowchart showing the control logic of the problem analysis in step S13 of FIG. Here, the specific operation of step S13 in FIG. 2 will be described.

  First, as shown in step S1301, the operation test management server 20 repeats the processing of steps S1302 to S1309 for each construction target server 11 (S1301).

  That is, first, in step S1302, the operation test management server 20 reads the analysis item table 80, and creates an analysis result setting table 140 for setting analysis results based on the analysis item table 80 (S1302).

  FIG. 14 is a diagram illustrating an example of the analysis result setting table 140 according to the embodiment of this invention. The analysis result setting table 140 is a table that is created for each construction target server 11 and stores the analysis result of the corresponding construction target server 11. In the following description, the same components as those in FIG.

  The analysis result setting table 140 shown in FIG. 14 includes a plurality of analysis objects 81, an analysis method 82 corresponding to each analysis object 81, an OK / NG condition 83 corresponding to each analysis object 81, and each analysis object 81. A measurement value 141 when analyzed by the analysis method 82 corresponding to the analysis target 81, a definition value 142 defined in the definition document 30, and an analysis result 143 indicating whether or not each analysis target 81 has a problem. Including. At the time when the analysis result setting table 140 is created, the measurement value 141, the definition value 142, and the analysis result 143 are blank.

  Returning to FIG. 13, as shown in step S <b> 1303, the operation test management server 20 performs the processing of steps S <b> 1304 to S <b> 1309 with the number of analysis items defined in the analysis item table 80 (in FIG. 8, columns 801, 802, 803). 3 times) (S1303). Hereinafter, the processing of steps S1304 to S1309 will be described.

  In step S1304, the operation test management server 20 analyzes the analysis target 81 in the column selected in step S1303 according to the analysis method 82 (S1304). Thereafter, in step S1305, the operation test management server 20 displays the measurement value obtained by the analysis and the definition value defined in the definition document 30 in the column of the measurement value 141 and the definition value 142 of the analysis result setting table 140, respectively. Setting is made (S1305).

  Thereafter, in step S1306, the operation test management server 20 confirms whether the analysis result is OK or NG based on the set measurement value 14 and definition value 142 and the OK / NG condition 83 (S1306).

  Thereafter, in step S1307, the operation test management server 20 determines whether the confirmation result in step S1306 is OK or NG (S1307). In the case of OK (YES in S1307), the operation test management server 20 sets “◯” in the column of the analysis result 143 (S1308). On the other hand, in the case of NG (NO in S1307), the operation test management server 20 sets “x” in the column of the analysis result 143 (S1309).

  Through the processing described above, the operation test management server 20 analyzes the problem for each construction target server 11 and the analysis result of each construction target server 11 in the analysis result setting table 140 created for each construction target server 11. Is set.

  FIG. 15 is a flowchart showing the control logic for creating display information of the problem / cause / solution in step S15 of FIG. Here, the specific operation of S15 in FIG. 2 will be described.

  First, as shown in step S1501, the operation test management server 20 creates a problem / cause / solution storage table 160 for storing problem / cause / solution information (S1501).

  FIG. 16 is a diagram showing an example of the problem / cause / solution storage table 160 according to the embodiment of this invention. The problem / cause / solution storage table 160 is a table for generating information on problems / causes / solutions to be displayed on the user terminal 3 (see display content 112 in FIG. 11).

  The problem / cause / solution storage table 160 shown in FIG. 16 includes an analysis target server 161, an analysis target 162, a measurement value 163, a definition value 164, a cause item 165, and a solution item 166. Including. Note that when the problem / cause / solution storage table 160 is created, the columns 161 to 166 are blank.

  Thereafter, as shown in step S1502, the operation test management server 20 repeats the processing of steps S1503 to S1510 for each construction target server 11 (S1502).

  That is, first, in step S1503, the operation test management server 20 reads the analysis result setting table 140 of the target construction target server 11 (S1503). Thereafter, as shown in step S1504, the operation test management server 20 repeats the processing of steps S1505 to S1510 (the number of analysis items in the analysis result setting table 140 (three times 1401, 1402, 1403 in FIG. 14)) ( S1504). Hereinafter, the processing of steps S1505 to S1510 will be described.

  In step S1505, the operation test management server 20 acquires the value of the analysis result 143 in the column selected in step S1504 from the analysis result setting table 140 (S1505). Thereafter, in step S1506, the operation test management server 20 determines whether the value of the acquired analysis result 143 is “◯” or “×” (S1506).

  If “◯” in step S1506 (YES in S1506), the process returns to step S1504 to repeat the process for the next analysis item. On the other hand, if “x” (NO in step S1506), the process advances to step S1507.

  In step S1507, the operation test management server 20 acquires the analysis target 81, measurement value 141, and definition value 142 in the column selected in step S1504 from the analysis result setting table 140 (S1507). Thereafter, the problem / cause / solution table 90 is read (S1508), and the corresponding row (column) is acquired from the read problem / cause / solution table 90 (S1509).

  For example, when the analysis target 81 acquired in step S1507 is the Full GC accumulation count of the column 1402, the column 902 related to the Full GC accumulation count is acquired in Step S1509. If the analysis target 81 acquired in step S1507 is the CPU usage rate of the column 1401, the column 901 relating to the CPU usage rate is acquired in step S1509.

  Thereafter, in step S1510, the operation test management server 20 includes information (analysis target 81, measurement value 141, and definition value 142) in the analysis result setting table 140 and information (cause item 92, The solution method item 93) is set in the analysis target 162, measurement value 163, definition value 164, cause item 165, and solution method item 166 of the problem / cause / solution storage table 160 (S1510).

  Through the processing described above, the operation test management server 20 analyzes the analysis result of each construction target server 11 set in the analysis result setting table 140 and the problem / cause set in the problem / cause / solution table 90. Based on the association information of the solution, information on the problem / cause / solution to be displayed on the user terminal 3 (see display content 112 in FIG. 11) is generated.

  As described above, according to the system test method of the present embodiment, a test target system can be automatically constructed on the operation test environment 10. In addition, analysis of a problem in the operation test environment 10, discovery of a problem, and presentation of a solution to the discovered problem to the user terminal 3 can be automatically performed. Furthermore, it is possible to prompt the user to select a solution, automatically create a definition document 30 that satisfies the solution selected by the user, and perform a retest based on the created definition document 30. Become.

  As mentioned above, although embodiment of this invention was described, the said embodiment showed one of the application examples of this invention, and in the meaning which limits the technical scope of this invention to the specific structure of the said embodiment. Absent. Various modifications can be made without departing from the scope of the present invention.

1 Computer System 3 User Terminal 10 Operation Test Environment 11 Construction Target Server 20 Operation Test Management Server 30 Definition Document 40 Test Conditions 50 Definition File 60 Construction Script 70 Operation Information 80 Analysis Item Correspondence Table 90 Problem / Cause / Solution Table 100 Test History pool 140 Analysis result setting table 160 Problem / cause / solution storage table 210 System construction processing unit 220 Test execution / analysis processing unit

Claims (10)

A test in a computer system having a test environment including one or more test servers and a test management server connected to the test environment via a network and managing an operation test of a test target system operating in the test environment A method,
Each of the one or more test servers and the test management server includes a processor that executes a program and a memory that stores a program executed by the processor.
The test management server has an analysis item table in which an analysis target to be analyzed in the operation test, an analysis method for the analysis target, and reference information related to the analysis of the analysis target are associated with each other.
The method
The test management server constructs the test target system on the test server based on a definition document in which operation settings of the test target system are defined;
The test management server operates the test target system based on test conditions in which the test content of the test target system is defined;
A procedure in which the test environment transmits operation information indicating an operation status of the test target system to the test management server;
The test management server analyzes the received operation information by an analysis method set in the analysis item table, and determines whether or not the analysis result satisfies the reference information associated with the analysis method When,
The test management server outputs a determination result according to the determination procedure;
A test method characterized by comprising: The test management server has a solution table in which one or more solutions when the criteria information is not satisfied are set;
When it is determined in the determining procedure that the reference information is not satisfied, in the outputting procedure, the test management server outputs the one or more solution methods set in the solution method table. The test method according to claim 1. The test management server presenting the output one or more solutions and prompting the user to select;
The test management server creates a new definition document that satisfies the selected solution based on the solution selected by the user from among the one or more solutions;
The test method according to claim 2, further comprising:   The test method according to claim 3, wherein the test management server further includes a procedure for reconstructing the test target system in the test server based on the created definition document. The test management server has a cause table in which one or more causes are set when the criteria information is not satisfied,
The test management server outputs the one or more causes set in the cause table in the outputting procedure when it is determined in the determining procedure that the reference information is not satisfied. Item 2. The test method according to Item 1.   The test method according to claim 1, wherein each of the one or more test servers is virtualized by logically dividing physical resources related to a processor and a memory. The test management server has a solution table in which one or more solutions when the criteria information is not satisfied are set;
When it is determined in the determining procedure that the reference information is not satisfied, in the outputting procedure, the test management server outputs the one or more solution methods set in the solution method table;
The method further comprises:
The test management server presenting the output one or more solutions and prompting the user to select;
A procedure in which the test management server newly creates a definition document satisfying each solution for each solution selected by the user among the one or more solutions;
The test management server constructs the same number of virtualized test servers as the created definition documents in the test environment, and based on the respective definition documents, the test management server assigns the test to each of the virtualized test servers. The procedure to build the target system,
The test method according to claim 6, further comprising:   The test method according to claim 1, wherein the test management server outputs history information including the received operation information, the analysis result, and the determination result. A test management server connected to a test environment including one or more test servers via a network and managing an operation test of a test target system operating in the test environment;
A processor for executing a program; and a memory for storing a program executed by the processor;
An analysis item table in which an analysis object to be analyzed in the operation test, an analysis method for the analysis object, and reference information related to the analysis of the analysis object are associated with each other;
The test management server
Based on a definition document in which operation settings of the test target system are defined, the test target system is constructed on the test server,
Based on the test conditions in which the test content of the test target system is defined, the test target system is operated,
Receiving operation information indicating the operation status of the test target system from the test environment;
The received operation information is analyzed by an analysis method set in the analysis item table, it is determined whether the analysis result satisfies the reference information associated with the analysis method, and the determination result is output. A test management server characterized by that. A test management program used in a test management server that is connected to a test environment including one or more test servers via a network and manages an operation test of a test target system operating in the test environment,
The test management server has an analysis item table in which an analysis target to be analyzed in the operation test, an analysis method for the analysis target, and reference information related to the analysis of the analysis target are associated with each other.
The test management program includes:
A procedure for constructing the test target system in the test server based on a definition document in which operation settings of the test target system are defined;
A procedure for operating the test target system based on test conditions in which the test content of the test target system is defined;
A procedure for receiving operation information indicating an operation status of the test target system from the test environment;
Analyzing the received operation information by an analysis method set in the analysis item table, and determining whether an analysis result satisfies the reference information associated with the analysis method;
A procedure for outputting a determination result according to the determination procedure;
Is executed by the test management server.