JP2016071400A - Test execution device, test execution method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To streamline log verification in an operational test of a system.SOLUTION: A test execution device 10 is for testing a test target system. The test execution device 10 includes: a time acquisition unit 22 that acquires a test start time; a test execution unit 23 that starts automatic execution of a test on the test target system 12 at the test start time; a log file acquisition unit 26 that acquires a log file that has been updated since the test start time from the system after completion of the test; a log extraction unit 27 that extracts a log that has been generated since the test start time from the acquired log file; a verification unit 29 that verifies whether or not a portion of the acquired log file excluding elements which values are determined at the execution of the test matches a predetermined expected value; and a verification result recording unit 30 that records a verification result in a verification result retaining unit 47.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a data processing technique, and more particularly to a technique for testing an information system.

  In order to guarantee the quality of the information system, operational tests before starting the service are indispensable. For example, an information system developer often performs an operation test by manually operating a user interface of the information system and confirming the operation result. However, in recent information systems with abundant operation variations, manual operation tests are limited.

  The present applicant has proposed a technique for automating an operation test of an information system and realizing an efficient operation test (see, for example, Patent Document 1).

International Publication No. 2010/116586

  In the operation test, as one result verification, a log of each device of the system and a log of each application operating on the device are checked to verify whether the operation of the device or application is as expected. Specifically, it is verified whether or not the content of the log matches a value expected in advance. However, the log may include variable elements such as a time stamp when the log is output. For example, when verifying a log acquired by a regression test including verification of this variable element, the verification results are inconsistent. For this reason, conventionally, after verifying whether or not the variable elements coincide with the expected value after manually removing such variable elements. This work was a heavy burden on the test personnel.

  The present invention has been made in view of these problems, and a main object thereof is to provide a technique for improving the efficiency of log verification in a system operation test.

  In order to solve the above problems, a test execution device according to an aspect of the present invention is a device for testing a system, and includes a time acquisition unit that acquires a test start time, and automatic execution of the system test from the test start time. The test execution unit to start, the acquisition unit that acquires the log file updated after the test start time from the system after the test ends, and the log that extracts the log output after the test start time from the acquired log file An extraction unit; and a verification unit that verifies whether or not a portion obtained by removing an element whose value is determined at the time of test execution from the extracted log matches a predetermined expected value.

  Another aspect of the present invention is a test execution method. This method is executed in an apparatus for testing a system, and is updated after a step of obtaining a test start time, a step of starting automatic execution of a system test from the test start time, and a test start time or later. To acquire the log file from the system after the test is completed, to extract the log output after the test start time from the acquired log file, and the element whose value is determined at the time of test execution from the extracted log Verifying whether or not the portion excluding is coincident with a predetermined expected value.

  Note that any combination of the above-described constituent elements and the expression of the present invention converted between a system, a program, a recording medium storing the program, and the like are also effective as an aspect of the present invention.

  According to the present invention, it is possible to provide a technique for improving the efficiency of log verification in a system operation test.

It is a figure which shows the structure of the test system which concerns on embodiment. It is a block diagram which shows the function structure of the test execution apparatus of FIG. It is a figure which shows the data structure of an update time holding | maintenance part. It is a figure which shows the data structure of an acquisition object list holding part. It is a figure which shows an example of the log output to the log file. It is a figure which shows an example of the shaping log output to the shaping log file.

  FIG. 1 shows the configuration of the test system 100. The test system 100 includes a test execution device 10 and a test target system 12. The test target system 12 is a system to be tested and includes a web application server 14. The test target system 12 may further include various servers and devices such as a mainframe (not shown), a form server, and a database server, and various information processing services may be realized by cooperation of these devices. .

  The web application server 14 provides various information processing services by the test target system to the client device. In the embodiment, the web application server 14 provides a web page to the client device, acquires information input to the web page from the client device, and executes business processing. As a modified example, not only a web page but also a client device displays a dedicated client screen, the client device transmits input information to the screen to the test target system 12, and the test target system 12 executes business processing. Also good.

  The test execution device 10 is an information processing device that executes a test of the test target system 12. Specifically, the test execution device 10 operates as a client device of the test target system 12 and inputs the business event information to the test target system 12 via a communication network such as a LAN, WAN, or the Internet. The operation of the system 12 is verified.

  The test execution device 10 acquires a web page from the web application server 14, and executes an operation test of the test target system 12 by reproducing a predetermined operation on the web page. Further, as one result verification of the operation test, the test execution device 10 confirms the log of each device of the test target system 12 and the log of each application operating on the device, and whether the operation of the device or application is as expected. To verify.

  In the embodiment, the test execution device 10 includes a first log file 16a (system.log) and a second log file 16b (error.log) output from the web application server 14, and an application operating on the web application server 14. The third log file 16c (apl01.log) and the fourth log file 16d (apl02.log) output are stored, and the test execution device 10 acquires only the log files that need to be verified. A method for determining a log file that needs to be verified will be described later. Hereinafter, the first log file 16a, the second log file 16b, the third log file 16c, and the fourth log file 16d are collectively referred to as “log file 16”.

  The acquired log file 16 includes elements whose values are determined at the time of test execution (in other words, elements that cannot be expected in advance), such as a time stamp indicating the time when the log is output. It is. The test execution device 10 creates a log (hereinafter referred to as “shaped log”) in which such elements are deleted from the log of the log file 16. On the other hand, the test execution device 10 holds an expected value log file in which a log (hereinafter referred to as “expected value log”) that is expected to be output to the log file 16 when an operation test is performed is written. . In the embodiment, the test execution device 10 includes the first expected value log file 18a (system_kitai.log) corresponding to the first log file 16a and the third expected value corresponding to the third log file 16c (apl01_kitai.log). Log file 18c. The test execution device 10 verifies whether the shaping log matches the expected value log. If they match, the test result is determined to be normal, and if they do not match, the test result is determined to be abnormal.

  FIG. 2 is a block diagram showing a configuration of the test execution device 10 of FIG. The test execution device 10 includes a control unit 20 and a data recording unit 40. The control unit 20 controls the operation of the test execution device 10 and executes various data processing related to the test of the test target system 12. The data recording unit 40 is a storage area for storing various data that are referred to and updated in the data processing by the control unit 20. Although not shown in FIG. 2, a predetermined web browser program is installed in the test execution device 10.

  Each block shown in the block diagram of the present specification can be realized in terms of hardware by an element such as a CPU and a memory of a computer or a mechanical device, and in terms of software, it can be realized by a computer program or the like. , Depicts functional blocks realized by their cooperation. Therefore, those skilled in the art will understand that these functional blocks can be realized in various forms by a combination of hardware and software. For example, an automatic test execution application including a software module corresponding to each functional block of the control unit 20 may be installed in the test execution device 10. Then, the function of each functional block of the control unit 20 may be exhibited by the CPU of the test execution device 10 reading out and executing each software module in the memory. Each functional block of the data recording unit 40 may be realized by storing data in a storage device such as a storage or a memory.

  The data recording unit 40 includes a start time holding unit 41, an update time holding unit 42, an acquisition target list holding unit 43, a log file holding unit 44, a shaped log file holding unit 45, an expected value log file holding unit 46, and a verification result holding unit. 47.

  The start time holding unit 41 holds the time when the operation test is started as “test start time”. The update time holding unit 42 holds update times of various log files of the test target system 12 after the operation test is completed. FIG. 3 shows the data structure of the update time holding unit 42. The update time holding unit 42 holds the log file names 120 of various log files and the update time 122 in association with each other. Here, the test target system 12 includes "system.log (first log file 16a)", "error.log (second log file 16b)", "apl01.log (third log file 16c)", "apl02 .log (fourth log file 16d) ". Returning to FIG.

  The acquisition target list holding unit 43 holds a list of log files that need to be verified as a result verification of the operation test among various log files of the test target system 12, that is, a list of log files that the test execution apparatus 10 should acquire. To do. FIG. 4 shows the data structure of the acquisition target list holding unit 43. The acquisition target list holding unit 43 holds the log file name 124 of the log file to be acquired. Here, among the four log files held by the test target system 12, “system.log (first log file 16a)” and “apl01.log (third log file 16c)” are log files to be acquired. ing.

  The log file holding unit 44 holds a log file acquired from the test target system 12. FIG. 5 shows an example of the log output to the log file held in the log file holding unit 44. The log shown in the figure has a delimiter character string “” (space), “time stamp 102” indicating the time when the log was output, “user ID 104” performing business, and uniquely set for each request. “Request ID 106”, “host name 108” of the server that performs log output, “class name 110” of the call destination, “method name 112” of the call destination, and “message body 114”.

  The shaping log file holding unit 45 holds the shaping log file in which the shaping log is written. FIG. 6 shows an example of the shaping log output to the shaping log file. FIG. 6 corresponds to a log shaping log surrounded by a dotted line in FIG. The shaping log in FIG. 5 includes a user ID 104 who performs business, a callee class name 110, a callee method name 112, and a message body 114. That is, the time stamp 102 indicating the time when the log is output, the request ID 106 uniquely set for each request, and the host name 108 of the server that performs log output are removed from the log of FIG.

  The expected value log file holding unit 46 holds the expected value log file in which the expected value log is written. The expected value log file is created by the tester before starting the operation test. The person in charge of the test may create an expected value log file based on a log output when a similar operation test is performed in the past. For example, a log is output to “system.log (first log file 16a)” and “apl01.log (third log file 16c)” out of four log files held by the test target system 12 by an operation test. Are expected, log files “system_kitai.log (first expected value log file 18a)” and “apl01_kitai.log (third expected value log file 18c)” corresponding to these are created in advance. The data structure of the expected value log is the same as the data structure of the shaping log.

  The verification result holding unit 47 holds verification result information indicating a result of verifying whether the shaping log matches the expected value log. For example, the test case ID that uniquely identifies the content of the behavior test, the file name of the expected value log file, the content of the expected value log, the file name of the shaped log file, the content of the shaped log, and the verification result (normal・ Abnormal judgment result) is held as verification result information.

  The control unit 20 includes an operation detection unit 21, a time acquisition unit 22, a test execution unit 23, an update time acquisition unit 24, an acquisition target list creation unit 25, a log file acquisition unit 26, a log extraction unit 27, a log shaping unit 28, a verification Unit 29, verification result recording unit 30, and verification result providing unit 31. The operation detection unit 21 detects an operation input by the user with respect to an input device such as a keyboard or a mouse.

  The time acquisition unit 22 acquires the system time of the test target system 12 when an operation for instructing execution of the test is input. The time acquisition unit 22 records this as the “test start time” in the start time holding unit 41.

  When the time acquisition unit 22 acquires the system time, the test execution unit 23 starts an automatic test for the test target system 12. The test execution unit 23 passes a predetermined URL to the web browser. The web browser acquires the operation target web page specified by the URL from the web application server 14 and displays it on the display device. The test execution unit 23 performs a predetermined operation on the web page displayed by the web browser. For the operation on the web page, a known method such as the method described in Prior Art Document 1 may be used.

  When the update time acquisition unit 24 detects that the operation test by the test execution unit 23 is completed, the update time acquisition unit 24 acquires update times of various log files of the test target system 12. The update time acquisition unit 24 records the log file names of various log files and their update times in the update time holding unit 42 in association with each other.

  The acquisition target list creation unit 25 creates a list of log files to be acquired as verification targets. Specifically, the acquisition target list creating unit 25 refers to the start time holding unit 41 and the update time holding unit 42 and creates a list of log files whose update time is newer than the test start time. That is, since the log at the time of the operation test is output to the log file whose update time is later than the test start time, the acquisition target list creation unit 25 includes them in the list as log files to be acquired. The acquisition target list creation unit 25 records the created list in the acquisition target list holding unit 43. For example, if the test start time is “2014-03-23 14: 25: 38.150”, the update time of “system.log” and “apl01.log” is newer than the test start time in the example of FIG. The file to be acquired.

  The log file acquisition unit 26 refers to the acquisition target list holding unit 43 and acquires log files included in the list from the test target system 12. The log file acquisition unit 26 stores the acquired log file in the log file holding unit 44.

  The log extraction unit 27 extracts a verification target log from the log output to the log file. Specifically, the log extraction unit 27 extracts logs written after the test start time from the logs output to the log file held in the log file holding unit 44. For example, when the test start time is “2014-03-23 14: 25: 38.150”, in the example of FIG. 3, a log surrounded by a broken line is extracted.

  The log shaping unit 28 creates a log from the log extracted by the log extraction unit 27 by removing elements whose values are determined at the time of test execution (in other words, elements that cannot be expected in advance) To do. In the embodiment, a shaping log that does not include the time stamp 102 indicating the time when the log is output, the request ID 106, and the host name 108 is created. In the embodiment, the log shaping unit 28 further encloses each element in quotation marks and converts the delimiter character string into “,” (comma). The log shaping unit 28 outputs the created shaping log to a shaping log file and stores it in the shaping log file holding unit 45.

  The verifying unit 29 verifies whether the shaped log output to the shaped log file matches the expected value log written in the corresponding expected value log file. The verification unit 29 determines that the test result is normal when the two match. On the other hand, if the two do not match, the test result is determined to be abnormal. In addition, as a mismatch case, the log that should be output is not output, the content of the output log is different from the expectation, the log that is not expected to be output is output, Can be considered. A typical example of a case where a log that is not expected to be output is output is a case where a log regarding an unexpected error is output.

  The verification result recording unit 30 records verification result information in the verification result holding unit 47. The verification result providing unit 31 causes the display device to display verification result information held in the verification result holding unit 47 when an operation for instructing display of the verification result is input. For example, when the verification result is abnormal, the mismatched portion between the expected value log and the shaped log may be highlighted and displayed. Further, when a log regarding an unexpected error is output, the log may be particularly emphasized.

The operation during the test with the above configuration will be described below.
The person in charge of the test inputs a test start instruction to the test execution device 10. The time acquisition unit 22 acquires the system time from the test target system 12 and sets it as the test start time. The test execution unit 23 executes an operation test of the test target system 12.

  When the update time acquisition unit 24 detects that the operation test by the test execution unit 23 is completed, the update time acquisition unit 24 acquires update times of various log files of the test target system 12. The acquisition target list creation unit 25 determines a log file whose update time is newer than the test start time among the various log files as an acquisition target, and the log file acquisition unit 26 acquires the determined log file.

  The verification unit 29 extracts a log written after the test start time from the log output to the log file, and creates a shaping log therefrom. The verification unit 29 verifies whether the shaping log matches the expected value log. If they match, the test result is determined to be normal, and if they do not match, it is determined to be abnormal. The verification result providing unit 31 displays verification result information on the display device in response to a user request.

  According to the test execution device 10 of the embodiment, a log whose update time is after the test start time is acquired from the test target system 12. That is, only the log file to which a log is output when the operation test is executed is acquired from the test target system 12. For this reason, for example, when all log files held by the test target system 12 are acquired, the time required to acquire the log file is shortened compared to the case where even the log file that does not need to be verified is acquired. Operation test can be realized.

  In addition, according to the test execution device 10, logs output after the test start time are extracted from the logs output to the log file. Therefore, the time required for verification can be shortened and an efficient operation test can be realized as compared with the case where all the logs output to the log file are to be verified.

  In addition, according to the test execution device 10, elements whose values are determined at the time of test execution (in other words, elements that cannot be expected in advance) such as a time stamp indicating the time when the log is output are removed. A formatted log is created. This eliminates the need for a tester to manually remove such elements, and allows an operation test to be automatically executed including log verification. This reduces the burden on the operator. In addition, an efficient operation test can be realized.

  Further, according to the test execution device 10, the system time of the test target system 12 when the execution of the operation test is started is acquired as the test start time. Therefore, for example, even if the system time is shifted between the test execution device 10 and the test target system 12, it is possible to acquire a log to be acquired properly and to extract a log to be extracted.

  The present invention has been described based on the embodiments. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  Although not particularly mentioned in the embodiment, elements included in the shaping log may be designated. For example, the log shaping unit 28 may accept the designation of the delimiter character string of the log output to the log file and the designation of the position of the element to be included in the shaping log, and create the shaping log based on this. For example, when the log shaping unit 28 receives the delimiter character string “” (space) and the element position “third column, sixth column, seventh column, eighth column” for the log in FIG. 28 creates the shaping log of FIG.

  It should also be understood by those skilled in the art that the functions to be fulfilled by the constituent elements described in the claims are realized by the individual constituent elements shown in the embodiments and the modification examples or by their cooperation.

  DESCRIPTION OF SYMBOLS 10 Test execution apparatus, 12 Test object system, 14 Web application server, 20 Control part, 21 Operation detection part, 22 Time acquisition part, 23 Test execution part, 24 Update time acquisition part, 25 Acquisition object list creation part, 26 Log file Acquisition unit, 27 log extraction unit, 28 log shaping unit, 29 verification unit, 30 verification result recording unit, 31 verification result providing unit, 40 data recording unit, 41 start time holding unit, 42 update time holding unit, 43 acquisition target list Holding unit, 44 log file holding unit, 45 shaped log file holding unit, 46 expected value log file holding unit, 47 verification result holding unit, 100 test system.

Claims (5)

A device for testing a system,
A time acquisition unit for acquiring a test start time;
A test execution unit for starting automatic execution of the test of the system from the test start time;
An acquisition unit for acquiring a log file updated after the test start time from the system after the end of the test;
A log extraction unit that extracts logs output after the test start time from the acquired log file;
A verification unit that verifies whether or not a part obtained by removing an element whose value is determined at the time of test execution from the extracted log matches a predetermined expected value;
A test execution device comprising:   The test execution apparatus according to claim 1, wherein the element whose value is determined at the time of the test execution includes at least information related to a time when the log is output.   The test execution device according to claim 1, wherein the time acquisition unit acquires a system time of the system. A method performed on an apparatus for testing a system, comprising:
Obtaining a test start time;
Starting automatic execution of the test of the system from the test start time;
Obtaining a log file updated after the test start time from the system after completion of the test;
Extracting the log output after the test start time from the acquired log file;
Verifying whether or not the portion of the extracted log excluding elements whose values are determined at the time of test execution matches a predetermined expected value;
A test execution method comprising: To test equipment to test the system,
The ability to get the test start time,
A function of starting automatic execution of the test of the system from the test start time;
A function to obtain a log file updated after the test start time from the system after the test is completed;
A function to extract logs output after the test start time from the acquired log file;
A function that verifies whether or not the part of the extracted log excluding the element whose value is determined at the time of test execution matches the predetermined expected value;
Computer program for realizing.

Images