JP2006268666A - Automatic test system with correction function, automatic test method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To automatically correct a script by analyzing the action of an application in a test environment employing the script to be applied in the test process of software development for operating the application. <P>SOLUTION: The automatic test system includes a correction function. In an automatic test for a system including a GUI, the automatic test system for performing the automatic test preserves a result (log information) obtained by comparing timing to control the GUI with image information, and a correction tool analyzes the log information and corrects the script of the automatic test system to exclude errors during the test. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a technique for analyzing a behavior of an application and automatically correcting the script in a test environment having a script for operating the application.

  In recent years, test work in software development has been increasingly automated. Test automation not only reduces test man-hours, but also improves the accuracy of test operations and results judgment, making it an effective tool indispensable for software development.

  In addition, visualization has progressed in applications, and styles using a GUI (Graphical User Interface) have been frequently used.

  Attempts to automate tests related to GUI applications have been made in response to such automation of tests and heavy use of GUI applications. For example, a technique has been proposed in which image information prepared in advance is compared with image information obtained when an application to be tested is actually operated to determine whether the behavior of the GUI application is good or not (patent) Reference 1).

However, in the conventional test system, when the image comparison timing set in the script does not coincide with the actual behavior of the GUI, the test cannot be continued and the test work is hindered. In order to make the interrupted test system work as expected, the script had to be modified manually and required a lot of man-hours.
JP-A-8-241191

  Therefore, in order to solve the above problem, in the test system of the present invention, various image patterns of the GUI are prepared, the behavior of the GUI when the test script is executed and the image pattern are compared, and the result is used as log information. A technique is provided for automatically correcting a test script for an error log by recording and analyzing the log information.

  The first invention is an automatic test system having a function of correcting a script for testing a graphical user interface system, and is registered in advance in an image screen and an execution screen displayed by executing the test script of the system. Result / image comparison means for comparing images, log output means for outputting the result / comparison result in the image comparison means as a log information file, referring to the log information file, and screen mismatch from the log information Log analysis means for analyzing the execution timing of the error log, and script correction means for specifying a correction value for the error log analyzed by the log analysis means and automatically correcting the test script according to the correction value; The present invention relates to an automatic test system.

  That is, according to the first invention, the result / image comparison unit compares the actual screen of the GUI system obtained by executing the GUI test script with the pre-registered image, and the log output unit compares the comparison result. The result is output as a log information file, and the log analysis means detects the error log from the log information file, analyzes when the screen mismatch occurred, and the script correction means based on the analysis result By correcting the test script with respect to the error log, the error is automatically eliminated, and the test operation can be realized as expected without the software development tester being aware of the error content.

  This makes it possible to smoothly carry out software testing work that has been complicated in the past, and facilitates maintenance of test scripts.

  According to a second aspect of the present invention, the correction value of the script correction means is set as a comparison period of screen switching when comparing the execution screen of the system and an image registered in the image file, or the number of comparisons. The present invention relates to the automatic test system according to the first invention.

  That is, according to the second invention, the script correction unit defines comparison period information such as 100 ms, 50 ms, and 10 ms, or comparison number information such as 10, 20, and 30 times as the comparison correction value of the switching screen. By doing so, the test script can be executed with the information intended by the tester, and a useless test is not performed, and the work efficiency of the test is improved.

  According to a third aspect of the present invention, the script correction means determines a correction value for correcting an error with reference to a user-defined file or a statistical data file in which determination results based on past correction values are stored. The present invention relates to an automatic test system according to the first or third invention.

  That is, according to the third aspect, the script correction means adopts a correction value defined in advance by the user or a correction value based on statistical data based on the result of success or failure when the correction value is applied in the past. A wide range of options will enable more appropriate test execution.

  According to the present invention, the log analysis unit detects the error log from the log information file, analyzes the timing at which the screen mismatch occurred, and the script correction unit applies the error log to the error log based on the analysis result. By correcting the test script, errors are automatically eliminated, and it becomes possible for the software development tester to realize the expected operation of the test script without being aware of the error content. This makes it possible to smoothly proceed with the software testing work that has been complicated in the past, and facilitates maintenance of the test script.

  In addition, by defining the comparison cycle and the number of comparisons in advance for the GUI test script by the tester, the test script can be executed with the information intended by the tester, and there is no need to perform a useless test. The work efficiency of the test will be improved.

  Furthermore, by adding a correction value based on statistical data based on the result of success or failure when a correction value has been applied in the past as an option, a more appropriate test can be executed.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
FIG. 1 shows a basic configuration of a GUI automatic test system having a correction function according to an embodiment of the present invention. The system of the present invention includes an automatic test system 1 and a GUI system 2 in which the screen is switched depending on various events to be subjected to the test evaluation.

  The automatic test system 1 includes an automatic test control unit 11 that controls the entire automatic test system, a test script file 100 (sample data 1-FIG. 2) for operating the GUI system 2, and a GUI execution unit 12 that controls the GUI system 2. As a result of the operation by the test script, a result / image comparison unit 14 for comparing the displayed screen 13 with the image information of the screen information file 200 prepared in advance, and the comparison result are shown in the log information file 300 (sample data 2-diagram 3), the log output unit 15 that outputs the log information file 200, the log analysis unit 16 that analyzes the log contents of the log information file 200 and analyzes which procedure caused the image comparison timing mismatch, and the correct correction value to the script Reflected (sample data 3-FIG. 4) is composed of a script correction unit 17.

  Sample data 1, sample data 2, and sample data 3 described above are shown in FIGS. 2, 3, and 4, respectively.

  FIG. 2 shows a test script example (sample data 1: before correction) of the GUI application according to the embodiment of the present invention. The test script is a script file in which a scenario for operating the GUI is written, and is executed by the automatic test apparatus 1 having a correction function. As a result of the execution, the displayed actual image does not match the pre-registered screen. When this occurs, the value of the script comparison period or the number of comparisons is corrected.

  The image information to be compared with the actual screen displayed as a result of executing the test script is the same image as the image displayed on the GUI system stored as an image file, and is prepared in advance by the tester.

  In the embodiment, a script for testing whether the GUI application built in the mobile phone is operating well is shown, and an example of checking the switching of the actual image in the “mode setting” of the mobile phone is taken up.

  The command in each line of the script first executes a test of each setting key in the mobile phone, and in line (a), the image os-01-3. This means that the img repetition test is executed 20 times with a cycle of 500 ms.

  FIG. 3 shows log contents (sample data 2) of the test script execution result of the GUI application according to the embodiment of the present invention. The sample data 2 is obtained by executing the test script of the sample data 1 output by the log output unit 15 as the log information file 300. As a result, the log content of the determination result of the displayed real screen is the date, test content, test target, And information on determination results. In the example, in line (b), the image repeat test results in an error, and the determination of “reception failure” is written.

  FIG. 4 shows a test script example (sample data 3: after correction) of the GUI application according to the embodiment of the present invention. The sample data 3 shows an example in which an error is analyzed by the log analysis unit 16, the correction value is specified by the script correction unit 17, and the specified correction value is reflected in the script.

  In the embodiment, in the row (a), the script is corrected so that the comparison period is 400 ms and the number of comparisons is 25.

FIG. 5 shows the configuration of the correction processing of the automatic test system according to the embodiment of the present invention.
This block diagram of correction processing of the automatic test system 1 shows the processing configuration of the script correction unit 17 assuming that the comparison period is corrected.

  The configuration of the correction process includes a correction means specifying unit 21 that selects a correction value based on user-defined or past statistical data, a user-defined correction value specifying unit 22 that specifies a correction value based on the user definition, and a correction value that determines the upper limit of the correction value See upper limit determination unit 23, user-defined upper limit value specifying unit 24 that specifies an upper limit value defined by the user, statistical data storage unit 25 that stores determination data based on past corrections in statistical data DB (database) 400, and statistical data file 400. The statistical data correction value specifying unit 26 and the statistical data upper limit specifying unit 27 for specifying the upper limit with reference to the statistical data file 400 are included.

  (A) in the figure is a user-defined correction value applied to the test script, and indicates that, for example, a comparison period value of 10 ms to 100 ms and a comparison count of 10 to 20 are set.

  (A) shows an example in which the upper limit value of the comparison period is set to 10 ms and the upper limit value of the number of comparisons is set to 30 from the numerical range of the user-defined correction value a.

  FIG. 6 shows a configuration of error determination processing in the automatic test system according to the embodiment of the present invention. This figure shows a configuration for concretely implementing means for performing a retry of a test item or system by analyzing a log or notifying information when an error occurs.

  The log analysis unit 16 that performs log analysis is within the correction limit when there is an error (when there is room for adjustment of the comparison period and number of comparisons), or when the correction limit is exceeded (when the upper limit of the comparison period and number of comparisons is exceeded) An error determination unit 20 that analyzes whether or not the correction limit is exceeded, a communication control unit 30 that notifies the tester of information when the correction limit is exceeded, a restart control unit 40 that restarts the system and OS (Operating System) at that time, and The scenario activation control unit 50 performs a test from the next item of the test item that has exceeded the correction limit at the time of restart.

  The communication control unit 30 includes a mail transmission unit 31 that transmits an error message by mail and a display unit 32 that displays the error message.

  Further, the restart control unit 40 includes a test script, a GUI system 2, a test script re-execution control unit 41 that controls restart of the personal computer, a GUI system restart control unit 42, and a PC restart control unit 43. The

Here, the principle of the automatic test system 1 having the test script correction function according to the present invention will be described. The following corrections are assumed in the following three cases based on the comparison determination result between the actual image and the registered image displayed in the GUI test script.
Case 1 (in the case of confirmation OK): This is an example when the automatic test system 1 operates normally. There is no problem with the test script, so no correction of the test script is performed.
Case 2 (in the case of confirmation NG due to a long comparison cycle): Corresponding to the case where it is determined that the comparison cycle is long based on the log contents of the log information file 300, and the script correction unit 17 sets the comparison cycle to The appropriate value of the comparison period is obtained by shortening in stages.
Case 3 (confirmation NG due to a small number of comparisons): Corresponding to a case where it is determined that a failure has occurred due to a small number of comparisons based on the log contents of the log information file 300, and the script correction unit 17 sets the number of comparisons By increasing in a stepwise manner, an appropriate value for the number of comparisons is obtained.

Furthermore, a test script and its correction procedure will be described.
<Test script>
A script file in which a scenario for operating a GUI is written, and can be freely created and changed by a tester depending on the purpose of the test. It is executed by the automatic test control unit 11 of the automatic test apparatus 1 and corrected by the script correction unit 17 in accordance with the result analyzed by the log analysis unit 16. An example of a test script will be described below.

// Check the operation when setting the mode
// Confirm that image a transitions to image b and image c by script operation
replywait wait 100 retry 10 b.img
// Compare image b and actual GUI image in 100ms cycle, 10 times
key # press 4000 count 1
// Press “#” key once for 4000ms
loop end
// End image comparison process
replywait wait 100 retry 10 c.img
// Compare image c and actual GUI image 10 times in 100ms cycle
key # press 4000 count 1 wait 1000
// Wait for 1000ms.
loop end
// End of image comparison processing The image information applied above is an image file that is displayed in the GUI system when the test script is executed, and is prepared in advance by the tester.

<Script correction procedure (period correction case)>
□ Test script before correction
replywait wait 100 retry 10 b.img (→ Let X be the comparison cycle)
key # press 4000 count 1
loop end
replywait wait 100 retry 10 c.img
key # press 4000 count 1 wait 1000
loop end
□ User-defined correction value Correction means Fixed time Comparison period information (fixed time) 10 (ms) (→ Define value as Y)
Comparison period information (fixed rate) 10%

The correction procedure under the above conditions is shown below.
(1) Obtain image comparison period X from test script X = 100 (ms)
(2) Since the user-defined correction value is defined as “fixed time”, the value Y of the comparison cycle information (fixed time) is acquired from the user-defined correction value. Y = 10 (ms)
(3) Calculate the correction value Z by the following formula
Fixed time: XY = Z
Constant ratio: X * Y / 100 = Z
(4) Since the user-defined correction value is defined as “fixed time”, the calculation formula for fixed time in (3) is used. Z = 100 (ms) −10 (ms) = 90 (ms)
(5) Reflect Z in the test script
replywait wait 90 retry 10 b.img
<Script correction procedure (number correction case)>
□ Test script before correction
replywait wait 100 retry 10 b.img (→ X is the number of comparisons)
key # press 4000 count 1
loop end
replywait wait 100 retry 10 c.img
key # press 4000 count 1 wait 1000
loop end
□ User-defined correction value correction means: constant ratio comparison number information (constant time): 20 times comparison number information (constant ratio) 200% (→ define value as Y)

The correction procedure under the above conditions is shown below.
(1) Obtain image comparison count X from test script
X = 10 (times)
(2) Since the user-defined correction value is defined as “fixed rate”, the value Y of the comparison count information (fixed rate) is obtained from the user-defined correction value.
Y = 200 (%)
(3) The correction value Z is calculated by the following formula: Fixed time: X + Y = C
Constant ratio: X * Y / 100 = Z
(4) Since the user-defined correction value is defined as “constant ratio”, a constant ratio calculation formula is used. Z = 10 (times) * 200 (%) / 100 = 20 (times)
(5) Reflect Z in the test script
replywait wait 100 retry 20 b.img

Next, an operation flow of the automatic test apparatus 1 having a correction function will be described with reference to FIGS.

  FIG. 7 shows an overall processing flow for correcting the GUI test script in the automatic test system according to the embodiment of the present invention.

  First, in step S11, the automatic test control unit 11 executes the GUI application of the GUI system 2 via the GUI execution unit 12 according to the test script. In step S12, as a result of executing the GUI, the result / image comparison unit 14 compares the actual image displayed on the screen 13 with the image of the image information file 200 prepared in advance. In step S13, the log output unit The execution result is output to the log information file 300.

  In step S14, the log analysis unit 16 analyzes the log content with reference to the output log information file 300. In step S15, it is determined whether there is an error in the log information. As a result, if the error information does not exist, the process ends. If the error information exists, the script correction unit 17 specifies the correction value in step S16, and the test script is determined according to the correction value in step S17. to correct. Thereafter, the automatic test control unit 11 executes the GUI again with the corrected test script.

  Next, processing operations of the log analysis unit and the script correction unit, which are main functions of the present invention, will be described with reference to FIGS.

  FIG. 8 shows a log analysis processing flow according to the embodiment of the present invention. This flow describes details of the log analysis processing in step S14 in the overall flow of FIG.

  In step S21, log analysis processing is performed with reference to the log information file 300. In step S22, it is determined whether there is an error in the log information. As a result, when it is detected that there is an error in the log information, in step S23, an image comparison process identification number that can identify which image is generated when the image is compared, and the image of the application within the image comparison time. The script correction unit 17 is notified of image change presence / absence information that can specify whether or not there has been a change.

  Here, the image change presence / absence information is obtained by preparing a pre-change image, a post-change image, and a processing completion (initial screen) image, and comparing the three patterns of images with the actual GUI image in the test script. It is done.

  FIG. 9 shows a processing flow of the script correction unit according to the embodiment of the present invention. This flow details the specification of the correction value in step S16 of FIG.

  In step S31, the script correction unit 17 determines whether or not the image has changed from the image change presence / absence information. If there is a change in the image, it is determined that the comparison period is too large and the image has been missed, and the process proceeds to step S22 to perform a process of correcting the comparison period. Therefore, it is determined that the image has been missed, and the process proceeds to step S36 to correct the number of comparisons.

  When correcting the comparison period, in step S32, the script correction unit 17 acquires the image comparison period X set in the test script, and in step S33, acquires the value Y of the comparison period information defined by the user. To do.

  Next, in step S34, a correction value Z set in the test script is calculated using the acquired X and Y. Here, the comparison cycle information may be a fixed time for subtracting from X or a fixed ratio with respect to X. In the case of a certain time, Y is subtracted from X, and in the case of a certain ratio, X is multiplied by Y to obtain a correction value.

  In step S35, the calculated Z is reflected as a correction value in the corresponding comparison processing location in the test script.

  When correcting the number of comparisons, in step S36, the script correction unit 17 acquires the image comparison number X set in the test script, and in step S37, the value Y of the comparison number information defined by the user. To get.

  Next, in step S38, the correction value Z set in the test script is calculated using the acquired X and Y. Here, the comparison number information may be a fixed time for adding to X or a fixed ratio with respect to Y. When the time is constant, Y is added to X, and when the ratio is constant, X is multiplied by Y to obtain the correction value Z.

  In step S35, the calculated Z is reflected as a correction value in the corresponding comparison processing location in the test script.

FIG. 10 shows a processing flow of restart by error determination according to the embodiment of the present invention.
First, in step S41, a test result is acquired from the log analysis unit. Next, in step S42, the type of error is determined.

  If the result is normal (comparison with the normal result image matches), the restart information is initialized (set to none) in step S43.

  If the result is abnormal (comparison with the normal result image does not match), the process proceeds to the process of the correction means specifying unit 21 in step S44, and the restart information is initialized (set to none) in step S45.

  Further, if the result is abnormal (the comparison with the normal result image is inconsistent and the comparison with the image at the time of error occurrence is identical), the content of the error result is notified to the designated partner in step S46. Notification methods include e-mail and notification to applications including dialog displays. In step S47, an error test item number is acquired. In step S48, the presence / absence of restart information is determined.

  As a result, if the restart information is “none”, the restart information is set to “present” in step S51, and the GUI system 2 is restarted in step S52. If the restart information is “Yes” in step S48, the restart information is set to “Yes” in step S49, and the GUI system is restarted after the OS is restarted in step S50.

  Note that the restart information is set to “present” before restarting. When the GUI system is restarted, the test script is corrected, and the test items that have already succeeded are not executed, but the test of the subsequent items is executed.

  FIG. 11 shows a processing flow for selecting correction means according to the embodiment of the present invention. First, in step S61, the correction unit is acquired from the user definition file in which the correction unit is designated by the user. In step S62, it is determined whether the correction is user-defined or statistical data. If the specified correction means is user-defined, in step S63, a correction value specifying request is made to the user-defined correction value specifying unit. If the specified correction means is statistical data, in step S64, a correction value specifying request is made to the statistical data correction value specifying unit.

  As another means, statistical data should be referred to preferentially, and if the correction value used in the previous test matches the correction value obtained from the statistical data, it is extracted from the statistical data in the previous test. The user-defined correction value is used in the next test, and if the correction values do not match, the statistical data is regarded as unused and the statistical data is acquired. This eliminates the need to determine whether correction values are acquired from statistical data or user-defined data, thereby eliminating the need for the user to define correction means.

FIG. 12 shows the data structure of the statistical data DB according to the embodiment of the present invention.
The statistical data DB (database) 400 stores data notified from the error determination unit as statistical data, and executes “item number”, “comparison period” for comparing GUI images, “number of comparisons”, and test script. It is composed of a plurality of records consisting of “result” that represents the check result, “execution date and time” of the test script, “operation” that represents the content of the test script processing item, and “scenario name” of the test script processing. Yes.

  For example, in the first record, the item of item No. 001 is based on scenario 1, the image comparison period is 100 ms and the number of comparisons is 10 times, and the operation is performed on the condition that the phone book key is pressed, and scenario 1 is November 26, 2004. The determination result is OK.

  FIG. 13 shows a correction flow based on statistical data according to the embodiment of the present invention. First, in step S71, statistical data is acquired from the statistical data DB 400. From the acquired data, for the same operation and the result is OK, an average value is obtained for each of the comparison periods or the number of comparisons.

  In step S72, it is determined whether it can be calculated as statistical data. If it is NG, a request is made to the user-defined correction value specifying unit in step S73. Also, in step 74, it is determined whether or not the current execution is a result of executing NG using the statistical data correction value. If the result is NG even using the statistical data, a request is made to the user-defined correction value specifying unit. Do.

  In step S72, if it is possible to calculate from the statistical data and the statistical data in step S74 is not corrected, in step S75, the result OK and the comparison period and the number of comparisons are obtained from the same operation data. The average value of is calculated.

  Next, in step S76, the test script is corrected using the calculated average value as a correction value.

FIG. 14 shows a script correction value changing process flow according to the embodiment of the present invention. This flow acquires the comparison period S set in the test script in the correction value control unit step S81. Next, in step S82, a correction value T to be newly set is acquired from the comparison cycle information. Then, in step S83, it is determined whether all correction value candidates have been referenced. If all references have been made, control is passed to the error route (notification, restart, scenario start) in step S86.

  If there is a correction value candidate, in step S84, the acquired comparison periods S and T are compared. If S is small, the process returns to step S82, and the next correction value is acquired and compared again. On the other hand, if S is large, in step S85, T is reflected as a correction value in the corresponding comparison processing location in the test script.

  FIG. 15 shows a processing flow of error detection by log analysis according to the embodiment of the present invention. First, in step S91, time information of the test script file 100 is acquired, and in step S92, it is determined whether or not correction has been made. If there is no correction, the process ends. If correction is present and the determination of whether correction is present is the first time, in step S93, an automatic test is performed using a test script. In step S94, the result / image comparison unit 14 and the log output unit In step S95, it is determined whether there is an error as a result of log analysis. As a result, if there is an error, the script correction unit 17 performs correction in step S96, and then returns to step S91 to repeatedly perform the retest after that.

FIG. 16 shows a processing flow for obtaining a correction value as an upper limit according to the embodiment of the present invention.
In step S <b> 101, the correction value upper limit determination unit determines whether the upper limit value is user-defined or statistical data is acquired. As a determination criterion, user-defined and statistical data determination flavs are set in advance or can be automatically determined.

  As a result of the determination, in either case of user definition or statistical data, it is determined in step S102 or step S105 whether to acquire the period upper limit value or the number upper limit value. As a result of the determination, in step S103, step S104, step S106, or step S107, any one of the user-defined cycle upper limit value, the user-defined count upper limit value, the statistical cycle upper limit value, and the statistical count upper limit value is acquired.

  As described above, the correction value upper limit determination unit 23, the user-defined upper limit value specifying unit 24 for specifying the user-defined upper limit value, and the statistical data upper limit value specifying unit 27 are provided, and by executing the respective processes, It is possible to prevent unnecessary processing from being embedded in the test script.

  FIG. 17 shows a correction processing flow in the case of horizontal development to another test scenario that has not been executed according to the embodiment of the present invention. In this embodiment, after the test scenario is executed, an operation related to the successful correction value is extracted from the statistical data storage unit 25 shown in FIG. 4, and the test script is horizontally developed, and another test script having the same operation procedure is extracted. It reflects the successful correction value.

  In step S111, the scenario success procedure, comparison cycle, and number of comparisons are acquired and compared with an existing test script. Next, in step S112, the test script file 100 is referenced to determine whether there is a matching script. If there is a match, the script is corrected in step S113.

  FIG. 18 shows a processing flow of a test script corresponding to multimedia according to the embodiment of the present invention.

  First, in step S121, a voice or a moving image accompanying GUI execution by a test scenario is captured. Next, in step S122, the acquired voice or moving image data is compared with basic sound information or basic moving image information. In step S123, if the comparison results match, the log output is performed as failure if the success does not match.

  In this embodiment, the automatic test system 1 is provided with means for sound comparison and moving image comparison, and similar to the image information that was the comparison target at the time of image comparison, the sound and the moving image that are expected after the test are each represented as a basic sound. A file to be stored as information and basic video information is made easy. When performing an automatic test, information is stored as sound information and moving image information collected as a result of the execution, and in each comparison, basic sound information and collected sound information, or basic moving image information and collected moving image information, respectively. Comparison is made separately or in both cases to make a correct / incorrect decision, making it possible to handle multimedia.

  FIG. 19 shows a comparison processing flow for replacing a plurality of images in the result / image comparison unit according to the embodiment of the present invention. First, in step S131, the result / image comparison unit compares the GUI execution result with the image information. Next, in step S19, it is determined whether the image comparison results match. In step S133, the image correction unit replaces the image information (image A → B). In this embodiment, image A, image B, and image C that meet all the preconditions are stored in the image information, and when an error is detected, the image to be compared is image A → image B → image C. To make a recomparison.

  FIG. 20 shows a processing flow for detecting a mismatch in the switching image according to the embodiment of the present invention. This flow shows a flow when the error determination unit 20 specifies the error type of the test result.

  In step S 141, the test result is acquired from the result / image comparison unit 14. Next, in step S142 and step S143, a match of image comparison is determined. This process is compared with the full screen image prepared in advance until the images match. If the images match, the image identification number is acquired and notified to the result / image comparison unit 14 in step S144. If it is determined in step S142 and step S143 that the images do not match, the comparison target image is changed in step S145, and the process returns to step S142 to execute the subsequent processing.

  The embodiments of the present invention described above are as shown in the following supplementary notes.

(Appendix 1) An automatic test system having a script correction function for testing a graphical user interface system,
A result / image comparison means for comparing the execution screen displayed by executing the test script of the system with an image registered in the image information file in advance;
Log output means for outputting the comparison result in the result / image comparison means as a log information file;
Log analysis means for referring to the log information file and analyzing the execution timing of an error log that has a screen mismatch from the log information;
Script correction means for specifying a correction value for the error log analyzed by the log analysis means, and automatically correcting the test script according to the correction value;
An automatic test system characterized by comprising:

  (Supplementary Note 2) The correction value of the script correction unit is set as a comparison period or a comparison count of screen switching when comparing the execution screen of the system and the image registered in the image file. The automatic test system according to appendix 1.

  (Supplementary note 3) The automatic test system according to Supplementary note 1 or 2, wherein in the script correction means, an upper limit value is set as the correction value for the comparison period of screen switching or the number of comparisons.

  (Additional remark 4) The said script correction | amendment means determines the correction value which corrects an error with reference to the statistical data file which accumulated the correction value defined by the user beforehand by the user definition file or the determination result by the past correction value. The automatic test system according to any one of appendices 1 to 3, characterized by:

  (Supplementary Note 5) The log analysis means has an error determination means for referring to the log information file to determine an error type as to whether or not the error content is unrecoverable. The automatic test system according to appendix 1, wherein retry of an item is interrupted.

  (Additional remark 6) The said script correction | amendment means repeats correction | amendment by the said comparison period or the said frequency | count of a comparison of an image until an error does not generate | occur | produce, The automatic test system of Additional remark 1 thru | or 4 characterized by the above-mentioned.

  (Supplementary note 7) The automatic test system according to supplementary note 4, wherein the correction value based on the statistical data is automatically calculated and determined from the accumulated past data by a statistical method.

  (Supplementary note 8) Extract the test scenario of the same pattern from other test scenarios that have not been executed, the corrected comparison period or the number of comparisons that no longer causes an error, and calculate the correction value and the value in the unexecuted test scenario. The automatic test system according to appendix 1, wherein the replacement is horizontally deployed.

  (Supplementary note 9) Having a multimedia-compatible function that compares the sound waveform of the basic sound information saved as a database with the sound sampled during the test, or compares the saved basic video information with the video image captured during the test The automatic test system according to appendix 1, characterized by:

  (Supplementary Note 10) When the expected image information is different even in the same operation procedure due to the preconditions of the test, a plurality of expected image information is stored in advance for each condition, and the expectation is triggered by an error in image comparison. The automatic test system according to appendix 1, wherein the image information is corrected regardless of the preconditions by changing the image information.

  (Additional remark 11) Even if the display switching of the system is not in time and transition to an unexpected screen, the error type should be clarified by preparing a full-screen image in advance so that the images can be collated. The automatic test system according to supplementary note 1, which is characterized.

(Supplementary note 12) An automatic test method having a correction function of a script for testing a graphical user interface system,
A result / image comparison step for comparing the execution screen displayed by executing the test script of the system with an image registered in the image information file in advance;
A log output step for outputting the comparison result in the result / image comparison step as a log information file;
A log analysis step of referring to the log information file and analyzing the execution timing of an error log that has a screen mismatch from the log information;
A script correction step for specifying a correction value for the error log analyzed by the log analysis step, and automatically correcting the test script according to the correction value;
An automatic test method characterized by comprising:

(Supplementary note 13) A program having a script correction function for testing a graphical user interface system,
On the computer,
A result / image comparison step for comparing the execution screen displayed by executing the test script of the system with an image registered in the image information file in advance;
A log output step for outputting the comparison result in the result / image comparison step as a log information file;
A log analysis step of referring to the log information file and analyzing the execution timing of an error log that has a screen mismatch from the log information;
A script correction step for specifying a correction value for the error log analyzed by the log analysis step, and automatically correcting the test script according to the correction value;
A program that executes

It is a figure which shows the basic composition of the GUI automatic test system provided with the correction function which becomes embodiment of this invention. It is a figure which shows the test script example (sample data 1: before correction | amendment) of the GUI application which becomes embodiment of this invention. It is a figure which shows the log content (sample data 2) of the test script execution result of the GUI application which becomes embodiment of this invention. It is a figure which shows the example test script (sample data 3: after correction | amendment) of the GUI application which becomes embodiment of this invention. It is a figure which shows the structure of the correction process in the automatic test system which becomes embodiment of this invention. It is a figure which shows the structure of the error determination process in the automatic test system which becomes embodiment of this invention. It is a figure which shows the whole processing flow which correct | amends the GUI test script in the automatic test system which becomes embodiment of this invention. It is a figure which shows the log analysis processing flow which becomes embodiment of this invention. It is a figure which shows the processing flow of the script correction | amendment part which becomes embodiment of this invention. It is a figure which shows the processing flow of restart by the error determination which becomes embodiment of this invention. It is a figure which shows the processing flow of correction | amendment means selection which becomes embodiment of this invention. It is a figure which shows the data structure of statistical data DB which becomes embodiment of this invention. It is a figure which shows the correction flow by the statistical data which becomes embodiment of this invention. It is a figure which shows the change process flow of the correction value of the script which becomes embodiment of this invention. It is a figure which shows the processing flow of the error detection by the log analysis which becomes embodiment of this invention. It is a figure which shows the processing flow of the correction value acquisition used as the upper limit which becomes embodiment of this invention. It is a figure which shows the correction | amendment processing flow in the case of carrying out horizontal expansion | deployment to the other test scenario which is not implemented which becomes embodiment of this invention. It is a figure which shows the processing flow of the test script matched with the multimedia which becomes embodiment of this invention. It is a figure which shows the comparison processing flow of replacement of the several image in the result and image comparison part which becomes embodiment of this invention. It is a figure which shows the processing flow for the mismatching detection of the switching image which becomes embodiment of this invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Automatic test system 2 GUI system 11 Automatic test control part 12 GUI execution part 13 Screen 14 Result / image comparison part 15 Log output part 16 Log analysis part 17 Script correction part 20 Error determination part 21 Correction means specification part 22 User-defined correction value Specification unit 23 Correction value upper limit determination unit 24 User-defined upper limit value specification unit 25 Statistical data storage unit 26 Statistical data correction value specification unit 27 Statistical data upper limit value specification unit 30 Communication control unit 31 Mail transmission unit 32 Display unit 40 Restart control unit 41 Test script re-execution control unit 42 GUI system restart control unit 43 PC restart control unit 50 Scenario activation control unit 100 Test script file 200 Image information file 300 Log information file 400 Statistical data DB

Claims (5)

An automatic test system with a script correction function for testing a graphical user interface system,
A result / image comparison means for comparing the execution screen displayed by executing the test script of the system with an image registered in the image information file in advance;
Log output means for outputting the comparison result in the result / image comparison means as a log information file;
Log analysis means for referring to the log information file and analyzing the execution timing of an error log that has a screen mismatch from the log information;
Script correction means for specifying a correction value for the error log analyzed by the log analysis means, and automatically correcting the test script according to the correction value;
An automatic test system characterized by comprising:   2. The correction value of the script correction unit is set as a comparison period of screen switching when comparing an execution screen of the system and an image registered in the image file, or the number of comparisons. Automatic test system as described in   The script correction means determines a correction value for correcting an error with reference to a user-defined file for a correction value defined in advance by a user or a statistical data file in which determination results based on past correction values are accumulated. The automatic test system according to claim 1 or 2. An automatic test method with a script correction function for testing a graphical user interface system,
A result / image comparison step for comparing the execution screen displayed by executing the test script of the system with an image registered in the image information file in advance;
A log output step for outputting the comparison result in the result / image comparison step as a log information file;
A log analysis step of referring to the log information file and analyzing the execution timing of an error log that has a screen mismatch from the log information;
A script correction step for specifying a correction value for the error log analyzed by the log analysis step, and automatically correcting the test script according to the correction value;
An automatic test method characterized by comprising: A program having a script correction function for testing a graphical user interface system,
On the computer,
A result / image comparison step for comparing the execution screen displayed by executing the test script of the system with an image registered in the image information file in advance;
A log output step for outputting the comparison result in the result / image comparison step as a log information file;
A log analysis step of referring to the log information file and analyzing the execution timing of an error log that has a screen mismatch from the log information;
A script correction step for specifying a correction value for the error log analyzed by the log analysis step, and automatically correcting the test script according to the correction value;
A program that executes

Images