JP2017016507A - Test management system and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a test management system that is capable of unifying test quality, shortening a delivery period, and lightening workload of a person in charge of test.SOLUTION: Test resource storage means 52 for storing test data and test scenario; job storage means 42 for combining a plurality of jobs including a test environment construction job which sets test data to a test object system 200 and a test job which executes test by using a test scenario, and storing a job net formed by specifying their execution sequence; and job execution means 23 for continuously executing a plurality of jobs according to the execution sequence specified by the job net, arraying each diagram showing each of the plurality of jobs according to the execution sequence to display on a screen, and displaying test progress are provided, and thus constituting a test management system 10.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a test management system and program configured by a computer that executes and manages an operation check test of a test target system, and can be used for, for example, a test continuously executed in development of a distributed application system. .

  In general, in the development of a complex or large-scale system such as a distributed application system, continuous integration (CI) in which tests including build and deployment are repeated is practiced.

  Conventionally, in order to realize such continuous integration, some tests have been automated using test tools. In addition, by creating a test procedure manual and conducting tests according to the test procedure document, variations in procedure among test personnel were suppressed, and test quality was made uniform and test reliability improved.

  Furthermore, when the test fails, in order to analyze the cause of the failure, the failure information is manually registered in the problem management table, which is useful for application correction and the like.

  In addition, using a large number of test results (for example, thousands to tens of thousands of test results), analysis such as distribution of test execution time and calculation of test success rate is performed manually to verify the test results. Was. In addition, the test data used in each test and the documents such as test procedure manuals are manually managed.

  On the other hand, in recent years, a tendency to use software that supports continuous integration (hereinafter referred to as a CI tool) is also increasing. For example, an open source CI tool called Jenkins has been developed. There are also tools that automate the execution of some tests (see Non-Patent Documents 1 and 2).

Hewlett-Packard Japan, “Quick Test Professional Manual”, [online], the company's homepage, [Search May 25, 2015], Internet <URL: http://h50146.www5.hp.com/doc/ manual / openview / qtp.html> Nomura Research Institute, Ltd., “Test Platform Package“ Tepura ””, [online], the company's homepage, [Search May 25, 2015], Internet <URL: http://teplapa.nri.co. jp / ＞

  As mentioned earlier, some tests have traditionally been automated using test tools, but each test tool supports a single test and automatically tests multiple tests. Could not run continuously. Therefore, since the test is not completely automated and is executed and confirmed manually, a so-called human error has occurred with a certain probability. In addition, rework accompanying it occurred, leading to a delay in the delivery date of the development target system. Furthermore, since it is a manual work, it takes a lot of time to repeat a series of tests, and the ratio of the time required for testing to the total development period is quite large. In addition, since the tests are performed manually, there are many cases where sufficient tests cannot be performed due to time and budget reasons.

  In addition, a test procedure document was prepared and a test was performed according to the test procedure. However, since this was a manual operation, personalization occurred, and the test procedure might differ depending on individual skills. As a result, test quality and reliability, as well as system development efficiency, may differ among multiple testers. And since it is a manual work, every time there is a personnel change, an educational cost is incurred, and various troubles tend to occur immediately after the change.

  Furthermore, each time a test fails, the failure information was manually registered in the issue management table and analyzed, and the failure reason obtained by the analysis was described. In the first half of the test phase, Since many failures occur, information on the huge number of failures was manually managed. In addition, related error logs and the like are placed on a file server separately from assignment registration, and linked. However, assignment registration and related error log linking is a heavy work, so it has not been performed sufficiently, and it took time to analyze the reason for failure. There were also many cases that could not be analyzed due to lack of information on failure.

  In addition, statistical analysis was performed using a large number of test results (for example, thousands to tens of thousands of test results), and the test results were verified. However, since this was a manual analysis work, the load was high. And since it was a heavy work, there was a case where analysis was not performed. Furthermore, since the test data used for each test and the documents such as test procedure manuals were manually managed, each test or their success / failure information and the test used for each test Data and documents were not linked sufficiently. In addition, since the person who performed the test was manually managed, it was similarly not sufficiently linked. For this reason, there has been a situation where effective analysis cannot be performed or analysis cannot be fully utilized.

  An object of the present invention is to provide a test management system and a program capable of achieving uniform test quality, shortening the delivery time, and reducing the work burden on a tester.

The present invention is a test management system configured by a computer that executes and manages an operation check test of a test target system,
Test resource storage means for storing test data to be stored or arranged in a test target system and a test scenario specifying a test content;
A plurality of jobs including a test environment construction job for storing or arranging the test data stored in the test resource storage unit in the test target system, and a test job for executing a test using the test scenario stored in the test resource storage unit A job storage means for storing a job net formed by specifying the execution order of these jobs in combination,
A job execution process for continuously executing a plurality of jobs in accordance with the execution order specified in the job net stored in the job storage means, and a plurality of figures indicating each of the plurality of jobs are arranged and displayed on the screen in the execution order. A job graphic display process that displays a job name and / or an explanatory text that briefly indicates the job contents in or near each graphic, and a test progress display process that indicates a job being executed using multiple graphics And a job execution means for executing.

  Here, it is described that “job storage means” stores a job net, and “job execution means” continuously executes a plurality of jobs constituting the job net. In other words, it is only necessary that the job net can be stored and executed.

  In such a test management system of the present invention, the job net can be stored by the job storage means, and the job net can be executed by the job execution means. A plurality of jobs can be continuously executed according to the designated execution order.

  For this reason, manual work is reduced for a series of tests, and the occurrence probability of so-called human error is suppressed, so that it is possible to shorten the delivery time and reduce the work load of the person in charge of the test.

  In addition, because there are many manual operations in the past, there were many cases where sufficient tests could not be performed due to time and budget reasons, but such a situation was realized by realizing continuous execution of job net configuration jobs. This makes it possible to avoid the problem and improve the quality and reliability of the test.

  In addition, the conventional test involves a lot of manual work in accordance with the test procedure manual, so personalization occurs and the test procedure may differ depending on individual skills, but continuous execution of job net configuration jobs As a result, the homogenization of test quality and reliability can be achieved, and differences in system development efficiency between individuals can be reduced and uniformized. . Further, since manual work is reduced, it is possible to solve various problems such as an educational cost each time a personnel change occurs or various troubles occur immediately after the change.

  In addition, job graphic display processing and test progress display processing are performed by the job execution means. By this visualization, the person in charge of the test can view the overall content of the test including the execution procedure and the progress of the test on the screen. Since it is possible to easily grasp and understand, the burden of confirmation and monitoring of the tester is reduced, and this also reduces the work load of the tester.

  Furthermore, test data and test scenarios used in the test are stored in the test resource storage means, and the job execution means uses the test data and test scenarios stored in the test execution means to perform the test. It is possible to eliminate the situation where a specific tester or application developer stores and manages test data and test scenarios related to himself / herself. For this reason, another person, for example, a person in charge of test or application development, or a person who temporarily replaces the test data or test scenario stored in the test resource storage means can reuse the next test. Since it is possible to implement and update them, the time and effort required for test preparation can be greatly reduced, and standardization of test quality and improvement of reliability can be achieved. become.

  In the past, one or more test tools (single-function test robots) that were individually activated and used manually by the tester for some processes in a series of tests If automatically called and started by executing one or a plurality of test jobs constituting the net, it is possible to realize automatic cooperation of an arbitrary number of existing or new test tools.

  Also, by using a test tool (single-function test robot) through the test management system, it is possible to record which test tool was used as the build information in the executed test. It is possible to grasp the operating status of the tool. This makes it possible to measure economic effects and return on investment. As a result, the test tool can be strengthened / improved according to the usage situation, and the object can be achieved.

In the test management system described above,
The test data or test scenario used to execute the configuration job stored in the test resource storage unit is updated for the job net configuration job that is specified in advance as an automatic execution target among the job nets stored in the job storage unit It is desirable to include test automatic execution means for executing processing for starting job execution processing by the job execution means when it is determined whether or not it has been updated.

  In this way, with a configuration with automatic test execution means, it is possible to automatically execute the test when the application etc. is updated. You will be able to test This automatic execution can automate the continuous execution of job net configuration jobs, including startup, thereby further reducing the delivery time and the workload of testers. become able to.

Furthermore, in the test management system described above,
Setting information storage means for storing the timer execution time for executing the job net stored in the job storage means by starting the timer in association with the identification information of the job net;
Monitor whether or not the timer execution time stored in the setting information storage means has arrived, and if it is determined that it has arrived, start job execution processing by the job execution means for the job net related to the timer execution time that has arrived It is desirable to have a configuration including timer execution means for executing the processing to be performed.

  In the case of the configuration including the timer execution means as described above, the test can be performed outside the working hours such as at night or when the user goes out by setting the timer execution time. In addition, the timer execution can realize automation including start-up for continuous execution of job net configuration jobs. For this reason, it becomes possible to further shorten the delivery time and reduce the work burden on the tester.

In the test management system described above,
Test result storage means for accumulating and storing test result data of repeated tests;
Test success / failure determination means for executing processing for determining success / failure of a test by the job execution means;
The success / failure determination result by the test success / failure determination means is used to identify the tester's identification information, test execution time, test execution time, and build information including the test data or test scenario information used for the test, or A test result registration unit that executes a process of storing the test result data in the test result storage unit together with the build identification information for associating the information,
A test result analyzing means for executing a statistical process including a trend analysis on a test time zone and / or a test person using a plurality of test result data accumulated and stored in the test result storing means;
The analysis result output means for executing the process of displaying the analysis result by the test result analysis means on the screen or printing, or sending the analysis result file storing the analysis result to the mail address of the person in charge of the test by e-mail. It is desirable to have a configuration.

  When statistical analysis of test results is performed and the analysis results are displayed, printed, or sent by e-mail, a large number of test results (for example, thousands to tens of thousands of tests) are used. Compared with the case where manual analysis is performed, the workload is reduced, so that sufficient analysis can be performed and verification of the test result with high accuracy can be realized.

  Also, the test result registration means makes the test success / failure determination result include information on the build (including identification information of the person in charge of the test, test execution time, test execution time, and test data or test scenario information used for the test). Or, together with build identification information (information for associating build information), it is saved in the test result storage means as test result data, so test data and test procedures used for each test as before Compared to the case where the documents are managed manually, each test or their success / failure information can be sufficiently linked to the test data and test scenario used in each test. It becomes like this. In addition, when and who performed the test, it is possible to sufficiently perform the linking as compared with the case where the test was manually performed as in the past. Therefore, effective analysis can be realized and the analysis result can be fully utilized.

  Furthermore, the test result data stored in the test result storage means is automatically created and registered by the test result registration means, and is not a manual data registration work. There is no variation or variation every time, and uniform expression is possible and standardization is possible. For this reason, improvement in test coverage and improvement in review accuracy can be expected. Further, automatic registration of test result data by the test result registration means can save labor and time when the person in charge of the test performs manual registration work.

In addition, in the case of a configuration provided with the test automatic execution means described above,
Test result storage means for accumulating and storing test result data of repeated tests;
Test success / failure determination means for executing processing for determining success / failure of a test by the job execution means;
The result of the success / failure determination by the test success / failure determination means is expressed as the identification information of the person in charge of the test, the test execution time, the test execution time, and the build information including the test data or test scenario information used for the test or the information of these builds. A test result registration unit that executes processing for storing the test result data in the test result storage unit together with the identification information of the build to be associated,
A test result analyzing means for executing a statistical process including a trend analysis on a test time zone and / or a test person using a plurality of test result data accumulated and stored in the test result storing means;
As a result of the analysis by the test result analysis means, a job net created by a certain tester or its constituent job is executed by the test automatic execution means or at the time of automatic execution by the test automatic execution means. If the test failure probability is high or the test execution time tends to be long, the automatic test execution means specified in the job net created by the tester or its constituent jobs Whether to cancel the selection information for automatic execution by, or invalidate the identification information of the job net or its constituent job stored in the setting information storage means as the automatic execution target by the test automatic execution means E-mail that instructs manual execution that avoids the time period for automatic execution by automatic test execution means Analysis result output that executes processing to send an e-mail to notify the tester's e-mail address that the job net or its constituent job is to be recreated by informing that a problem has occurred during execution by the automatic test execution means It is good also as a structure provided with means.

  When the statistical analysis of the test results is performed and the analysis results are linked to the processing by the automatic test execution means, the analysis results can be used more fully. If the automatic test execution means regularly monitors the update of the application or the like, a temporal element occurs there, and the connection with the statistical analysis relating to the time zone becomes strong.

Furthermore, in the case of a configuration provided with the timer execution means described above,
Test result storage means for accumulating and storing test result data of repeated tests;
Test success / failure determination means for executing processing for determining success / failure of a test by the job execution means;
The result of the success / failure determination by the test success / failure determination means is expressed as the identification information of the person in charge of the test, the test execution time, the test execution time, and the build information including the test data or test scenario information used for the test or the information of these builds. A test result registration unit that executes processing for storing the test result data in the test result storage unit together with the identification information of the build to be associated,
A test result analyzing means for executing a statistical process including a trend analysis on a test time zone and / or a test person using a plurality of test result data accumulated and stored in the test result storing means;
As a result of analysis by this test result analysis means, if a job net created by a certain tester or its constituent job is executed by the timer execution means or at a specific night time, the test failure will occur. If the probability is high or the test execution time tends to be long, when the test person tries to set a timer for the job net created by the test person or its constituent job, or when a specific When setting the timer for the night time period, store the identification information of the person in charge of the test in the setting information storage means as the timer setting warning target person information that warns that a malfunction occurs, or , E-mail instructing manual execution without timer setting, or execution by timer execution means The e-mail urging informs that it will live recreate a job net or a configuration job, may be configured to include an analysis result output means for executing a process of transmitting to the mail address of the tester.

  When the statistical analysis of the test results is performed in this way and the analysis results are linked to the processing by the timer execution means, the analysis results can be utilized more fully. The timer execution means starts the job net at the set timer execution time, but usually the timer execution time is often set at a specific night time period, so statistical analysis on the time period The connection with is strong.

In the test management system described above,
Test success / failure determination means for executing processing for determining success / failure of a test by the job execution means;
A task registration unit that executes a process of storing the detailed information of the failure about the test determined to be failed by the test success / failure determination unit in association with the identification information of the job net or its constituent job in the task storage unit;
Application developers who created test data or test scenarios used to execute job nets related to failed tests or their configuration jobs, and / or test personnel who created job nets related to failed tests or their configuration jobs A test result transmission means for executing processing for sending an email indicating that the test has failed to an email address;
Failed for the test executed by the job net related to the browsing request or its constituent job, receiving the browsing request signal for analyzing the cause of failure sent from the developer terminal operated by the application developer via the network It is desirable to have a problem management means for executing processing for displaying the detailed information on the developer terminal on the screen.

  In the case of the configuration including the problem registration means, detailed information on the failure is automatically stored in the problem storage means, so it is possible to efficiently manage a huge number of failure information. It becomes. For this reason, compared to the case where failure information is manually registered in the problem management table and analyzed, and the failure reason obtained by the analysis is described as before, the workload is reduced and the analysis time of the failure reason is compared. In addition to shortening the time, making it easier and quicker to investigate the cause, it is also possible to prevent the occurrence of cases where analysis cannot be performed due to a lack of information regarding failures.

In the test management system described above,
In the job storage means,
A job net including parallel processing formed by specifying an execution order of jobs including simultaneously executing a plurality of jobs is stored,
Job execution means
When a job net including parallel processing is executed, a plurality of jobs subject to parallel processing are executed concurrently as job execution processing, and a plurality of jobs subject to parallel processing are indicated as job graphic display processing. Each figure is arranged on the screen in a state where the entire flow of the job net is branched, and as a test progress display process, it is configured to indicate that a plurality of jobs being processed in parallel are being executed concurrently. It is desirable.

  When the configuration is such that a plurality of jobs can be processed in parallel, the processing time of the entire job net can be shortened.

Furthermore, in the test management system described above,
Job execution means
As test progress display processing, the execution time of the job when the execution of the job is started, or the execution time required for the execution of the job when the execution of the job is completed, in or near the graphic indicating the job It is desirable that at least one of these be displayed.

  As described above, when the test progress display process is configured to display the execution time or execution time of the job in or near the figure, the test progress display based on whether or not the characters (including numbers) are displayed. In addition, the meaning content of the characters (including numbers) can be transmitted to the tester.

  The program of the present invention is for causing a computer to function as the test management system described above.

  The above-mentioned program or a part thereof is, for example, a magneto-optical disk (MO), a read-only memory (CD-ROM) using a compact disk (CD), a CD recordable (CD-R), a CD rewritable (CD -RW), read-only memory (DVD-ROM) using digital versatile disk (DVD), random access memory (DVD-RAM) using DVD, flexible disk (FD), magnetic tape, hard disk, It can be recorded on storage media such as read-only memory (ROM), electrically erasable and rewritable read-only memory (EEPROM), flash memory, and random access memory (RAM) for storage and distribution. For example, LAN, MAN, WAN, Internet , An intranet, a wired network such as an extranet or wireless communications network, and further is capable of transmitting using a transmission medium such as a combination thereof, also can be delivered by placing the carrier. Furthermore, the above program may be a part of another program, or may be recorded on a recording medium together with a separate program.

  As described above, according to the present invention, a plurality of jobs including a test environment construction job and a test job can be continuously executed as a job net according to a designated execution order. This has the effect of shortening the work load of the tester.

1 is an overall configuration diagram of a test management system according to an embodiment of the present invention. The figure of the flowchart which shows the flow of execution and management of the test by the test management system of the said embodiment. The figure which shows an example of the job figure display process of the said embodiment, and a test progress display process. The figure which shows an example in case a parallel process is included in the job figure display process and the test progress display process of the said embodiment.

  An embodiment of the present invention will be described below with reference to the drawings. FIG. 1 shows an overall configuration of a test management system 10 of the present embodiment. FIG. 2 is a flowchart showing the flow of test execution and management by the test management system 10. 3 and 4 show an example of job graphic display processing and test progress display processing by the test management system 10.

  In FIG. 1, a test management system 10 is a system that executes and manages an operation check test of the test target system 200. In the present embodiment, as an example, the test target system 200 is a distributed application system using the web (Web), and includes a web server 210, an application server 220, and a data base server 230 connected via a network 201 such as an in-house LAN or an intranet. It is assumed that each is configured and placed in a development environment. The application server 220 is a generic term for a web application server and a batch server, and the data base server 230 is assumed to include a data base 231 and a relational database management system (RDBMS).

  The test management system 10 includes a test overall management server 20, a repository server 50, an issue management server 60, a user information management server 70, a keystroke test machine 80, and a test person terminal 90 operated by a test person. These are connected by a network 1 such as an in-house LAN or an intranet and configured as a Web system. Further, a developer terminal 100 and a mail server 110 operated by an application developer (programmer) are connected to the network 1. The mail server 110 may be installed outside and connected via an external network such as the Internet.

  Furthermore, the test management system 10 and the test target system 200 are connected by a network 2 such as the Internet or an intranet. The network 2, the network 1 for constructing the test management system 10, and the network 201 for constructing the test target system 200 are described separately with different reference numerals in the system configuration. However, it may be a shared network. However, it is desirable that the communication path from the key-pressing test machine 80 to the test target system 200 and the communication path in the test target system 200 be in the same state as the production environment. Further, it does not matter whether any of the networks 1, 2, 201 is wired, wireless, or a mixed type of wired and wireless.

  The test supervision management server 20 is composed of one or a plurality of computers, and is connected to the processing means 20A for executing various processes necessary for test execution and management, and is connected to the processing means 20A for test execution and management. It includes a setting information storage means 41 for storing various necessary data, a job storage means 42, a test result storage means 43, a project information storage means 44, a build history storage means 45, and an analysis result storage means 46. .

  The processing unit 20A includes a setting unit 21, a job creation unit 22, a job execution unit 23, a timer execution unit 24, an automatic test execution unit 25, a test success / failure determination unit 26, a test result registration unit 27, and a task. The registration unit 28, the test result transmission unit 29, the test result analysis unit 30, and the analysis result output unit 31 are included. The processing means 20A may be constructed using an existing CI tool, or may be newly constructed. When using an existing CI tool, for example, improvement of Jenkins, plug-in to Jenkins, etc. Can be constructed.

  In response to a transmission request from the tester terminal 90, the setting unit 21 transmits display screen display data to the tester terminal 90 via the network 1, and from the tester terminal 90 via the network 1. The received setting information (setting information input by the tester using the setting screen) is received, and the received setting information is stored in the setting information storage unit 41. In this embodiment, the setting information is saved in a setting file and stored in a predetermined folder (a recording medium such as a hard disk constituting the setting information storage unit 41).

  Here, the setting information includes, for example, a timer execution time (year / month / day and time) for timer execution by the timer execution unit 24 for a job or a job net including a plurality of jobs. In the present embodiment, the timer execution time and the identification information (for example, name) of the job or job net that is the target of the timer execution are associated and stored in the setting file. A plurality of sets of job / job net identification information (for example, name) and timer execution time may be stored in the setting file, or separate setting files may be prepared for each group. Note that the identification information (for example, name) of the first job among a plurality of jobs constituting the job net can be substituted for the identification information (for example, name) of the job net.

  In addition, as setting information, identification information (for example, name) of a job (including job net configuration job) that is automatically executed when the application is updated by the test automatic execution unit 25 is saved in a setting file. Also good. Note that whether or not to be subject to automatic execution accompanying the update of an application or the like may be stored in a job file.

  The job creation means 22 receives, via the network 1, display data for a job creation screen for performing new creation / editing / modification / deletion of a job or job net in response to a transmission request from the tester terminal 90. Jobs and job nets transmitted from the test staff terminal 90 via the network 1 (jobs newly created / edited / modified by the test staff using the job creation screen) And the job net) are received, and the received job or job net is stored in the job storage means 42. In the present embodiment, newly created, edited, and modified jobs (including jobs that make up a job net) are saved in a job file and recorded in a predetermined folder (such as a hard disk that makes up the job storage unit 42). Media).

  Here, a job is a group of instructions written in a predetermined language for executing a test (including preparation). For example, a Jenkins job is written in a markup language called XML. The job file includes, for example, a test procedure (instructions composing the job itself and text data explaining it in text), a test execution content (instructions composing the job itself and text data explaining it in text), a test Test data and test scenarios (names and storage locations in the repository server 50 (URL, etc.)) used in the test, version information of test data and test scenarios used in the test (actual deployment), tester (job creation) And an automatic e-mail destination when a test fails, etc.) and an application developer (a programmer involved in a development project related to a test, and an automatic e-mail destination when a test fails. ) Etc. are registered and saved. Since a list of persons involved in the development project is stored in the project information file stored in the project information storage means 44, the tester (job creator) can select application development from the list. Select a person and save it in the job file.

  Jobs registered in the job storage means 42 include, for example, (1) “test job” that executes various tests such as unit test, screen keying test, and database (DB) data matching and outputs the results. There are a “test environment construction job” for constructing a test environment necessary for executing a test, such as DB preparation and application server setting, and (3) a “build job” for creating a binary from source code.

  A job net is formed by combining a plurality of jobs and specifying the execution order of these jobs. There are four job net creation methods as follows.

  <First creation method of job net> When creating a job net for executing jobs A and B in this order, the job A to be executed is described and specified in the calling job A in advance. There is a way.

  In this case, when job A is executed, job B is automatically executed. Therefore, if job A is to be executed alone, it is necessary to prepare job A without job B description designation. In addition, when performing parallel processing in which job B and job C (a plurality of jobs) are executed in parallel after job A is executed, job B and job C to be executed next by caller job A are performed. Both of these (multiple jobs) can be described and specified.

  Execution of the entire job net created by the first creation method can be realized simply by issuing an execution command for the first (sometimes referred to as the head) job A. Therefore, in the following description, when a description to start execution of a job net is made, a description to issue an execution command for the first job may be substituted. Further, it is possible to execute from the middle of the job net. For example, if an execution command for the second job B is issued, the subsequent jobs C, D,... Can be continuously executed.

  <Second creation method of job net> When creating a job net that executes job A and job B in this order, what happens to called job B (when job A is executed) There is a method of describing whether (Job B) is called.

  In this case, when job A is executed, job B is automatically executed. Therefore, when job A is to be executed alone, it is necessary to save or delete job B to another location. In addition, after executing job A, when performing parallel processing in which job B and job C (multiple jobs) are executed in parallel, both called job B and job C (multiple jobs) In all cases, it is only necessary to describe when (job B or job C) is called (when job A is executed).

  It should be noted that when job A is executed, job B called by it must be searched (for jobs other than job A, it is not determined whether these are jobs called by job A execution). Therefore, it takes time if there are many related jobs. When performing parallel processing, when job A is executed, it is necessary to search for a plurality of jobs B, C,... That are called by the job A. It is necessary to determine whether or not the job is a job called by executing job A, and it is not necessary to find one or several jobs called by executing job A. In short, even if you find a few, they may still exist, so in the end you will decide on all the rest.

  Execution of the entire job net created by the second creation method can be realized simply by issuing an execution command for the first (sometimes referred to as the head) job A. This is similar to the case of the first creation method described above. Therefore, in the following description, when a description to start execution of a job net is made, a description to issue an execution command for the first job may be substituted. Further, it is possible to execute from the middle of the job net. For example, if an execution command for the second job B is issued, the subsequent jobs C, D,... Can be continuously executed. This is also the same as in the case of the first creation method described above.

  <Third Method for Creating Job Net> When creating a job net that executes job A and job B in this order, an execution order management job for scheduling that calls and executes job A and job B in this order is executed. There is a method of preparing and creating a job file in which the execution order management job is stored. Therefore, in addition to the job file for job A and the job file for job B, a job file for an execution order management job for forming a job net exists.

  In this case, job A and job B are automatically and continuously executed by executing the execution sequence management job. Therefore, if job A is to be executed alone, simply execute without using the execution sequence management job. It is only necessary to execute job A. Further, when job A and job B are successively executed in this order, the process proceeds to the execution of job B on condition that job A is completed. However, after job A is executed, job B and job C (multiple jobs) are executed. When performing parallel processing that is executed in parallel, the process proceeds to execution of jobs B and C on the condition that job A ends, so that job C should not be executed on condition that job B ends. In addition, the execution of job B may be such that the end of job C is not a condition.

  In the first and second job net creation methods described above, there is only one job A to be executed first, and parallel processing is performed from the next job B, C, D,. However, in the third creation method, since the execution order management job for scheduling is executed, a plurality of jobs (the execution order management job is not included because it is for scheduling) are executed simultaneously from the beginning. Parallel processing can be realized.

  Execution of the entire job net created by the third creation method can be realized simply by issuing an execution command for an execution order management job. Therefore, in the following description, when a description to start execution of a job net is made, a description to issue an execution command to an execution order management job may be substituted. Also, in the following, when selecting, specifying, executing, starting up, etc. the entire job net, the “first (first) job” is set with the job net created by the first and second creation methods described above in mind. Although there is a description of selection / designation / execution / startup, the description is equivalent to selection / designation / execution / startup of the execution order management job, and can be replaced. When executing from the middle of a job net, it is easy to prepare another execution order management job that describes scheduling from the middle, but start after the execution starts in the execution order management job A function for selecting a job (whether the job starts from the first job or the job starts from any job in the middle) may be prepared.

  <Fourth creation method of job net> When creating a job net that executes job A and job B in this order, these jobs are connected so that job A and job B are executed in this order. There is a method of creating a new combined job and preparing a job file storing the combined job. Therefore, in addition to the job file for job A and the job file for job B, there are job files for combined jobs that are job nets. In other words, the combined job is based on the premise that there is a single job before combining, and it is possible to execute a single job A or a single job B, and a combined job that is a job net. In the job graphic display processing (see FIGS. 3 and 4) on the screen of the test person in charge terminal 90 when executing, job A and job B, which are single jobs before combination, are displayed as independent jobs. It will be in the state. However, the stand-alone job A and the stand-alone job B are not called and executed, but only appear as they appear on the display. What is being executed is a combined job.

  In this case, if any job is modified, it is necessary to recreate a combined job including the job. Further, as described above, the combined job here is not a single job but an aggregate of a plurality of jobs, so it is not an image of creating a very long job in which many statements are described. For this reason, each job can have a length according to a predetermined rule. Therefore, it is possible to avoid a situation in which jobs with different granularities are disturbed, and the first to third creations. As in the case of the method, appropriate job management (including uniformization / standardization of job granularity, rule creation of job creation, creation guidance, etc.) can be performed.

  The execution of the entire job net created by the fourth creation method can be realized simply by issuing a combined job execution command. Accordingly, in the following description, when a description to start execution of a job net is made, a description to issue an execution command for a combined job may be substituted. Also, in the following, when selecting, specifying, executing, starting up, etc. the entire job net, the “first (first) job” is set with the job net created by the first and second creation methods described above in mind. Although there is a description of selection / designation / execution / startup, the description is equivalent to selection / designation / execution / startup of a combined job and can be replaced. When executing from the middle of a job net, it is easy to prepare another combined job that combines jobs from the middle. However, after the start of execution in the combined job, the start job (the first job A function for selecting whether to start execution from a job or whether to start execution from any job in the middle may be prepared.

  In response to a transmission request from the tester terminal 90, the job execution unit 23 transmits display data for the execution job selection screen to the tester terminal 90 via the network 1 and from the tester terminal 90 to the network. Performs “execution job selection acceptance process” to receive selection information (selection information of a single job or job net to be executed selected by the tester using the execution job selection screen) transmitted via 1 Display the figure indicating the single job specified in the received selection information on the screen, or arrange the figures indicating the multiple jobs that make up the job net specified in the selection information in the order of execution. “Job graphic display processing” (FIG. 3, FIG. 3) for transmitting display data for display to the tester terminal 90 via the network 1 4), and the tester clicks an execution command (execution button provided in or near the figure displayed on the screen) transmitted from the tester terminal 90 via the network 1. (Execution command for a single job or job net) transmitted by the user and execute a single job using a job file stored in the job storage unit 42 based on the received execution command or designated by a job net In addition to performing “Job Execution Processing” to continuously execute multiple jobs in accordance with the execution order being displayed, the job being executed is displayed using graphics displayed on the screen in parallel with the execution of this single job or job net. “Test progress display processing” (see FIGS. 3 and 4) is performed.

  Here, in the execution job selection acceptance process by the job execution means 23, when accepting selection of a job net to be executed, if a job net has been created by the first and second creation methods described above, Selection of the first job among a plurality of jobs constituting the job net is accepted. If a job net has been created by the third creation method, selection of an execution order management job is accepted. If a job net has been created by the fourth creation method, selection of a combined job is performed. Accept.

  In the job graphic display processing by the job execution means 23, a descriptive text that simply indicates the job name and / or the contents of the job in or near the graphic indicating the job (the text data of this descriptive text is stored in the job file) Is displayed.) In the example of FIG. 3, the characters “backup of the old environment” (may be a job name or an explanatory text that simply shows the contents of the job) are displayed in the graphic 301 displayed on the screen 300. Yes. When the selection information received in the above-described execution job selection reception process is the designation of a single job, the graphic display process indicating the job may be omitted. However, even when the graphic display process indicating the job is omitted, the execution button for starting the job execution is displayed on the screen.

  Further, in the job graphic display process by the job execution means 23, if the selection information received in the execution job selection reception process described above is a job net designation, each job file for a plurality of jobs constituting the job net Analyzing the execution order with reference to the above, creating the display data for displaying each figure representing multiple jobs in the execution order and displaying them on the screen, and testing the created display data via the network 1 To the user terminal 90.

  The analysis of the execution order in the job graphic display processing is performed according to the first to fourth creation methods of the job net described above. For example, in the case of the first creation method, a description specifying the second job B to be executed is extracted from the first job A, and then the third job to be executed from the job B. A chained extraction process such as extracting a description specifying C is performed. In the case of the second creation method, if the second job having a description that it is called when the first job A is executed is searched for, and it is job B, then that 2 If the third job is described as being called when the third job B is executed, and if it is the job C, then when the third job C is executed, the third job C is executed. A chained search process such as searching for the fourth job with a description that is called is performed. In the case of the third creation method, since there is a description of scheduling in the execution order management job, job graphic display processing is performed according to the description, but the execution order management job itself is not a target of graphic display on the screen. (It may be displayed graphically but is not essential). In the case of the fourth creation method, the combined job includes various information such as a description of the job name and / or job contents in addition to the command statement for each constituent job before combining. At the same time, since the execution order information is included, the job graphic display processing is performed using the information. However, the combined job itself is not the target of graphic display on the screen (graphic display is possible, but it is essential. is not).

  In the example of FIG. 3, a graphic 301 indicating a “backup of old environment” job, a graphic 302 indicating a “test data import” job, and a graphic 303 indicating a “screen keystroke test” job are displayed on the screen 300. Then, a graphic 304 indicating a “DB match before and after keystroke” job and a graphic 305 indicating a “return environment” job are displayed side by side in a state in which the execution order is indicated by an arrow graphic. FIG. 4 is a display example in the case of parallel processing. A graphic 401 indicating a job of “DB data backup” is displayed on the screen 400 as the first job, branches from here, and as a first flow, A graphic 402 indicating a job of “DB data import (1)” and a graphic 403 indicating a job of “screen keystroke test (1)” are displayed side by side. As a second flow, “DB data import” A graphic 404 indicating the job “(2)” and a graphic 405 indicating the job of “screen key test (2)” are displayed side by side.

  Further, in the job graphic display process by the job execution means 23, a process of providing an execution button for issuing an execution command for starting execution of a job or job net is performed in or near the graphic indicating the job. As shown in the examples of FIGS. 3 and 4, in the case of a job net, execution buttons 306 and 406 are provided in or near the figures 301 and 401 indicating the first job. In the present embodiment, no execution button is provided in or near the graphic indicating the job in the middle of the job net. Therefore, when starting execution from a job in the middle of a job net, the job graphic display process in which the selection of a job in the middle is accepted in the execution job selection acceptance process described above and the middle job is set as a new head job. And an execution button is provided in or near the graphic indicating the new head job. In the examples of the screens 300 and 400 shown in FIGS. 3 and 4, an execution button is also provided on a graphic other than the graphics 301 and 401 indicating the first job so that execution can be instructed from an intermediate job. May be.

  In the job execution process by the job execution means 23, when executing a job net, the execution order analyzed by the above-described job graphic display process is temporarily stored in the main memory or the like, and a plurality of jobs are used by using the execution order. May be performed continuously, or the job to be executed next may be recognized (including a search) each time the jobs constituting the job net are executed. In the case of automatic execution associated with timer execution or application update, job graphic display processing is not performed. Therefore, execution order analysis for job graphic display is not performed. You can analyze the execution order of these items, temporarily store the analyzed execution order in the main memory, etc., and use it to continuously execute multiple jobs, or each job that makes up the job net , The job to be executed next may be recognized (including search) each time. In addition, in the process of executing each job, information on the start and end of execution of each job (timing information such as now start and end), execution time of each job (date and time when the execution started), The execution time (required processing time) of each job is handed over to the following test progress display processing routine.

  In the test progress display processing by the job execution means 23, the test progress display is performed in accordance with the timing of the start and end of execution based on the information on the start and end of execution of each job obtained in the course of executing each job. Display data to be transmitted is transmitted to the test staff terminal 90 via the network 1. In the test progress display, for example, only the graphic indicating the currently running job blinks, which changes the color of the graphic indicating the currently executing job to a color different from the color of the graphic indicating the currently not executing job. For example, only the figure indicating the job currently being executed is surrounded by a special frame (for example, a red frame or a thick frame), and the color of the figure indicating the job that has been executed is changed (for example, changed to a dark color). It can be realized by color coding or the like. In short, it is sufficient that the person in charge of the test can visually grasp which job is currently being executed. In the examples of FIGS. 3 and 4, the color of the figure indicating the job that is currently being executed (the figure with diagonal lines in the figure) has changed.

  In the test progress display process by the job execution unit 23, as shown in the examples of FIGS. 3 and 4, the execution time (year / month / day and time) when the execution of each job is started and the execution required for the execution of each job are performed. Display data for displaying the time in a graphic representing each job or in the vicinity thereof is transmitted to the test staff terminal 90 via the network 1 in accordance with the timing of the start or end of execution. Thereby, in addition to the above-described visual identification processing other than characters (including numbers) by coloring, frame enclosing, blinking, color coding, etc., visual identification processing based on the presence or absence of characters (including numbers) can be realized. Furthermore, the meaning content (including information about the time and time itself) can be transmitted to the person in charge of the test. In the example of FIG. 3, the job execution time “2015/05/25 18:47:42” and the job execution time “0.58 seconds” are displayed in the graphic 301.

  In the test progress display processing by the job execution means 23, when only the job execution time is displayed among the job execution time and the job execution time, the job execution time is displayed on the figure. If the job corresponding to the last figure is an active job and only the job execution time is displayed, the job execution time corresponds to the last figure displayed. If the next job of the job is an active job and both are displayed, the job corresponding to the graphic that displays the job execution time and does not display the job execution time Can be grasped as a running job.

  Further, the job execution means 23 of the present embodiment is provided with a key-pressing test machine 80 in the test management system 10 of the present embodiment. Are omitted from the test management server 20 and the key-pressing test machine 80. For example, in the case of Jenkins, by installing the Jenkins slave agent in the key-pressing test machine 80, it can be used as auxiliary processing means for the job execution means 23. It becomes a slave machine having the server 20 as a master. The auxiliary processing means provided in such a machine (slave machine) different from the test management server 20 may not be provided depending on the content of the test, the configuration of the test target system 200, and the like.

  The job execution means 23 communicates with the tester terminal 90 via the network 1 when the tester operates the tester terminal 90 and manually executes a job or job net. . Therefore, the job execution unit 23 does not perform manual execution by the operation of the tester when the timer execution unit 24 performs timer execution or when the test automatic execution unit 25 performs automatic execution accompanying update of an application or the like. Therefore, various processes for displaying the screen on the tester terminal 90 are not performed.

  The timer execution unit 24 monitors whether or not the timer execution time stored in the setting information storage unit 41 has arrived. If it is determined that the timer execution time has arrived, a job or job net related to the timer execution time that has arrived is determined. A process for starting the job execution process by the execution unit 23 is executed. In this timer execution, there is no communication with the tester terminal 90.

  More specifically, the timer execution unit 24, for example, sets one or a plurality of sets of job names (the job names of the first job constituting the job net) stored in the setting file stored in the setting information storage unit 41. And the timer execution time for starting execution of the job is read at a predetermined time interval and placed in the main memory, and whether or not the read one or more timer execution times have arrived is shorter. If there is a job (including the first job that makes up the job net) whose timer execution time has arrived after executing the monitoring process that is determined by the interval, the job name (job identification information) is displayed as the job to be executed. The job name is transferred to the job execution means 23 to start job execution processing.

  The test automatic execution means 25 is a test data (including an application) stored in the test resource storage means 52 or a test scenario (a job designated as a monitoring target among jobs and job nets stored in the job storage means 42). Or test data used for execution of a job net or a test scenario) is updated, and if it is determined that it has been updated, a process for starting job execution processing by the job execution means 23 is executed. is there. In the case of automatic execution accompanying this update, there is no communication with the test staff terminal 90.

  More specifically, for example, the test automatic execution means 25 performs the following process. First, when the application developer corrects the test data (including the application) or the test scenario, the application developer operates his / her developer terminal 100 to correct the corrected application (here, a source code program). Is stored and registered in the test resource storage means 52. At this time, the registered application (source code program) and the like are registered and version-managed by the test resource management means 51 including version information. On the other hand, when creating a job or job net, the person in charge of the test indicates selection information as to whether or not the job or job net is subject to automatic execution accompanying the update of the application or the like by the automatic test execution means 25. As in the case of setting the timer execution time (year / month / day and time), the setting means 21 can be used to set the application or the like. The identification information (for example, name) of the job or job net to be automatically executed along with the update is saved in the setting file and stored in the setting information storage unit 41. When many job files are stored in the job storage unit 42, the processing time is shorter when the job storage unit 42 stores the setting files as in the latter case. In the case of job nets, test data (including applications) and test scenarios used in jobs other than the first job may be modified. Set all the configuration jobs to be monitored.

  The test automatic execution means 25 is periodically (in this state) set by the tester in the state of setting the jobs (including job net configuration jobs) to be automatically executed along with the update of the application or the like. For example, a process of searching for such a job to be monitored is performed twice a day). In this search process, as described above, the job file stored in the job storage unit 42 may be read and searched, or the setting file stored in the setting information storage unit 41 may be read and searched. Then, from the job file of the job (including job net configuration job) to be monitored, the test data (including application) used (deployed) in the previous test and the version information of the test scenario get. For example, assume that it is version 4. Then, the automatic test execution means 25 inquires of the repository server 50 via the network 1 about the latest version of the test data (including applications) and the test scenario. In the repository server 50, when the inquiry is received by the test resource management means 51, the test resource storage means 52 is referred to, the latest version of the test data (including application) and the test scenario is acquired, and the acquired version information Is transmitted to the test management server 20 via the network 1. For example, assume that it is version 5. When the test management server 20 receives the information that the latest version is version 5 from the automatic test execution means 25, the test data (including the application) and test scenario used (deployed) in the previous test are the versions. 4 is recognized, it can be determined that the update (version up) has been performed. When it is determined that there has been an update, the job execution processing by the job execution means 23 (in the case of a job net configuration job, execution processing of the job net including the configuration job) is automatically started (ie, The version 5 is acquired from the repository server 50 and deployed, and the test data (including application) and test scenario used (deployed) in this test are version 5, so that version 5 is a job file. To record. On the other hand, if the response from the repository server 50 is version 4 in response to an inquiry about the latest version to the repository server 50, the automatic test execution means 25 may determine that the update (version up) has not been performed. Therefore, job execution processing (automatic execution) by the job execution means 23 is not performed.

  To achieve the above, the job file contains test data (including applications) used in the most recent test (including applications) and test scenario version information, as well as a repository server. Description of acquiring latest test data (including application) and test scenario from 50 and deploying to test target system 200, and whether test data (including application) and test scenario to be used have been updated There is a description in the job that the repository server 50 confirms whether or not.

  The test success / failure determination means 26 determines the success / failure of the test by the job execution means 23 (including a test executed by the timer execution means 24 and a test executed automatically by the test automatic execution means 25). The process is executed. For example, in the case of a keystroke test, the test execution result is returned from the test target system 200 to the test overall management server 20 via the keystroke test machine 80, so that the test success / failure determination unit 26 refers to the execution result. And determine the success or failure of the test. More specifically, for example, in the case of a test of a transfer process to a savings account by a customer, since the key-pressing test machine 80 is a virtual customer terminal, the key-pressing test is performed from the test target system 200 that is a bank system. Since the execution result transmitted to the machine 80 is message information for reporting a defect in the transfer process to the customer, etc., the success / failure of the test is determined based on the content of the message information. One keystroke is one test scenario, and there is one success or failure for one test scenario. A large number of test scenarios (including keystroke input data) are used in one job execution. If one of the many test scenarios used in the job fails, The job has failed.

  The test result registration unit 27 uses the test success / failure determination result by the test success / failure determination unit 26 as test result data together with build identification information (for example, a build number that is incremented by 1 when job execution is repeated) as test result storage unit. The process memorize | stored in 43 is performed. In this embodiment, test result data for one test (one build) is stored in one test result file and stored in a predetermined folder (a recording medium such as a hard disk constituting the test result storage unit 43). To do. Therefore, when the test (build) is repeated, a plurality of test result files are stored in the test result storage unit 43. In each test (build), the test result data at that time may be added to the same test result file. In that case, one test result file for the same type of test, It is stored in the test result storage means 43.

  Further, the test result registration means 27 includes the identification information (name, employee code, etc.) of the person in charge of the test (job creator), and the test required for processing for each test (one build once). Execution time (job execution time, in the case of job nets, the job execution time of each component job or in addition to the execution time of the entire job net), test execution time (date and time when the job was started) In the case of a job net, the execution time of each component job), test data (including applications) actually used (deployed), test scenarios, and build (job execution) information such as version information, A process of storing the information in the build history storage unit 45 in association with the build identification information (for example, the build number) is also executed. The identification information of the person in charge of the test (job creator) is stored in the job file. In the present embodiment, build information for a plurality of repeated tests (multiple builds) is stored in a single build history file as a build history, and a predetermined folder (such as a hard disk constituting the build history storage unit 45) In the recording medium. Therefore, every time a build (job execution) is performed, the latest build information is added to the same build history file. Note that an individual build file may be created and stored in a predetermined folder for each test once (one build at a time).

  In this embodiment, as described above, the test result registration unit 27 uses the test result storage unit 43 as a result of the success / failure determination by the test success / failure determination unit 26 as test result data together with build identification information (for example, a build number). However, the result of the success / failure determination by the test success / failure determination means 26 is stored in the build information itself stored in the build history storage means 45 (test person identification information, test execution time, test execution time, etc. ) And test result data may be stored in the test result storage means 43.

  The assignment registering means 28 associates detailed information on failure of the test determined to be failed by the test success / failure determination means 26 with identification information (for example, job name in the case of a job net), and assigns the assignment. The processing to be stored in the problem storage means 62 of the management server 60 is executed. Here, the detailed information on the failure is information for the application developer to identify the cause of the failure, such as an error code. The failed detailed information includes identification information (file name, scenario name, etc.) about the failed test scenario, version number, and storage location (directory) in the test resource storage means 52 of the repository server 50. . The task registration unit 28 performs task registration when the test fails, whereas the test result registration unit 27 described above registers both failure and success information for use in statistical analysis.

  The test result transmission means 29 is an application developer who created test data (including applications) and test scenarios used in failed jobs (configuration jobs in the case of job nets) and / or failed jobs. (In the case of a job net, a process of automatically sending an email indicating that the test has failed to the mail address of the tester who created the job, that is, automatically sending it to the mail server 110 via the network 1) It is something to execute.

  The identification information (name, employee code, etc.) of the application developer or tester (job creator) serving as the destination is specified in the job file of the failed job. In this embodiment, the address is not directly stored in the job file, but identification information (name, employee code, etc.) of the application developer or tester (job creator) is used. Then, the mail addresses of these persons are acquired from the user information storage means 72 of the user information management server 70, and an e-mail is automatically transmitted to the mail addresses. However, the mail addresses are directly stored in the job file. You may keep it.

  The test result analyzing unit 30 uses a plurality of test result data (stored in the test result file) accumulated and stored in the test result storage unit 43, and uses the test time zone and / or the person in charge of the test (job creator). ) And various statistical processes including trend analysis are performed, and the process of storing the obtained analysis result data in an analysis result file and storing it in the analysis result storage means 46 is executed.

  As for the execution timing of the automatic analysis by the test result analyzing means 30, for example, the automatic analysis is started by a manual operation from an administrator terminal (terminal having the same function and arrangement status as the test person terminal 90) (not shown). Alternatively, it may be performed daily or periodically by night batch processing or the like, or may be performed each time a test is performed and new test result data (test result file) is generated.

  As for the population to be automatically analyzed (size / range / content of the set of data to be analyzed), for example, all test result data (test result file) may be used as the population, and for each job type (The job type can be defined by putting the job type information in the job.) The mother for each tester or application developer A group may be formed (the identification information of the tester or application developer is included in the job).

  Furthermore, the contents and method of the automatic analysis by the test result analysis means 30 are, for example, (Analysis Example 1) Trend analysis regarding the test time zone and / or tester (job creator), (Analysis Example 2) Tester And an analysis of the relationship between the timer execution unit 24 and the timer execution, (analysis example 3) an analysis of the relationship between the person in charge of the test and the automatic execution associated with the update of the application or the like by the test automatic execution unit 25.

  As for (Analysis Example 1), for example, a certain test person, or a certain time zone, or a certain test person has a high probability of test failure or a long test execution time. For example, analysis of whether or not there is a tendency. The test failure / success can be acquired from a plurality of test result data (test result files) accumulated and stored in the test result storage means 43. The tester identification information (name, employee code, etc.), test The execution time (time zone) and the test execution time can be acquired from the build history stored in the build history storage unit 45 using the build identification information (for example, the build number) included in the test result data. The time zone distribution is divided into a predetermined number of unit periods according to the calendar, such as 24 hours, 1 week, 1 month, 1 year, etc. , Evening features, Friday features, Monday morning features, Wednesday night features, early month features, end of month features, summer features, end of year features, and the like. Whether or not the probability of test failure is high may be set in advance as a threshold. For example, the probability of failure is 50% or more, or the probability of failure of the entire population is three times or more. It is sufficient to determine whether or not the probability of failure in a certain time zone of a tester satisfies that threshold. Further, whether or not the test execution time is long may be set in advance, for example, a threshold value that requires more than five times the average value of the test execution time of the entire population. It may be set and it may be determined whether or not the test execution time or the like for a certain time zone of a certain tester satisfies that condition.

  For (Analysis Example 2), first, whether or not the timer is executed is recorded in the build history file of the build history storage unit 45 as the build information, or the test result file of the test result storage unit 43 Can be recorded. Even without these records, the timer is often executed at night, which is outside the working hours, so it is executed at a specific night time zone (for example, from 10:00 to 5:00 in the early morning). If so, it can be estimated that the timer is executed by the timer execution means 24.

  Then, if a job created by a tester (in the case of a job net, its constituent job) is executed by the timer execution means 24 or executed at a specific night time, the probability of test failure Analyze whether the test is high or the test execution time tends to be long.

  For (Analysis Example 3), first, whether or not the execution is automatic execution accompanying the update of the application or the like is recorded as build information in the build history file of the build history storage means 45 or stored as a test result. It can be recorded in the test result file of the means 43. Even if these records are not recorded, automatic execution by the automatic test execution means 25 is performed periodically (for example, twice a day). It can be estimated that it is automatic execution.

  A job created by a person in charge of a test (in the case of a job net, its constituent job) is executed by the automatic test execution means 25 or in the automatic execution time zone by the automatic test execution means 25. And whether the test failure probability is high or the test execution time tends to be long.

  The analysis result output means 31 stores the analysis result data (including graphs, judgment results, etc.) by the test result analysis means 30 in response to a request from the test person or system administrator, or a management person not shown in the figure. Displayed on the screen of the tester terminal (terminal having the same function and arrangement status as the tester terminal 90), printed by these terminals or their own printing device, or stored in the analysis result storage means 46 The analysis result file containing the analysis result data is automatically transmitted to the mail address of the person in charge of the test by electronic mail, that is, the process of automatically transmitting to the mail server 110 via the network 1 is executed. At this time, the e-mail address of the test person in charge of the transmission destination is stored in the user information management server 70 using the identification information (name, employee code, etc.) of the test person determined as the transmission target person in the analysis. It can be obtained from the means 72. If the email address of the tester is stored in the build history storage means 45 as the build history together with the identification information (name, employee code, etc.) of the tester, the tester to be sent in the analysis can be selected. At the determined stage, the e-mail address can also be specified, and may be so. However, in this embodiment, the e-mail address of each person may be changed. The latest e-mail address is obtained from the user information storage means 72.

  Further, the analysis result output means 31 causes the timer execution means 24 to send a job (in the case of a job net, a constituent job) created by a tester in the analysis of the above analysis example 2 by the test result analysis means 30. A job created by a tester if it runs or runs at a specific nighttime time and there is a high probability of test failure or a tendency to increase test execution time When the tester tries to set a timer for a job net (or its constituent job), or warns that a problem will occur when trying to set a timer for a specific night time zone The setting information storage means 41 may store the identification information (name, employee code, etc.) of the person in charge of the test as warning target information at the time of timer setting. Is an e-mail instructing manual execution without setting a timer, or notifying that a problem occurs in execution by the timer execution means 24 and prompting the user to re-create a job (in the case of a job net, its constituent job) A process of automatically transmitting an electronic mail to the mail address of the person in charge of the test, that is, automatically transmitting the electronic mail to the mail server 110 via the network 1 is executed.

  When the analysis result output unit 31 performs the process of storing the timer setting warning target person information in the setting information storage unit 41 as described above, the test person in charge tries to set the timer. Or when trying to set a timer for a specific night time zone (in the latter case, it will not warn you if you try to set a timer for a time zone other than a specific night time zone) The setting means 21 determines whether or not the identification information of the person in charge of the test is stored in the setting information storage means 41 as timer setting warning target person information. Then, the setting unit 21 transmits warning display data indicating that a failure has occurred to the tester terminal 90 operated by the tester via the network 1.

  Further, the analysis result output means 31 is a test automatic execution means 25 in which the job (in the case of a job net, its constituent job) created by a certain tester in the analysis of the above-described analysis example 3 by the test result analysis means 30. If the test failure probability is high or the test execution time tends to be long when the test automatic execution means 25 executes the test, the test execution time tends to be long. Cancel the selection information for automatic execution by the automatic test execution means 25 specified in the job created by the person in charge (or its constituent job in the case of a job net), or the automatic test execution means Identification of a job (in the case of a job net, its constituent job) stored in the setting information storage means 41 as a target of automatic execution by 25 Information (job name, etc.) is invalidated, or an e-mail instructing manual execution avoiding the automatic execution time zone by the automatic test execution means 25, or a problem occurs in execution by the automatic test execution means 25 Is sent to the mail address of the person in charge of the test, that is, automatically sent to the mail server 110 via the network 1. Execute the process.

  As described above, the analysis result output unit 31 uses the selection information to be automatically executed by the test automatic execution unit 25 specified in the job (in the case of a job net, its constituent job). Cancel or invalidate identification information (job name, etc.) of a job (in the case of a job net, its constituent job) stored in the setting information storage unit 41 as a target of automatic execution by the test automatic execution unit 25 When the process is performed, the automatic test execution means 25 updates the job (or the configuration job in the case of a job net) without performing a monitoring process as to whether or not the application to be used has been updated. Automatic execution is not performed.

  The setting information storage unit 41 stores various setting information set by the setting unit 21. For example, the setting information storage means 41 includes a timer execution time (year / month / day and time) for executing a job or job net by the timer execution means 24 and identification information of the job or job net to be executed by the timer. A setting file stored in association with (for example, name) is stored. The setting information storage unit 41 stores the identification information (for example, name) of jobs (including job net configuration jobs) to be automatically executed when the test automatic execution unit 25 updates the application and the like. Remember the file.

  The job storage means 42 stores jobs and job nets. In the present embodiment, a job file in which a job (including job net configuration jobs) is stored is stored.

  The test result storage means 43 accumulates and stores test result data of tests that are repeatedly performed. In the present embodiment, the test result storage unit 43 increments the success / failure determination result determined by the test success / failure determination unit 26 by 1 when the build identification information (for example, job execution is repeated). A test result file of each test stored as test result data is stored together with the build number). The test result storage means 43 stores the result of the success / failure determination by the test success / failure determination means 26 as test result data together with the build information itself (the identification information of the person in charge of the test, the test execution time, the test execution time, etc.). Also good.

  The project information storage means 44 records a list of identification information (for example, name, employee code, etc.) of persons involved in the development project (application developer, tester, etc.). The mail address is not stored in the project information storage unit 44 but is stored in the user information storage unit 72 of the user information management server 70. However, the project information storage unit 44 stores the mail address. Also good.

  The build history storage unit 45 accumulates and stores information on each build for repeated tests in association with identification information (for example, build number) of each build. The build information (job execution information) for one test at a time includes, for example, identification information (name, employee code, etc.) of a tester (job creator), test execution time required for processing ( Job execution time, in the case of job nets, the job execution time of each component job or the execution time of the entire job net), test execution time (date and time when the job was started) In this case, the execution time of each configuration job), actually used (deployed) test data (including applications), test scenarios, version information thereof, and the like are included. In the present embodiment, the build history storage unit 45 stores a build history file in which build information for a plurality of repeated tests (a plurality of builds) is collectively stored as a build history. Individual build files may be created and stored for each test once (one build at a time).

  The analysis result storage means 46 stores analysis result data (including graphs, determination results, etc.) obtained by the test result analysis means 30. In the present embodiment, the analysis result storage means 46 stores an analysis result file containing analysis result data.

  The repository server 50 includes one or a plurality of computers, and includes a test resource management unit 51 and a test resource storage unit 52.

  The test resource management means 51 executes file management processing such as registration, update, and deletion of a file containing various data as test resources while performing version management. In addition, the test resource management unit 51 executes a process of responding to an inquiry about the latest version of the test data or test scenario from the automatic test execution unit 25.

  The test resource storage means 52 stores various test data including an upper layer application to a lower layer operating system arranged in the test target system 200 and a test scenario defining the test contents together with version information. It is. In the present embodiment, the test scenario is created as an Excel file and includes input data for keystrokes. Usually, a large number of test scenarios (Excel files) are used in one test.

  The assignment management server 60 is constituted by one or a plurality of computers, and includes assignment management means 61 and assignment storage means 62.

  The assignment management means 61 is a detailed failure information about a test transmitted from the test overall management server 20 via the network 1 (failure that is determined as failure by the test success / failure determination means 26 and transmitted by the assignment registration means 28. The detailed information) is associated with the identification information (for example, job name in the case of a job net) and stored in the problem storage unit 62.

  The assignment management unit 61 also receives an issue browsing request signal for analyzing the cause of failure by the application developer transmitted from the developer terminal 100 via the network 1, and receives a job (job network) related to the browsing request. The failure detailed information about the test executed according to (2) is acquired from the assignment storage means 62, and the display data of the acquired detailed failure information is transmitted to the developer terminal 100 via the network 1. Then, a process of accepting an operation of recording a correction part and closing an issue management table by the application developer is executed.

  The issue storage means 62 stores detailed information on failure of the test (information for identifying the cause of failure by the application developer, such as an error code) as a job (in the case of a job net, a configuration job). Identification information (for example, job name), status (“New” indicating that it has just been registered, “executing” indicating that correction work is being performed, and “close” indicating completion of correction work). ), Date (changes with status changes such as “New” → “running” → “closed”, etc.), issue correction person (at the time of initial registration, it is the test management server 20 and the subsequent access It is stored in association with the application developer. In this embodiment, a large number of test scenarios are used by one job execution (thus, a large number of keystrokes are made), and there is one success or failure for one test scenario. The record of the issue database as means 62 is created for each failed test scenario, and there may be a plurality of failed test scenario records for the same job. The failed test scenario is included in the detailed failure information. Identification information (file name, scenario name, etc.), version number, and storage location (directory) in the test resource storage means 52 of the repository server 50, so that the test scenario can be specified.

  The user information management server 70 is configured by one or a plurality of computers, and includes a user information management unit 71 and a user information storage unit 72.

  The user information management means 71 accepts access from various internal terminals such as the test staff terminal 90 and the developer terminal 100, and includes registration, change, deletion, etc. of each employee's information in the user information storage means 72. In addition to executing registration processing, in response to requests from terminals and servers, search processing and transmission processing of information on each employee is executed.

  The user information storage means 72 stores various information of all employees including application developers and testers (job creators) belonging to various projects, such as e-mail address, date of birth, age, department, occupation, class, etc. Is stored in association with identification information (name, employee code, etc.) of each employee.

  The keystroke test machine 80 is configured by a computer and includes auxiliary processing means (not shown) of the job execution means 23. The key-pressing test machine 80 is a virtual customer terminal that communicates with the test target system 200 via the network 2 when, for example, testing a transfer process to a savings account by a customer. For example, in the case of Jenkins, by installing the Jenkins slave agent in this key-pressing test machine 80, it is possible to function as a slave machine of the test management server 20 and to function as auxiliary processing means of the job execution means 23. A plurality of keystroke test machines 80 may be provided.

  The test staff terminal 90 and the developer terminal 100 are configured by a computer, and include a display unit such as a liquid crystal display and a CRT display and an input unit such as a mouse and a keyboard, and appropriately include a printing apparatus. Any number of these terminals 90 and 100 may be connected to the network 1. Moreover, it is only necessary to install a general-purpose Web browser, and it is not necessary to install dedicated software necessary for the test. Furthermore, these terminals 90 and 100 may be portable devices such as a tablet terminal, a personal digital assistant (PDA), and a mobile phone (including PHS).

  The mail server 110 is configured by a computer installed inside or outside the company, and transmits and receives electronic mail. However, the mail server 110 may have a normal structure that also transmits and receives electronic mail for purposes other than testing.

  In the above, the test overall management server 20, the repository server 50, the problem management server 60, and the user information management server 70 are not limited to those realized by one computer or one CPU, respectively. It may be realized by performing distributed processing with a CPU.

  Each means 21 to 31 included in the processing means 20A constituting the test overall management server 20, the test resource management means 51 constituting the repository server 50, the assignment management means 61 constituting the assignment management server 60, the user information management server The user information management means 71 constituting the server 70 is a central processing unit (CPU) provided inside the computer main body constituting each server 20, 50, 60, 70, and one that defines the operation procedure of this CPU. Alternatively, it is realized by a plurality of programs (in the case of the job execution means 23, a job is also included).

  Further, the storage means 41 to 46 constituting the test overall management server 20, the test resource storage means 52 constituting the repository server 50, the assignment storage means 62 constituting the assignment management server 60, and the users constituting the user information management server 70 The information storage means 72 is preferably realized by, for example, a hard disk, but other recording media such as a flash memory may be adopted as long as there is no problem in storage capacity, access speed, and the like.

  In this embodiment, the operation check test of the test target system 200 is executed by the test management system 10 as follows.

  First, the application developer operates the developer terminal 100 to register and store the created test data and test scenario in the test resource storage means 52 of the repository server 50 together with version information. The test data to be registered includes an application (here, a source code program). This registration process with version management is received and executed by the test resource management means 51. Even if version 1 is registered, automatic execution by the test automatic execution means 25 is not performed.

  Next, the tester operates the tester terminal 90 to create a job (including a job net configuration job), and registers and stores the created job (job file) in the job storage unit 42. Keep it. This registration process is received and executed by the job creation means 22 of the test overall management server 20.

  In addition, when the tester wants to execute the created job (the head job in the case of a job net) by the timer execution unit 24 at night or the like, the tester operates the tester terminal 90 to execute the timer execution time. Is set in the setting information storage means 41 and stored. This setting process is received and executed by the setting means 21 of the test overall management server 20. Here, the following description will be given on the assumption that manual execution is performed without timer execution.

  In FIG. 2, the tester operates the tester terminal 90 and transmits a transmission request for an execution job selection screen showing a list of created jobs to the test management server 20 via the network 1. In the test overall management server 20, when the job execution unit 23 receives this transmission request, it transmits display data for the execution job selection screen to the test staff terminal 90 via the network 1. Then, since an execution job selection screen is displayed on the screen of the test person in charge terminal 90, the test person in charge executes the job to be executed (if a job net is executed on this execution job selection screen, The first job) is selected, and the selection information is transmitted to the test overall management server 20 via the network 1. Here, it is assumed that the first job is selected as an execution target in order to execute the job net. In the test management server 20, the job execution unit 23 receives the job selection information. Up to this point, the execution job selection reception process by the job execution unit 23 has been described.

  Then, display data for displaying, on the screen, the graphics indicating the plurality of jobs constituting the job net specified by the selection information received from the tester terminal 90 by the job execution means 23 in the order of execution. Is transmitted to the person in charge of test 90 via the network 1. Then, on the screen of the test person in charge terminal 90, a display in which a plurality of figures 301 to 305 indicating the job components of the job net are arranged as shown in the example of FIG. Since the contents of the job graphic display processing relating to the job net have already been described in detail, the description thereof is omitted here. The example of FIG. 3 is given as an image, and does not completely correspond to the description of the flowchart of FIG.

  Subsequently, the test person in charge is provided in the graphic indicating the first job of the job net displayed on the screen on the test person terminal 90 or an execution button provided in the vicinity thereof (in the example of FIG. 3, in the graphic 301). The execution button 306) is clicked and an execution command for starting execution of the first job is transmitted to the test management server 20 via the network 1 (step S1). In the test overall management server 20, the job execution unit 23 receives the execution command transmitted from the tester terminal 90, and based on the received execution command, the job file of the first job stored in the job storage unit 42 is stored. The first job is used to execute (step S1). From here, the job execution processing related to the job net by the job execution means 23 is started. Here, it is assumed that the head job is a test environment construction job.

  Then, in accordance with the description contents of the test environment construction job, the job execution means 23 uses the version information of the test data (including the application) designated by the test environment construction job from the test resource storage means 52 of the repository server 50. Acquired together (step S2).

  Further, according to the description contents of the test environment construction job, the job execution unit 23 backs up the old environment for the test target system 200 (step S3). Here, the test target system 200 is placed in the development environment, but the data (including applications and the like) of the servers 210, 220, and 230 in the development environment are saved.

  Subsequently, according to the description contents of the test environment construction job, the test data (data in the database 231, software under the application, OS, etc.) other than the application out of the test data acquired in step S 2 by the job execution unit 23, The test target system 200 is set by FTP transmission or the like via the network 2 (step S4).

  Then, the build job (job file) stored in the job storage unit 42 is automatically started and executed according to the execution order designated by the job net, and the job execution unit 23 executes the build job according to the description content of the build job. Among the test data acquired in step S2, the application is arranged in the test target system 200 by FTP transmission or the like via the network 2 (step S5). The acquisition of the application from the test resource storage unit 52 of the repository server 50 may be performed in step S5 instead of step S2.

  After that, according to the execution order specified in the job net, the test job (job file) of the keystroke test stored in the job storage means 42 is automatically started and executed, and the job is executed according to the description content of the test job. The means 23 acquires the test scenario designated by the test job from the test resource storage means 52 of the repository server 50 together with the version information (step S6). Usually, since a large number of test scenarios are used in one test job, a large number of test scenarios are acquired together with version information.

  Subsequently, according to the description contents of the test job of the key-pressing test, the job execution unit 23 transmits the test scenarios (Excel files) one by one to the key-pressing test machine 80 via the network 1 together with the execution command of the test scenario. (Step S7). At this time, the version information of the test scenario is grasped by the test management server 20, so it is not necessary to transmit it to the key-pressing test machine 80. Note that sending one by one means sending the next test scenario when the execution result is received.

  Further, when the keystroke test machine 80 receives a test scenario and its execution command transmitted from the test overall management server 20 by an auxiliary processing means (not shown) of the job execution means 23, the test scenario is executed. The key-entry input data included in the test scenario is transmitted to the test target system 200 via the network 2 (step S8).

  Subsequently, the test target system 200 receives the keying input data, executes a test target process such as a transfer process to a savings account, and sends the execution result to the key pressing test machine 80 via the network 2. Transmit (step S9).

  Then, in the key-pressing test machine 80, the auxiliary processing means (not shown) of the job execution means 23 receives the execution result transmitted from the test target system 200, and the received execution result is supervised via the network 1. Transfer to the management server 20 (step S10).

  Subsequently, when the test management server 20 receives the execution result transferred from the key-pressing test machine 80, the test success / failure determination unit 26 automatically determines the success / failure of the test by using the execution result (step). S11).

  Thereafter, after completing the execution of all the test scenarios related to one test (execution of one test job) or each time the execution of each test scenario is completed, the assignment registering unit 28 performs the test success / failure determination unit 26. The failure detailed information about the tests determined to have failed (all of the failed test scenarios) is transmitted to the problem management server 60 via the network 1. In the assignment management server 60, the assignment management unit 61 receives the detailed information on the failure transmitted from the test overall management server 20, and uses the received detailed information on the failure as identification information (for example, job name) of the test job. And stored in the task storage means 62 (step S11).

  In addition, after the execution of all test scenarios related to one test is completed, the test result registration unit 27 displays the success / failure determination result by the test success / failure determination unit 26 together with the build identification information (for example, the build number). The data is stored in the test result storage means 43 as data (step S11). In addition, the test result registration means 27 identifies the identification information (name, employee code, etc.) of the person in charge of the test (job creator), the test execution time required for processing, the test execution time, and the test data actually used (expanded). Build (job execution) information such as (including applications) and test scenarios and their version information is stored in the build history storage means 45 in association with build identification information (for example, build number) (step S11).

  Further, the test result transmission means 29 uses the test data (including the application) used in the failed job, the application developer who created the test scenario, and / or the email address of the tester who created the failed job. Then, an electronic mail indicating that the test has failed is automatically transmitted, that is, automatically transmitted to the mail server 110 via the network 1 (step S12).

  Then, the environment return job (job file) stored in the job storage means 42 is automatically started and executed according to the execution order specified by the job net, and the job execution means is executed according to the description contents of the environment return job. 23, the test target system 200 is returned to the old environment backed up in step S3 (step S13).

  Thereafter, the application developer operates the developer terminal 100 in order to analyze the cause of failure, and transmits an issue browsing request signal to the issue management server 60 via the network 1 (step S14). In the assignment management server 60, the assignment management means 61 receives the assignment browsing request signal transmitted from the developer terminal 100, and stores detailed information on the failure of the test executed by the job related to the browsing request as the assignment storage means. The data for display of the detailed information of failure acquired from 62 is transmitted to the developer terminal 100 via the network 1 (step S14).

  When the application developer analyzes and identifies the cause of the failure, the test data used in the failed job (including the application) and the test scenario are corrected, and the corrected test data and the like are tested by the repository server 50. It is stored in the resource storage means 52 (step S15). Then, there is a case where automatic execution is performed by the automatic test execution means 25 accompanying the update of the application or the like by registration of the corrected test data (including the application) and the test scenario, that is, version upgrade (automatic execution setting is set). If it has been done, it will run automatically). In addition, the application developer may modify the test data set in the test target system 200 (step S15).

  Finally, the application developer operates the developer terminal 100 to access the issue management server 60. In the assignment management server 60, the assignment is accepted by the assignment management means 61, and the application developer records the corrected part and closes the assignment management table (process for setting the status to “closed”) (step S16). .

  According to this embodiment, there are the following effects. That is, the test management system 10 can store all the jobs forming the job net by the job storage unit 42 (if the job net is created by the third creation method described above, An order management job is also stored, and if it is generated by the above-described fourth generation method, a combined job can also be stored.) Since the job execution means 23 can execute the job net. A plurality of jobs including a test environment construction job and a test job can be continuously executed according to a designated execution order.

  For this reason, it is possible to reduce manual work and suppress the probability of occurrence of so-called human error in a series of tests, so that the delivery time can be shortened and the work load on the tester can be reduced.

  In addition, since there are many manual operations in the past, there are many cases where sufficient tests cannot be performed due to time and budget reasons. However, the test management system 10 realizes continuous execution of job net configuration jobs. , Can avoid such situation, can improve the quality and reliability of the test.

  Furthermore, since the conventional test involves a lot of manual work according to the test procedure manual, personalization occurs, and the test procedure may differ depending on individual skills. Such a personalization can be suppressed by realizing the continuous execution of the constituent jobs. For this reason, the test quality and reliability can be homogenized, and further, the difference between individuals can be reduced and the system development efficiency can be made uniform. Further, since manual work is reduced, it is possible to solve various problems such as an educational cost each time a personnel change occurs or various troubles occur immediately after the change.

  In addition, since the job execution means 23 performs job graphic display processing and test progress display processing (see FIGS. 3 and 4), the tester can make the overall contents of the test including the execution procedure by this visualization. In addition, since the progress of the test can be easily grasped and understood on the screen of the tester terminal 90, the burden of checking and monitoring the tester can be reduced. The burden can be reduced.

  Further, in the test management system 10, test data and test scenarios used in the test are stored in the test resource storage means 52, and the job execution means 23 performs a test using the test data and test scenarios stored therein. As a result, it is possible to eliminate the situation where a specific tester or application developer stores and manages test data and test scenarios related to himself / herself as in the past. For this reason, another person, for example, a person in charge of test or application development, or a person who temporarily replaces the test data or test scenario stored in the test resource storage means 52 reuses the next test. Since it is possible to perform and update operations, it is possible to significantly reduce the time and labor required for test preparation, and to standardize test quality and improve reliability.

  In the past, one or more test tools (single-function test robots) that were individually activated and used manually by the tester for some processes in a series of tests By automatically calling and starting by executing one or a plurality of test jobs constituting the net, it is possible to realize automatic cooperation of any number of existing or new test tools.

  Further, by using a test tool (single-function test robot) through the test management system 10, which test tool is used in the executed test is stored in the build history storage means 45 as build information. The operating status of each test tool in the test management system 10 can be grasped. This makes it possible to measure economic effects and return on investment. As a result, it is possible to enhance and improve the test tool according to the usage situation.

  Further, since the test management system 10 includes the automatic test execution means 25, the test can be automatically executed as the application is updated. For this reason, non-execution of a test due to time constraints or forgetfulness can be avoided, and a sufficient test can be performed. This automatic execution can automate the continuous execution of job net configuration jobs, including startup, thereby further reducing the delivery time and the workload of testers. it can.

  Further, since the test management system 10 includes the timer execution unit 24 and the setting information storage unit 41, the test is performed outside the working hours such as at night or when the user goes out by setting the timer execution time. can do. Therefore, for example, the development work can be performed during the day, and the test can be performed after the end of work (evening, midnight, etc.). In addition, the timer execution can realize automation including start-up for continuous execution of job net configuration jobs. For this reason, it is possible to further shorten the delivery time and reduce the work load on the tester.

  Further, since the test management system 10 includes a test success / failure determination unit 26, a test result registration unit 27, a test result storage unit 43, a test result analysis unit 30, and an analysis result output unit 31, statistical analysis of the test results is performed. It can be done automatically and the results of the analysis can be displayed, printed, or emailed. For this reason, compared with the conventional case where a large number of test results (for example, thousands to tens of thousands of test results) are manually analyzed, the workload can be reduced, so that sufficient analysis is performed. And verification of test results with high accuracy can be realized.

  Further, the test result registration means 27 stores the test success / failure determination result together with the build identification information (build number, etc.) as test result data in the test result storage means 43, and further, the build information (the tester's information) Identification information, test execution time, test execution time, and test data or test scenario information used for the test are stored in the build history storage means 45 in association with the build identification information (build number, etc.). Therefore, compared to the conventional case where documents such as test data and test procedure manuals used at the time of each test are manually managed, each test or the information on the success / failure of each test and each test. The test data and test scenario used at the time of testing can be sufficiently linked. In addition, when and who performed the test, it is possible to sufficiently perform the linking as compared with the case where the test was manually performed as in the past. Therefore, effective analysis can be realized and the analysis result can be fully utilized.

  Furthermore, the test result data stored in the test result storage unit 43 and the build information stored in the build history storage unit 45 are automatically created and registered by the test result registration unit 27, and are manually generated data. Since it is not a registration work, there is no variation between testers or every variation in registration contents, and uniform expression is possible and can be standardized. For this reason, improvement of test coverage and review accuracy can be expected. Further, the automatic registration of test result data and build information by the test result registration means 27 can save labor and time when the person in charge of the test performs manual registration work.

  In the test management system 10, since the analysis result by the test result analysis unit 30 is linked to the processing by the timer execution unit 24 and the test automatic execution unit 25, the analysis result can be utilized more fully. .

  Further, since the test management system 10 includes the test success / failure determination unit 26, the task registration unit 28, the task storage unit 62, the test result transmission unit 29, and the task management unit 61, detailed information on the failure is stored in the task storage unit 62. By automatically storing the information, an enormous number of failure information can be managed efficiently. For this reason, compared to the case where failure information is manually registered in the problem management table and analyzed, and the failure reason obtained by the analysis is described as before, the workload is reduced and the analysis time of the failure reason is compared. Can be shortened, the cause investigation can be facilitated and accelerated, and the occurrence of a case where the analysis cannot be performed due to lack of information regarding the failure can be prevented.

  Further, the test management system 10 forms a job net designating the job execution order including the simultaneous execution of a plurality of jobs, and stores the job net including the parallel processing in the job storage unit 42. The job execution means 23 can execute job execution processing, job graphic display processing, and test progress display processing corresponding to the job net including this parallel processing (see FIG. 4). For this reason, it is possible to shorten the processing time of the entire job net.

  The job execution means 23 displays the execution time and execution time of the job in or near the graphic indicating the job as the test progress display process, so depending on whether or not the characters (including numbers) are displayed. The test progress display can be realized, and the meaning content of the characters (including numbers) can be transmitted to the tester.

  Note that the present invention is not limited to the above-described embodiment, and modifications and the like within a scope where the object of the present invention can be achieved are included in the present invention.

  For example, in the above embodiment, the test management system 10 includes a plurality of servers 20, 50, 60, and 70 connected by the network 1, but the test management system of the present invention is configured as described above. For example, the plurality of servers 20, 50, 60, and 70 may be combined into one server (for example, the test management server 20). The functions may be distributed.

  In the above-described embodiment, the test management system 10 is configured to include the key-pressing test machine 80. However, the installation of the key-pressing test machine 80 can be omitted.

  Furthermore, in the above-described embodiment, the test management system 10 includes the test staff terminal 90 and is connected to each server 20, 50, 60, 70 via the network 1 (server / client type). System), but may be a stand-alone system constructed by a single computer. However, considering that there are a plurality of parties such as a tester, application developer, system administrator, etc., a case where the person is in a different department or a remote place, etc., each terminal 90, 100, etc. via the network 1 It is preferable to make the system accessible.

  In FIG. 1, the developer terminal 100 and the mail server 110 are described outside the test management system 10, but these may be included in the test management system 10 as long as processing related to testing is performed. .

  As described above, the test management system and program of the present invention are suitable for use in tests that are continuously executed in the development of distributed application systems, for example.

DESCRIPTION OF SYMBOLS 1 Network 10 Test management system 23 Job execution means 24 Timer execution means 25 Test automatic execution means 26 Test success / failure determination means 27 Test result registration means 28 Problem registration means 29 Test result transmission means 30 Test result analysis means 31 Analysis result output means 41 Setting Information storage means 42 Job storage means 43 Test result storage means 46 Analysis result storage means 90 Tester terminal 100 Developer terminal 200 Test target system

Claims (10)

A test management system configured by a computer that executes and manages an operation check test of a test target system,
Test resource storage means for storing test data to be stored or arranged in the test target system and a test scenario specifying a test content;
A test environment construction job for storing or arranging the test data stored in the test resource storage means in the test target system, and a test job for executing a test using the test scenario stored in the test resource storage means Job storage means for storing a job net formed by combining a plurality of jobs to be included and specifying the execution order of these jobs;
A job execution process for continuously executing the plurality of jobs in accordance with the execution order specified in the job net stored in the job storage means, and a screen in which a plurality of figures showing each of the plurality of jobs are arranged in the execution order Job graphic display processing that displays a description of the job name and / or job contents in or near each graphic, and a test that indicates the job being executed using the multiple graphics A test management system comprising job execution means for executing progress display processing.   The test data used for the execution of the configuration job stored in the test resource storage unit for the configuration job of the job net specified in advance as an automatic execution target among the job nets stored in the job storage unit Or, it is characterized by monitoring whether or not the test scenario has been updated, and when it is determined that the test scenario has been updated, it comprises test automatic execution means for executing processing for starting the job execution processing by the job execution means, The test management system according to claim 1. Setting information storage means for storing a timer execution time for executing the job net stored in the job storage means by starting a timer in association with identification information of the job net;
It is monitored whether or not the timer execution time stored in the setting information storage means has arrived, and if it is determined that it has arrived, the job execution means by the job execution means for the job net related to the timer execution time that has arrived The test management system according to claim 1, further comprising: a timer execution unit that executes a process for starting a job execution process. Test result storage means for accumulating and storing test result data of repeated tests;
Test success / failure determination means for executing a process for determining success / failure of the test by the job execution means;
The success / failure determination result by the test success / failure determination means is used to identify the tester's identification information, test execution time, test execution time, and build information including the test data or test scenario information used for the test, or A test result registration unit that executes a process of storing the test result data in the test result storage unit, together with the build identification information for associating information,
Test result analysis means for performing statistical processing including trend analysis on a test time zone and / or a tester using a plurality of the test result data accumulated and stored in the test result storage means;
An analysis result output means for executing a process of displaying an analysis result by the test result analysis means on a screen or printing, or sending an analysis result file storing the analysis result to an e-mail address of a tester by e-mail; The test management system according to claim 1, further comprising a test management system. Test result storage means for accumulating and storing test result data of repeated tests;
Test success / failure determination means for executing a process for determining success / failure of the test by the job execution means;
The success / failure determination result by the test success / failure determination means is used to identify the tester's identification information, test execution time, test execution time, and build information including the test data or test scenario information used for the test, or A test result registration unit that executes a process of storing the test result data in the test result storage unit, together with the build identification information for associating information,
Test result analysis means for performing statistical processing including trend analysis on a test time zone and / or a tester using a plurality of the test result data accumulated and stored in the test result storage means;
As a result of the analysis by the test result analysis means, a job net created by a tester or a constituent job thereof is executed by the test automatic execution means or executed at a time of automatic execution by the test automatic execution means. If the test failure probability is high or the test execution time tends to be long, the test specified in the job net created by the person in charge of the test or its constituent job is used. Release the selection information for automatic execution by the automatic execution means, or the job net stored in the setting information storage means as the automatic execution target by the test automatic execution means or the identification information of the constituent job Disable the manual execution by avoiding the time period for automatic execution by the automatic test execution means. Or an e-mail notifying that a problem has occurred during execution by the automatic test execution means and prompting the re-creation of the job net or its constituent job is sent to the e-mail address of the person in charge of the test The test management system according to claim 2, further comprising: an analysis result output unit that executes processing. Test result storage means for accumulating and storing test result data of repeated tests;
Test success / failure determination means for executing a process for determining success / failure of the test by the job execution means;
The success / failure determination result by the test success / failure determination means is used to identify the tester's identification information, test execution time, test execution time, and build information including the test data or test scenario information used for the test, or A test result registration unit that executes a process of storing the test result data in the test result storage unit, together with the build identification information for associating information,
Test result analysis means for performing statistical processing including trend analysis on a test time zone and / or a tester using a plurality of the test result data accumulated and stored in the test result storage means;
As a result of analysis by this test result analysis means, if a job net created by a certain tester or its constituent job is executed by the timer execution means or at a specific night time, the test fails. When the test person tries to set the timer for the job net created by the test person or its constituent job, or the test execution time becomes longer When setting a timer for a specific night time period, the setting information storage means stores the identification information of the person in charge of the test as warning target person information at the time of timer setting that warns that a malfunction occurs. Or by e-mail instructing manual execution without setting a timer or the timer execution means. An analysis result output means for executing a process of sending an e-mail notifying that a problem occurs in execution and prompting the re-creation of the job net or its constituent job to the e-mail address of the person in charge of the test The test management system according to claim 3. Test success / failure determination means for executing a process for determining success / failure of the test by the job execution means;
A task registration unit that executes a process of storing the detailed information of the failure about the test determined to be failed by the test success / failure determination unit in association with the identification information of the job net or its constituent job in the task storage unit;
The application developer who created the test data or test scenario used to execute the job net related to the failed test or its configuration job, and / or the test staff who created the job net related to the failed test or its configuration job A test result transmission means for executing processing for sending an e-mail indicating that the test has failed to the e-mail address of the person,
A browsing request signal for analyzing the cause of failure transmitted from the developer terminal operated by the application developer via the network is received, and a test executed by a job net related to the browsing request or a configuration job thereof The test management system according to claim 1, further comprising: problem management means for executing a process of displaying detailed information of failure on the developer terminal. In the job storage means,
A job net including parallel processing formed by specifying an execution order of jobs including simultaneously executing a plurality of jobs is stored,
The job execution means includes:
When executing the job net including the parallel processing, as the job execution processing, a plurality of jobs to be processed in parallel processing are concurrently executed, and as the job graphic display processing, the processing target to be processed in parallel processing is executed. Each figure indicating a plurality of jobs is arranged on the screen in a state where the entire flow of the job net is branched, and as the test progress display process, the plurality of jobs being processed in parallel are being executed concurrently. The test management system according to claim 1, wherein the test management system is configured as follows. The job execution means includes:
As the test progress display process, the execution time of the job when the execution of the job is started and the execution time required for the execution of the job when the execution of the job is completed are shown in or near the graphic indicating the job. It is set as the structure which displays at least one of them. The test management system in any one of Claims 1-8 characterized by the above-mentioned.   A program for causing a computer to function as the test management system according to claim 1.

Images