JP5213671B2 - Test case selection method and selection system - Google Patents

Description

  The present invention relates to a test case selection method and a selection system for testing a program, and in particular, it is possible to perform an efficient test when testing a program when continuous integration is used for program development. The present invention relates to a test case selection method and a selection system.

  As one of program development techniques, a technique called continuous integration is known. This development method called continuous integration is an application of the idea of “creating something that always moves”, which is one of the features of iterative development typified by the so-called agile method, to program production. In this method, the master source management is gathered in one place, the program is always run by repeating the cycle of compiling, building, and arranging many times a day, and performing automatic tests. Is to put in.

  As described above, the method called continuous integration makes it possible to reduce the cost of integration, and to quickly notice function degradation due to mixing of defects or correction, resulting in corrections during program development. It is possible to drastically reduce the cost.

  In general, a program using a continuous integration method is developed using a source configuration management tool for managing a master source and then using a continuous integration tool. On the other hand, automatic test of a program is generally performed using a test framework corresponding to various languages and systems.

  The continuous integration method is merely a program development method, not a test method from the viewpoint of quality assurance. In other words, the continuous integration method is intended to prevent major rework and prevent obvious development regressions, so the automated testing required here is strictly covering the application. It is not necessary.

  The main purpose of continuous integration is to eliminate defects and prevent degradation. To achieve this goal to the fullest extent, it is desirable to integrate whenever there is a change in the source. In fact, a continuous integration tool is usually equipped with a mechanism that operates by detecting a source configuration change managed by the source configuration management tool.

  However, when the above-mentioned continuous integration tool is actually used, there is a problem that a great deal of time is spent on automatic testing, which can be said for compiling, building and deploying. This is because there are usually several tests for one function, and there may be several tens or more depending on the function, and when the test must be performed using an external resource (such as a database). This is basically because it takes time.

  If you proceed with the development of the program without being conscious of the fact that it takes a lot of time for the tests described above, new changes will frequently occur in the source before the previous tests are completed, and the test results When things go wrong, it's difficult to know what went wrong, causing problems with development stagnation.

In order to avoid the problems as described above, various techniques have been proposed in which the number of test cases is reduced by finding a changed part and executing only the test cases related thereto. For example, Patent Document 1 discloses a technique for reducing the number of test cases by uniquely weighting test cases when modifications are made during program development, and Non-Patent Document 1 discloses a call tree. A technique for reducing the number of test cases has been proposed by investigating the impact and conducting an impact investigation.
JP 2008-129661 A "Regression test selection method based on virtual machine intermediate language", Transactions of Information Processing Society of Japan Vol.45, No.8 (20040815) pp. 2043-2054, Information Processing Society of Japan ISSN: 03875806

  However, the techniques described in Patent Document 1 and Non-Patent Document 1 described above are effective for skipping a test case related to a part that has not been changed, but consider the importance and execution time of the test case itself. However, the test cannot be performed efficiently.

  The object of the present invention is to solve the above-mentioned problems of the prior art, and use the importance level and the execution performance time given to the test case itself, and perform a test to be performed when the program is developed using the continuous integration method. It is an object of the present invention to provide a test case selection method and a selection system that make it possible to select test cases so that they can be efficiently implemented.

  According to the present invention, the object is to provide a test case selection method for selecting a test case so that a test performed when a program is developed using a continuous integration technique can be efficiently performed. The repository in the processing apparatus holds the importance and execution time as attributes of the test case, and the information processing apparatus includes a continuous integration tool and a test case selection unit, and the test case selection unit is a developer When a source change notification is received from the source configuration management tool that received the source registration from the continuous integration tool, the change file passed from the continuous integration tool is analyzed, and the analysis result is stored in the repository. The actual passed from the continuous integration tool And severity and implementation time threshold, is achieved by selecting a test case to be executed from the importance and perform actual times as an attribute of the stored test cases to the repository.

  According to the present invention, it is possible to efficiently perform a test in continuous integration by using the importance level and execution result time assigned to the test case itself, the importance level set at the time of test execution, and the execution time threshold value. The test case can be selected as follows.

  Embodiments of a test case selection method and selection system during a program test according to the present invention will be described below in detail with reference to the drawings.

  FIG. 1 is a block diagram showing a configuration of a test case selection system for selecting a test case when testing a program according to an embodiment of the present invention.

  The system shown in FIG. 1 includes a source management machine and a test machine. Each of the source management machine and the test machine is an information processing apparatus configured to include at least a storage device such as a CPU, a main memory, and an HDD, and various functions required in the embodiment of the present invention are used as programs. Various functions in the present invention are constructed by being stored in the storage device, and loading these programs into the main memory and executing them by the CPU.

  As described above, the source configuration management tool 101 provided in the source machine has a function of performing configuration management between the source of the target application and the source of the test case. In addition, the continuous integration tool 102 in the test machine has a connection with the source configuration management tool 101 in order to perform continuous integration, and compiles a test target application, or causes a compiler (not shown) to compile, In addition, the test case selection unit 103 is activated.

  In the above description, the test case is a unit for collating the result desired by the user with the actually returned result when a method is called (executed) by giving a certain argument. A set of multiple program steps.

  The test case selection unit 103 and the repository 104 will be described later with reference to FIG. 2, but the test case selection unit 103 and the repository 104 are the main components of the present invention.

  The automatic test framework 105 in the test machine is a library that supports writing and executing automatic tests. The test target application 106 is built by the continuous integration tool 102.

  The terminal 107 is a development machine used by a developer, and checks in the created source. The terminal 108 sets parameters (execution importance and execution time threshold) at the time of test execution. The arrow extending from the terminal 108 reaches the test case selection unit 103 through the continuous integration tool 102. This means that the parameter itself is registered in the continuous integration tool 102, but when the test is executed, the test case selection is performed. It means that it is passed to the part 103. The terminal 109 displays a list of test execution results.

  The terminals 107 to 109 described above are information processing apparatuses represented by a PC, and the terminals 108 and 109 may be one terminal.

  FIG. 2 is a block diagram showing the internal structure of the test case selection unit 103. The test case selection unit 103 includes an analysis unit 201, an update unit 202, and a test unit 103.

  The analysis unit 201 has a function of analyzing the code of the target source or test case and acquiring information on the class or method being used.

  The update unit 202 has a function of writing analysis results, execution result times, and the like in the repository 104.

  The test unit 203 compares the execution importance / execution time threshold passed from the continuous integration tool 102 with the importance / execution performance time read from the repository 104 to determine a test case to be executed.

  Next, a record of information stored in the repository 104 will be described with reference to FIGS. 3 to 5. In the example shown here, Java (registered trademark) is used as a language and a test framework is provided for easy understanding. This is an example using JUnit. Of course, other languages and other tools can be replaced accordingly.

  FIG. 3 is a diagram for explaining a record in the repository in which test target application class usage information is stored. In this record, a target class name 301 and a used class name 302 are stored.

  In the target class name 301, the class name of the application to be tested is stored in FQDN (Fully Qualified Class Name). In the illustrated example, “org.example.TestedClass” is illustrated as a specific application class name.

  In the used class name 302, the class name of the class directly used in the above-described target class is stored in FQDN. A plurality of data may be stored in this item. In the illustrated example, “net.example.utils.Util.checkMyTest” and “org.example.testing.Dollers” are illustrated as specific class names.

  FIG. 4 is a diagram for explaining the records in the repository in which the test case execution information is stored. In this record, a test case class name 401, a test case method name 402, a use target class name 403, an importance 404, and an execution result time 405 are stored. The test case class name 401 and the test case method name 402 are the test case names assigned to the test cases.

  In the test case class name 401, the class name in which the test case method is stored is stored in FQDN. In the illustrated example, “org.example.ci.TestSample” is illustrated as a specific test case class name 401.

  The test case method name 402 stores the method name of the test case. In the illustrated example, “TestSample.Test” is illustrated as a specific test case method name 402.

  In the use target class name 403, the class name inside the target application used inside the test case method is stored in FQDN. A plurality of pieces of data may be stored in the item of the usage target class name 403. In the illustrated example, “org.example.TestedClass” and “net.example.utils.Util” are illustrated as specific class names 403 to be used.

  An importance value 404 stores an integer value for determining whether or not to execute the test case. Normally, this value is set to about 1 to 5. However, when it is desired to set the importance level in more detail, the range of the importance level can be made wider. It is assumed that this importance is as important as a smaller value and the test should be executed. In the illustrated example, “15” is illustrated as the specific importance 404.

  In the execution result time 405, the time when the test case is actually executed is stored in unit milliseconds. A plurality of pieces of data may be stored in the execution performance time 405 item. In the illustrated example, the first to third execution result times “10 milliseconds”, “15 milliseconds”, and “12 milliseconds” are illustrated as specific execution result times 405.

  FIG. 5 is a diagram for explaining the records in the repository in which the test case update information is stored. In this record, a test case class name 501, a test case method 502, and a method checksum 503 are stored.

  The test case class name 501 and the test case method 502 are the same as those of the record described with reference to FIG. The method checksum 503 is a checksum derived from the contents of the test case method. Here, a message digest by MD5 is used. The information of the checksum 503 indicates whether or not the test case has been changed, and is information that is compared during the test.

  FIG. 6 is a sequence chart for explaining the entire processing operation in the embodiment of the present invention. Next, this will be explained. In FIG. 6, SCM indicates the source configuration management tool 101, CI indicates the continuous integration tool 102, and test FW indicates the test framework 105. Moreover, the part enclosed by the thick black frame in FIG. 6 is an original operation | movement in embodiment of this invention.

(1) The test case selection unit 103 is executed by the continuous integration tool 102, triggered by a change notification from the source configuration management tool 101 that has received source registration from the developer to the continuous integration tool 102. At this time, the name of the changed file is passed from the continuous integration tool 102 to the selection unit 103 (steps 601 to 603).

(2) The selection unit 103 acquires the file from the source configuration management tool 101 based on the passed file name, and analyzes the acquired file. Details of this file analysis will be described later with reference to FIG. 7 (steps 604 and 605).

(3) The analysis result is stored in the test target application class usage information of the repository described with reference to FIG. 3 (step 606).

(4) After the analysis of all files is completed by the test case selection unit 103, the test execution is started. Therefore, first, the continuous integration tool 102 obtains a file from the source configuration management tool, and builds the target application as the test target application 106 (steps 611 and 612).

(5) Thereafter, the test case selection unit 103 is executed by the continuous integration tool 102. At this time, the implementation importance level and the implementation time threshold are passed from the continuous integration tool 102 to the test case selection unit 103. The execution importance level that is passed is that only test cases having a lower importance level are used as execution candidates. Further, the execution time threshold value is that the test case execution average time exceeding this is not executed (step 621).

(6) The test case selection unit 103 acquires a record in which the file name passed earlier matches the class name to be used in the test case execution information from the repository, and executes the test case from the acquired test cases Select. Details of the process for selecting a test case to be executed will be described later with reference to FIG. 8 (steps 622 and 623).

(7) The test case selection unit 103 passes the selected test case method name to the test framework 105 to instruct test execution, and the test framework 105 calls the actual test case method from the test target application 106 to perform the test. Are executed (steps 624 to 626).

(8) Information on the result of the test and the time required for the test is passed from the test framework 105 to the test case selection unit 103, and the test case selection unit 103 determines the execution time of the successful test case in the repository. The test case execution information is additionally written (steps 627 and 628).

  FIG. 7 is a flowchart for explaining the file analysis processing operation performed by the selection unit 103 at step 605 in the sequence shown in FIG. 6, which will be described next.

(1) Generally, since the source configuration management tool 101 stores a test code together with the target application file, the test case selection unit 103 performs analysis of the file acquired from the source configuration management tool 101. First, it is determined whether the acquired file is a target application or a test code (step 701).

(2) If it is determined in step 701 that the acquired analysis target file is the target application, the selection unit analyzes the source and acquires the name of the class directly used by the file. At this time, only the classes in the target application are acquired (step 702).

(3) Then, it is checked whether or not the information of the file is already stored in the test target application class use information described with reference to FIG. 3, and if it is stored, the information is once erased. Next, the information acquired this time is stored in the test target application class usage information, and the processing here is terminated (step 703).

(4) If it is determined in step 701 that the acquired analysis target file is a test case, the entire file and the checksum of each method are acquired, and then the test case update information described with reference to FIG. The test case class information is obtained from the data and compared (step 704).

(5) It is determined from the checksum comparison in step 704 whether or not the method has been changed. If the method has been changed, the information in the test case update information is deleted, new information is set, The process here ends. If there is no change in the method, the processing here is terminated without doing anything (steps 705 and 706).

  FIG. 8 is a flowchart for explaining the test case selection processing operation performed by the selection unit 103 at step 623 in the sequence shown in FIG. 6. Next, this will be described.

(1) When the test case selection unit 103 starts the test case selection process, first, the test case selection unit 103 acquires information on the target test case from the test case execution information described with reference to FIG. The execution importance thus obtained is compared with the importance of the acquired test case (steps 801 and 802).

(2) In the comparison process in step 802, if the importance of the test case is smaller than the implementation importance passed at the time of startup, the average of the execution time threshold passed at startup and the actual time of the acquired test case Are compared (step 803).

(3) In the comparison process in step 802, when the importance of the test case is equal to or greater than the execution importance passed at the time of starting, or in the comparison in step 803, the average of the actual time of the test case However, if it is equal to or larger than the execution time threshold value passed at the time of activation, the test case is determined not to be a test execution target, and the process is terminated (step 804).

(4) In the comparison process in step 803, when the average of the actual time of the test case is smaller than the execution time threshold value passed at startup, that is, the importance of the test case passed at startup If the average of the actual time of the test case is smaller than the execution time threshold value passed at the time of activation, the test case is regarded as a test execution target and the process is terminated (step 805). .

  Each process in the above-described embodiment of the present invention is configured by a program and can be executed by an information processing apparatus, and these programs are provided by being stored in a recording medium such as an FD, CDROM, or DVD. And can be provided by digital information over a network.

It is a block diagram which shows the structure of the test case selection system which selects a test case at the time of the test of the program by one Embodiment of this invention. It is a block diagram which shows the internal structure of a test case selection part. It is a figure explaining the record in the repository in which test object application class use information is stored. It is a figure explaining the record in the repository in which test case execution information is stored. It is a figure explaining the record in the repository in which test case update information is stored. It is a sequence chart explaining the whole processing operation in the embodiment of the present invention. 7 is a flowchart for explaining a file analysis processing operation performed by a selection unit in step 605 in the sequence illustrated in FIG. 6. 7 is a flowchart for explaining a test case selection processing operation performed by a selection unit in step 623 in the sequence illustrated in FIG. 6.

Explanation of symbols

DESCRIPTION OF SYMBOLS 101 Source configuration management tool 102 Continuous integration tool 103 Test case selection part 104 Repository 105 Automatic test framework 106 Test object application 107-109 Terminal 201 Analysis part 202 Update part 203 Test part

Claims (2)

In the test case selection method that selects the test case so that the test performed when developing the program using the continuous integration method can be performed efficiently,
The repository in the information processing apparatus holds the importance and execution performance time as the attribute of the test case, and includes a continuous integration tool and a test case selection means in the information processing apparatus,
The test case selection means analyzes the change file passed from the continuous integration tool when the source configuration management tool that received the source registration from the developer is notified of the source change to the continuous integration tool. Analysis results should be stored in the repository, and then executed based on the execution importance and execution time thresholds passed from the continuous integration tool, and the importance and execution actual time as attributes of the test cases stored in the repository Select a test case, and compare the execution importance passed with the importance of the acquired test case, and compare the passed execution time threshold with the average of the actual execution time of the acquired test case. , The importance of the test case is less than the implementation importance passed at startup, and the test case Average execution actual time, if smaller than passed execution time threshold during startup, the selection method of test cases and selects the test case as to the test case the test executed. In the test case selection system that selects test cases so that the tests performed when developing programs using the continuous integration method can be performed efficiently,
The repository in the information processing apparatus holds the importance and execution performance time as the attribute of the test case, and includes a continuous integration tool and a test case selection means in the information processing apparatus,
The test case selection means performs analysis of the change file passed from the continuous integration tool when the source configuration management tool that received the source registration from the developer is notified of the source change to the continuous integration tool. Section, an update section for storing the analysis result in the repository, and an implementation importance level and an execution time threshold value passed from the continuous integration tool and an attribute of the test case stored in the repository. A test unit that selects a test case to be executed based on time, the test unit compares the passed execution importance with the importance of the acquired test case, Compare the average execution time of the acquired test cases, and the importance of the test case is passed at startup. If the average execution time of the test case is smaller than the execution time threshold passed at startup, the test case is selected as the test execution target. Characteristic test case selection system.

Images