JP2017194894A - Test data generation program and test data generation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently generate a set of test data having high condition coverage.SOLUTION: Provided is a test data generation program for generating test data for a designated test scenario, said program having the following realized by a computer: a code insertion function for inserting, into a program to be tested, a code that outputs a branch result including an evaluation value for quantitatively evaluating the determination of branching; a first test data generation function for generating test data for a designated test scenario, and generating a branch result outputted by the inserted code; and a second test data generation function for generating a target branch result that matches the generated branch result in path up to the middle, and generating test data that satisfies the target branch result, using first test data that matches the target branch result in path, except for a final determination branch, and second test data having a branch result of the final determination branch in a genetic algorithm.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a test data generation program and a test data generation method, and more particularly to a technique for generating new test data by combining test data having different characteristics in a genetic algorithm of SBST (Search-Based Software Testing) Called the crossover method).

  In the test in the software development process, a test case is necessary to confirm whether or not the program is executed as designed. If test cases can be prepared automatically instead of manually, development costs can be improved.

  The test case is composed of a test scenario representing a test procedure and test data which is specific data for realizing the test scenario.

  In order to generate test cases, methods for automatically generating test scenarios and test data based on program design are being studied.

  Non-Patent Document 1 proposes to generate a test scenario and test data (integer type, character string type) with input value definition and processing flow diagram as input.

  Non-Patent Document 2 proposes to generate test data using a genetic algorithm as a search algorithm for SBST.

  Non-Patent Document 3 proposes a test data generation technique that combines Concolic Testing and SBST.

Akira Zhang and Takashi Hoshino, "Test Item Extraction and Test Data Generation Method Using Design Model", IEICE Tech. Volume 109, SS2009-7, pp.37-42, 2009 "Evolutionary Generation of Whole Test Suites", 2011 11th International Conference on Quality Software, pp.31-40, 2011 "A Hybrid Directed Test Suite Augmentation Technique", 2011 22nd International Symposium on Software Reliability Engineering, pp.150-159, 2011

  In order to confirm whether or not the program is executed as designed, both the test scenario for the test execution procedure and the test data that is the specific input value used in the test scenario Necessary. Furthermore, it is necessary to increase the branch coverage rate, which is the rate at which branches in the program are tested, and the condition coverage rate, which is the rate at which conditions in the program are tested.

  According to the technique of Non-Patent Document 1, since it is necessary to manually create an input value definition and a processing flow diagram, it is difficult to efficiently generate a test scenario and test data.

  Further, according to the technique of Non-Patent Document 2, test data having a high branch coverage can be automatically generated, but the condition coverage cannot be increased.

  Further, according to the technique of Non-Patent Document 3, it takes time to generate test data because the two methods of Concolic Testing and SBST are only executed in order.

  An object of the present invention is to efficiently generate a set of test data having a high condition coverage when a certain test scenario is given.

A test data generation program according to an aspect of the present invention includes:
A test data generation program that generates test data for a specified test scenario.
A code insertion function for inserting a code that outputs a branch result including an evaluation value for quantitatively evaluating the determination of branching into a test target program;
A first test data generation function for generating test data for a specified test scenario and generating a branch result output by the inserted code;
A target branch result whose path is partially matched with the generated branch result is generated, the first test data whose path is matched with respect to the target branch result except for the final determination branch, and the branch result of the final determination branch A second test data generating function for generating test data satisfying the target branch result using the second test data having
It is characterized by realizing.

A test data generation method according to an aspect of the present invention includes:
A test data generation method in a test data generation device for generating test data for a specified test scenario,
The test data generation device inserts a code that outputs a branch result including an evaluation value for quantitatively evaluating branch determination into a test target program; and
The test data generation device generates test data for a specified test scenario, and generates a branch result output by the inserted code;
The test data generation device generates a target branch result that matches the route to the generated branch result halfway, and first test data that matches the path of the target branch result except for a final decision branch; Using the second test data having a branch result of the final decision branch in a genetic algorithm to generate test data that satisfies the target branch result;
It is characterized by having.

  According to the present invention, it is possible to efficiently generate a set of test data with a high condition coverage when a certain test scenario is given.

Figure showing an example of the branch result Diagram showing the relationship between branch coverage and condition coverage Functional block diagram of a test case generation apparatus according to an embodiment of the present invention View transition diagram and VC scenario example A diagram showing the predetermined coverage criteria for state transition diagrams Schematic of characteristic-specific crossover method used in test case generation apparatus according to an embodiment of the present invention Schematic flowchart of a test case generation method according to an embodiment of the present invention Detailed flowchart of step S001 of FIG. Detailed flowchart of step S002 of FIG. Detailed flowchart of step S003 of FIG. Detailed flowchart of step S005 of FIG. Detailed flowchart of step S007 of FIG. Detailed flowchart of step S008 of FIG. Detailed flowchart of step S205 of FIG. Detailed flowchart of step S305 of FIG. Detailed flowchart of step S306 in FIG.

  Embodiments of the present invention will be described below with reference to the drawings.

  In an embodiment of the present invention, a test case generation program and a test case generation method for generating a test case that is a combination of a test scenario and test data of a program will be described. For convenience of explanation regarding the test case generation program, a test case generation apparatus in which each function of the program is realized on a computer will be described as an example.

  The following embodiment will be described from the generation of a test scenario. However, the embodiment of the present invention is a test data generation program and a test data generation method for generating test data for a specified test scenario when the test scenario is specified. Can also be realized.

  The test case generation device generates a test scenario using a program of an application having a user interface such as a Web application as a test target. Such an application is designed according to MVC (Model View Controller), and includes at least a view and a controller. Model is an element for expressing the data and logic handled by the application, View is an element for input / output and display, and Controller executes the necessary logic according to the input from View This is an element for requesting View and requesting the result display to View. In a program designed according to MVC, connections between views can be extracted using controller information. The test case generation device generates a VC scenario representing the connection between the view and the controller based on a predetermined coverage standard for the state transition diagram from the view transition diagram representing the connection between the views.

  The test case generation device generates test data that satisfies the generated VC scenario while moving the test target program. On the other hand, only the VC scenario information that consists of a list of View and Controller names does not tell you what operation to perform. Therefore, the test case generation device generates a test scenario by adding View operation information to the VC scenario by a crawler, a user operation, or the like.

  Next, the test case generation apparatus specifies one scenario among the generated test scenarios and generates test data as follows.

  The test case generation device generates test data using a test data generation method that generates test data by analyzing an evaluation value for a branch. A specific example of such a test data generation method is SBST. SBST is a test data generation method for generating test data by analyzing an evaluation value for quantitatively evaluating branch determination. For example, an evaluation function (0−input value) is designed for a branch with an input value> 0, and test data that satisfies the branch condition is generated by analyzing the evaluation value of the evaluation function.

  A specific example using SBST will be described below.

  In SBST, branch evaluation values are used. Therefore, first, the test case generation device inserts a code that outputs a branch result including an evaluation value for quantitatively evaluating branch determination into the test target program. . Then, the test case generation apparatus generates test data for a specified test scenario by using Concolic Testing or SBST, and generates a branch result output by the inserted code. Note that “Concolic Testing” is a test data generation method for generating test data by analyzing a constraint equation obtained by replacing input values with symbols. For example, instead of using a specific numerical value for a branch with an input value> 0, test data that satisfies the branch condition is generated by replacing the numerical value with the symbol x and analyzing the constraint expression x> 0.

  Next, the test case generation device generates a target branch result that matches the route to the generated branch result halfway, and generates test data that satisfies the target branch result. A genetic algorithm is used as a test data search algorithm that satisfies the target branch result. The genetic algorithm considers a set of test data as a set of genes, and generates new test data by manipulating the genes by selection, crossover, and mutation. The test case generation device generates new test data by a characteristic-specific crossover method that combines test data having different characteristics in a genetic algorithm. Specifically, the test case generation device performs tests with different characteristics such as first test data whose path matches the target branch result except for the final determination branch, and second test data having the branch result of the final determination branch. By using the data, test data that satisfies the target branch result is generated.

  However, if Concorlic Testing or SBST is performed on the source code of the scale that is the subject of the integration test, a calculation explosion occurs. Therefore, test data may be generated with the specified user description class as the analysis target. In other words, the calculation explosion may be avoided by not generating a test case that improves the coverage for a part not described by the user, such as a framework or a library.

  Finally, the test case generation device outputs a test case that is a combination of a test scenario and test data.

<Explanation of terms>
First, terms used in the embodiments of the present invention will be described.

  The test suite is composed of a plurality of test cases and a branch result when each test case is executed. In the embodiment of the present invention, a test suite composed of a test case as a final output and a branch result of the test case is called an output test suite, and a test suite used for executing SBST is called a material test suite. . For example, the test suite is defined as [test case 1, branch result 1], [test case 2, branch result 2],.

  A test case possesses information necessary for executing a program test, and is composed of one test scenario and one test data.

  The test scenario represents a procedure for executing a test, and includes a VC scenario and a set of steps as View operation information. For example, the test scenario is defined as [VC scenario], [Step 1, Step 2,...].

  Test data is a set of specific values for realizing a test scenario. In SBST, when a genetic algorithm is used to determine this specific value, the test data is considered a set of genes. For convenience, test data is defined as a set of genes even when Concolic Testing or the like is used. For example, the test data is defined as gene 1, gene 2,.

  A view transition diagram is a transition diagram showing the connection between views. In a program designed according to MVC, the Controller communicates the input from the View to the Model and the execution result of the Model to the View. Therefore, it is possible to generate a transition diagram of the View that is the connection between the Views using the Controller information.

  A VC scenario is a list of view names and controller names generated from a view transition diagram. A VC scenario begins with a View name and ends with a View name. For example, a VC scenario is defined as View name → Controller name → View name → ... → View name.

  A step represents a minimum unit of a test scenario that is a procedure for executing a test. In the case of a Web application, a step is composed of an id assigned to a form or button in html and a proc indicating an operation.

  A gene represents each element of test data. In the case of a Web application, a gene is composed of an id of a form in html and a value that is a specific value.

  For example, consider an example of testing a Web application that inputs an ID and PW in View 1 and presses a “login” button to which “id” is assigned. In this example, one test case includes a combination of the following test scenario and test data.

test scenario:
VC scenario: View1 → Controller1 →…
Step 1: id = "ID", proc = "input"
Step 2: id = "PW", proc = "input"
Step 3: id = "transmit", proc = "press"
...
test data:
Gene 1: id = "ID", value = "aaaa"
Gene 2: id = "PW", value = "1234"
...
That is, in this test case, (1) “aaaa” is input to the form with id = “ID” in View1, and (2) “1234” is input to the form with id = “PW”. (3) This means that the button with id = "transmit" is pressed.

  The branch result is composed of branch result elements that are the result of each branch on the code when the test case is executed. For example, the branch result is defined as a branch result element 1, a branch result element 2,.

  The branch result element includes a determination branch, a determination order, a determination result, and an evaluation value. The decision branch indicates a decision target branch, that is, which branch is decided. The determination order represents the number of the determination branch determined among all branches in the program. The determination result represents the determination result (True / False) of the determination branch. The evaluation value represents a value that quantitatively represents the determination of branching.

  The evaluation value will be described more specifically. The evaluation value represents the degree of satisfying the branch condition or the degree of not satisfying the branch condition, and can be acquired by the evaluation function set for the determination branch. For example, if there is a branch of if (x> 0), the same True is returned when x = 1 and x = 100, but when x = 100 is more than 0 when x = 1 Since it is large, it can be evaluated that the degree of satisfying the branch condition is stronger. The evaluation value has a negative value when the branch condition is satisfied, and has a positive value when the branch condition is not satisfied. The evaluation function is set for the branch. For example, if the branch condition is a == b, the evaluation function may be defined as abs (ab), and if the branch condition is a! = B, the evaluation function The function may be defined as -abs (ab), and if the branch condition is a <b or a <= b, the evaluation function may be defined as ab and the branch condition is a> b or a> = If b, the evaluation function may be defined as ba.

  The final branch result element indicates a branch result element having the largest determination order value among the branch result elements of a certain branch result. A set of branch results excluding the final branch result element is defined as an intermediate branch result. The determination branch, determination order, determination result, and evaluation value of the final branch result element are defined as a final determination branch, a final determination order, a final determination result, and a final evaluation value, respectively.

  FIG. 1 shows an example of the branch result. Taking the branch result α consisting of three branch result elements whose decision branches are branch a, branch b, and branch c as an example, the final branch result element is the branch result element of branch c [c, 3, True , -2], and the intermediate branch result is [a, 1, True, -1], [b, 2, False, 1] which are the branch result elements of branch a and branch b.

  Path match means that when branch result A and branch result B exist, all branch result elements of branch result A match all of the branch result elements of branch result B, except for the evaluation value. It is defined that the branch result A matches the route with respect to the branch result B.

  Taking the branch result α, the branch result β, and the branch result γ shown in FIG. 1 as an example, the branch result element of the branch result α matches one of the branch result elements of the branch result β except for the evaluation value. Therefore, the branch result α matches the path with respect to the branch result β. Since the branch result element of the branch result α does not match any of the branch results γ except for the evaluation value, the branch result α does not match the path with respect to the branch result γ.

  Coverage indicates the rate at which the source code of a program is tested. The unit is%. There are a plurality of coverage standards. For example, there are a branch coverage ratio and a condition coverage ratio, but unless otherwise specified, the branch coverage ratio is referred to as coverage in the embodiment of the present invention. A collection of test data with high coverage is called comprehensive test data or test data with high completeness.

  FIG. 2 shows the relationship between the branch coverage rate and the condition coverage rate. The branch coverage is also called C1 coverage and indicates the rate at which branches in the program are tested. For example, in order to set C1 coverage to 100% in a program having two branches A and B as shown in FIG. 2, when branch A = False and branch B = True, branch A = True and branch B We need to test two cases of = False.

  The condition coverage rate is also called C2 coverage, and indicates the rate at which the conditions in the program are tested. For example, in a program having two branches A and B as shown in FIG. 2, in order to set C2 coverage to 100%, when branch A = False and branch B = True, branch A = True and branch B It is necessary to test four cases: = False, branch A = True and branch B = True, and branch A = False and branch B = False.

<Configuration of test case generator>
Next, the configuration of the test case generation device according to the embodiment of the present invention will be described. FIG. 3 is a functional block diagram of the test case generation apparatus 100 according to the embodiment of the present invention.

  The test case generation device 100 may be a computer including a processor such as a CPU (Central Processing Unit), a memory device such as a RAM (Random Access Memory) and a ROM (Read Only Memory), a storage device such as a hard disk, and the like. For example, the function and processing of the test case generation device may be realized by a processor executing data or a program stored in a storage device or a memory device.

  The test case generation device 100 receives a source code and outputs a test case that is a combination of a test scenario and test data.

  The test case generation device 100 includes a VC scenario generation unit 101, a test scenario generation unit 103, and a test data generation unit 105. The embodiment of the present invention may be realized as a test data generation apparatus including the test data generation unit 105 without including the VC scenario generation unit 101 and the test scenario generation unit 103. For example, a test scenario generated from a process flow diagram or the like may be specified, and the test data generation apparatus may generate test data for the specified test scenario.

  The VC scenario generation unit 101 generates a View transition diagram using information on the Controller in the source code when an application program designed according to MVC is a test target. An example of a view transition diagram is shown on the left side of FIG. The transition diagram is composed of transitions that are lines representing the relation between states. The view in the source code is considered as one state, and the controller is also considered as one state. Since the controller communicates the input from the view to the model and the execution result of the model to the view, the transition diagram of the view that is the connection between the views can be extracted using the information of the controller. Further, the VC scenario generation unit 101 generates a VC scenario representing the connection between the view and the controller based on a predetermined coverage standard for the state transition diagram from the view transition diagram. As shown on the left side of Fig. 4, when the transition diagram of View, which is the connection of View1, View2, and View3, can be generated by Controller1, Controller2, and Controller3, as shown on the right side of Fig. 4, one view name starts. Multiple VC scenarios that end with the View name can be generated. Since the number of VC scenarios becomes enormous depending on the number of states, VC scenarios are generated based on a predetermined coverage standard for the state transition diagram.

  FIG. 5 is a diagram showing coverage criteria for the state transition diagram. For example, there are All-states, All-transitions, All-transition-pairs, All-loop-free-paths, All-one-loop-paths, All-paths, etc. as predetermined coverage criteria for the state transition diagram. All-states is a criterion that all states are included in some scenario. For example, taking the state transition diagram shown in FIG. 5 as an example, the scenario of state A → state B → state D → state E → state B → state C satisfies All-states. All-transitions is a standard that covers scenarios with a length of 1 or less (one transition). For example, taking the state transition diagram shown in FIG. 5 as an example, the scenarios of state A → state B, state B → state C, state B → state D, state D → state E, state E → state B are all- Satisfies transitions. All-transition-pairs is a standard that covers all paths (transition pairs) whose length is 2 or less. For example, taking the state transition diagram shown in FIG. 5 as an example, in addition to the scenario that satisfies All-transitions, state A → state B → state C, state A → state B → state D, state B → state D → state The scenario of E, State D → State E → State B, State E → State B → State D, State E → State B → State C satisfies All-transition-pairs. All-loop-free-paths is a standard that covers all scenarios (called prime paths) that do not appear the same state more than once and are not partially included in other scenarios. All-loop-free-paths does not guarantee All-transitions. For example, taking the state transition diagram shown in FIG. 5 as an example, the scenario of state A → state B → state C, state A → state B → state D → state E satisfies All-loop-free-paths. All-one-loop-paths is a standard that covers only one loop once in addition to All-loop-free-paths. For example, taking the state transition diagram shown in FIG. 5 as an example, the scenario of state A → state B → state C, state A → state B → state D → state E → state B → state C is All-one-loop. Satisfies -paths. All-paths is a standard that covers all scenarios. Since there are usually loops, there are an infinite number of scenarios that satisfy All-paths, which is not useful.

  The test scenario generation unit 103 adds View operation information to the generated VC scenario. For example, view operation information that satisfies the generated VC scenario is added by a crawler, a user operation, or the like. View operation information is represented by a set of steps. Specifically, the test scenario generation unit 103 acquires the View operation information that performs the same transition as the generated VC scenario and the View operation information by moving the test target program by a crawler, a user operation, or the like. The initial test data that is the test data at that time is acquired. The test scenario generation unit 103 generates an initial test case in which View operation information and initial test data are added to the VC scenario.

  The test data generation unit 105 specifies one scenario among the generated test scenarios and generates test data. Here, a configuration example is described in which test data and branch results are first generated by Concolic Testing, and new test data is generated by SBST using the test data and branch results obtained by Concolic Testing. The test data generation method for generating data and branch results is not limited to Concolic Testing. When a combination of Concolic Testing and SBST is used, the test data generation unit 105 includes a code insertion unit 111, a Concolic Testing unit 113, and an SBST unit 115.

  The code insertion unit 111 inserts into the test target program a code that outputs a branch result including an evaluation value for quantitatively evaluating branch determination. In this manner, the code insertion unit 111 inserts a code for acquiring information necessary for SBST in advance while executing the concolic testing. The code insertion unit 111 may insert a code that outputs a branch result at a branch point in the user description class to be analyzed. The code insertion unit 111 may insert a code for analyzing the branch of the program into the user description class to be analyzed. However, a part not described by the user such as a framework or a library is to be analyzed. It is not necessary to insert the code for analyzing the branch of the program.

  The Concolic Testing unit 113 generates test data by analyzing a constraint equation obtained by replacing input values with symbols, and generates a branch result output by the inserted code. As described above, the concoli testing unit 113 collects branch results, which are information necessary for SBST, while generating test data. That is, the Concolic Testing unit 113 adds the specified test scenario, the test data, and the branch result to the material test suite used for executing SBST in the process of generating test data for the specified test scenario. To do. Since there is a possibility that a branch result that matches the route may exist in the material test suite, the Concolic Testing unit 113 determines that the specified test is performed when the coverage is improved by the test data generated for the specified test scenario. The scenario, the test data, and the branch result are added to the output test suite.

  The SBST unit 115 uses a genetic algorithm to generate test data. In order to perform selection, crossover, and mutation of the genetic algorithm, the SBST unit 115 includes a selection unit 121, a crossover unit 123, and a mutation unit 125. In particular, the crossover unit 123 generates new test data by a characteristic-specific crossover method that combines test data having different characteristics. Specifically, the SBST unit 115 generates a target branch result that matches the route to the middle of the branch result generated by the Concolic Testing unit 113, and matches the route to the target branch result except for the final decision branch. Test data satisfying the target branch result is generated by crossing the test data of 1 and the second test data having the branch result of the final determination branch at the crossover unit 123.

  FIG. 6 shows a schematic diagram of the characteristic-specific crossover method. First, it is assumed that there is test data α passing through a route of branch A → branch B → branch C → branch D → result C. When test data that passes the path of branch A → branch B → branch C → branch D → result D is required as test data to increase the condition coverage rate, branch A → branch B → branch C → branch D → result Set D as the target branch result. In this case, test data satisfying the target branch result can be generated by crossing with test data β passing through the last path of the target branch result on the basis of the test data α that matches the path halfway. Note that the test data β may pass through any route up to the final determination branch of the target branch result, and the test data β has the branch result of the final determination branch of the target branch result. If so, the final determination result of the target branch result may be different. For example, in the example of FIG. 6, the test data β may be False because the branch result of branch D is not necessarily true. Even if the branch result of the branch D of the test data β is False, if excellent test data is selected as the test data β according to the evaluation value of the branch D, test data that satisfies the target branch result is generated. Is likely to be.

  In this way, the SBST unit 115 generates a target branch result that partially matches the output test suite branch result, and the test data that matches the target branch result does not exist in the output test suite. Test data that satisfies the target branch result is generated using the test suite and added to the output test suite. In addition, the crossover unit 123 of the SBST unit 115 can perform a test with only the first test data whose path matches the target branch result except the final determination branch and the second test data having the branch result of the final determination branch. Considering that data cannot be generated, third test data obtained by removing the first test data and the second test data from the material test suite may be used. The SBST unit 115 sorts the branch results of the material test suite by the evaluation value of the branch result element having the final determination branch of the target branch result, and sequentially selects the first test case and the second test case from the sorted material test suite. And the third test case are selected, and the combination of the first test case and the third test case and the combination of the second test case and the third test case are crossed at the crossover part 123. By doing so, test data satisfying the target branch result may be generated.

<Procedure for test case generation method>
Next, a test case generation method according to an embodiment of the present invention will be described. FIG. 7 is a flowchart of a test case generation method according to an embodiment of the present invention.

  First, in step S001, the VC scenario generation unit 101 receives a source code of a test target program as an input, and generates a view transition diagram representing a connection between views using information of a controller in the source code.

  Step S001 will be further described with reference to FIG.

  In step S101, the VC scenario generation unit 101 receives the source code of the test target program as an input, and extracts a view in the source code.

  In step S102, the VC scenario generation unit 101 receives the source code of the test target program as an input, and detects the connection between the views using the information of the controller. Specifically, the VC scenario generation unit 101 selects one of the extracted views as the target view, and acquires the controller that is the transition destination of the view specified by the target view name. Then, the transition destination view of the acquired transition destination controller is set as the transition destination view of the target view. By repeating the above process for each view extracted in step S101, all the transition destination view names of each target view and the controller names existing between them are output.

  In step S103, the VC scenario generation unit 101 receives, as inputs, all transition destination view names of each target view and a controller name existing therebetween, and creates a view transition diagram. In the View transition diagram, not only the View name but also the Controller name is recorded. When the View transition diagram is generated, the process returns to Step S002 of FIG.

  Next, in step S002 of FIG. 7, the VC scenario generation unit 101 receives a view transition diagram as an input, and generates a VC scenario for the view transition diagram based on a predetermined coverage criterion for the state transition diagram. .

  Step S002 will be further described with reference to FIG.

  In step S104, the VC scenario generation unit 101 treats the view transition diagram as a state transition diagram, and creates a plurality of VC scenarios based on a predetermined coverage criterion for the state transition diagram such as All-states. When the VC scenario is generated, the process returns to step S003 in FIG.

  Next, in step S003 of FIG. 7, the test scenario generation unit 103 receives a VC scenario as an input, and generates a test case that satisfies the VC scenario for each VC scenario using a technique such as a crawler. Then, the test scenario and test data for realizing the VC scenario are output as one test case (initial test case).

  Step S003 will be further described with reference to FIG.

  In step S105, the test scenario generation unit 103 receives the test target program and the VC scenario as input, actually operates the test target program by a crawler or the like, and accesses the html file represented by the top view of the VC scenario. Move to. The operation of the program to be tested can be replaced by a user operation in addition to the crawler.

  In step S106, the test scenario generation unit 103 creates an empty test case TC0. The empty test case TC0 does not contain specific values for the VC scenario, which is a test scenario, and a set of steps, and does not contain test data.

  In step S107, the test scenario generation unit 103 receives the test target program and the test case TC0 as inputs, actually operates the test target program by a crawler or the like, and the form input when the test target program is operated. Add the id and its value to the test data of test case TC0, and add the id of the entered form and the id of the pressed button to the test scenario of test case TC0. The operation of the program to be tested can be replaced by a user operation in addition to the crawler. Also, obtain the transition of View or Controller being operated and add it to the test scenario of test case TC0. When transitioning to a new view, the process proceeds to the next step S108.

  In step S108, the test scenario generation unit 103 receives the test case TC0 and the VC scenario as inputs, and determines whether the transition of the view and controller of the VC scenario and the test case TC0 have undergone the same transition halfway or the end. To do. When the same transition is not performed, the process proceeds to step S109, and when the same transition is performed, the process proceeds to step S110.

  In step S109, the test scenario generation unit 103 receives the test case TC0 as an input, and if the transition of the VC scenario and the transition of the test case TC0 are different, the test case TC0 does not satisfy the VC scenario, and thus the test case TC0 is discarded. The process proceeds to step S105.

  In step S110, the test scenario generation unit 103 receives the test case TC0 and the VC scenario as inputs, and determines whether the transition of the view and controller of the VC scenario and the test case TC0 have made the same transition to the end. If the same transition has not been made to the end, that is, the same transition has been made to the middle, but the same transition has not been made to the end, it is necessary to further operate the program to be tested, so step S107 Migrate to When the same transition is made, the process proceeds to step S111.

  In step S111, the test scenario generation unit 103 receives the test case TC0 as an input, and outputs the test case TC0 as an initial test case. When the initial test case is output, the process returns to step S005 in FIG.

  Next, in step S005 of FIG. 7, the code insertion unit 111 of the test data generation unit 105 receives the byte code of the test target program as an input, and performs test data on the byte code on the method in the user description class Insert code to obtain information necessary for generation. A source code may be received as an input instead of a byte code. In Concolic Testing, since it is necessary to analyze the constraint expression obtained by replacing the input value with a symbol, a code for replacing the input value with the symbol is inserted. In addition, the code insertion unit 111 inserts a code for acquiring information necessary for SBST in advance while executing the Concolic Testing. In SBST, since it is necessary to analyze the evaluation value of the branch, a code for outputting the value of the evaluation function for the branch is inserted. The code insertion unit 111 outputs a bytecode inserted into the user description class.

  The code insertion unit 111 may receive a user description class name to be analyzed as an input, or may receive a user description class name that is not an analysis target as an input. The user description class name to be analyzed and the user description class name not to be analyzed may be specified by a namespace. In this case, the code insertion unit 111 inserts a code for acquiring information necessary for generating test data into the bytecode on the method in the user description class to be analyzed.

  Step S005 will be further described with reference to FIG.

  In step S112, the code insertion unit 111 may receive as input the user description class name to be analyzed input by the user. Further, the code insertion unit 111 may receive, as an input, a user description class name that is input by the user and is not to be analyzed. When the user description class name is designated by the name space, the code insertion unit 111 obtains the name of the class having the designated name space by examining the byte code. However, if there is a user class name that is not an analysis target, the code insertion unit 111 excludes a user description class having a user class name that is not an analysis target.

  In step S113, the code insertion unit 111 examines each line of the bytecode on the method in the user description class to be analyzed, and inserts a code for acquiring information necessary for test data generation. The code insertion unit 111 is a code for substituting the input value with a symbol in the line where the operation is performed on the input value in the user description class to be analyzed and the branch line in order to acquire the information necessary for the Concolic Testing. Insert. In addition, the code insertion unit 111 inserts a code that outputs a branch result at a branch point in the user description class to be analyzed in order to acquire information necessary for SBST. Do nothing for lines that do not require code to be inserted on methods in user-written classes. When the code is inserted, the process returns to step S006 in FIG.

  Next, one test case is designated from the initial test cases, and test data is generated as follows.

  In step S006 of FIG. 7, the concoli testing unit 113 of the test data generation unit 105 creates an empty output test suite. The empty output test suite does not contain specific values for the test scenario VC scenario and the set of steps, and does not contain test data or branch results.

  Next, in step S007 of FIG. 7, the Concolic Testing unit 113 receives the inserted bytecode, the initial test case, and the output test suite as inputs, and generates test data according to the Concolic Testing based on the initial test case. And add the resulting test case to the output test suite.

  Step S007 will be further described with reference to FIG.

  In step S114, the concolic testing unit 113 receives the initial test case as an input, duplicates the initial test case, and defines it as a test case TCN.

  In step S115, the concolic testing unit 113 receives the test case TCN and the bytecode into which the code has been inserted as input, and generates new test data according to the concolic testing. In Concolic Testing, it is necessary to actually execute the test case TCN. However, since the test case TCN includes the test scenario including the View operation information and specific test data, create a script and execute the script. By using Selenium etc., which is the tool to be executed, it is possible to automatically execute Concolic Testing. Further, since a code for outputting a branch result, which is information necessary for SBST, has been inserted by the code insertion unit 111, the Concolic Testing unit 113 can acquire the branch result together with the test data.

  In step S116, the concoli testing unit 113 receives the generated test data and the initial test case as inputs, replaces the generated test data with the test data in the initial test case, and outputs a new test case TCN.

  In step S117, the concolic testing unit 113 receives the test case TCN, the branch result, and the material test suite as inputs, and adds the test case TCN and the branch result to the material test suite.

  In step S118, the concolic testing unit 113 receives the test case TCN and the output test suite as inputs, and determines whether the coverage is improved by adding the test case TCN to the output test suite. If the coverage is improved, the process proceeds to step S119. If the coverage is not improved, the process proceeds to step S120.

  In step S119, the concolic testing unit 113 receives the test case TCN, the branch result, and the output test case as inputs, and adds the test case TCN and the branch result to the output test suite.

  In step S120, if the analysis has not been completed, the concolic testing unit 113 proceeds to step S115 and repeats generation of test data. When the analysis is completed, the process returns to step S008 in FIG.

  Next, in step S008 of FIG. 7, the SBST unit 115 of the test data generation unit 105 inputs the byte code, the output test suite, and the material test suite into which the code for outputting the branch result, which is information necessary for SBST, is inserted. Receive as. The SBST unit 115 executes SBST for analyzing the program by actually moving the test target program with each gene of the test data as input for the byte code into which the code is inserted, and the test data generated by the Concolic Testing unit 113 passes. Test data that passes through a branch that cannot be generated is generated. The SBST unit 115 strengthens the output test suite by adding test data for improving the coverage to the output test suite. Note that the test data used in the SBST unit 115 is selected from the material test suite.

  Step S008 will be further described with reference to FIG.

  In step S201, the SBST unit 115 sets the number of branch result elements of the branch result BR0 to n for each branch result BR0 of the output test suite.

  In step S202, the SBST unit 115 receives the branch result BR0 as an input, and creates a new branch result BR in which all branch result elements having a determination order value greater than n are deleted from the branch result BR0.

  In step S203, the SBST unit 115 receives the branch result BR as an input, creates a new branch result obtained by inverting the final determination result with respect to the branch result BR, and sets it as the target branch result. For example, when the final determination result of the branch result BR is True, the final determination result is False, and when the final determination result of the branch result BR is False, the final determination result is True.

  In step S <b> 204, the SBST unit 115 determines whether a branch result whose path matches the target branch result exists in the output test suite. As described with reference to FIG. 1, the evaluation value may be different in this case in order to determine the route match of the branch result. When the branch result that matches the path with the target branch result does not exist in the output test suite, the process proceeds to step S205, and when it exists, the process proceeds to step S206.

  In step S205, the SBST unit 115 receives the material test suite and the target branch result as inputs, and generates test data that satisfies the target branch result using SBST.

  Step S205 will be further described with reference to FIG.

  In step S301, G is the maximum number of generations when executing SBST using a genetic algorithm, and 1 is the current generation g.

  In step S302, the SBST unit 115 receives the material test suite and the target branch result as inputs, and sorts each branch result of the material test suite by the evaluation value of the branch result element having the final determination branch of the target branch result. For example, if the final decision branch of the target branch result is True, the smaller the evaluation value, the better the decision. If the final decision branch of the target branch result is False, the higher the evaluation value, the better It is judged that. However, the branch result that does not have the final decision branch of the target branch result is judged to be the least excellent. The SBST unit 115 sorts the branch results determined to be excellent and the corresponding test cases in order from the top.

  In step S303, the SBST unit 115 generates an empty test suite TS. The empty test suite TS does not contain specific values for test cases and does not contain branch results.

  In step S304, the selection unit 121 of the SBST unit 115 receives the sorted material test suite and the test suite TS as inputs, and selects a plurality of test cases in order from the sorted material test suite. The number of test cases to be selected is a predetermined constant. The selection unit 121 generates a test case obtained by manipulating the gene of the selected test case. For example, the selection unit 121 may randomly add 1 or subtract 1 for one gene of a selected test case if it is an int type. For example, the selection unit 121 may randomly add one character or delete one character as long as it is a String type for one gene of the selected test case. The selection unit 121 performs this operation for all genes, thereby generating new test cases for each test case by the number of genes, and adding new test cases to the test suite TS.

  In step S305, the crossover unit 123 of the SBST unit 115 receives the sorted material test suite and the test suite TS as inputs, classifies the material test suite into three test suites according to the branch result characteristics with respect to the target branch result, and A new test case in which genes of test cases with different characteristics are mixed is generated by the crossover method according to characteristics.

  Step S305 will be further described with reference to FIG.

  In step S401, the crossover unit 123 creates empty test suites TS1, TS2, and TS3, respectively. The empty test suites TS1, TS2, and TS3 do not contain specific values for test cases and do not contain branch results.

  In step S402, the crossover unit 123 determines whether or not the intermediate branch result of the target branch result matches the path for each branch result in the sorted material test suite. If the route matches, the process proceeds to step S403. If the route does not match, the process proceeds to step S404.

  In step S403, the crossover unit 123 receives the material test suite and the test suite TS1 as inputs, and when the branch result in the material test suite matches the intermediate branch result of the target branch result, the branch result in the material test suite is obtained. And the corresponding test case is added to the test suite TS1.

  In step S404, the crossover unit 123 determines whether or not a final determination branch of the target branch result exists for each branch result in the sorted material test suite. If there is a final determination branch, the process proceeds to step S405, and if the path does not match, the process proceeds to step S406.

  In step S405, the crossover unit 123 receives the material test suite and the test suite TS2 as inputs, and when the branch result in the material test suite has the final determination branch of the target branch result, the branch result in the material test suite and the correspondence Add a test case to test suite TS2.

  In step S406, the crossover unit 123 receives the material test suite and the test suite TS3 as inputs, and adds the remaining branch results and corresponding test cases to the test suite TS3.

  In step S407, the crossover unit 123 receives the test suites TS1, TS2, and TS3 as inputs, and for each branch result of the test suites TS1, TS2, and TS3, the evaluation value of the branch result element having the final determination branch of the target branch result Sort by. For example, if the final decision branch of the target branch result is True, the smaller the evaluation value, the better the decision. If the final decision branch of the target branch result is False, the higher the evaluation value, the better It is judged that. However, the branch result that does not have the final decision branch of the target branch result is judged to be the least excellent. The crossover unit 123 sorts the branch results determined to be excellent and the corresponding test cases in order from the top.

  In step S408, the crossover unit 123 receives the sorted test suites TS1, TS2, and TS3 as inputs, generates a new test suite TS1 ′ that combines the test suite TS1 and the test suite TS3, and also tests the test suite TS2 and the test suite TS2. A new test suite TS2 ′ is generated by combining the suite TS3. Here, combining test suite TS1 and test suite TS3 means adding test suite TS3 directly under test suite TS1, and combining test suite TS2 and test suite TS3 is test suite TS2. This means that the test suite TS3 is added as is.

  In step S409, the number n of gene manipulations by the characteristic crossover method is set to zero.

  In step S410, the crossover unit 123 receives the test suites TS1 ′ and TS2 ′ as inputs, selects test cases in order from the top of the test suites TS1 ′ and TS2 ′, and each test data of the test case selected from TS1 ′. For a gene, search for the gene having the same gene id in the test data of the test case selected from TS2 ', and substitute the value of that gene. For example, if one gene of a test case selected from the test suite TS1 ′ is [id = 1, value = 0], a gene having id = 1 is searched from the test case of the test suite TS2 ′, and [id = When a gene of 1, value = 10] is found, a gene of [id = 1, value = 10] is created. As a result, a test case genetically manipulated by the characteristic crossover method is generated.

  In step S411, the crossover unit 123 receives the genetically manipulated test case and the test suite TS as inputs, and adds the genetically manipulated test case to the test suite TS.

  In step S412, 1 is added to n, and the above process is repeated until n reaches a predetermined value in step 413. If n reaches a value determined in advance in step S413, the process returns to step S306 in FIG.

  Next, in step S306 of FIG. 14, the mutation unit 125 of the SBST unit 115 receives the sorted material test suite and the test suite TS as inputs, and performs a plurality of tests in order from the sorted material test suite. Select a case. The number of test cases to be selected is a predetermined constant. The mutation unit 125 generates a new test case in which the gene of the selected test case is mutated. For example, the mutation unit 125 generates a new test case in which a new value is given with a certain probability for each gene of the selected test case. The mutation unit 125 generates a plurality of new test cases by mutation and adds new test cases to the test suite TS.

  Step S306 will be further described with reference to FIG.

  In step S601, the mutation unit 125 receives the sorted material test suite as an input, and extracts a plurality of test cases in order from the sorted material test suite.

  In step S602, the mutation unit 125 receives the extracted test case as an input, and changes the value value of the gene to a random value with a certain probability for each gene of each test data of the test case. For example, the mutation unit 125 changes the value = "aaa" of the gene to value = "hoge" with a certain probability. The value to be changed may be any value. In addition, the gene is not changed with the remaining probability. The probability is a predetermined constant.

  In step S603, the mutation unit 125 receives the genetically manipulated test case and the test suite TS as inputs, and adds the genetically manipulated test case to the test suite TS. When the genetically manipulated test case is added to the test suite TS, the process returns to step S307 in FIG.

  Next, an output test suite is generated for each test case of the test suite TS as follows.

  In step S307 of FIG. 14, the SBST unit 115 receives the test case in the test suite TS as an input, and causes the test target program to execute the test case in the test suite TS. Since the code insertion unit 111 has inserted the code for outputting the branch result, the corresponding branch result can be acquired for the test case.

  In step S309, the SBST unit 115 receives the test case in the test suite TS, the corresponding branch result, and the material test suite as inputs, and adds the test case and the corresponding branch result to the material test suite.

  In step S310, the SBST unit 115 receives the test case in the test suite TS, the corresponding branch result, and the output test suite as input, and the path match with the branch result obtained by the SBST unit 115 executing the test case. It is determined whether or not a branch result to be present exists in the output test suite. If it does not exist, the process proceeds to step S311. If it exists, the process returns to step S307, the process for the next test case is performed, and if the process for all test cases is completed, the process proceeds to step S313.

  In step S311, the SBST unit 115 receives the test case in the test suite TS, the corresponding branch result, and the output test suite as inputs, and adds the test case and the corresponding branch result to the output test suite.

  In step S312, the SBST unit 115 determines whether or not the branch result obtained by the SBST unit 115 executing the test case matches the path of the target branch result. If the routes do not match, the process proceeds to step S313. If the routes match, the process returns to step S206 in FIG.

  In step S313, g = g + 1 is set, and in step S314, the process returns to step S302 until g reaches the maximum generation number G, and the above processing is repeated. When g reaches the maximum number of generations, the process returns to step S206 in FIG.

  Next, n = n-1 is set in step S206 in FIG. 13, and the process returns to step S202 until n becomes 0 in step S207, and the above processing is repeated. When n becomes 0, the process returns to step S201, the process for the next branch result is performed, and when the process for all the branch results is completed, the process returns to step S009 of FIG.

  Next, in step S009 of FIG. 7, the test data generation unit 105 outputs all the test cases in the output test suite to a file or the like.

With the above procedure, the test case generation device 100 can output a test case that is a combination of a test scenario and test data.
<Effect of the embodiment of the present invention>
According to the embodiment of the present invention, it is possible to efficiently generate a set of test data having a high condition coverage when a certain test scenario is given.

  In particular, by combining test data having different characteristics, it is possible to generate test data having a high condition coverage rate considering the combination of branches. As a result, the ability to detect bugs resulting from the combination of branches is improved.

  In addition, test data with higher coverage can be generated by combining two methods having different properties of Concolic Testing and SBST. However, if the two methods are executed in sequence, it takes a considerable amount of time to generate the test data. Therefore, by acquiring the necessary information in advance while executing Concolic Testing, the efficiency of the test data in SBST can be obtained. Search is possible. As a result, the ability to detect bugs due to structural defects in the program is improved.

  In addition, by generating test scenarios from View transition diagrams, it is possible to automatically generate test cases with high completeness for screen transitions in Web applications. Since the view in the Web application has a granularity very close to the screen, by extracting the test scenario using the predetermined coverage criteria for the state transition diagram for the view transition diagram, it is close to the granularity of the screen transition, Test scenarios with high completeness for screen transitions can be automatically generated.

  Furthermore, when generating test data using Concolic Testing or SBST, it is possible to avoid computational explosions by not analyzing branches at locations not specified by the user, such as frameworks and libraries. Test level test cases can also be generated. Instead of generating test cases that improve coverage for places that the user does not describe, such as frameworks and libraries, instead of using computational resources to generate test cases with high coverage for the user-written classes to be analyzed Can be assigned.

  For convenience of explanation, the test case generation device according to the embodiment of the present invention is described using a functional block diagram. However, the test case generation device according to the embodiment of the present invention may be hardware, software, or their It may be realized in combination. For example, the embodiment of the present invention is a program for causing a computer to realize each function of the test case generation apparatus according to the embodiment of the present invention, and each procedure of the test case generation method according to the embodiment of the present invention for the computer. It may be realized by a program that executes In addition, the functional units may be used in combination as necessary. Further, the test case generation method according to the embodiment of the present invention may be performed in an order different from the order shown in the embodiment.

  As described above, the method for efficiently generating a set of test data having a high condition coverage when a certain test scenario is given has been described, but the present invention is not limited to the above embodiment, Various modifications and applications are possible within the scope of the claims.

DESCRIPTION OF SYMBOLS 100 Test case production | generation apparatus 101 VC scenario production | generation part 103 Test scenario production | generation part 105 Test data production | generation part 111 Code insertion part 113 Concolic Testing part 115 SBST part 121 Selection part 123 Crossover part 125 Mutation part

Claims (5)

A test data generation program that generates test data for a specified test scenario.
A code insertion function for inserting a code that outputs a branch result including an evaluation value for quantitatively evaluating the determination of branching into a test target program;
A first test data generation function for generating test data for a specified test scenario and generating a branch result output by the inserted code;
A target branch result whose path is partially matched with the generated branch result is generated, the first test data whose path is matched with respect to the target branch result except for the final determination branch, and the branch result of the final determination branch A second test data generating function for generating test data satisfying the target branch result using the second test data having
Test data generation program for realizing In the process of generating test data for a specified test scenario, the first test data generation function adds the specified test scenario, the test data, and the branch result to the material test suite, and adds the specified test scenario to the specified test scenario. If the test data generated by the test results in improved coverage, add the specified test scenario, the test data, and the branch result to the output test suite.
The second test data generation function generates a target branch result that partially matches the branch result of the output test suite, and the test data that matches the target branch result does not exist in the output test suite The test data generation program according to claim 1, wherein test data that satisfies the target branch result is generated using the material test suite and added to the output test suite.   The second test data generation function sorts the branch results of the material test suite by the evaluation values of the branch result elements having the final determination branch of the target branch result, and sequentially from the sorted material test suite, Selecting a first test case whose path matches with the target branch result except a final decision branch, a second test case having a branch result of the final decision branch, and a remaining third test case; Test data satisfying the target branch result is generated by crossing the combination of the first test case and the third test case with the combination of the second test case and the third test case. The test data generation program according to claim 2. If the program to be tested consists of at least View and Controller,
On the computer,
A VC scenario generation function that generates a transition diagram of a view using controller information, and generates a VC scenario that represents a connection between the view and the controller based on a predetermined coverage standard for the state transition diagram from the transition diagram of the view,
A test scenario generation function for generating a test scenario in which View operation information is added to the generated VC scenario;
Further realized,
The test data generation program according to any one of claims 1 to 3, wherein the designated test scenario is selected from test scenarios generated by the test scenario generation function. A test data generation method in a test data generation device for generating test data for a specified test scenario,
The test data generation device inserts a code that outputs a branch result including an evaluation value for quantitatively evaluating branch determination into a test target program; and
The test data generation device generates test data for a specified test scenario, and generates a branch result output by the inserted code;
The test data generation device generates a target branch result that matches the route to the generated branch result halfway, and first test data that matches the path of the target branch result except for a final decision branch; Using the second test data having a branch result of the final decision branch in a genetic algorithm to generate test data that satisfies the target branch result;
A test data generation method comprising:

Images