JP2015069400A - Software test system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a software test system capable of efficiently conducting a regression test for failure prevention on source-code-changed software.SOLUTION: The software test system includes: dependency analysis means analyzing a structure of a source code for specifying a control flow to determine a dependent relation among variables; changed-portion extraction means determining a changed portion from source codes before and after the change; execution-range determination means determining a test execution range so that the control flow for which the source code is changed includes the changed portion; input-variable determination means determining input variables for the test from among variables having the dependent relation on the variables included in the changed portion; test-basis setting means setting a basis for a coverage of the test; test-case generation means generating the test case on the basis of the execution range, the input variables, and the test basis; and test-case output means outputting the test case.

Description

  The present invention relates to a system for testing software, and more particularly to a technique for generating test data.

  In software development, a regression test is performed when the specification is changed or the implementation (source code) is changed. The regression test is performed again by using a test case performed on the software before the change, and it is confirmed whether the function is not impaired by the specification or the implementation change. Here, the test case refers to test data composed of a set of input values, execution preconditions, expected results, and post-execution conditions, and one test performed using the data. A “test case” may be used to refer to a part of the data set, such as only an input value. In this case, the expected result can be regarded as an undescribed test case. In this specification, a “test case” may be used in this sense.

  In order to perform the regression test efficiently in a limited time, it is desirable not to execute all the test cases but only to the test cases related to the software changes. As a publicly known technique for realizing this, in Patent Document 1, only a test case including a software change point in an execution path is extracted from an existing test case. In Patent Literature 2, the influence range on the source code due to the software change is analyzed, and the necessity of executing the test case given from the outside is determined by checking whether the test case is involved in the influence range.

JP 2008-003985 A JP 2010-067188 A

  Depending on the changes to the software, existing test cases are not sufficient, and new test cases need to be created. Patent documents 1 and 2 do not correspond to this point.

  As a technique for automatically creating a new test case, there is a method of generating a test case by analyzing a source code. There is also a method of generating a test case so as to satisfy a predetermined test coverage standard (C0, C1, etc.). If the coverage standard of the test case generation method applied before the software specification change is Cb, and the coverage standard of the test case generation method applied after the specification change is Ca, if Cb ≦ Ca, the test after the specification change is the coverage It can be said that this is equivalent to performing a regression test with the reference Cb. In addition, there is a test case generation method (input value determination method) for determining input values with completeness using an orthogonal table or the Pairwise method. If such a method is applied before and after the specification change, It can be said that the test after the specification change is equivalent to the regression test. From the above, it can be said that the test case generation technology also supports regression testing.

  As another method for generating a test case, there is a test case generation applying model checking. This generates a comprehensive test case in consideration of all input conditions and execution timing, so that a test case with extremely high coverage can be obtained.

  However, the problem of test case generation is that the higher the required coverage standard, the greater the number of test cases that are generated and the greater the number of man-hours required for test execution. For example, it is practically difficult to apply test case generation using model checking to the entire software.

  Therefore, an object of the present invention is to provide a software test system capable of efficiently performing a regression test for preventing defects on software whose source code has been changed.

  In order to solve the above problems, the present invention provides a dependency analysis unit that analyzes a structure of a source code that defines a control flow to obtain a dependency relationship between variables, and a change location extraction that obtains a change location from the source code before and after the change. Means for determining a test execution range so as to include the changed portion with respect to the control flow in which the source code has been changed, and among variables having dependency relations with variables included in the changed portion. The test variable is generated based on the input variable determination means for determining the input variable of the test, the test reference setting means for setting the reference of the coverage range of the test, the execution range, the input variable, and the test reference. Test case generating means and test case output means for outputting the test case.

  According to the present invention, it is possible to efficiently carry out a regression test for preventing problems on software whose software source code has been changed.

The function block diagram of Example 1 of the software test system which concerns on this invention. 1 is a system configuration diagram of Embodiment 1. FIG. 9 is a processing flow for generating a test case according to the first embodiment. Sample source code used in the first embodiment. 2 is a program dependence graph of sample source code used in the first embodiment. 3 is an example of a test case generated in the first embodiment. Sample source code used in Example 2. 10 is an example of a test case generated in the second embodiment. 10 is an execution start point table used in the third embodiment. 12 is an input variable table used in the fourth embodiment. 12 is an input variable table used in the fourth embodiment. 10 is an example of a test case generated in the fifth embodiment.

  Hereinafter, embodiments of the present invention will be described.

  FIG. 1 is a basic functional configuration diagram for realizing the software test system 20 according to the first embodiment of the present invention, and is common to all the embodiments in this specification. The software test system 20 is realized as a software tool. Each function is implemented as software, and is executed on a computer including a display screen, a main arithmetic device, a main storage device (main memory), a storage medium such as a hard disk, and an input device such as a keyboard and a pointing device.

  The source file input unit 10 reads a source code file before and after the change of the software to be tested from a computer storage medium. Source code is grouped as a “project” before and after the change.

  The syntax analysis unit 11 analyzes the syntax of the source code file input from the source file input unit 10 and generates syntax tree (Abstract Syntax Tree) data. In the embodiment of the present specification, source code written in C language is handled.

  The dependency relationship analysis unit 12 analyzes and generates control flow and dependency relationship data in the source code to be tested from the syntax tree data obtained by the syntax analysis 11. Specifically, a program dependency graph (PDG) including both data is analyzed.

  The change location extraction unit 13 extracts a difference (that is, a change location) before and after the change of the source code file input from the source file input unit 10.

  The change location dependency extraction unit 14 extracts a control flow that passes through the change location from the control flow analyzed by the dependency analysis unit 12. Further, on the dependency relationship analyzed by the dependency relationship analyzing unit 12, a portion corresponding to the changed portion discovered by the changed portion extracting unit 13 is specified, and further, a portion to which the dependency relationship is connected is obtained from the portion, and the changed portion is determined. Extract as related dependencies.

  The execution range determination unit 15 determines a test case execution range, that is, a test execution start point and an execution end point in the source code, from the control flow related to the change part extracted by the change part dependency relationship extraction unit 14.

  The input variable determination unit 16 determines a variable to be input to the execution range of the test case from the dependency relationship related to the changed part extracted by the changed part dependency relationship extracting unit 14.

  The test case generation unit 17 determines the coverage of the test of the test set in the test criterion setting unit 18 (that is, the coverage criterion), the test case execution range determined by the execution range determination unit 15, and the input variable determination. Test case data is generated from the input variables determined by the unit 16.

  The test case output unit 19 outputs the test case data from the test case generation unit 17 as a file or a screen display.

  FIG. 2 shows a system configuration when the software test system 20 is realized. The software described in FIG. 1 is stored as a test case generation program 2131 in, for example, the ROM 213 on the computer 21, read by the CPU 211, and executed using the RAM 212 as a temporary storage area. The computer 21 executes the process defined by the test case generation program 2131, that is, the process of each unit described in FIG. A source code file to be tested is stored in the hard disk 214 on the computer 21. User operations such as analysis execution instructions can be performed from the input device 22 such as a keyboard.

  The output from the test case output unit 19 is output to the display device 23 such as a display or the hard disk 214, for example, but is output via an external network (not shown) or a data file to an external storage medium such as a flash memory. It may be output by writing in a format.

  FIG. 3 is a flow of processing executed by the functional configuration of FIG. After starting the processing, in step 31, the source file input unit 10 reads the source code file before and after the change of the software to be tested, and the syntax analysis unit 11 analyzes the source code file and generates syntax tree data.

  In step 32, the dependency relationship analysis unit 12 analyzes the control flow and dependency relationship of the source code input from the syntax tree data.

  In step 33, the changed part extracting unit 13 extracts a difference (that is, a changed part) before and after the input source code is changed. As a method of taking the difference, there is a method of comparing texts and dependencies.

  In step 34, the changed part dependency relationship extracting unit 14 extracts a part including the changed part before and after the change of the source code from the control flow and the dependency relation in the source code to be tested.

  In step 35, the execution range determination unit 15 determines the execution range of the test case by selecting the execution start point and the execution end point from the control flow related to the changed portion. Further, the input variable determination unit 16 determines a variable to be an input of the test case on the dependency relationship related to the changed part.

  In step 36, the test case generation unit 17 generates test case data by determining combinations of input values and processing execution timing.

  In step 37, the test case output unit 19 outputs a test case. The process ends as described above.

  FIG. 4 is an example of source code of software to be tested. The source code 41 is a part of the source code to be tested. The leftmost number is the line number. Further, the source code 42 is shown in which a part of the source code 41 is changed, and the constant “2” on the 15th line is replaced with “3” as the changed part. These source codes 41 and 42 are read into the source file input unit 10.

  FIG. 5 shows the source code 42 shown in FIG. 4 in the program dependence graph generated by the dependency analysis unit 12 and related to the code change part by the change part dependency relation extraction part 14 (connected by the control flow or the dependency relation). Only) are extracted. In such a method of extracting a graph, a node corresponding to a code change location is searched on the program dependency graph from information on the description position of the code change location, and the control flow and the dependency relationship are searched forward and backward from that node. Follow each one.

  FIG. 5 shows a control flow graph 51 and a dependency relationship graph 52 separately for easy viewing. The difference before and after the code change extracted by the change location extraction unit 13 is linked to the control flow graph 51 and the dependency relationship graph 52.

  Hereinafter, a specific example of test case generation for the changed source code 42 will be described.

  The execution range determination unit 15 searches the flow backward from one code change point (15th line) in order to determine the execution start point on the control flow graph. In a control flow in which a certain function A is called from the function B, the search is terminated when no code change portion is encountered during the control flow search in the function B, and the start point of the function A is set as the execution start point. For the determination of the execution start point, a function call tree may be used instead of the control flow.

  In the control flow graph 51, since there is no code change part in the function f11, the start point (13th line) of the function f12 is set as the execution start point. The execution end point is the end point of the function that is set as the execution start point, or the end point of the function that includes the code change point that passes last in the control flow of the execution range. Here, the end point (24th line) of the function f12 is set as the execution end point. However, other than this, the execution end point may be set by the test reference setting unit 18 and may be instructed from the test reference setting unit 18 to the execution range determination unit 15.

  Next, the input variable determination unit 16 searches the dependency relationship graph 52 for the dependency relationship from the change point (15th line) to the rear and the front. The search range is within the control flow from the execution start point to the execution end point determined by the execution range determination unit 15 for the same change point (line 15), and when a statement not included in this control flow is encountered. Abandon the search.

  In the present embodiment, an undefined variable (that is, an unassigned variable) on the dependency relationship graph is handled as an input variable of the test case. Since the variables c and v are undefined in the range searched in the dependency graph 52, these two are input variables.

  Next, the test case generation unit 17 generates test case data. The test case execution range and input variables have already been determined, and here the values of the input variables are determined. In this embodiment, it is assumed that C1 (branch coverage) is set as a coverage reference in the test reference setting unit 18. From the control flow graph 51, values for selecting true and false in the branch condition are calculated. Since c> 0 (17th line) is the branch condition, the variable c = {0, 1} is set to satisfy C1. In this embodiment, it is assumed that a variable not related to the branch condition takes two values at random. For example, the variable v = {1, 2} is determined. The input variable value of the test case is a combination of variable c = {0, 1} and variable v = {1, 2}.

  FIG. 6 shows test case data output by the test case output unit 19. Each row of the table 61 corresponds to one test case, and includes an execution start point, an execution end point, an input variable, and its value. For test execution, it is necessary to add expected results separately.

  As described above, a test case can be generated for the change in the source code and the affected range, and the regression test corresponding to the change in the source code can be easily executed. Since the test case generation unit 17 generates a test case by tracing the control flow and the dependency relationship of the program, not only the change of the source code but also the influence range can be set as the test target.

  That is, the software test system 20 according to the present embodiment has a specification / implementation in which changes in the software specification or implementation (source code) and the range affected by the change, that is, the change may cause a difference in processing contents and processing results. Since test case generation can be applied in a limited manner, it is effective for both the limitation of the number of test cases and the prevention of problems caused by changes in software specifications and implementation. In addition, since a specification change leads to a mounting change, it is only necessary to generate a test case for the mounting change point and its influence range even when the specification is changed.

  Hereinafter, an example of test case generation when “use model checker” is set as the test case generation method (that is, the test reference input value generation method) in the test reference setting unit 18 will be described. Other conditions are the same as those in the first embodiment.

  FIG. 7 is a part of source code included in the same project as the software of FIG. Assume that this source code has not changed. The function intr is set to be executed by an interrupt using a compiler directive (pragma) in the source code. This setting information is given to the control flow data by the dependency analysis unit 12.

  The change location dependency extraction unit 14 also extracts a program dependency graph of a function designated as an interrupt. For this reason, the program dependence graph of the function intr is also extracted as related to the code change point. The processing contents of the execution range determination unit 15 and the input variable determination unit 16 are the same as those in the first embodiment.

  The test case generation unit 17 generates a test case using a model checker. The source code is converted into a specification description language (Promel in SPIN) of a model checker (SPIN or the like) using a predetermined rule via AST. Further, the function f12 that is the execution start point and the location corresponding to the function intr that is executed by the interrupt are also designated as the execution start point in the model check. For example, when SPIN is used, a process for executing a portion corresponding to the function f12 and a process for executing a portion corresponding to the function intr are assigned as the Promela code. As for the input variables, possible values are obtained as in the first embodiment, and a code in which one value is selected nondeterministically is described before execution of the portion corresponding to the function f12. If a specification description language can specify that a variable can take all possible values of its type, it may be specified as such. Here we take the former method.

  For the specification description created as described above, a model check is executed after specifying properties so that an execution trace is generated as a counterexample. Input variable values and process execution timings are extracted from the generated counterexample.

  FIG. 8 shows an example of a test case output from the test case output unit 19. However, of the counter examples generated by the model checker, redundant ones are excluded as execution traces. The table 81 describes a function executed by an interrupt and data related to the execution timing. For example, Test Case 5 describes that the function intr is executed as an interrupt when the 17th line of the function f12 is executed.

  With the above, it is possible to generate test cases for changes and their impact on software source code including multiple processes (tasks and interrupts), and to easily execute regression tests corresponding to source code changes. .

  In the following, a method different from that of the first embodiment will be described for the test case execution range determination by the execution range determination unit 15. The conditions for test case generation are the same as in the first embodiment.

  FIG. 9 is an execution start point table 91 in which function candidates that become program execution start points in the test are set for the source code of FIG. The execution start point table 91 is given to the execution range determination unit 15.

  The execution range determination unit 15 performs the following process to determine the execution range. In the control flow graph 51 of FIG. 5, the backward search is performed from the code change portion (15th line). A match with the execution start point table 91 is determined for each statement in the control flow graph 51. Then, the function f12 matches and is extracted as an execution start point. When the execution start point is extracted, the search for the control flow graph 51 is terminated. If there is no statement that matches the execution start point table 91, the function at the start point of the control flow is set as the execution start point.

  Other processes are the same as those in the first embodiment, and as a result, the test case of FIG. 6 is generated.

  As described above, it is possible to generate test cases for changes in the source code and the range of influence thereof, and the generation has the following characteristics. By using the execution start point table 91, the execution start point can be limited to a specific statement. This allows, for example, the execution start point to be the same as the actual operation of the program, the beginning of a function that starts a periodic task or interrupt processing, and makes it easier to prepare the test preconditions. There are advantages such as being possible. Also, by making the test execution start point the same before and after the code change, it becomes easier to compare the test results before and after the code change.

  In the following, a method different from that of the first embodiment will be described for the input variable determination by the input variable determination unit 16. The conditions for test case generation are the same as in the first embodiment.

  FIG. 10 is an input variable table 101 in which variable candidates that are input when a part or all of the program is executed in the test and the choices and ranges of the values are set for the source code of FIG. The input variable table 101 is given to the input variable determination unit 16.

  The input variable determination unit 16 performs the following processing to determine the input variable. In the dependency relationship graph 52 of FIG. 5, the backward search is performed from the code change portion (15th line). During the search, in each statement of the dependency relationship graph 52, a match between the variable described in the statement and the variable in the input variable table 101 is determined. Then, the variable v matches and is extracted as an input variable. It can also be seen that the range of values is limited to [1,2].

  In the search in a certain path on the dependency relationship graph 52, the variable used in the statement at the search position matches the variable in the input variable table 101, or the search position exceeds the execution range determined by the execution range determination unit 15. By the way, the process ends. After the search of the dependency relationship graph 52, the variable c that is found during the search on the dependency relationship graph 52 and is undefined is extracted as an input variable in the same manner as in the first embodiment.

  When determining the value of the input variable, the test case generation unit 17 selects a value that can be taken within the range of the variable v designated by the input variable determination unit 16. As the selection method, all may be selected, or, for example, if “Use Pairwise method” is set in the test reference setting unit 18, a combination may be selected by a designated method.

  Other processes are the same as those in the first embodiment, and as a result, the test case of FIG. 6 is generated.

  As described above, it is possible to generate test cases for changes in the source code and the range of influence thereof, and the generation has the following characteristics. By using the input variable table 101, it is possible to define the values that can be taken by the input variables (including values outside the specification range) and generate a comprehensive test case within the range of the values that can be taken. Conversely, it is possible to prevent unnecessary test cases from being generated with values that the input variables do not take, and to gain benefits such as shortening the time required for testing.

  Hereinafter, a method different from that of the first embodiment will be described with respect to determination of the test case execution range by the execution range determination unit 15 and input variable determination by the input variable determination unit 16. The conditions for test case generation are the same as in the first embodiment.

  In the present embodiment, unlike the first to fourth embodiments, the input variable determination unit 16 determines the input variable before the execution range determination unit 15 determines the test case execution range. The input variable determination unit 16 searches the dependency graph 52 of FIG. 5 in order to determine the input variable. This process is the same as in the fourth embodiment, except that the search range is not limited to the test case execution range determined by the execution range determination unit 15.

  Next, the execution range determination unit 15 performs the following processing to determine the execution range. In the control flow graph 51 of FIG. 5, the backward search is performed from the code change portion (15th line). The search is terminated at the point where the input variable determined by the execution range determination unit 16 is used even once for each input variable, and the start point of the function including the statement at that time is defined as the execution start point. To do. The execution end point is determined in the same manner as in the first embodiment.

  Next, the input variable determination unit 16 extracts the variable c, which is found during the search on the dependency relationship graph 52 and remains undefined, as an input variable as in the first embodiment.

  FIG. 11 shows an input variable table 111 given to the input variable determination unit 16 in the present embodiment. The input variable determination unit 16 searches the dependency graph 52, extracts the variable reg1 as an input variable, and sets the value candidate as {1, 3}. At the end of the search of the dependency graph 52, the variable c remains as an undefined variable.

  Next, the execution range determination unit 15 searches the control flow graph 51, aborts the search at the statement on the ninth line where the variable reg1 is used, and sets the function f11 including this statement as the execution start point. Next, the input variable determination unit 16 confirms that the statement that uses the variable c is included in the execution range, and adds the variable c to the input variable. The method of determining the value of the variable c depends on the coverage standard and test case generation method set in the test standard setting unit 18 as in the first embodiment.

  Through the above processing, the test case output unit 19 outputs the test case data shown in FIG. Each row of the table 121 corresponds to one test case, and the input value is a combination of the value {1, 3} of the variable reg1 and the value {0, 1} of the variable c.

  As described above, it is possible to generate test cases for changes in the source code and the range of influence thereof, and the characteristics of the generation are the same as in the fourth embodiment. However, as compared with the fourth embodiment, the ratio of the input variables for which possible values are defined is equal or increased. As a result, it is possible to improve the completeness of the test and reduce unnecessary test cases.

  In each of the embodiments described above, it is possible to generate test cases regarding software changes and their influence ranges, and it is possible to realize a regression test after supplementing the newly required test cases by code changes. In the software test system 20, since the method of generating test cases limited to the change point of the software source code and the influence range is defined and the test case is generated in a limited range, the test case for a wider range as in the entire software is taken. Compared to the generation method, the time required for test case generation and the time required for test execution can be reduced.

  Similar to the software test system 20, as another method for obtaining a test case by limiting to the software change point and its influence range, a test case is generated for a wider range, for example, the entire software, and the test case A method for extracting only the changes related to the change of the source code and the influence range thereof can be considered. In this method, the number of test cases to be generated is significantly larger than that of the software test system 20, and it is necessary to separate the necessary ones from the many test cases. It is considered that a larger amount) is required than the software test system 20. The software test system 20 improves this point.

  DESCRIPTION OF SYMBOLS 10 ... Source file input part, 11 ... Syntax analysis part, 12 ... Dependency analysis part, 13 ... Change location extraction part, 14 ... Change location dependency extraction part, 15 ... Execution range determination part, 16 ... Input variable determination part, DESCRIPTION OF SYMBOLS 17 ... Test case production | generation part, 18 ... Test reference | standard setting part, 19 ... Test case output part, 20 ... Software test system

Claims (4)

Dependency analysis means for analyzing the structure of the source code that defines the control flow and obtaining the dependency relationship between variables,
Change location extraction means for obtaining a change location from the source code before and after the change,
Execution range determination means for determining an execution range of a test so as to include the changed portion with respect to the control flow in which the source code is changed;
An input variable determining means for determining an input variable of the test from variables having a dependency relationship with a variable included in the changed portion;
Test standard setting means for setting a standard for the coverage of the test;
A software test system comprising: a test case generation unit that generates the test case based on the execution range, the input variable, and the test criterion; and a test case output unit that outputs the test case. The software test system of claim 1,
The execution range determining means has execution start point candidates in the source code as execution start point data,
A software test system, wherein an execution start point included in the execution start point data is searched from a control flow including the changed part to determine the execution range. The software test system of claim 1,
The input variable determination means has input variable candidates in the source code as input variable data,
A software test system, wherein the input variable is determined by searching for a variable included in the input variable data from a variable having a dependency relationship with a variable included in the change portion. The software test system of claim 3,
The execution range determination means determines the execution range so that the input variable appears on the control flow.