JP2016115175A - Software test apparatus and software test program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To extract a necessary additional test case for software to be inspected, and to automatically output determination on success or failure thereof.SOLUTION: A software test apparatus operates as follows: inputting software to be inspected (21); generating a control flow graph formed by representing all possible paths for a program to be executed in a graphic form (23); executing an existing test case 22 on the control flow graph, and recording paths used on the control flow graph (24); generating an additional test case to be added for satisfying a predetermined complete criterion when the predetermined complete criterion based on control path test is not satisfied only with the existing test case (25); generating an expected result to be obtained as a result of executing the program in a condition in the additional test case (254); and calculating a test result of the additional test case, on the basis of an execution result of the additional test case obtained by executing only the additional test case in a test case execution section, and the expected result (26).SELECTED DRAWING: Figure 1

Description

  Embodiments described herein relate generally to a software test apparatus and a software test program.

  In software test automation, it is necessary to constantly maintain test cases and test codes in accordance with changes in software to be inspected.

  Therefore, techniques for automatically designing test cases and techniques for automatically generating test codes have been proposed in order to reduce maintenance costs.

  However, in test case design, there is a problem that there is no effective means for automatically generating expected results. This is due to the following reasons.

  Even if the expected result is generated from the inspection target software, it is not known whether the expected result is correct. Further, if information for generating an expected result is input in addition to the software to be inspected, the information is exactly what the software to be inspected should be, and there is no point in developing the software to be inspected.

  There are two implementations with the same specification, and there is a method for judging pass / fail according to whether the output for the input is the same, but it is not only costly, but when the output is different, it is not known which is wrong.

JP 2012-37938 A JP 2012-185539 A

  The problem to be solved by the present invention is to provide a software test apparatus and a software test program for extracting additional test cases necessary for software to be inspected and automatically outputting the pass / fail judgment.

  The software test apparatus according to the embodiment generates a control flow graph that generates a control flow graph in which all paths that may pass through the path when the program is executed with respect to the input inspection target software. And a test case execution unit that acquires an existing test case that has already been confirmed to execute the program, executes the existing test case on the control flow graph, and records a path that has passed on the control flow graph; and An additional test case generation unit that generates an additional test case that is added to satisfy the predetermined coverage criterion when the existing test case alone does not satisfy the predetermined coverage criterion according to the control path test method, and the additional test case When the program is executed under the written conditions, the expected result that should be obtained as a result is generated. A test result of the additional test case is calculated based on an execution result and an expected result of the additional test case obtained by executing only the additional test case by the expected result generation unit to be generated and the test case execution unit. A pass / fail determination unit.

It is a block diagram which shows schematic structure of the software test apparatus which concerns on embodiment of this invention. It is a figure showing an example of inspection object software. It is a figure which shows an example of the existing test case. It is the figure which showed the control flow graph of the test object software shown in FIG. It is a figure showing the data structure of the control flow graph shown in FIG. It is a figure which shows an example of the path | route (path | pass) to record. It is a figure which shows an example of a lack path | pass. It is a figure which shows an example of the restrictions (node formula) corresponding to the shortage path shown in FIG. It is a figure which shows an example of an input value in order to pass an insufficient path | pass. It is the figure which showed an example of the extraction of the path | pass used as teacher data with reference to the path | pass and insufficient path | pass which passed on the control flow graph. It is a figure which shows an example of the teacher data extracted from the existing test case. It is a figure which shows an example of the expected result produced | generated from the regression model. It is a figure explaining that the test result of a test case is obtained based on the execution result and expectation result of an additional test case. It is a figure which shows an example of the output of a test result list. It is a flowchart which shows the flow of the production | generation process of the expected result in the software test apparatus which concerns on embodiment, and the calculation process of a test result. It is a figure which shows the source code of the test object software in a present Example. It is a figure which shows the existing test case of the test object software shown in FIG. It is a figure which shows the source code changed with the specification change of test object software. It is a figure which shows the input value and expectation result of an additional test case. It is a figure which shows the execution result of an additional test case. It is a figure explaining investigation of the error of the source code based on a test result.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same portions are denoted by the same reference numerals, and redundant description is omitted.

  First, main terms used in the present embodiment will be described.

  “Software test” refers to an operation of executing a computer program and confirming whether or not it operates correctly. In general, it is necessary to perform tasks such as designing test cases, checking results by executing tests, and creating reports (pass / fail list).

  A “test case” is a description of what input is given to a program and what output should be obtained as a result (expected result).

  The “expected result” is the result that should be obtained when the program is executed under the conditions written in the test case.

  “Test automation” refers to automating a part of software test work.

  “Test code” means a test case described as an executable program. Implemented to run tests and check results.

  The “control path test method” is one of software test methods, and refers to a test that provides test data so that all codes in a program are executed. A type of white-box test that analyzes the logical structure of software. In order to check whether the logical structure of a program is written as intended, the test is performed considering combinations of test data that can execute as many codes as possible. .

  “Regression model” refers to an expression estimated by a statistical method among expressions expressing the relationship between two variables. Also called regression equation.

  In the present embodiment, when the control path test method does not satisfy the predetermined criteria, an additional test case is found, the expected result of the additional test case is generated as a probability density function, and combined with the pass / fail result of the existing test case, Calculate test results for additional test cases.

  FIG. 1 is a block diagram showing a schematic configuration of a software test apparatus according to an embodiment of the present invention. This apparatus is realized using a general-purpose computer (for example, a personal computer (PC) or the like) and software operating on the computer. The computer includes an engineering workstation (EWS) suitable for CAD (Computer Aided Design) and CAE (Computer Aided Engineering). In this embodiment, there is a possibility that a test target software and an existing test case whose execution of the program has already been confirmed are input to such a computer, and the test target software passes through the path when the program is executed. A function that generates a control flow graph that represents all paths in a graph, a function that executes an existing test case on the control flow graph, records a path that passes through the control flow graph, and a control path that uses only the existing test case A function that generates additional test cases that are added to satisfy the specified coverage criteria when the specified coverage criteria by the test method are not met, and when the program is executed under the conditions written in the additional test cases, The ability to generate expected results to be obtained as a result, and additional tests obtained by executing only additional test cases The function of calculating the test result of the additional test case based on the execution result and the expected result of the strike case can also be implemented as a software test program for realizing.

  As shown in FIG. 1, the software test apparatus 100 according to the present embodiment mainly includes a test target software input unit 21, an existing test case holding unit 22, a control flow graph generation unit 23, a test case execution unit 24, and an additional test case. The generation unit 25, the pass / fail determination unit 26, and the test result list output unit 27 are configured.

  The inspection target software input unit 21 inputs inspection target software to be inspected by the software test apparatus 100. The input inspection target software is sent to the control flow graph generation unit 23. The input format of the inspection target software is preferably source code, binary, or byte code. FIG. 2 is a diagram illustrating an example of inspection target software. The inspection target software shown in FIG. 2 includes three functions, and the third function indicates that the first and second functions are referred to.

  The existing test case holding unit 22 holds an existing test case. Here, the existing test case has confirmed the execution of the program, and there is no problem in the expected result to be obtained. Existing test cases are recorded separately as a set of “input” and “expected result”. The input consists of “function to be operated” and “input value”, and the expected result is expressed as “expected value”. For example, the existing test case is input to the existing test case holding unit 22 from the outside, or is held in the existing test case holding unit 22 in advance. Note that it is unknown whether the inspection target software passes the existing test case, and there may be a test case that fails. In this case, the existing test case is not used for learning data (described later).

  FIG. 3 is a diagram illustrating an example of an existing test case. In the example shown in FIG. 3, functions (clip_0to100 and clip_0to200), input values (−10, 83, −5, 130, and 250) and expected results (0, 83, 0, and 130) are operated for five test cases. , 200). A plurality of functions to be operated can exist in one program and are used as a starting point of a shortage path (described later). The data of the existing test case is sent to the test case execution unit 24.

  The control flow graph generation unit 23 generates a control flow graph of the inspection target software received from the inspection target software input unit 21. The control flow graph is a graph that represents all paths that may be passed when a program is executed, that is, a path that may be passed. Details of the control flow graph will be described later. Here, a path means a place where a program for executing a command is located, and “passing a path” means setting a path to execute a command.

<Control flow graph>
The control flow graph is a graph that represents all paths that may be passed when a program is executed, that is, a path that may be passed. FIG. 4 is a diagram showing a control flow graph of the inspection target software shown in FIG. In the example shown in FIG. 4, there are all paths that may pass the path from the start point “1” to the end point “5” and all paths that may pass the path from the start point “6” to the end point “10”. It is shown. The function to be operated is a node in which no arrow is connected from any node in the control flow graph.

  FIG. 5 is a diagram showing the data structure of the control flow graph shown in FIG. The data structure consists of “node information” and “edge information”. The node information includes a node ID for each node and the contents of each node (node expression). The edge information consists of a combination of a start point node ID and an end point node ID. In the example shown in FIG. 5, the node contents (node expressions) are shown for the node IDs “1” to “15” as the node information. Further, as edge information, the relationship of the end node ID when each node ID is set as the start node ID is shown. For example, regarding the node ID “3”, it is understood that the node ID “4” or the node ID “5” is a candidate as the end node ID. Similarly, regarding the node ID “15”, it is understood that the node ID “3” or the node ID “8” is a candidate as the end node ID.

  The test case execution unit 24 acquires an existing test case from the existing test case holding unit 22, executes the test case on the control flow graph, and records a path (path) passed on the control flow graph. FIG. 6 is a diagram illustrating an example of a route that has been recorded. In the example illustrated in FIG. 6, when compared with the control flow graph illustrated in FIG. 4, a route (path) having the node ID “3” as the start point and the node ID “4” as the end point is not a route that has passed. Recognize.

  The additional test case generation unit 25 generates an additional test case. The additional test case is a test case added so as to satisfy a predetermined coverage standard when the predetermined “coverage standard” is not satisfied in the control path test method. Here, as a predetermined coverage criterion, for example, “condition coverage” that covers test cases that are true or false for each condition, or test cases that are executed even once for all branches. Covering “branch coverage” is preferred. More specifically, the additional test case is specified by describing “input value”, “function to be operated”, and “expected result” that pass through a shortage path (described later). One additional test case is generated for each missing path. The predetermined coverage standard is input from the outside to the additional test case generation unit 25 or stored in advance in the additional test case generation unit 25.

  The additional test case generation unit 25 can be divided into a shortage path generation unit 251, an input value generation unit 252, a regression model generation unit 253, and an expected result generation unit 254.

  The insufficient path generation unit 251 compares a predetermined coverage criterion with a path that has passed through execution of an existing test case, and generates an insufficient path. Here, the missing path is searched so as to pass through nodes, edges, or combinations of nodes, edges, or combinations that need to be passed according to the coverage criteria, that did not pass even if all existing test cases were executed. Route. FIG. 7 is a diagram illustrating an example of a shortage path. From the recorded route (path) shown in FIG. 6, it is known that the route (path) having the node ID “3” as the start point and the node ID “4” as the end point is not a route that has passed. . In the example illustrated in FIG. 7, a path (path) having a node ID “1” as a start point and a node ID “5” as an end point, and passing from the node ID “3” to the node ID “4”. However, it turns out that it is a shortage path.

  For each insufficient path, the input value generation unit 252 solves the “node expression” that passes through the shortage path as a constraint on the satisfiability problem, and generates an “input value” to pass through the shortage path. As a solution to the satisfiability problem, a specific set of values that satisfy the constraints is obtained. The node formula is created when the control flow graph is generated. Specifically, the node formula is the contents of the node in the node information shown in FIG.

  FIG. 8 is a diagram illustrating an example of a constraint (node expression) corresponding to the shortage path illustrated in FIG. In the example shown in FIG. 8, expressions of seven nodes are listed. FIG. 9 is a diagram illustrating an example of input values for passing through a shortage path. In the example shown in FIG. 9, the input value = 101 is obtained as a solution obtained by solving the expression of seven nodes as a satisfiability problem. The specific values that satisfy the above-described constraints may be two or more specific values that satisfy the constraints when there are two or more input values in the node expression. In the example shown in FIG. , Clip_0to100 has one input value, so there is only one input value as a solution. The above input value = 101 is treated as a one-dimensional vector.

  The regression model generation unit 253 extracts, for each insufficient path, a test case group in which the start point of the path (path) passed on the control flow graph and the start point of the insufficient path are extracted from the existing test cases. A regression model is generated by using such teacher data. FIG. 10 is a diagram illustrating an example of extraction of paths used as teacher data with reference to the paths and insufficient paths that have passed on the control flow graph. FIG. 11 is a diagram illustrating an example of teacher data extracted from an existing test case. In the example shown in FIG. 11, two existing test cases are extracted as teacher data.

  The generated regression model (1) does not assume the shape of the mapping (linear, etc.), (2) mapping from a multidimensional vector to a multidimensional vector is possible, and (3) the prediction can be obtained with a probability density function. Has characteristics.

  As the regression model, for example, Gaussian process regression or a related vector machine can be used. Gaussian process regression is based on a Gaussian process that assumes that the corresponding predicted value is a random variable that follows a Gaussian distribution for a given input value.

For example, if the regression model equation is expressed by a Gaussian process regression equation, it is as follows.

In the above,
y ^new is a probability density function, and x ^new is an input value of an additional test case.

X ^new is a vector because there is not always one input value.

  X is a matrix in which input values of existing test cases are arranged, and y is a vector in which expected values of existing test cases are arranged.

  Depending on the test, the value (return value, variable, etc.) at multiple locations becomes a success condition, so multiple expected values may be required, but in that case, for each return value and variable, Create a model. β, θ, θ1, and θ2 are constants, and these constants are obtained by an annealing method or the like so that the regression model fits the existing test case.

  The expected result generation unit 254 generates an “expected result” from the input value generated by the input value generation unit 252 and the regression model generated by the regression model generation unit 253 for each insufficient path. The expected result in the present embodiment is generated as a probability density function having a width calculated from the average value and the variance. FIG. 12 is a diagram illustrating an example of the expected result generated from the regression model. In the example shown in FIG. 12, the expected result is generated as a probability density function f (x).

  The pass / fail determination unit 26 acquires the execution result of the additional test case obtained by executing only the additional test case from the test case execution unit 24, and acquires the probability density function of the expected result from the expected result generation unit 254. The pass / fail determination unit 26 calculates pass / fail based on the execution result and the expected result. In this embodiment, since the expected result is represented by a probability density function, the pass / fail determination unit 26 calculates the test result. The test result is a probability that the additional test case is acceptable, and is calculated by substituting a value obtained by executing the additional test case into the probability density function. FIG. 13 is a diagram illustrating that the test result of the test case is obtained based on the execution result and the expected result of the additional test case.

  The test result list output unit 27 acquires the test results of the additional test cases from the pass / fail determination unit 26 and outputs a list of the pass / fail results of the existing test cases and the additional test cases. FIG. 14 is a diagram illustrating an example of the output of the test result list. As shown in FIG. 14, the test results are shown as pass / fail for the existing test case and the probability for the additional test case.

  Next, the flow of expected result generation processing / test result calculation processing in the software test apparatus 100 configured as described above will be described. FIG. 15 is a flowchart illustrating a flow of expected result generation processing / test result calculation processing in the software test apparatus 100 according to the embodiment.

  First, the inspection target software input unit 21 inputs inspection target software (step S1501).

  Next, the control flow graph generation unit 23 generates a control flow graph in which all the paths that may be passed through the path when the program is executed are expressed as a graph for the input inspection target software (step) S1502).

  Next, the test case execution unit 24 executes the existing test case that is held in the existing test case holding unit 22 and has already been confirmed to execute the program on the control flow graph, and records the path that has passed on the control flow graph. (Step S1503).

  Subsequently, the shortage path generation unit 251 compares the predetermined coverage criterion with the path that has passed through the execution of the existing test case, and generates a shortage path (step S1504). Missing paths were searched to pass through nodes, edges, or combinations of nodes, edges, or combinations that need to pass through a given coverage criterion that did not pass through all existing test cases. A route.

  Next, the input value generation unit 252 generates an input value for each insufficient path by giving an equation of a passing node as a constraint for the satisfiability problem (step S1505).

  Next, the regression model generation unit 253 extracts a test case group having the same execution path start point from existing test cases for each insufficient path, and generates a regression model by using it as teacher data (step S1506). ).

  Subsequently, the expected result generation unit 254 generates a probability density function of the expected result from the input value and the regression model (step S1507).

  Next, the test case execution unit 24 executes only the additional test case and passes the execution result to the pass / fail determination unit (step S1508).

  Subsequently, the pass / fail determination unit 26 calculates a pass / fail or test result from the execution result and the expected result or the probability density function of the expected result (step S1509).

  Next, the test result list output unit 27 outputs a list of pass / fail results of the existing test case and the additional test case (pass / fail for the existing test case, test result for the additional test case) (step S1510). The test result calculation process ends.

[Example]
Next, examples of the present embodiment will be described. The present embodiment is inspection object software that calculates cosx by using linear interpolation in increments of π / 12. FIG. 16 is a diagram illustrating the source code of the inspection target software in the present embodiment. As shown in FIG. 16, this embodiment corresponds to an angle in the range of 0 ° to 360 ° (2π).

  FIG. 17 is a diagram illustrating a recording state of an existing test case of the inspection target software illustrated in FIG. As shown in FIG. 17, expected results are recorded for input values of 0, π / 12, 3π / 4, π, 7π / 6, 4π / 3, 11π / 6, and 2π.

  FIG. 18 is a diagram illustrating the source code that has been changed due to the specification change of the inspection target software. In the present embodiment, additional source code accompanying the specification change is added so as to correspond to an angle in a range other than 0 ° to 360 ° (2π). In the software test apparatus 100, the source code shown in FIG. 18 is input and an existing test case is executed. Then, a path that has never passed (the condition of the two added while statements becomes true) is added as an additional test case. The expected result of the additional test case is calculated by giving the input value of the test case to the regression model given the existing test case as teacher data.

  FIG. 19 is a diagram illustrating input values and expected results of additional test cases. In the example shown in FIG. 19, the expected result f (x) is obtained as a normal distribution with respect to the input value (x).

The test result of the additional test case is calculated by giving the input value (x) of the additional test case to the software to be tested and executing it, and substituting the expected result into the execution result x of the following equation. Here, Gaussian process regression is used as the regression model.

  Here, μ = average value of expected results, and σ = variance.

  FIG. 20 is a diagram illustrating an execution result of the additional test case. As shown in FIG. 20, when the input value x = −0.1, the cosx execution result 0.98696... Is obtained, and the test result in that case is 72.20%. On the other hand, when the input value x = −0.1, the cosx execution result −0.98479... Is obtained, and the test result in that case is 0.49%.

  FIG. 21 is a diagram for explaining an examination of source code errors based on the test results. In this embodiment, the value of the test result is low in the additional test case that passes through the second while statement. Therefore, a software developer or a person in charge of testing can perform defect correction activities more efficiently by checking from an additional test case having a low test result value. As shown in FIG. 21, an incorrect source code can be extracted efficiently.

  As described above, according to the present embodiment, wrong source code can be extracted efficiently, so that software defect correction activities can be performed more efficiently.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 100 ... Software test apparatus 21 ... Test object software input part 22 ... Existing test case holding part 23 ... Control flow graph generation part 24 ... Test case execution part 25 ... Additional test case generation Unit 251 ... shortage path generation unit 252 ... input value generation unit 253 ... regression model generation unit 254 ... expected result generation unit 26 ... pass / fail judgment unit 27 ... test result list output unit

Claims (13)

A control flow graph generation unit that generates a control flow graph that represents all paths that may pass through the path when the program is executed with respect to the input inspection target software; and
A test case execution unit that acquires an existing test case that has already been confirmed to execute the program, executes the existing test case on the control flow graph, and records a route that has passed on the control flow graph;
An additional test case generation unit that generates an additional test case that is added to satisfy the predetermined coverage criterion when the existing test case alone does not satisfy the predetermined coverage criterion by the control path test method;
An expected result generation unit that generates an expected result to be obtained as a result when the program is executed under the conditions written in the additional test case;
A pass / fail determination unit that calculates a test result of the additional test case based on the execution result of the additional test case and the expected result obtained by executing only the additional test case by the test case execution unit. Software test device. An inspection target software input unit for inputting the inspection target software;
An existing test case holding unit for holding the existing test case;
The software test device according to claim 1. The test result list output part which acquires the test result of the said additional test case from the said pass / fail determination part, and outputs the list of the pass / fail result of the said existing test case and the said additional test case is provided. Software testing equipment.   The software test apparatus according to claim 3, wherein the pass / fail result is a pass / fail result for the existing test case and a test result for the additional test case.   5. The software test apparatus according to claim 1, wherein an input format of the inspection target software is any one of a source code, a binary, and a byte code.   The software test apparatus according to claim 1, wherein the predetermined coverage criterion is condition coverage or branch coverage.   The software test apparatus according to claim 1, wherein the expected result is generated as a probability density function having a width calculated from an average value and a variance. The additional test case generation unit includes:
Compare the predetermined coverage criteria and the path passed by the execution of the existing test case, and generate a shortage path, a shortage path generation unit,
An input value generation unit that generates an input value for passing through the shortage path by solving for each short path by giving an equation of a node that passes through the shortage path as a constraint of the satisfiability problem;
For each of the insufficient paths, a test case group in which the start point of the route passed on the control flow graph and the start point of the insufficient path are the same is extracted from the existing test case as teacher data, and the teacher data is A regression model generation unit for generating a regression model to be used;
The expected result generation unit configured to generate the expected result from the input value generated by the input value generation unit and the regression model generated by the regression model generation unit for each of the insufficient paths. The software test apparatus according to any one of claims 1 to 5.   The software test apparatus according to claim 8, wherein the additional test case is specified by an input value passing through the shortage path, a function to be operated, and the expected result.   The software test apparatus according to claim 9, wherein the function to be operated is used as a starting point of the insufficient path.   The missing path passes through a node, an edge, or a combination of nodes, an edge, or a combination thereof that needs to pass through the predetermined coverage criteria, but failed to pass even if all the existing test cases are executed. The software test apparatus according to claim 8, wherein the software test apparatus is a searched route.   The software test apparatus according to claim 8, wherein one additional test case is generated for each of the insufficient paths. To software test equipment that inputs software to be inspected and existing test cases that have already been confirmed to execute the program,
A function for generating a control flow graph that represents all paths that may pass through the path when the program is executed for the inspection target software; and
A function of executing the existing test case on the control flow graph and recording a route passed on the control flow graph;
A function of generating an additional test case that is added to satisfy the predetermined coverage criterion when the existing test case alone does not satisfy the predetermined coverage criterion by the control path test method;
A function for generating an expected result to be obtained as a result when the program is executed under the conditions written in the additional test case;
A function for calculating a test result of the additional test case based on an execution result and an expected result of the additional test case obtained by executing only the additional test case;
Software test program for realizing.