JP6748357B2 - Analysis device, analysis program, and analysis method - Google Patents

Description

The present invention relates to an analysis device, an analysis program, and an analysis method.

In the field of software development, in order to improve the quality of software, the software program may be tested. In addition, the effectiveness of the test itself may be evaluated. For example, coverage (coverage rate) is an evaluation index of the test. Coverage is the percentage of times a test executed a given location (line, branch, condition, etc.) in a program. Improve coverage to reduce untested code.

For example, there is a proposal of a verification support device that automatically generates input data in order to improve coverage. In this proposal, a verification support device parses a source code to generate a graph structure including a data processing node and a branch processing node, extracts a program execution path based on the graph structure, and executes the execution path to the program. Input data for generating is generated.

Further, there is also a proposal of an exhaustive testing apparatus that enables exhaustive testing without using a device such as a debugger, even for software in which source code is not provided and only binary data is available.
By the way, as another evaluation method of the test, a method called mutation analysis may be used. Mutation analysis is a method of embedding a bug in a program by intentionally changing one or more elements of the program and checking how much the test can detect the bug. In the mutation analysis, for example, the output when a predetermined input is given in a state where the bug is embedded in the program is checked, and it may be checked whether the test can detect the bug.

JP, 2016-18390, A JP, 2016-146032, A

In order to perform mutation analysis, it is possible to compile the modified source code in which the modified content is applied to the modified part of the source code and execute the test by the compiled program. However, if the source code to which the changed contents (mutation contents) are applied is separately compiled and the test is executed individually for each combination of changed parts (mutation parts), there is a problem that analysis takes time.

In one aspect, the invention aims to speed up the analysis.

In one aspect, an analysis device is provided. The analysis device has a storage unit and a calculation unit. The storage unit stores a plurality of changed portions of the program when performing a test including a plurality of test cases for confirming the output with respect to the input of the program, and information on the changed contents of each of the plurality of changed portions. The arithmetic unit applies the change contents to the first combination of the changed parts and executes the test, and executes the executed changed part on a node other than the end node, the changed contents of the changed part on the edge, and the execution result on the end node. A test in which the information of the decision tree is generated for each test case, and the change contents are applied to the second combination of the changed portions for which the test result has not been acquired, based on the plurality of generated decision trees. Determine the result.

In one aspect, the analysis can be speeded up.

It is a figure which shows the analysis apparatus of 1st Embodiment. It is a figure which shows the hardware example of the analysis apparatus of 2nd Embodiment. It is a figure which shows the example of mutation analysis. It is a figure showing an example of a function of an analysis device. It is a figure which shows the example of a source code. It is a figure which shows the example of the source code after substitution. It is a figure which shows the example of a test program. It is a figure which shows the example of a mutation operator list. It is a figure which shows the example of metamutation information. It is a figure which shows the example of an intermediate code. It is a figure which shows the example of an execution state set. It is a figure which shows the example of a mutant list. It is a figure which shows the example of a test result table. It is a figure which shows the example of a decision tree. It is a figure which shows the example of the merge result of each decision tree. It is a figure which shows the code example of a node edge. It is a figure which shows the data example of a decision tree. It is a figure which shows the output example of a result report. It is a flow chart which shows the example of processing of an analysis device. It is a flow chart which shows an example of decision tree creation processing. 6 is a flowchart showing an example of initialization processing. It is a flow chart which shows an example of a decision tree internal node addition processing. It is a flow chart which shows an example of decision tree leaf node addition processing. It is a flow chart which shows the current position specific processing example of a decision tree. It is a flow chart which shows an example of decision tree merge processing. It is a flow chart which shows an example of result report output processing. It is a figure which shows the example (1) of creation of a decision tree. It is a figure which shows the example (2) of creation of a decision tree. It is a figure which shows the example (3) of creation of a decision tree. It is a figure which shows the example (4) of creation of a decision tree. It is a figure which shows the example of a decision tree creation (the 5). It is a figure which shows the example (6) of creation of a decision tree. It is a figure which shows the example of a decision tree creation (the 7). It is a figure which shows the example of a decision tree creation (the 8). It is a figure which shows the example (the 9) of creation of a decision tree. It is a figure which shows the example (10) of creation of a decision tree. It is a figure which shows the merge example (the 1) of a decision tree. It is a figure which shows the merge example (the 2) of a decision tree. It is a figure which shows the merge example (the 3) of a decision tree. It is a figure which shows the merge example (the 4) of a decision tree. It is a figure which shows the example (the 1) which traces a decision tree. It is a figure which shows the example (the 2) which traces a decision tree. It is a figure which shows the acquisition example of a test result. It is a figure which shows the example of the mutant which is not performed in SSE. It is a figure which shows the comparative example of test result acquisition.

Hereinafter, the present embodiment will be described with reference to the drawings.
[First Embodiment]
FIG. 1 is a diagram showing an analyzing apparatus according to the first embodiment. The analysis apparatus 1 tests a software program. The analysis device 1 performs a test and also evaluates the test itself by a mutation analysis method. The analysis apparatus 1 provides the user (software developer) with the evaluation result of the test and assists the user in improving the test.

The analysis device 1 includes a storage unit 1a and a calculation unit 1b.
The storage unit 1a may be a volatile storage device such as a RAM (Random Access Memory) or a non-volatile storage device such as an HDD (Hard Disk Drive) or a flash memory. The calculation unit 1b may include a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and the like. The calculation unit 1b may be a processor that executes a program. The processor may include a set of multiple processors (multiprocessor). The analysis device 1 may be called a “computer”.

The storage unit 1a stores the source code P1. The source code P1 is an example of a software program to be tested. The source code P1 includes, for example, an integer type function func. An integer type value (x) is input to the function func.

The storage unit 1a has a source code P2 in which a changed portion (mutation portion) in which a bug is embedded in the source code P1 is replaced with a predetermined metamutation function (hereinafter, may be simply referred to as a metafunction) for performing a mutation. Memorize Such a method of replacing the changed portion with the meta function and subsequently providing the mutation information is also called MSG (Mutant Schema Generation). In the example of FIG. 1, each metafunction has three arguments. The first argument is the identifier of the changed part. The second argument is an input value (“x” in this example). The third argument is a value used in the calculation. For example, the source code P2 is generated by the calculation unit 1b based on the source code P1.

Specifically, in the source code P2, the description "x<0" on the second line of the source code P1 is replaced with the description of the meta function "bop_lt(1,x,0)". The first argument “1” of the meta function bop_lt(1, x, 0) is the identifier of the changed part. The second argument “x” is an input value. The third argument “0” is a value used for a comparison operation on the input value. For example, the meta-function bop_lt changes a relational operator (which may also be called a comparison operator) “<” into “>=”. The change content of the changed part "1" is that "<" is set to ">=", and the changed part "1" is not changed (here, "<" remains "<"). is there. Here, not changing is referred to as “nop” (Non-OPeration).

In the source code P2, the description "x+1" on the third line of the source code P1 is replaced with the description of the meta function "bop_add(2, x, 1)". The meta function bop_add changes the arithmetic operator "+" (addition) to "-" (subtraction). That is, the changed contents of the changed portion "2" are "+" to "-" and "nop".

In the source code P2, the description "x>0" on the fourth line of the source code P1 is replaced with the description of the meta function "bop_gt(3,x,0)". The meta function bop_gt changes the relational operator “>” to “<=”. That is, the changed contents of the changed portion “3” are “>” is changed to “<=” and “nop”.

Further, in the source code P2, the description "x*2" on the fifth line of the source code P1 is replaced with the description of the meta function "bop_mul(4,x,2)". The meta-function bop_mul changes the arithmetic operator “*” (multiplication) to “/” (division) or “%” (modulo operation). That is, the changed contents of the changed portion “4” are “*” is changed to “/” and “%”, and “nop”. Note that the return value of the function func's return instruction is rounded to an integer (for example, the result of the operation "-1/2" is rounded to an integer "0").

As described above, the storage unit 1a stores information on a plurality of changed portions of the program when the program is tested and the change contents of each of the plurality of changed portions. The calculation unit 1b executes the test using the source code P2, so that the changed content is applied to the changed portion of the source code P2 according to the execution path. Therefore, the changed content is applied to a combination of changed portions along the execution path (one or more changed portions are elements). The calculation unit 1b may execute the test using the intermediate code generated by compiling the source code P2 (the method may be referred to as Bitcode Mutation). In addition, the technique of dynamically providing the mutation information to the execution state of the program by the intermediate code is also called Online Adaptation Technique. Further, a method of dividing the execution state at the execution location and sharing the execution up to the execution location to reduce the execution cost is also called SSE (Split-Stream Execution).

The storage unit 1a also stores a test program (not shown in FIG. 1) that describes the contents of the test. The test includes a plurality of test cases that confirm the output to the input of the program. In FIG. 1, the test includes, by way of example, three test cases.

The first test case (testcase1) is func(0)==0. That is, the first test case is confirmation whether or not the output becomes the confirmation value 0 when x=0 is input to the function func.

The second test case (testcase2) is func(1)==2. That is, the second test case is confirmation whether the output becomes the confirmation value 2 when x=1 is input to the function func.

In the third test case (testcase3), func(-1)==0. That is, the third test case is confirmation whether the output becomes the confirmation value 0 when x=−1 is input to the function func.

The calculation unit 1b executes the test using the source code P2 and evaluates the test by mutation analysis.
For example, when the change content is applied to a certain combination of changed parts, if at least one of the three test cases produces an output (Fail) different from the confirmation value, the test causes the combination of the corresponding changed parts to be changed. Therefore, the bug can be properly detected. Also, when the change contents are applied to a certain combination of changed parts and all three test cases have the same output (Pass) as the confirmation value, in the test, a bug is generated for the combination of the corresponding changed parts. Will not be detected properly. A test result indicating that a bug can be detected is referred to as "Killed", and a test result indicating that a bug cannot be detected is referred to as "Unkilled".

The calculation unit 1b refers to the storage unit 1a, applies the changed content to the first combination of changed portions, and executes the test. While executing the test, the computing unit 1b sets the executed changed portion as a node other than the terminal node, the changed content of the changed portion (including the case of not changing (including nop)) as the edge, and the execution result as the terminal node. Information on the decision tree is generated for each test case. That is, the operation unit 1b branches during the execution of each test case into each change content (nop if no change is made) at the changed place in the program, and the execution result is stored in the terminal node. To generate. The execution result indicates either the same as the confirmation value (that is, Pass) or the difference from the confirmation value (that is, Fail). The decision tree is represented by a directed graph.

According to the above example, the calculation unit 1b generates information on the decision tree T1 for the first test case. The calculation unit 1b generates information on the decision tree T2 for the second test case. The calculation unit 1b generates information on the decision tree T3 for the third test case. For example, the node corresponding to the changed portion of the identification information “1” is the node m1. The node corresponding to the changed portion of the identification information “2” is node m2. A node corresponding to the changed portion of the identification information “3” is a node m3. A node corresponding to the changed portion of the identification information “4” is a node m4. The structure of the decision tree T1 will be specifically described below.

The node m1 has an edge corresponding to “nop” and an edge corresponding to “>=”. In the decision tree T1, the edge of the edge corresponding to “nop” of the node m1 is the node m3. The tip of the edge corresponding to “>=” of the node m1 is the node m2.

The node m2 has an edge corresponding to “nop” and an edge corresponding to “−”. In the decision tree T1, the end of the edge corresponding to “nop” and the end of the edge corresponding to “−” of the node m2 are the end nodes corresponding to “Fail”.

The node m3 has an edge corresponding to “nop” and an edge corresponding to “<=”. In the decision tree T1, the tip of the edge corresponding to “nop” of the node m3 is the terminal node corresponding to “Pass”. The tip of the edge corresponding to “<=” of the node m3 is the node m4.

The node m4 has an edge corresponding to "nop", an edge corresponding to "/", and an edge corresponding to "%". In the decision tree T1, the end of the edge corresponding to “nop” of the node m4, the end of the edge corresponding to “/”, and the end of the edge corresponding to “%” are all terminations corresponding to “Pass”. It is a node.

The decision trees T2 and T3 also have a structure according to the execution paths by the second and third test cases.
By the above test execution, the calculation unit 1b can obtain a test result indicating "Killed" or "Unkilled" for some combinations of changed portions. However, there is also a combination of changed parts for paths that are not executed in any test case or all test cases. If there is a combination of changed parts for a route that is not executed in a certain test case, and the execution results of other test cases are all "Pass", whether the test result for that combination is "Killed" or "Unkilled" Become unknown. Further, when there is a combination of changed parts for a route that is not executed in all the test cases, it is unclear whether the test result for that combination is "Killed" or "Unkilled".

In the example of FIG. 1, in the test execution described above, the computing unit 1b identifies, for example, "nop" for the changed portions of the identification information "1" and "2", "<=" for the changed portions of the identification information "3", The test result cannot be obtained for the combination (the combination A) in which the changed portion of the information “4” is “/”. This is because, in the first test case, the execution result is “Pass” when the change content of the combination A is applied to the source code P1 (or the source code P2), and in the second and third test cases, the combination A This is because the changes in the above are not applied at the same time.

Further, the calculation unit 1b, for example, identifies the changed portion of the identification information “1” as “>=”, the changed portion of the identification information “2” as “nop”, the changed portion of the identification information “3” as “<=”, and identifies the changed portion. The test result cannot be obtained for the combination (the combination B) in which the changed portion of the information “4” is “/”. This is because, in the first and second test cases, the changed contents of the combination B are not applied at the same time, and in the third test case, the changed contents of the combination B are applied to the source code P1 (or source code P2). This is because the execution result will be "Pass" when doing so.

When the calculation unit 1b applies the change content to the second combination of the changed portions for which the test result has not been acquired, based on the plurality of generated decision trees (that is, the decision trees T1, T2, T3) Determine the test results of. Specifically, the arithmetic unit 1b treats Fail as “0” and Pass as “1”, and integrates (merges) the decision trees T1, T2, and T3 with a logical product, so that after integration (after merging). A decision tree T4 is generated. The operation unit 1b can utilize the merge algorithm of the binary decision diagram (OBDD) for the merge processing of the decision trees T1, T2, T3. For example, the calculation unit 1b first merges the decision trees T2 and T3 to generate a merged decision tree, and further merges the created merged decision tree and the decision tree T1 to obtain the decision tree T4. To generate.

The decision tree T4 has nodes m1, m2, m3, m4 and nodes corresponding to “Pass” and “Fail”, like the decision trees T1, T2, T3. However, in FIG. 1, in the decision tree T4, there are two nodes (node m2) corresponding to the changed portion of the identification information “2” (hereinafter referred to as node m2(1) and node m2(2). Distinguish). In the decision tree T4, the edge of the edge corresponding to the "nop" of the node m1 is the node m2(1). The tip of the edge corresponding to “>=” of the node m1 is the node m2(2).

In the decision tree T4, the tip of the edge corresponding to “nop” of the node m2(1) is the node m3. The tip of the edge corresponding to "-" of the node m2(1) is the terminal node corresponding to "Fail". Further, both the tip of the edge corresponding to “nop” of the node m2(2) and the tip of the edge corresponding to “−” of the node m2(2) are end nodes corresponding to “Fail”.

In the decision tree T4, the edge of the edge corresponding to the "nop" of the node m3 is the node m4. The end of the edge corresponding to “<=” of the node m3 is the terminal node corresponding to “Fail”.

In the decision tree T4, the edge of the edge corresponding to “nop” of the node m4 is the terminal node corresponding to “Pass”. The tip of the edge corresponding to “/” and the tip of the edge corresponding to “%” of the node m4 are both terminal nodes corresponding to “Fail”.

Then, the calculation unit 1b determines the test result for the second combination (for example, the combination A and B described above) for which the test result has not been acquired by tracing the decision tree T4.
As an example, consider the combination A described above. In the decision tree T4, the computing unit 1b sequentially traces the edge corresponding to "nop" of the node m1, the edge corresponding to "nop" of the node m2(1), and the edge corresponding to "<=" of the node m3. Then, the terminal node corresponding to "Fail" is reached. Therefore, the calculation unit 1b determines the test result for the combination A as "Killed".

As another example, consider the combination B described above. In the decision tree T4, the calculation unit 1b sequentially traces the edge corresponding to “>=” of the node m1 and the edge corresponding to “nop” of the node m2(2), thereby obtaining the end node corresponding to “Fail”. Reach Therefore, the calculation unit 1b determines the test result for the combination B as "Killed".

The calculation unit 1b outputs (stores) the determined test result to the storage unit 1a. The calculation unit 1b may display the determined test result on a display device (not shown in FIG. 1) connected to the analysis device 1. In addition, the calculation unit 1b considers the test result determined by the decision tree T4, and determines the ratio of the combination in which the test result is “Killed” among the plurality of combinations of the changed portions to be tested, as an evaluation index of the test. May be calculated and output.

For example, it is conceivable to carry out the mutation according to the execution path by the SSE method as described above. This method can reduce the execution cost, but may not try to change a combination of some changes. Therefore, a test result may not be obtained for a certain combination of changed parts (for example, the above-mentioned combinations A and B).

In this case, the analysis device 1 prepares the source code to which the change contents are individually applied for the combination of the changed portions that has not been tried for each test case, compiles the same, and executes the test to execute the test for the corresponding combination. It is possible to supplement the test results. However, if the source code with the bugs embedded in this way is individually compiled (mutant generation) and the test is executed, there is a problem that analysis takes time.

Therefore, the analysis device 1 generates the decision trees T1, T2, T3 at the time of executing the test, generates the decision tree T4 based on the decision trees T1, T2, T3, and traces the decision tree T4 to obtain the test result. Determine the test result for the combination of changes that has not been obtained. Here, in the processing of the analysis device 1, processing costs such as generation of the decision trees T1, T2, T3 and merging occur. However, it is considered that the processing costs of generating and merging the decision trees T1, T2, T3 are sufficiently small as compared with the processing cost of individually generating and executing the mutants.

In this way, the analysis device 1 determines the test result for the combination of the changed portions for which the test result has not been acquired based on the decision trees T1, T2, T3, and thus does not have to individually generate and execute the mutants. , The execution time can be reduced accordingly. Thereby, the analysis device 1 can speed up the analysis. Below, a more specific example is shown and the function of the analysis apparatus 1 is demonstrated still in detail.

[Second Embodiment]
FIG. 2 is a diagram illustrating a hardware example of the analysis device according to the second embodiment. The analysis device 100 includes a processor 101, a RAM 102, an HDD 103, an image signal processing unit 104, an input signal processing unit 105, a medium reader 106, and a communication interface 107. Each unit is connected to the bus of the analysis device 100.

The processor 101 controls information processing of the analysis device 100. The processor 101 may be a multiprocessor. The processor 101 is, for example, a CPU, DSP, ASIC, FPGA, or the like. The processor 101 may be a combination of two or more elements among a CPU, DSP, ASIC, FPGA, and the like.

The RAM 102 is a main storage device of the analysis device 100. The RAM 102 temporarily stores at least part of an OS (Operating System) program and an application program to be executed by the processor 101. The RAM 102 also stores various data used for processing by the processor 101.

The HDD 103 is an auxiliary storage device of the analysis device 100. The HDD 103 magnetically writes and reads data to and from the built-in magnetic disk. The HDD 103 stores an OS program, application programs, and various data. The analysis apparatus 100 may include another type of auxiliary storage device such as a flash memory or an SSD (Solid State Drive), or may include a plurality of auxiliary storage devices.

The image signal processing unit 104 outputs an image to the display 11 connected to the analysis device 100 according to an instruction from the processor 101. As the display 11, a CRT (Cathode Ray Tube) display, a liquid crystal display, or the like can be used.

The input signal processing unit 105 acquires an input signal from the input device 12 connected to the analysis device 100 and outputs the input signal to the processor 101. As the input device 12, for example, a pointing device such as a mouse or a touch panel, a keyboard or the like can be used.

The medium reader 106 is a device that reads a program or data recorded in the recording medium 13. As the recording medium 13, for example, a magnetic disk such as a flexible disk (FD: Flexible Disk) or an HDD, an optical disk such as a CD (Compact Disc) or a DVD (Digital Versatile Disc), or a magneto-optical disk (MO: Magneto-Optical disk). Can be used. Further, as the recording medium 13, for example, a non-volatile semiconductor memory such as a flash memory card can be used. The medium reader 106 stores the program or data read from the recording medium 13 in the RAM 102 or the HDD 103 according to an instruction from the processor 101, for example.

The communication interface 107 communicates with other devices via the network 14. The communication interface 107 may be a wired communication interface or a wireless communication interface.

FIG. 3 is a diagram illustrating an example of mutation analysis. Here, the mutation analysis by the analysis device 100 is illustrated using the program P10 to be tested. The program P10 includes a description of the function abs. Here, the mutation analysis is a method of embedding a bug by changing one element of the program and checking how much the embedded bug can be detected by the test. A program with a bug embedded is called a "mutant". The mutant is represented by, for example, a set (mutation descriptor) of (mutation location, mutation content).

Specifically, the program P11 is a mutant in which the description of “<=” included in the second line of the program P10 is changed to “>”. The identification name of the program P11 is "mutant1".

The program P12 is a mutant in which the description of "0" included in the second line of the program P10 is changed to "1". The identification name of the program P12 is "mutant2".

Further, the program P13 is a mutant in which the description of "-" included in the third line of the program P10 is changed to "---". The identification name of the program P13 is "mutant3".

In the example of FIG. 3, consider two test cases. The first test case (Testcase1) is abs(2)==2. That is, the first test case confirms whether or not the output becomes “2” when x=2 is input to the function abs. If abs(2)==2 holds, it is "Pass", and if not, "Fail".

The second test case (Testcase2) is abs(-2)==2. That is, the second test case confirms whether the output becomes “2” when x=−2 is input to the function abs. If abs(−2)==2 is satisfied, it is “Pass”, and if not, “Fail”.

In the program P11, the execution result of both the first and second test cases is Fail. In the program P12, the execution result of the first test case is Pass, and the execution result of the second test case is Pass. In the program P13, the execution result of the first test case is Pass and the execution result of the second test case is Fail.

In this case, the bugs embedded in the program P11 can be detected by the first test case and the second test case. Therefore, the test result for the program P11 is Killed.

Further, in the first test case and the second test case, the bug embedded in the program P12 cannot be detected. Therefore, the test result for the program P12 is Unkilled.

In addition, the second test case makes it possible to detect a bug embedded in the program P13. Therefore, the test result for the program P13 is Killed.
For example, the analysis device 100 calculates and outputs the ratio of detected bugs (mutation score). In this case, the mutation score is 2/3=0.667. Software developers can improve test reliability by improving test cases to improve mutation scores.

Note that the example of FIG. 3 illustrates the programs P11, P12, and P13 to which one change (mutation) is applied. Such a program is called a first order mutant (FOM). Further, by combining FOM, it is possible to consider a program to which two or more mutations are applied. A program to which two or more mutations are applied is called a high order mutant (HOM).

In the mutant analysis, if the mutants are comprehensively generated and the test is executed, the test takes a long time. For example, considering the time required for compilation and test execution, the total test time for a plurality of mutants is expressed by the following equation.

Total time = (number of mutants) * (mutation time + compile time) + (number of mutants) * (number of test cases) * (average execution time per test case)
Therefore, there is a method (MSG) that suppresses the compile cost by replacing the normal mutation by replacing the mutation part of the program with a metafunction that performs the mutation and giving the mutation parameter afterwards. .. In MSG, the program after replacement with the metafunction may be compiled once. For MSG, the document “Mutation Analysis Using Mutant Schemata, Untch, Offutt, and Harrold, ISSTA, 1993” can be referred to.

There is also a method (Bitcode Mutation) that reduces the number of compilations by performing mutation on the bit code compiled from the source code. For the Bitcode Mutation, refer to the document “Mujava: An Automated Class Mutation System, Mar, Offutt, and Kwon, STVR, 2005”.

Furthermore, a method called Online Adaptation Technique may be used to dynamically provide the execution state with mutation information, thereby avoiding execution of redundant mutants and shortening the execution time. In particular, in SSE, the execution state can be divided at the execution points of the program, and the execution up to the execution points can be shared to reduce the execution cost. In SSE, the execution state is divided into multiple stages in consideration of HOM generation.

However, by using Online Adaptation and SSE, the execution cost can be reduced, but the redundant HOM is not executed, and the Killed/Unkilled test result for the HOM may become unknown. Note that a mutant that is not executed in SSE is illustrated in FIG. However, the omitted HOM may be killed. Therefore, the analysis apparatus 100 provides a function for efficiently obtaining the test result for the HOM for which the test result has not been obtained.

FIG. 4 is a diagram illustrating an example of functions of the analysis device. The analysis device 100 includes a storage unit 110, a metafunction replacement unit 120, a compiler 130, a decision tree creation unit 140, a decision tree merge unit 150, and a result output unit 160.

The storage unit 110 is realized by a predetermined storage area of the RAM 102 or the HDD 103. The metafunction replacing unit 120, the compiler 130, the decision tree creating unit 140, the decision tree merging unit 150, and the result output unit 160 are realized by the processor 101 executing the programs stored in the RAM 102.

The storage unit 110 stores the source code and the source code after replacement. The replaced source code is a description in which the description of the mutation portion in the source code is replaced with a metafunction. The storage unit 110 also stores a program generated by compiling the replaced source code. The program is, for example, an LLVM (Low Level Virtual Machine) bit code. Further, the storage unit 110 stores a test program describing the test content and a mutation operator list indicating the mutation content. The source code, the replaced source code, the LLVM bit code, the test program, and the mutation operator list may be stored in a storage device external to the analysis device 100.

The storage unit 110 stores metamutation information, a mutant list, an execution state set, a decision tree, and a test result table. The metamutation information is information for managing the position of the source code replaced with the metafunction. The mutant list is information on a list of mutants. The execution state set is information used to record the execution state associated with program execution. The decision tree is information in which the execution result of the program by the test is recorded together with the mutant portion traced by the execution. The test result table is information for recording the test result for each mutant.

The meta-function replacing unit 120 replaces the description related to the predetermined operation of the source code with the meta-function to generate the replaced source code and stores it in the storage unit 110. Further, the meta function replacement unit 120 generates meta mutation information and a mutant list according to the replacement, and stores the meta mutation information and the mutant list in the storage unit 110.

The compiler 130 compiles the source code after replacement stored in the storage unit 110 to generate an intermediate code program (LLVM bit code), and stores the intermediate code program (LLVM bit code) in the storage unit 110.

The decision tree creating unit 140 executes a test based on the test program on the intermediate code program stored in the storage unit 110. The decision tree creating unit 140 executes the test using the SSE method. At this time, the decision tree creating unit 140 creates a decision tree and records the execution state of the program while performing mutation on the mutation location according to the mutation operator list stored in the storage unit 110. The decision tree creating unit 140 creates a decision tree for each test case. The decision tree creation unit 140 stores the created decision tree information in the storage unit 110.

The decision tree merging unit 150 merges the decision trees for each test case stored in the storage unit 110. The decision tree merging unit 150 uses the OBDD merge algorithm for merging the decision trees. Regarding the merge algorithm, reference is made to the document “Merging Ordered Binary Decision Diagrams, http://www.cs.engr.uky.edu/~lewis/papers/merge-alg.pdf, September 7, 2016 search”. .. The decision tree merging unit 150 treats “Fail” as “0” and “Pass” as “1”, and determines the decision tree f(m1, m2,... mn) (n is a positive integer) and the decision tree g( , mn) to create a decision tree h(m1, m2,..., mn) (mn indicates a node). More specifically, the decision tree merging unit 150 selects nodes in the decision trees f and g with depth priority, assigns mutation operators, and creates the nodes and edges of the decision tree h in that order. When at least either the decision tree f or the decision tree g becomes Fail, the decision tree merging unit 150 creates an edge from the node of the decision tree h at that point in time to the Fail node (details will be described later). The decision tree merging unit 150 stores information on the decision tree obtained by the merging in the storage unit 110.

The result output unit 160 determines the test result for each mutant by referring to the mutant list stored in the storage unit 110 and following the merged decision tree stored in the storage unit 110 to determine the mutation score. Calculate. The result output unit 160 causes the display 11 to display a result report including the calculation result of the mutation score. The result output unit 160 causes the display 11 to display a list of mutants that could not be detected (cannot be killed) in the test. When any one of the mutants displayed in the list is selected by the user, the result output unit 160 refers to the metamutant information stored in the storage unit 110 to find the corresponding source code corresponding to the selected mutant. Present the location to the user.

FIG. 5 is a diagram showing an example of the source code. The source code 111 is input to the analysis device 100 by the user and stored in the storage unit 110. The source code 111 includes a description of an integer type function func in C language. The function func is a function that performs an operation on an integer type input value (x).

FIG. 6 is a diagram showing an example of the source code after replacement. The replaced source code 112 is generated by the metafunction replacing unit 120 based on the source code 111 and stored in the storage unit 110. The replaced source code 112 includes a metafunction. In this example, the metafunction has three arguments. The first argument is an identifier (MMID: Meta Mutation IDentifier) of the changed part. The second argument is an input value (“x” in this example). The third argument is a value used in the calculation.

Specifically, in the post-replacement source code 112, the description “x<0” on the second line of the source code 111 is replaced with the description of the meta function “bop_lt(1,x,0)”. The first argument “1” of the meta function bop_lt(1,x,0) is the MMID. The second argument “x” is an input value. The third argument “0” is a value used for a comparison operation on the input value. The meta function bop_lt performs a mutation that changes the relational operator “<” to “>=”.

In the post-replacement source code 112, the description "x+1" on the third line of the source code 111 is replaced with the description of the metafunction "bop_add(2,x,1)". The meta function bop_add performs a mutation that changes the arithmetic operator "+" to "-".

Further, in the post-replacement source code 112, the description “x>0” on the fourth line of the source code 111 is replaced with the description of the metafunction “bop_gt(3,x,0)”. The meta function bop_gt performs a mutation that changes the relational operator “>” to “<=”.

Further, in the source code 112, the description “x*2” on the fifth line of the source code 111 is replaced with the description of the metafunction “bop_mul(4,x,2)”. The meta function bop_mul performs a mutation that changes the arithmetic operator “*” to “/” and “%”.

Note that the return value of the function func's return instruction is rounded to an integer (for example, the result of the operation "-1/2" is rounded to 0).
FIG. 7 is a diagram showing an example of a test program. The test program 113 is input to the analysis device 100 by the user and stored in the storage unit 110. The test program 113 includes three test cases. The first test case (testcase1) checks whether func(0)==0 holds (whether the output of the function func when 0 is input to the function func matches the check value 0). Indicates that The second test case (testcase2) indicates checking whether func(1)==2 holds. The third test case (testcase3) indicates checking whether func(-1)==0 holds.

FIG. 8 is a diagram showing an example of a mutation operator list. The mutation operator list 114 is input to the analysis device 100 by the user and stored in the storage unit 110. The mutation operator list 114 is a list of mutation operators applied during the test.

OSSN is a shift operator mutation, and indicates that a mutation of a shift operator (for example, a mutation that changes a left shift operation “<<=” to a right shift operation “>>>=”) is performed.

ORRN is Relational operator mutation, and indicates that mutation of a relational operator (for example, mutation that changes “>” to “<=”) is performed.
OIDO is Increment/Decrement Replacement, and performs a mutation that changes addition and subtraction (for example, a mutation that changes “++” or “+1” into “−−” or “−1”). Show.

OAAN is Arithmetic operator mutation, and indicates that a mutation that changes multiplication into an operation that obtains a division or a remainder (for example, a mutation that changes “*” to “/” or “%”) is performed.

The mutation operator list 114 may be considered as information for specifying the content of the mutation during the test.
FIG. 9 is a diagram showing an example of metamutation information. The metamutation information 115 is generated by the metafunction replacing unit 120 when the source code 112 after replacement is generated, and is stored in the storage unit 110. The metamutation information 115 includes items of MMID and source code position information.

The MMID is registered in the MMID item. In the item of source code position information, the file name of the source code 111 and the position information replaced with the meta function in the source code 111 are registered. Here, the file name of the source code 111 is “ac”.

For example, the metamutation information 115 includes information that the MMID is "1" and the source code position information is "ac: 2 lines 6 digits-2 lines 11 digits". This indicates that the description of “x<0” corresponding to 2nd row 6th digit-2nd row 11th digit in the source code 111 is replaced with the metafunction corresponding to the MMID “1”.

In the metamutation information 115, the MMID and the source code position information are registered in association with each other in the same manner for other mutation locations.
FIG. 10 is a diagram showing an example of the intermediate code. The intermediate code 116 is generated by the source code 112 after replacement being compiled by the compiler 130, and is stored in the storage unit 110. One line of intermediate code 116 corresponds to one instruction. The instruction of the intermediate code 116 is identified by an instruction ID (IDentifier). A variable represented by a combination of a “%” symbol such as “%1” or “%2” and a numerical value in the intermediate code 116 indicates a register, and the execution result of each instruction is stored.

For example, the instruction with the instruction ID “3” corresponds to the metafunction bop_lt(1, x, 0) on the second line of the source code 112 after replacement. A calculation result according to the mutation content is set in "%2". The set value of "%2" is "1" if the operation result is true, and "0" if it is false.

The instruction with the instruction ID "4" sets "1" to "%3" when the set value of "%2" is not equal to 0, and "%3" when the set value of "%2" is equal to 0 Is set to "0".
The instruction with the instruction ID “5” jumps to the instruction ID “6” if the setting value of “%3” is “1”, and to the instruction ID “10” if the setting value of “%3” is “0”. Indicates jumping.

Similarly, the intermediate code 116 includes an instruction group (instruction IDs “6” to “9”) corresponding to the third line of the source code 112 after replacement. The intermediate code 116 includes an instruction group (instruction ID “10” to “13”) corresponding to the fourth line of the source code 112 after replacement. The intermediate code 116 includes an instruction group (instruction ID “14” to “17”) corresponding to the fifth line of the source code 112 after replacement. However, the instruction corresponding to “return” in the replaced source code 112 is shared by the instruction IDs “21” to “23”. That is, the execution result is determined by comparing the value stored in the "%13" register with the confirmation value of the test case (Pass if both values match, Fail if they do not match).

FIG. 11 is a diagram showing an example of the execution state set. The execution state set 117 is generated by the decision tree creating unit 140 and stored in the storage unit 110. The execution state set 117 includes an execution state ID, an instruction ID, a mutation descriptor, and a memory item.

In the item of execution state ID, identification information of execution state (execution state ID) is registered. The execution state is represented by a combination of register setting values. In the instruction ID, the instruction ID of the instruction in the intermediate code 116 in which the state branch due to mutation occurs is registered. A mutation descriptor indicating the content of the mutation applied to the instruction corresponding to the corresponding instruction ID is registered in the mutation descriptor item. In the memory item, a combination of register setting values representing the execution state of the program (intermediate code 116) is registered.

For example, in the execution state set 117, information that the execution state ID is “2”, the instruction ID is “3”, the mutation descriptor is “1, >=”, and the memory is “%1=0” is registered. .. This indicates that in the execution state indicated by the execution state ID “2”, a state branch due to mutation occurs in the instruction with the instruction ID “3”. The execution state indicated by the execution state ID “2” indicates that the set value of the register is “%1=0” and the mutation descriptor at the location is (1, >=). Show. Here, the numerical value on the left side of the mutation descriptor is the MMID. The operator on the right side of the mutation descriptor indicates the mutation content.

In the execution state set 117, a plurality of state branches currently occurring in association with the test execution may be recorded. In the example of FIG. 11, in addition to the execution state ID “2”, the execution states corresponding to the execution state IDs “3”, “4”, and “5” are recorded (see the execution state of FIG. 31, which will be described later). Equivalent to).

FIG. 12 is a diagram showing an example of the mutant list. The mutant list 118 is generated by the meta function replacement unit 120 and stored in the storage unit 110. The mutant list 118 includes items of mutant ID and mutation descriptor.

Mutant identification information (mutant ID) is registered in the mutant ID item. A mutation descriptor is registered in the mutation descriptor item. Mutants are represented by a combination of mutation descriptors. The highest order of the mutant to be tested is specified by the user. As an example, it is assumed that the highest order of mutants, “3”, is specified (indicating that a mutant of third order or lower is created).

In this example, there are five types of FOM. The mutant ID is represented by a combination of the letter “M” and a subscript indicating a combination of numerical values for identifying the FOM (one or more numerical values are elements). The mutant with the mutant ID “M ₁ ”corresponds to the mutation descriptor (1, >=). Here, in the following, the mutant with the mutant ID “M ₁ ”may be expressed as “mutant M ₁ ”.

Mutant M ₂ is mutation descriptor (2, -) corresponding to. Mutant M ₃ corresponds to mutation descriptor (3,<=). Mutant M ₄ corresponds to mutation descriptor (4,/). Mutant M ₅ corresponds to the mutation descriptor (4%). Mutants M ₁ , M ₂ , M ₃ , M ₄ and M ₅ are FOMs.

The HOM is represented by a combination of FOM mutation descriptors. For example, the mutant M _1,2 corresponds to the mutation descriptor {(1, >=), (2, −)}. Further, for example, the mutants M _1,2,3 correspond to mutation descriptors {(1, >=), (2, −), (3, <=)}. In the mutant list 118, other mutants are similarly associated with the mutation descriptor. In this example, the total number of mutants is 21.

For example, the meta-function replacing unit 120 specifies a list of FOMs input by the user (a list showing the correspondence between mutant IDs of the mutants M ₁ , M ₂ , M ₃ , M ₄ , and M ₅ and mutation descriptors) and designation. The mutant list 118 may be created based on the order.

FIG. 13 is a diagram showing an example of the test result table. The test result table 119 is stored in the storage unit 110 and updated by the decision tree creating unit 140 and the result output unit 160. The test result table 119 includes items of mutant ID, Test1, Test2, Test3, and mutant survival result.

The mutant ID is registered in the mutant ID item. The execution result of the first test case (testcase1) is registered in the item Test1. The execution result of the second test case (testcase2) is registered in the item Test2. The execution result of the third test case (testcase3) is registered in the Test3 item. Information (test result) indicating whether or not the mutant has been killed by the test program 113 is registered in the mutant survival result item. As described above, if the execution result of at least one test case is “Failed”, the mutant survival result is “Killed”. If the execution result of any test case is “Passed”, the mutant survival result is “Unkilled”.

Here, the test result table 119 shows a state in which the result of the normal mutant analysis by SSE is registered.
For example, in the test result table 119, information that the mutant ID is “M ₁ ”, Test ₁ is “Failed”, Test 2 is “Passed”, Test 3 is “Passed”, and mutant survival result is “Killed” is registered. This indicates that the mutant M ₁ is detected by the test program 113.

In the test result table 119, execution results and mutant survival results of each test case are similarly registered for other mutant IDs. However, a place where "-" (hyphen symbol) is not set indicates that the execution result for the corresponding test case and the mutant survival result have not been acquired because the normal SSE does not try. Locations for which mutant survival results have not been obtained will be supplemented by the procedure described below.

FIG. 14 is a diagram showing an example of a decision tree. FIG. 14A shows a decision tree T10 for the first test case (testcase1). FIG. 14B shows a decision tree T20 for the second test case (testcase2). FIG. 14C shows a decision tree T30 for the third test case (testcase3). The decision trees T10, T20, T30 are represented by a directed graph.

Here, the decision tree T10 is represented as Test1(m1, m2, m3, m4). m1, m2, m3 and m4 are node identifiers. For example, the node with the identifier m1 is represented as “node m1”. The numerical value attached to the node identifier corresponds to the MMID. The edge starting from each node represents the mutation content at the mutation location indicated by the MMID of the corresponding node. When no mutation is performed, it corresponds to the edge of "nop".

FIG. 15 is a diagram showing an example of the result of merging decision trees. The decision tree T40 is a logical product of the decision trees T10, T20, T30 (Test1(m1, m2, m3, m4)∧Test2(m1,m2,m3,m4)∧Test3(m1,m2,m3,m4)). It is the result of merging with. The decision tree merging unit 150 generates a decision tree T40 by using the above-mentioned OBDD merge algorithm (details of the decision tree merge processing procedure will be described later).

FIG. 16 is a diagram showing a code example of a node/edge. The code C1 is a code for managing the nodes and edges belonging to the decision trees T10, T20, T30, T40, respectively. For example, the node has the MMI D (corresponding to mmid in FIG. 16) and is the next node according to the mutation operator (“nop”, “lt”, “le”, etc. defined in the 10th line of the code C1). Has a pointer to (corresponding to an edge). The terminal node (referred to as a leaf node) has either true (corresponding to "Pass") or false (corresponding to "Fail") depending on a boolean type variable "isPassed".

FIG. 17 is a diagram showing a data example of a decision tree. The data D1 indicates information equivalent to the decision tree T10. Specifically, in the node m1, the pointer to the node m3 is set for the mutation operator [nop]. In the node m1, a pointer to the node m2 is set for the mutation operator [ge] (corresponding to the operator “>=”).

At the node m2, a pointer to the leaf node of Fail is set for the mutation operator [nop]. At the node m2, a pointer to the leaf node of Fail is set for the mutation operator [sub] (corresponding to the operator "-").

In the node m3, a pointer to the leaf node of Pass is set for the mutation operator [nop]. In the node m3, a pointer to the node m4 is set for the mutation operator [le] (<=).

In the node m4, a pointer to the leaf node of Pass is set for the mutation operator [nop]. In the node m4, a pointer to the leaf node of Pass is set for the mutation operator [div] (corresponding to the operator “/”). At the node m4, a pointer to the leaf node of Pass is set for the mutation operator [rem] (corresponding to the operator "%").

The information of the decision trees T20, T30, T40 is also managed by the same data structure as the decision tree T10.
FIG. 18 is a diagram showing an output example of the result report. The result report R1 is an example of a result report output by the result output unit 160. The result report R1 represents, as an example, the result of the mutant analysis for the function func1. The result report R1 includes a calculation result of a mutation score (for example, "75%" (9/12) is displayed) and a list of undetected mutants (Unkilled mutants). One element of the list includes the source code location, the original code, and the mutant code. When a plurality of mutants are generated at the corresponding position, a plurality of mutants for the corresponding position may be displayed. The user can select any mutant from the list. The result output unit 160 causes the relevant part of the source code corresponding to the mutant selected by the user to be displayed in the result report R1. For example, the result output unit 160 assists the user in recognizing a problematic part by highlighting the corresponding part or by underlining the corresponding part. The user can check the result report R1 and review the test contents to improve the mutation score.

Next, a processing procedure of mutation analysis by the analysis device 100 will be described.
FIG. 19 is a flowchart showing a processing example of the analysis device. Hereinafter, the process illustrated in FIG. 19 will be described in order of step number.

(S1) The meta function replacement unit 120 receives an input of the source code 111. Then, the metafunction replacing unit 120 embeds the metafunction in the source code 111. Specifically, the meta-function replacing unit 120 replaces the code corresponding to the operation using the predetermined operator determined in advance with the code of the meta-function corresponding to the operation. As a result, the meta function replacement unit 120 generates the replaced source code 112 and stores it in the storage unit 110. The meta function replacement unit 120 generates the mutant list 118 according to the embedded meta function and stores it in the storage unit 110. Further, the meta function replacement unit 120 generates meta mutation information 115 and stores it in the storage unit 110. At this time, the meta-function replacement unit 120 may receive the input of the mutation operator list 114, the FOM list, and the order of the mutation by the user, and specify the mutation descriptor of each mutant.

(S2) The compiler 130 compiles the replaced source code 112 into the intermediate code 116. The compiler 130 stores the intermediate code 116 generated by the compilation in the storage unit 110.

(S3) The decision tree creating unit 140 performs a decision tree creating process. Specifically, the decision tree creating unit 140 uses the intermediate code 116 to execute a test by the test program 113 and generate information on the decision trees T10, T20, T30. The decision tree creating unit 140 stores information on the decision trees T10, T20, T30 in the storage unit 110. The decision tree generator 140 uses the SSE technique for test execution. The decision tree creating unit 140 updates the test result table 119 stored in the storage unit 110 along with the test execution. Further, the decision tree creating unit 140 receives in advance a user's input of the mutation operator list 114 indicating the content of the mutation to be made to function at the time of testing, and stores it in the storage unit 110. In this example, it is assumed that at least ORRN, OIDO, and OAAN are specified by the mutation operator list 114. Details of the decision tree creation process will be described later.

(S4) The decision tree merging unit 150 performs a decision tree merge process. Specifically, the decision tree merging unit 150 merges the decision trees T10, T20, and T30 to generate information on the decision tree T40. The decision tree merging unit 150 stores the information of the decision tree T40 in the storage unit 110. Details of the decision tree merge processing will be described later.

(S5) The result output unit 160 performs result report output processing. Specifically, the result output unit 160 refers to the information of the decision tree T40 stored in the storage unit 110 and complements the registered content of the test result table 119. The result output unit 160 generates a result report based on the test result table 119 and causes the display 11 to display the result report.

FIG. 20 is a flowchart showing an example of decision tree creation processing. Hereinafter, the process illustrated in FIG. 20 will be described in order of step number. The process described below corresponds to step S3 in FIG.
(S11) The decision tree creating unit 140 performs initialization processing. Specifically, the decision tree creating unit 140 prepares to newly create a decision tree based on a new test case. Details of the initialization process will be described later.

(S12) The decision tree creating unit 140 selects an execution state by referring to the execution state set 117 stored in the storage unit 110. When step S12 is first executed for a certain test case, the initial execution state prepared in the above-described initialization processing is selected. The initial execution state is the execution state ID “1”, and the initial instruction ID, mutation descriptor, and memory (group of registers) are not set. Note that the decision tree creating unit 140 updates the instruction ID for the execution state ID of interest and the recorded content in the memory as the intermediate code 116 is executed.

(S13) The decision tree creating unit 140 extracts the next instruction from the intermediate code 116 stored in the storage unit 110. As mentioned above, an instruction is identified by an instruction ID.
(S14) The decision tree creating unit 140 determines whether the fetched instruction is a metafunction call. If the call is a meta function, the process proceeds to step S16. If not, the process proceeds to step S15.

(S15) The decision tree creating unit 140 executes the original instruction extracted in step S13. Then, the decision tree creating unit 140 advances the process to step S19.
(S16) The decision tree generator 140 determines whether the MMID included in the metafunction is an MMID that has not been recorded in the current execution state. If the MMID is unrecorded in the current execution state, the process proceeds to step S17. If the MMID is not unrecorded in the current execution state (that is, the MMID is recorded in the current execution state), the process proceeds to step S18.

(S17) The decision tree creating unit 140 performs decision tree internal node addition processing. Details of the decision tree internal node addition processing will be described later.
(S18) The decision tree creating unit 140 executes the instruction of the mutation descriptor (may be "nop") corresponding to the fetched instruction. That is, the decision tree generator 140 applies the mutation content indicated by the mutation descriptor to the fetched instruction and executes the corresponding instruction. When "nop" is applied, the instruction is executed without performing mutation on the instruction (the original instruction fetched in step S13).

(S19) The decision tree creating unit 140 refers to the intermediate code 116 and determines whether or not the next instruction is empty. If it is empty (that is, the instruction ID “24” is reached), the process proceeds to step S20. If not empty (that is, the instruction ID “24” has not been reached), the process proceeds to step S13.

(S20) The decision tree creation unit 140 performs decision tree leaf node addition processing. Details of the decision tree leaf node addition processing will be described later.
(S21) The decision tree creating unit 140 determines whether the execution state set 117 is empty. If it is empty, the process proceeds to step S22. If not empty, the process proceeds to step S12.

(S22) The decision tree generator 140 determines whether or not the next test is empty. If the next test is empty (ie, all test cases have been executed), the process ends. If the next test is not empty (that is, there is an unexecuted test case), the process proceeds to step S11.

FIG. 21 is a flowchart showing an example of initialization processing. Hereinafter, the process illustrated in FIG. 21 will be described in order of step number. The process described below corresponds to step S11 of FIG.
(S31) The decision tree generator 140 inputs the next test to the intermediate code 116. The next test refers to the test case to be executed next among the three test cases included in the test program 113. For example, the decision tree creating unit 140 selects the next test in the order of the first test case (testcase1), the second test case (testcase2), and the third test case (testcase3).

(S32) The decision tree creating unit 140 prepares an initial execution state. As described above, the initial execution state is the execution state ID “1”, and initially, the instruction ID, the mutation descriptor, and the memory (set of registers) are not set. The decision tree creating unit 140 registers the initial execution state in the execution state set 117.

(S33) The decision tree creating unit 140 newly prepares an empty decision tree. The prepared decision tree does not have a node or an edge at the stage of step S33.
FIG. 22 is a flowchart showing an example of decision tree internal node addition processing. In the following, the process illustrated in FIG. 22 will be described in order of step number. The process described below corresponds to step S17 in FIG.

(S41) The decision tree creating unit 140 copies the original execution state by the number of mutation operators corresponding to the meta function, generates a new execution state, and registers it in the execution state set 117. The decision tree creating unit 140 can specify the mutation operator and the number of mutation operators (here, “nop” may not be included) from the mutation operator list 114 stored in the storage unit 110.

(S42) The decision tree creating unit 140 records the MMID and the mutation operator corresponding to the metafunction in the mutation descriptor of the new execution state.
(S43) The decision tree creation unit 140 executes the current position identifying process (M) with M as the mutation descriptor of the current execution state. The details of the current position specifying process of the decision tree will be described later. However, if the decision tree is empty in this test case, the current position indicates the starting point of the decision tree.

(S44) The decision tree creating unit 140 adds the MMID as a node to the current position of the decision tree, and adds the corresponding mutation operator as an edge from the node. When the decision tree is empty, the current position is the starting point of the decision tree. In that case, in step S44, the node added first (at the starting point of the decision tree) to the test case this time is the root node. After that, internal nodes are sequentially added to the current position of the decision tree (beyond the edge).

FIG. 23 is a flowchart showing an example of decision tree leaf node addition processing. The process illustrated in FIG. 23 will be described below in order of step number. The process described below corresponds to step S20 of FIG.

(S51) The decision tree creating unit 140 records the execution result in the test result table 119. As illustrated in the test result table 119, the execution result is represented by a combination of a mutant ID, a test ID (test case ID), and a test pass/fail (Pass or Fail). However, the decision tree creating unit 140 records the test pass/fail in the test result table 119 when no mutation is performed for the addition of the leaf node this time (when the original program is executed instead of the mutant). You don't have to.

(S52) The decision tree creation unit 140 executes the current position identifying process (M) with the mutation descriptor of the current execution state as M. The details of the current position specifying process of the decision tree will be described later.

(S53) The decision tree creation unit 140 adds a leaf node representing a test pass/fail (Pass or Fail) to the current position of the decision tree.
(S54) The decision tree creating unit 140 deletes the current execution state from the execution state set 117.

FIG. 24 is a flowchart showing an example of the current position identification processing of a decision tree. In the following, the process illustrated in FIG. 24 will be described in order of step number. The processing described below corresponds to step S43 in FIG. 22 and step S52 in FIG. 23 (however, it is also used when outputting a result report described later).

(S61) The decision tree creating unit 140 selects the root node of the decision tree (decision tree of interest) corresponding to the current test case.
(S62) The decision tree generator 140 determines whether or not the MMID of the currently selected node is included in the input mutation descriptor M. If included, the process proceeds to step S63. If not included, the process proceeds to step S65.

(S63) From the mutation descriptor M, the decision tree creating unit 140 acquires the mutation operator corresponding to the MMID of the currently selected node.
(S64) The decision tree creating unit 140 selects an edge corresponding to the acquired mutation operator. Then, the decision tree creating unit 140 advances the process to step S66.

(S65) The decision tree creating unit 140 selects an edge corresponding to “nop”.
(S66) The decision tree creating unit 140 determines whether the edge selected in step S64 or step S65 has no node ahead of the edge, or whether the node ahead of the edge is a leaf node. .. If there is no node at the end of the corresponding edge or if the node at the end of the corresponding edge is a leaf node, the process proceeds to step S68. When there is a node other than the leaf node at the tip of the relevant edge, the process proceeds to step S67.

(S67) The decision tree generator 140 selects the node at the tip of the corresponding edge. Then, the decision tree creating unit 140 advances the process to step S62.
(S68) The decision tree generator 140 sets the tip of the corresponding edge as the current position.

In this way, the decision tree creation unit 140 creates the decision trees T10, T20, T30 for each test case.
FIG. 25 is a flowchart showing an example of decision tree merge processing. Hereinafter, the process illustrated in FIG. 25 will be described in order of step number. The process described below corresponds to step S4 in FIG. Here, the input of the decision tree merge processing is (T ₁ , T ₂ , n). T ₁ represents the first decision tree (or a subtree within the decision tree). T ₂ represents the second decision tree (or a subtree within the decision tree). If the subtree has only leaf nodes, T ₁ and T ₂ may represent leaf nodes. n is the MMID. In this example, the initial value of n is 1. This decision tree merge processing is recursively called and executed in steps S79 and S84 shown below.

(S71) The decision tree merging unit 150 prepares the decision tree Tout to be output. The initial value of the decision tree Tout is empty.
(S72) The decision tree merging unit 150 determines whether T ₁ or T ₂ is a leaf node of Fail. If T ₁ or T ₂ is a Fail leaf node, the process proceeds to step S73. If neither T ₁ nor T ₂ is a Fail leaf node, the process proceeds to step S74.

(S73) The decision tree merging unit 150 sets Tout as a leaf node of Fail. Then, the decision tree merging unit 150 advances the processing to step S86.
(S74) The decision tree merging unit 150 determines whether T ₁ is a leaf node of Pass. If T ₁ is a Pass leaf node, the process proceeds to step S75. If T ₁ is not a Pass leaf node, the process proceeds to step S76.

(S75) The decision tree merging unit 150 sets Tout=T ₂ . Then, the decision tree merging unit 150 advances the processing to step S86.
(S76) The decision tree merging unit 150 determines whether T ₂ is a leaf node of Pass. If T ₂ is a Pass leaf node, the process proceeds to step S77. If T ₂ is not a leaf node of Pass, the process proceeds to step S78.

(S77) The decision tree merging unit 150 sets Tout=T ₁ . Then, the decision tree merging unit 150 advances the processing to step S86.
(S78) The decision tree merge unit 150 determines whether or not Nn does not exist in T ₁ and T ₂ . Here, Nn indicates a node whose MMID is n (may be referred to as a node Nn). If Nn does not exist for both T ₁ and T ₂ , the process proceeds to step S79. If Nn exists in both or either of T ₁ and T ₂ , the process proceeds to step S80.

(S79) The decision tree merge unit 150 sets Tout=decision tree merge process (T ₁ , T ₂ , n+1). That is, the decision tree merge unit 150 adds 1 to n, executes the decision tree merge process, and sets the result as Tout. Then, the decision tree merging unit 150 advances the processing to step S86.

(S80) The decision tree merging unit 150 adds the node Nn to Tout.
(S81) The decision tree merging unit 150 sets m as a mutation operator (including nop) when the MMID is n, and repeatedly executes the processing up to step S85 for each m.

(S82) The decision tree merging unit 150 acquires T ₁ | _[n,m] . T ₁ | _{[n, m]} is a tree replaced Nn in T ₁ (the subtree was rooted) e _n in T _1, ahead of the subtree of _m. Here, e _n,m indicates the edge of the mutation operator m which is output from the node Nn. If there is no Nn in T _1, T ₁ | and _{[n, m] = T 1} .

(S83) The decision tree merging unit 150 acquires T ₂ | _[n,m] . T ₂ | _{[n, m]} is a tree by replacing the Nn in T ₂ (the subtree that is rooted) e _n in T _2, ahead of the subtree of _m. If there is no Nn in _{T 2, T 2 | [n} , m] and = T ₂ to.

(S84) The decision tree merge unit 150 adds the output of the decision tree merge process (T ₁ | _[n,m] , T ₂ | _[n,m] , n+1) to the destination of e _n,m in Tout. ..
(S85) When the decision tree merging unit 150 finishes the processing from step S81 onward for all m for MMID=n, the processing proceeds to step S86.

(S86) The decision tree merging unit 150 outputs Tout.
In this way, the decision tree merging unit 150 merges two decision trees by recursively executing the decision tree merging process. When merging the three decision trees T10, T20, T30, the decision tree merging unit 150 first merges the two decision trees (for example, the decision trees T20, T30) by the procedure of the decision tree merge process. Next, the decision tree merging unit 150 merges the decision tree obtained as the result of the merging and the remaining one decision tree (for example, decision tree T10) by the procedure of the above decision tree merging process. In this way, the decision tree merging unit 150 obtains a decision tree T40 which is the merged result of the three decision trees T10, T20, T30. The same applies when merging four or more decision trees. For example, when four decision trees are merged, the decision tree merge process may be performed on the decision tree T40 that is the result of merging the three decision trees and the remaining one decision tree.

FIG. 26 is a flowchart showing an example of the result report output processing. In the following, the process illustrated in FIG. 26 will be described in order of step number. The process described below corresponds to step S5 of FIG.

(S91) In the test result table 119, the result output unit 160 sets the mutant survival result of the mutant whose execution result by any test case includes Fail as Killed.

(S92) The result output unit 160 repeatedly executes the processes up to step S97, where M is the mutation descriptor of the mutant whose survival result is undetermined.
(S93) The result output unit 160 identifies the leaf node by the current position identifying process (M) for the decision tree T40. Specifically, the result output unit 160 causes the decision tree creating unit 140 to execute the decision tree current position specifying process (M) for the decision tree T40, and obtains the result (the leaf node is specified by the process). Will be). The procedure for determining the current position of the decision tree is similar to the procedure illustrated in FIG. The result output unit 160 may perform the current position specifying process of the decision tree.

(S94) The result output unit 160 determines whether or not the previous leaf node at the current position of the decision tree T40 is a Fail leaf node. If it is a Fail leaf node, the process proceeds to step S95. If it is not a Fail leaf node (that is, if it is a Pass leaf node), the process proceeds to step S96.

(S95) The result output unit 160 determines the mutant survival result of the mutant corresponding to the mutation descriptor M to be Killed, and records Killed in the relevant portion of the test result table 119. The reason for reaching Fail in the decision tree T40 is that the mutant corresponding to the mutation descriptor M becomes Fail in at least one of the three test cases. Then, the process proceeds to step S97.

(S96) The result output unit 160 determines that the mutant survival result of the mutant corresponding to the mutation descriptor M is Unkilled, and records Unkilled in the relevant part of the test result table 119. The reason for reaching Pass in the decision tree T40 is that the Mutant corresponding to the mutation descriptor M is Pass in all three test cases.

(S97) When the result output unit 160 finishes the processes from step S92 onward for all Ms, the process proceeds to step S98.
(S98) The result output unit 160 calculates the mutation score (MS) based on the mutant survival result in the test result table 119. The mutation score is the ratio of the number of Killed to the total number of mutants.

(S99) The result output unit 160 restores the mutant position information using the metamutation information 115 stored in the storage unit 110.
(S100) The result output unit 160 outputs the mutation score (MS) and the Unkilled mutant as a result report. For example, the result output unit 160 causes the display 11 to display a result report. Moreover, when the Unkilled mutant is selected by the user, the corresponding portion of the source code corresponding to the mutant is displayed on the display 11.

In this way, the result output unit 160 outputs a result report according to the first test result acquired by executing the test and the second test result determined based on the decision trees T10, T20, T30. .. As described above, the result report may include information (for example, a mutation score (MS)) about a mutant detectable by a plurality of test cases among a plurality of mutants (a plurality of combinations of mutation points). In addition, the result report may include information about mutants that cannot be detected by a plurality of test cases (for example, a list of Unkilled mutants or information on a relevant part of the source code).

Next, a specific example of the processing of the analysis device 100 according to the above procedure will be described. First, a method of creating the decision tree T10 according to the procedure of FIG. 20 will be specifically described.
FIG. 27 is a diagram illustrating an example (part 1) of creating a decision tree. Below, the 1st test case (testcase1) corresponding to test ID "Test1" is assumed. According to the test program 113, func(0)==0 is confirmed in the first test case. In the example of creating the decision tree after FIG. 27, the source code 111 is also illustrated so that the processing content by the intermediate code 116 can be easily understood. The instruction being referred to in the intermediate code 116 and the corresponding portion of the source code 111 corresponding to the instruction are shown surrounded by a rectangle.

First, the decision tree creating unit 140 selects the initial execution state (execution state of execution state ID “1”), and then reaches the instruction of instruction ID “3” in the intermediate code 116. The instruction with the instruction ID “3” includes the meta function bop_lt. MMID=1 indicated by the first argument of the meta function bop_lt is an MMID that has not been recorded in the execution state of the execution state ID “1”. Therefore, the decision tree creating unit 140 duplicates the execution state of the execution state ID “1” at this stage and sets the execution state of the execution state ID “2” (instruction ID=3 at this stage) to the execution state set 117. to add. In the figure, the execution state of the execution state ID “X” is represented as “StateX”. The mutation descriptor in the execution state of the execution state ID “1” is “nop” (indicating that the change is not made). The mutation descriptor in the execution state of the execution state ID “2” is (1, >=).

Further, the decision tree creating unit 140 adds the node m1 with MMID=1 as the root node to the decision tree T10. The decision tree creating unit 140 adds an edge corresponding to “nop” and an edge corresponding to “>=” to the node m1.

The decision tree creating unit 140 selects the edge corresponding to the "nop" of the node m1 and sets the tip of the edge as the current position. In this case, the decision tree creating unit 140 executes the instruction with the instruction ID “3” in the original operation “<”.

FIG. 28 is a diagram illustrating an example (No. 2) of creating a decision tree. The decision tree creating unit 140 continues execution of the intermediate code 116 and reaches the instruction with the instruction ID “11”. The instruction with the instruction ID “11” includes the meta function bop_gt. MMID=3 indicated by the first argument of the meta function bop_gt is an MMID that has not been recorded in the execution state of the execution state ID “1”. Therefore, the decision tree creating unit 140 duplicates the execution state of the execution state ID “1” at this stage and sets the execution state of the execution state ID “3” (instruction ID=11 at this stage) to the execution state set 117. to add. The mutation descriptor in the execution state of the execution state ID “3” is (3,<=).

Further, the decision tree creating unit 140 adds the node m3 with MMID=3 to the end of the edge corresponding to the "nop" of the node m1. The decision tree creating unit 140 adds an edge corresponding to “nop” and an edge corresponding to “<=” to the node m3.

The decision tree creating unit 140 selects the edge corresponding to the "nop" of the node m3 and sets the tip of the edge as the current position. In this case, the decision tree creating unit 140 executes the instruction with the instruction ID “11” in the original operation “>”.

FIG. 29 is a diagram illustrating an example (part 3) of creating a decision tree. The decision tree creating unit 140 continues to execute the intermediate code 116, and confirms that %13=0 in the execution state of the execution state ID “1”, and func(0)==0 holds. Therefore, the decision tree creating unit 140 adds the leaf node of Pass to the end of the edge corresponding to “nop” of the node m3. Then, the decision tree creating unit 140 deletes the execution state with the execution state ID “1” from the execution state set 117.

Next, the decision tree creating unit 140 selects the execution state with the execution state ID “3” from the execution states registered in the execution state set 117. Here, the decision tree creating unit 140 uses the SSE method. The order of execution state selection by the decision tree creating unit 140 is based on the depth-first search algorithm (since the execution state IDs are assigned in ascending order, the execution state IDs are selected in descending order). However, another algorithm may be used to select the next execution state.

The decision tree creating unit 140 refers to the information of the execution state ID “3”, returns to the instruction of the instruction ID “11” in the intermediate code 116, and selects the edge corresponding to the mutation descriptor (3,<=). Then, the tip of the edge is set as the current position. In this case, the decision tree creating unit 140 changes the instruction with the instruction ID “11” into the operation of “<=” and executes the operation.

FIG. 30 is a diagram showing an example (part 4) of creating a decision tree. The decision tree creating unit 140 continues execution of the intermediate code 116 and reaches the instruction with the instruction ID “15”. The instruction with the instruction ID “15” includes the meta function bop_mul. MMID=4 indicated by the first argument of the meta function bop_mul is an MMID that has not been recorded in the execution state of the execution state ID “3”.

Therefore, the decision tree creating unit 140 duplicates the execution state of the execution state ID “3” at this stage and sets the execution state of the execution state ID “4” (instruction ID=15 at this stage) to the execution state set 117. to add. The mutation descriptor in the execution state with the execution state ID “4” is {(3,<=),(4,/)}.

Further, the decision tree creating unit 140 duplicates the execution state of the execution state ID “3” at this stage, and sets the execution state of the execution state ID “5” (instruction ID=15 at this stage) to the execution state set 117. to add. The mutation descriptor in the execution state of the execution state ID “5” is {(3,<=),(4,%)}.

Further, the decision tree creating unit 140 adds the node m4 with MMID=4 to the end of the edge corresponding to “<=” of the node m3. The decision tree creating unit 140 adds an edge corresponding to “nop”, an edge corresponding to “/”, and an edge corresponding to “%” to the node m4.

The decision tree creating unit 140 selects an edge corresponding to “nop” of the node m4 and sets the tip of the edge as the current position. In this case, the decision tree creating unit 140 executes the instruction with the instruction ID “15” in the original operation “*”.

FIG. 31 is a diagram illustrating an example (No. 5) of creating a decision tree. The decision tree creating unit 140 continues execution of the intermediate code 116, and confirms that %13=0 and func(0)==0 holds in the execution state of the execution state ID “3”. The decision tree creating unit 140 records “Passed” for the mutant ID “M ₃ ”and the test ID “Test1” of the test result table 119. The decision tree creating unit 140 adds a leaf node of Pass to the end of the edge corresponding to "nop" of the node m4. Then, the decision tree creating unit 140 deletes the execution state with the execution state ID “3” from the execution state set 117.

Next, the decision tree creating unit 140 selects the execution state with the execution state ID “5” from the execution states registered in the execution state set 117.
The decision tree creating unit 140 refers to the information of the execution state ID “5”, returns to the instruction of the instruction ID “15” in the intermediate code 116, and selects the edge corresponding to the mutation descriptor (4%). , The tip of the edge is the current position. In this case, the decision tree creating unit 140 changes the instruction with the instruction ID “15” into the operation of “%” and executes the operation.

FIG. 32 is a diagram illustrating an example (No. 6) of creating a decision tree. The decision tree creating unit 140 continues to execute the intermediate code 116, and confirms that %13=0 and func(0)==0 holds in the execution state of the execution state ID “5”. The decision tree creating unit 140 records “Passed” for the mutant ID “M _3,5 ” and the test ID “Test1” in the test result table 119. The decision tree creating unit 140 adds the leaf node of Pass to the end of the edge corresponding to “%” of the node m4. Then, the decision tree creating unit 140 deletes the execution state with the execution state ID “5” from the execution state set 117.

Next, the decision tree creating unit 140 selects the execution state with the execution state ID “4” from the execution states registered in the execution state set 117.
The decision tree creating unit 140 refers to the information of the execution state ID “4”, returns to the instruction of the instruction ID “15” in the intermediate code 116, and selects the edge corresponding to the mutation descriptor (4,/). , The tip of the edge is the current position. In this case, the decision tree creating unit 140 changes the instruction of the instruction ID “15” into the operation of “/” and executes it.

FIG. 33 is a diagram showing an example (No. 7) of creating a decision tree. The decision tree creating unit 140 continues execution of the intermediate code 116, and confirms that %13=0 and func(0)==0 holds in the execution state of the execution state ID “4”. The decision tree creating unit 140 records “Passed” for the mutant ID “M _3,4 ”and the test ID “Test 1” of the test result table 119. The decision tree generator 140 adds a leaf node of Pass to the end of the edge corresponding to “/” of the node m4. Then, the decision tree creating unit 140 deletes the execution state with the execution state ID “4” from the execution state set 117.

Next, the decision tree creating unit 140 selects the execution state of the execution state ID “2” registered in the execution state set 117.
The decision tree creating unit 140 refers to the information of the execution state ID “2”, returns to the instruction of the instruction ID “3” in the intermediate code 116, and selects the edge corresponding to the mutation descriptor (1, >=). Then, the tip of the edge is set as the current position. In this case, the decision tree creating unit 140 changes the instruction with the instruction ID “3” to the operation “>=” and executes the operation.

FIG. 34 is a diagram illustrating an example (No. 8) of creating a decision tree. The decision tree creating unit 140 continues execution of the intermediate code 116 and reaches the instruction with the instruction ID “7”. The instruction with the instruction ID “7” includes the meta function bop_add. MMID=2 indicated by the first argument of the meta function bop_add is an MMID that has not been recorded in the execution state of the execution state ID “2”. Therefore, the decision tree creating unit 140 duplicates the execution state of the execution state ID “2” at this stage and sets the execution state of the execution state ID “6” (instruction ID=7 at this stage) to the execution state set 117. to add. The mutation descriptor in the execution state with the execution state ID “6” is {(1, >=), (2, −)}.

In addition, the decision tree creating unit 140 adds the node m2 with MMID=2 to the end of the edge corresponding to “>=” of the node m1. The decision tree creating unit 140 adds an edge corresponding to “nop” and an edge corresponding to “−” to the node m2.

The decision tree creating unit 140 selects an edge corresponding to "nop" of the node m2, and sets the tip of the edge as the current position. In this case, the decision tree creating unit 140 executes the instruction with the instruction ID “7” in the original operation “+”.

FIG. 35 is a diagram showing an example (No. 9) of creating a decision tree. The decision tree creating unit 140 continues execution of the intermediate code 116, and confirms that %13=1 and func(0)==0 is not established in the execution state of the execution state ID “2”. The decision tree creating unit 140 records “Failed” for the mutant ID “M ₁ ”and the test ID “Test ₁ ” in the test result table 119. The decision tree creating unit 140 adds a Fail leaf node to the end of the edge corresponding to the "nop" of the node m2. Then, the decision tree creating unit 140 deletes the execution state with the execution state ID “2” from the execution state set 117.

Next, the decision tree creating unit 140 selects the execution state of the execution state ID “6” registered in the execution state set 117.
The decision tree creating unit 140 refers to the information of the execution state ID “6”, returns to the instruction of the instruction ID “7” in the intermediate code 116, and selects the edge corresponding to the mutation descriptor (2, −). , The tip of the edge is the current position. In this case, the decision tree creating unit 140 changes the instruction with the instruction ID “7” into the operation with “−” and executes the operation.

FIG. 36 is a diagram illustrating an example (No. 10) of creating a decision tree. The decision tree creating unit 140 continues to execute the intermediate code 116, and in the execution state of the execution state ID “6”, %13=−1, and confirms that func(0)==0 is not established. The decision tree creating unit 140 records “Failed” for the mutant ID “M _1,2 ” and the test ID “Test1” in the test result table 119. The decision tree creating unit 140 adds the leaf node of Fail to the end of the edge corresponding to "-" of the node m2. Then, the decision tree creating unit 140 deletes the execution state with the execution state ID “6” from the execution state set 117.

At this stage, the execution states registered in the execution state set 117 disappear. Therefore, the decision tree creating unit 140 completes the execution of the test case with the test ID “Test1”. The decision tree creating unit 140 also completes the creation of the decision tree T10. The decision tree creating unit 140 selects the test case with the next test ID “Test2”. The decision tree creating unit 140 creates decision trees T20 and T30 in the same manner for other test cases.

Next, an example of merging two decision trees will be described. Below, the case where the decision trees T20 and T30 are merged is illustrated.
FIG. 37 is a diagram showing an example (part 1) of merging decision trees. FIG. 37A shows the decision tree T20 (Test2(m1, m2, m3, m4)) created by the decision tree creating unit 140. FIG. 37B shows the decision tree T30 (Test3(m1, m2, m3, m4)) created by the decision tree creating unit 140. The decision tree creating unit 140 prepares the decision tree Tout(1). Here, Tout that appears due to the recursive execution of the decision tree merging process is represented by Tout(1), Tout(2),... The decision tree Tout(1) is empty at this stage.

First, the input of the decision tree merge processing is (T ₁ , T ₂ , n)=(Test2(m1, m2, m3, m4), Test3(m1, m2, m3, m4), 1). Neither T ₁ =Test2 (m1, m2, m3, m4) nor T ₂ =Test3 (m1, m2, m3, m4) is a tree of Fail leaf nodes only (step S72 No). Moreover, neither T ₁ =Test2 (m1, m2, m3, m4) nor T ₂ =Test3 (m1, m2, m3, m4) is a tree having only Pass leaf nodes (No in step S74 and step S76). No). Furthermore, at T ₁ =Test2 (m1, m2, m3, m4) and T ₂ =Test3 (m1, m2, m3, m4), a node m1 corresponding to N1 exists (step S78 No).

Therefore, the decision tree merging unit 150 adds the node m1 to Tout(1) (step S80).
FIG. 38 is a diagram illustrating an example (part 2) of merging decision trees. The mutation operators in the node m1 are “>=” and “nop”. Therefore, the decision tree merging unit 150 executes steps S81 to S85 for these two mutation operators. First, consider “>=”.

In this case, T ₁ | _[1,>=] =Test2| _[1,>=] (m2,m3,m4) is the subtree at the end of the edge corresponding to “>=” of the node m1 of the decision tree T20. It is T21 (step S82).

Further, T ₂ | _[1,>=] =Test3| _[1,>=] (m2, m3, m4) is a subtree T31 at the end of the edge corresponding to “>=” of the node m1 of the decision tree T30. (Step S83).
The decision tree merging unit 150 executes the decision tree merging process with the input (T ₁ | _[1,>=] , T ₂ | _[1,>=] , 2). The decision tree merging unit 150 determines the decision tree merge process (T ₁ | _{[1, >=]} ) before the edge (e _1,>= ) corresponding to “>=” of the node m1 in the decision tree Tout(1) _. , T ₂ | _{[1, >=]} , 2) output Tout(2) is added (step S84).

Next, consider “nop”.
In this case, T ₁ | _[1,nop] =Test2| _[1,nop] (m2, m3, m4) is the subtree T22 ahead of the edge corresponding to “nop” of the node m1 of the decision tree T20. (Step S82).

Further, T ₂ | _[1,nop] =Test3| _[1,nop] (m2, m3, m4) is the subtree T32 at the end of the edge corresponding to the “nop” of the node m1 of the decision tree T30 ( Step S83).

The decision tree merging unit 150 executes the decision tree merging process with the input (T ₁ | _[1,nop] , T ₂ | _[1,nop] , 2), and outputs the decision tree Tout( 1) It is added to the end of the edge (e _1,nop ) corresponding to the “nop” of the node m1 in (step S84).

Next, an example of merging the above-mentioned subtrees T21 and T31 (decision tree merge processing (T ₁ | _[1,>=] , T ₂ | _[1,>=] , 2)) will be described.
The decision tree creating unit 140 prepares the decision tree Tout(2). The decision tree Tout(2) is empty at this stage.

Neither T ₁ | _[1,>=] nor T ₂ | _[1,>=] is a tree of Fail leaf nodes only (step S 72 No). Further, neither T ₁ | _[1,>=] or T ₂ | _[1,>=] is a tree having only leaf nodes of Pass (step S 74 No and step S 76 No). Furthermore, a node m2 corresponding to N2 exists in T ₁ | _{[1, >=]} (No in step S78).

Therefore, the decision tree merging unit 150 adds the node m2 to the decision tree Tout(2) (step S80).
FIG. 39 is a diagram illustrating an example (part 3) of merging decision trees. The mutation operators at node m2 are "-" and "nop". Therefore, the decision tree merging unit 150 executes steps S81 to S85 for these two mutation operators. First, consider "-".

In this case, T ₁ | _[1,>=],[2,-] =Test2| _[1,>=],[2,-] (m3,m4) is "-" of the node m2 of the subtree T21. It is the partial tree T21a at the end of the edge corresponding to (step S82). The subtree T21a has only a leaf node of Fail as an element.

Further, the subtree T31 does not have the node m2. Therefore, T ₂ | _[1,>=],[2,-] =Test3| _[1,>=],[2,-] (m3,m4) is a subtree having the same structure as the subtree T31. It is T31a (step S83).

The decision tree merging unit 150 takes the input as (T ₁ | _[1,>=],[2,-] , T ₂ | _[1,>=],[2,-] , 3) and makes a decision tree merge process. To execute. The decision tree merging unit 150 ends the decision tree merging process (T ₁ | _{[1, >=],[]} before the edge (e _2,− ) corresponding to “−” of the node m2 in the decision tree Tout(2) _. The output Tout(3) of _2,-] , T ₂ | _{[1,>=], [2,-]} , 3) is added (step S84).

In this case, T ₁ | _{[1, >=], [2,-]} indicates a subtree T21a having only Fail leaf nodes, and thus Tout(3) is a Fail leaf node (step S72 Yes, S73, S86).

Next, consider “nop”.
In this case, T ₁ | _{[1,>=],[2,nop]} =Test2| _{[1,>=],[2,nop]} (m3,m4) is the “nop” of the node m2 of the subtree T21. It is the partial tree T21b at the end of the edge corresponding to (step S82). The subtree T21b has only leaf nodes of Pass as elements.

Further, the subtree T31 does not have the node m2. Therefore, T ₂ | _{[1,>=],[2,nop]} =Test3| _{[1,>=],[2,nop]} (m3,m4) is a subtree having the same structure as the subtree T31. It is T31b (step S83).

The decision tree merging unit 150 takes the input as (T ₁ | _{[1,>=],[2,nop]} , T ₂ | _{[1,>=],[2,nop]} ,3) _{, and performs} the decision tree merge process. To execute. The decision tree merging unit 150 precedes the edge (e _2,nop ) corresponding to the “nop” of the node m2 in the decision tree Tout(2) with the decision tree merge process (T ₁ | _{[1, >=],[ 2,nop]} , T ₂ | _{[1,>=],[2,nop]} , 3) output Tout(4) is added (step S84).

In this case, T ₁ | _{[1,>=],[2,nop]} and T ₂ | _{[1,>=],[2,nop]} are not trees of Fail leaf nodes only (step S72 No).
On the other hand, T ₁ | _{[1,>=],[2,nop]} indicates a subtree T21b having only leaf nodes of Pass, so Tout(4)=T ₂ | _{[1,>=],[2 , nop]} (step S74 Yes, S75, S86).

FIG. 40 is a diagram showing an example (part 4) of merging decision trees. The tree T51 is a tree in the middle of merging the decision trees T20 and T30, and shows the stage in which the result of the merge processing illustrated in FIG. 39 (decision tree Tout(2)) is added to the decision tree Tout(1). ..

The decision tree merging unit 150 similarly performs the merge process on the edge of the edge corresponding to the "nop" of the node m1 in the tree T51, and obtains the decision tree T52 in which the decision trees T20 and T30 are merged. The decision tree T52 is the output result of Tout(1).

The decision tree merging unit 150 obtains the decision tree T40 by executing the decision tree merge process on the decision tree T52 and the decision tree T10.
FIG. 41 is a diagram illustrating an example (part 1) of tracing a decision tree. The result output unit 160 obtains the mutant survival result of each mutant based on the execution result (Passed or Failed) of the test case for each mutant recorded in the test result table 119. However, the result output unit 160 cannot obtain the mutant survival result for the mutant for which the execution result “Failed” is not recorded in the test result table 119. Therefore, the result output unit 160 determines the mutant survival result for the mutant for which the mutant survival result has not been obtained by tracing the decision tree T40.

For example, in the test result table 119 illustrated in FIG. 13, only “Passed” is recorded in the test case of the test ID “Test1” for the secondary mutant M _3,4 , and the mutant survival result is obtained. Not not.

Therefore, the result output unit 160, the mutant M _{3, 4,} to determine the mutant survival results by the decision tree T40. The code 111a is a code example corresponding to the mutant M _3,4 . In the code 111a, the operator ">" at the location (fourth line) corresponding to the MMID "3" of the source code 111 is changed to the operator "<=". In the code 111a, the operator “*” at the location (fifth line) corresponding to the MMID “4” of the source code 111 is changed to the operator “/”.

First, the result output unit 160 selects the node m1 that is the root node of the decision tree T40. Mutant M _3,4 does not include the mutation operator of MMID “1” (that is, the mutation operator for MMID “1” is “nop”). Therefore, the result output unit 160 traces the “nop” edge of the node m1 and selects the node m2.

Mutant M _3,4 does not include the mutation operator of MMID “2” (that is, the mutation operator for MMID “2” is “nop”). Therefore, the result output unit 160 follows the “nop” edge of the node m2 and selects the node m3.

The mutation operator of the MMID “3” of the mutant M _3,4 is “<=”. Therefore, the result output unit 160 traces the edge of “<=” of the node m3 to reach the leaf node of Fail. Having reached the leaf node of Fail, the result output unit 160 determines that the mutant M _3,4 is killed. Therefore, the result output unit 160 registers the mutant survival result “Killed” for the mutants M _{3,4 in the} test result table 119.

Incidentally, with respect to test case of the test ID "Test2" (func (1 ) == 2), when actually executing the code 111a, func (1) = 0 so the execution result for mutant M _{3, 4} is "Fail" Is. Further, when the code 111a is actually executed for the test case (func(−1)==0) of the test ID “Test3”, func(−1)=0, so the execution result for the mutant M _3,4 is “ Pass”. Since the test case of the test ID "Test2" is "Fail", the mutant survival result "Killed" obtained by tracing the decision tree T40 is a correct test result. It means that the analysis apparatus 100 has obtained the test result for the mutant M _3,4 without actually executing the test cases of the test IDs “Test2” and “Test3”.

FIG. 42 is a diagram illustrating an example (part 2) of tracing a decision tree. Next, consider the third-order mutants M _1,3,4 . In the test result table 119 illustrated in FIG. 13, for the third-order mutants M _1,3,4 , only “Passed” is recorded in the test case of the test ID “Test3”, and the mutant survival result is obtained. Not not.

Therefore, the result output unit 160 determines the mutant survival result for the mutants M _{1,3,4 using} the decision tree T40. Code 111b is a code example corresponding to mutants M _1,3,4 . In the code 111b, the operator "<" at the location (second line) corresponding to the MMID "1" of the source code 111 is changed to the operator ">=". Further, in the code 111b, the operator ">" at the location (4th line) corresponding to the MMID "3" of the source code 111 is changed to the operator "<=". Further, in the code 111b, the operator "*" at the portion (fifth line) corresponding to the MMID "4" of the source code 111 is changed to the operator "/".

First, the result output unit 160 selects the node m1 that is the root node of the decision tree T40. The mutation operator of the MMID “1” of the mutant M _1,3,4 is “>=”. Therefore, the result output unit 160 follows the “>=” edge of the node m2 and selects the node m2.

Mutants M _1,3,4 do not include the mutation operator of MMID “2” (that is, the mutation operator is “nop” for MMID “2”). Therefore, the result output unit 160 follows the “nop” edge of the node m2 to reach the Fail leaf node. Having arrived at the Fail leaf node, the result output unit 160 registers the mutant survival result “Killed” for the mutants M _1,3,4 of the test result table 119.

When the code 111b is actually executed for the test case of test ID “Test1” (func(0)==0), func(0)=1, so the execution result for the mutants M _1,3,4 is “ Fail”. When the code 111b is actually executed for the test case of test ID “Test2” (func(1)==2), func(1)=2, so the execution result for the mutants M _1,3,4 is “ Pass”. Since the test case of the test ID "Test1" is "Fail", the mutant survival result "Killed" obtained by tracing the decision tree T40 is a correct test result. It means that the analyzer 100 has obtained the test results for the mutants M _1,3,4 without actually executing the test cases of the test IDs “Test1” and “Test2”.

FIG. 43 is a diagram illustrating an example of acquisition of test results. In this way, the result output unit 160 traces the decision tree T40 to complement the mutant survival result for the mutant for which the mutant survival result has not been obtained, and obtains the test result table 119a.

Specifically, in the test execution by SSE, as shown in the test result table 119 of FIG. 13, among the 21 mutants, the mutants M ₁ , M ₂ , M ₃ , M ₄ , M ₅ , M _1, Mutant survival results of ₂ , M _1,3 , M _1,3,5 are obtained. On the other hand, no mutant survival results can be obtained for the remaining 13 mutants.

Therefore, the result output unit 160 obtains the mutant survival result of each mutant by tracing the decision tree T40 for each mutant. As a result, the result output unit 160 can obtain the mutant survival result for the mutants that have not obtained the mutant survival result including the above-mentioned mutants M _3,4 and M _1,3,4 . At this time, the result output unit 160 can also obtain mutant survival results for mutants that have not been tried in all test cases (for example, mutants M _1,4 , M _1,2,3, etc.).

That is, the analysis device 100 selects a combination in which the mutant survival result (test result) cannot be acquired from the execution results of the partial test case among the combinations of mutant parts (changed parts) that are tried by some test cases. On the other hand, test results can be obtained efficiently. In addition, the analysis apparatus 100 can efficiently obtain the test result even for the combination of mutant points that are not tried in the plurality of test cases currently prepared.

In the example of the test result table 119a, all the mutant survival results of the third or lower mutants designated by the user are “Killed”. Therefore, the result output unit 160 outputs the mutation score MS=21/21=100%. Therefore, it can be seen that the current test program 113 can detect all mutants of the third order and below. If the mutation score MS is less than 100%, the user can improve the test program so as to improve the mutation detection rate (mutation score), for example.

FIG. 44 is a diagram illustrating an example of a mutant that is not executed in SSE. In SSE, as described above, the intermediate code 116 obtained by compiling the source code 111 is mutated while branching the execution state in the order of instruction execution. For this reason, among the mutants of the order designated by the user, the execution of a relatively high-order mutant may be omitted.

In FIG. 44, as an example, the code corresponding to the mutant M ₁ 111x, code corresponding to the mutant M ₃ 111y, code 111a corresponding to the mutant M _{3, 4,} and corresponds to the mutant M _{1, 3,4} The code 111b is shown. The code 111x is obtained by changing the operator "<" in the portion (second line) corresponding to the MMID "1" of the source code 111 to the operator ">=". The code 111y is obtained by changing the operator ">" in the portion (4th line) corresponding to the MMID "3" of the source code 111 to the operator "<=".

In this case, the mutants M ₁ , M ₃ , M _3,4 are executed but the mutants M _1,3,4 are not executed for the test case (func(0)=0) of the test ID “Test1”. .. This is because, in the test case with the test ID “Test1”, the three mutation locations corresponding to MMID=1, 3, 4 are not reached at the same time. That is, when the mutant M ₁ is executed, the mutation points corresponding to the mutants M ₃ and M ₄ are deviated from the execution path, so that the change content corresponding to the mutants M ₃ and M ₄ is further added to the mutant M ₁ . It does not apply. In addition, when the mutant M ₃ is executed, the mutation portion corresponding to the mutant M ₁ is deviated from the execution path, so that the modification corresponding to the mutant M ₁ is not applied to the mutant M ₃ . .. Thus, there may be mutants that are not implemented in SSE. In addition, there may be a mutant which is executed in some test cases but whose execution result is Pass and whose mutant survival result cannot be determined.

The mutant M _{3,4 is} also executed in some test cases (here, only the test case of the test ID “Test1”), but the execution result is Pass. Therefore, the analysis device 100, than the current test case only run in SSE can not determine the mutant survival results for mutant M _{3, 4.}

FIG. 45 is a diagram illustrating a comparative example of test result acquisition. In the example of the test result table 119, the mutant survival result is not obtained for a total of 13 mutants. In order to obtain the mutant survival results for these mutants, the relevant mutants are individually compiled 13 times in total, and the test execution of the relevant mutants is calculated for each test case for which an execution result is not obtained. It is also possible to do it 36 times. However, if the compilation and the test execution are individually performed for each mutant in this way, there is a problem that the test takes a long time.

Therefore, the analysis apparatus 100 creates the decision trees T10, T20, T30 in the process of creating the test result table 119, and creates the decision tree T40 by merging the decision trees T10, T20, T30. The analysis apparatus 100 traces the decision tree T40 to determine the mutant survival result for 13 mutants for which the mutant survival result is not obtained in the test result table 119. Accordingly, the analysis apparatus 100 can obtain the mutant survival result by tracing the merged decision tree T40 13 times in total.

In the processing of the analysis device 100, processing costs such as generation of the decision trees T10, T20, T30 and merging occur. However, compared with the processing cost of individually compiling and executing each mutant, the processing costs of generating and merging the decision trees T10, T20, T30 are considered to be sufficiently low.

In this way, the analysis apparatus 100 can determine the test result only by tracing the decision tree T40 created based on the decision trees T10, T20, T30. Therefore, it is not necessary to compile and execute each mutant individually, and the execution time can be reduced accordingly. As a result, the mutation analysis can be speeded up. Further, the accuracy of the test evaluation result by the mutation analysis can be improved, and the user's work of improving the reliability of the test itself can be supported. By contributing to the improvement of test reliability, it is possible to contribute to the improvement of software quality.

The information processing according to the first embodiment can be realized by causing the calculation unit 1b to execute a program. The information processing of the second embodiment can be realized by causing the processor 101 to execute a program. The program can be recorded in a computer-readable recording medium 13.

For example, the program can be distributed by distributing the recording medium 13 recording the program. Alternatively, the program may be stored in another computer and distributed via the network. For example, the computer stores (installs) the program recorded in the recording medium 13 or the program received from another computer in a storage device such as the RAM 102 or the HDD 103, reads the program from the storage device, and executes the program. Good.

1 Analysis Device 1a Storage Unit 1b Arithmetic Unit P1, P2 Source Code T1, T2, T3, T4 Decision Tree

Claims (7)

A storage unit that stores a plurality of changed portions of the program when performing a test including a plurality of test cases for confirming an output with respect to an input of the program and information of change contents of the plurality of changed portions,
The test is executed by applying the change contents to the first combination of change parts, the executed change part is a node other than the terminal node, the change contents of the change part is an edge, and the execution result is the Information on a decision tree as a terminal node is generated for each test case, and the change contents are applied to a second combination of the changed portions for which a test result has not been acquired, based on the plurality of generated decision trees. An operation unit that determines the test result in the case of
An analyzer having a. The calculation unit generates a combined decision tree by combining the plurality of decision trees, and traces the combined decision tree for the second combination to perform the test for the second combination. The analysis device according to claim 1, which determines a result. The second combination cannot acquire the test result from the execution result of the some test cases among the combinations of the changed portions tried by some test cases of the plurality of test cases. The analysis device according to claim 1 or 2, comprising a combination. The analysis device according to any one of claims 1 to 3, wherein the second combination includes a combination of the changed portions that is not tried in the plurality of test cases. The calculation unit is configured to select a plurality of combinations of the changed parts according to a first test result acquired by executing the test and a second test result determined based on the plurality of decision trees. The analysis device according to any one of claims 1 to 4, which outputs at least one of information about a combination that can be detected by a test case and information about a combination that cannot be detected by the plurality of test cases. A first combination of changed portions by referring to information of a plurality of changed portions of the program and change contents of each of the plurality of changed portions when performing a test including a plurality of test cases for confirming output with respect to input of the program The test is executed by applying the change content to a node other than the end node, the change content of the change location being an edge, and the execution result being the end node. Generate information for each test case,
Determining the test result when the change content is applied to the second combination of the change points for which the test result has not been acquired, based on the plurality of generated decision trees,
An analysis program that causes a computer to perform processing. Computer
A first combination of changed portions by referring to information of a plurality of changed portions of the program and change contents of each of the plurality of changed portions when performing a test including a plurality of test cases for confirming output with respect to input of the program The test is executed by applying the change content to a node other than the end node, the change content of the change location being an edge, and the execution result being the end node. Generate information for each test case,
Determining the test result when the change content is applied to the second combination of the change points for which the test result has not been acquired, based on the plurality of generated decision trees,
analysis method.

Images