CN102193864B - Test Case Collection Optimization Method Based on Coverage-Based Error Location Technology - Google Patents

Abstract

The invention discloses a test case set optimization method of a coverage-based error positioning technology. The method comprises the following steps of: identifying accidental right test cases from a given test case set T by clustering, wherein the accidental right test cases mean that the error statements are executed and the execution results are still 'pass'; and processing the identified accidental right test cases to obtain an optimized test case set for coverage-based error positioning. The method has the advantages of optimizing the quality of the test case set at the same time of reducing the size of the original test case set, and reducing the interference of the accidental right test cases to the coverage-based error positioning so as to improve the efficiency and the accuracy of automatic error positioning and save the time cost of seeking errors by a programmer.

Description

Test Case Collection Optimization Method Based on Coverage-Based Error Location Technology

technical field

The invention belongs to the technical field of software testing, in particular to the field of automatic testing, which is used for the coverage-based error location technology CBFL, and is a test case set optimization method for the coverage-based error location technology.

Background technique

As an important technology of automated testing, coverage-based error localization technology CBFL is gradually becoming mature and widely accepted. However, the effectiveness of CBFL error localization is affected by the occasional correct test cases.

CBFL records the execution information of each statement in the program in the passed and failed test cases: executed or not executed, using formulas to calculate the suspiciousness of each statement. The CBFL method considers that those statements executed in the failed test cases are more likely to be error statements. However, the PIE model proposed by Jeffrey M.Voas believes that three conditions must be met to make the program invalid: 1) the wrong statement is executed; 2) the program reaches the "infected" state; 3) the "infected" state follows Control flow is propagated to the output. Therefore, the fact that the erroneous statement is executed is only a necessary condition for failure to occur, but not a sufficient condition. A test case that only satisfies condition 1) but still passes is called a test case with weak chance correctness, while a test case that satisfies conditions 1) and 2) but still passes is called a test case with strong chance correctness. The occasional correct test cases referred to below refer to the former.

In 2009, Wes Masri et al. proposed several factors that affect the effect of CBFL error localization, and the test cases that are accidentally correct are listed among them. That is to say, when there are occasionally correct test cases in the test case set, the suspiciousness of the wrong statement will be reduced by its influence.

In recent years, cluster analysis has been introduced into the field of test case selection. By clustering the execution profiles of test cases, test cases with similar execution paths/behavioral characteristics are gathered into the same cluster. Taking advantage of this feature, Vangala et al. identified redundant test cases in the test case set by clustering the original test case set. Coincidentally, Dickinson et al. also used cluster analysis to select test cases and proposed the concept of cluster filtering. This technique clusters the execution profiles of test cases first, and then uses some sampling technique to extract some test cases from each cluster to represent the entire cluster. This method of using partially sampled test cases to represent the entire test case set greatly reduces the number of test cases.

Contents of the invention

The problem to be solved by the present invention is: for the coverage-based error localization technology CBFL, identify and process occasionally correct test cases in the original test case set, thereby improving the effect of automatic error localization based on coverage.

The technical solution of the present invention is: a test case set optimization method based on the error location technology of coverage, for a given test case set T, identify the correct test case by clustering, and the correct test case by chance means that the wrong statement is Execution, but the execution result is still "passed" test cases, the identified occasional correct test cases are processed, and the optimized test case set is used for error location based on coverage:

1) Run the test case set T on the target program that needs to be tested. While running the test cases, collect the execution profile information of the test cases. After running the test case set T, determine the execution result of each test case according to the output results "Pass" or "Fail";

2) clustering the obtained execution profile information; the clustering method of the present invention is not limited;

3) Identify test cases that are contingency correct: After performing clustering on the profile information, the test cases are divided into several clusters, and the "passed" tests that are gathered together with the "failed" test cases in one cluster The use cases are identified as occasional correct test cases, and they are added to the set Ticc, which is the set of identified occasional correct test cases;

4) Handling of occasional correct test cases: choose one of the following two ways to process the recognition results in the set Ticc:

41) filter strategy, the test case in Ticc is deleted from test case collection T;

42) relabeling strategy, the execution result judgment label of the test case in the Ticc is changed from "passed" to "failed";

After the above-mentioned processing is performed on the test cases identified as accidental correctness, an optimized test case set T' is obtained.

Because the identification method of the occasional correct test case of the present invention utilizes the coverage information of the test case, this method is only used for CBFL, but it is applicable to all error location methods belonging to CBFL.

Cluster analysis has been applied in the field of test case selection, and the present invention also introduces this technology in the problem of accidental correctness. Although both belong to the category of test case set optimization, the purpose of the former is to reduce the size of test case set, which can help programmers to find errors as early as possible and reduce the time of regression testing, while the purpose of the present invention is to improve test case set The ability to help error location reduces the time for programmers to find errors, thereby reducing the cost of debugging errors. The clustering goals of the two are different, and the processing after clustering is also different.

Since the test cases in a class cluster have similar execution paths, it is not difficult to infer that the passed test cases in a class cluster may pass through the error statement just like the failed test cases in this class cluster, but due to They do not all meet the three conditions proposed in the PIE model, so no failure is triggered. Therefore, these passing test cases are highly likely to be correct by chance.

The beneficial effects of the present invention are: while reducing the size of the original test case set, the quality of the test case set is optimized, and the interference effect of occasional correct test cases on error positioning based on coverage is reduced, thereby improving automation error The efficiency and accuracy of positioning saves the time and cost of programmers looking for errors.

Description of drawings

Fig. 1 is a schematic flow chart of the present invention.

Fig. 2 is in the embodiment of the present invention, the accuracy of identifying the correct test case of contingency, p represents the ratio of the number of clusters, Fig. 2 (a)-(e) represent respectively p=1%, 2%, 4%, 6 %,8%. .

Fig. 3 is in the embodiment of the present invention, applies filtering strategy, to the improving effect of the method for error localization based on coverage, p represents the ratio of the number of clusters, Fig. 3 (a)-(e) represent p=1% respectively , 2%, 4%, 6%, 8%.

Figure 4 is the embodiment of the present invention, applying the relabeling strategy, the improvement effect of the method based on the coverage of the error location, p represents the ratio of the number of clusters, Figure 4 (a)-(e) respectively represent p=1 %, 2%, 4%, 6%, 8%.

Detailed ways

The present invention applies clustering analysis to the selection of test cases with correct chance, deletes the selected test cases from the original test case set or changes the execution result labels of these test cases, and optimizes the original test cases by the above two methods Set, the existing automatic error location technology is applied to the optimized test case set, so that the efficiency and accuracy of error location are improved.

For a given test case set T, this test case set consists of 2 parts: Tp and Tf, the object of the present invention is to identify Tcc from Tp, the result of recognition is Ticc, and each test case in Ticc has many The big possibility belongs to Tcc, where:

T: the set of test cases used for a given program

Tp: the collection of passed test cases

Tf: collection of failed test cases

Tcc: a collection of test cases that are accidentally correct

Ticc: the set of identified test cases that are incidentally correct

In practical application, need to finish the work of the present invention by 5 steps, as Fig. 1:

1) Run the test case set on the target program that needs to be tested. While running the test case, collect the execution profile information of the test case. After running the test case set, according to the output result, it is judged that the execution result of each test case is "passed" or "failed";

2) Clustering the execution profiles, the clustering method is not limited, and the test cases with similar execution profiles can be clustered;

3) Identify test cases that are occasionally correct;

4) Handle occasional correct test cases;

5) Apply the existing coverage-based automatic error location technology on the optimized test case set.

Among them, steps 1), 2), and 5) have ready-made methods and applications in automatic error location and cluster analysis, and will not be repeated here. The detailed explanation mainly focuses on steps 3) and 4) here.

After clustering the execution profiles, the test cases are divided into several clusters. Each class cluster may contain passed and failed test cases, and the present invention gathers those passed test cases in a class cluster with the failed test cases as occasional correct test cases, and adds them into the set Ticc. Elements in Ticc are highly likely to be correct test cases by chance for the following reasons:

(1) A test case that has executed an erroneous statement does not necessarily cause failure, but the opposite is not true, and a test case that fails must have executed an erroneous statement;

(2) Test cases with similar execution profiles will be divided into a class cluster, which means that the selected passed test cases and failed test cases have similar execution paths.

In summary, those selected test cases have a high probability of executing the wrong statement, but still giving the correct output, which is consistent with the definition of being correct by chance.

To deal with Ticc, the present invention proposes 2 different strategies:

●Filter strategy. The test cases in Ticc will be deleted from the original test case set, so as to reduce the interference effect of these test cases on automatic error localization.

● Relabeling strategy. The result judgment label of the test case in Ticc will be changed from "Passed" to "Failed". In this way, the suspiciousness of the erroneous statement is increased, which may lead to an increase in the suspiciousness ranking of the statement.

The implementation of the present invention will be described below through specific examples.

The Siemens test case set is a test case set widely used in the academic field. Many methods of evaluating software testing and bug discovery use it as the target program for experimentation. Table 1 lists the details of the Siemens program.

Table 1 Details of the Siemens program

Taking the Siemens program as an example, the embodiment of the present invention is as follows:

1. Execute test cases and collect execution profiles.

Use gcov (GNU call-coverage profiler) as a tool to collect execution profiles. To do this, compile options need to be added to the automation scripts that execute the test cases. gcov can obtain statement-level program coverage information. After the test case ti∈T is executed, a .gcov file will be generated. This file records the number of times each line of code is executed. For the test case ti, its statement coverage profile pi=<e1, e2, e3, ..., en>, where n represents the number of code lines of a given program, if this line of code has been executed, then ei =1, otherwise ei=0.

2. Cluster analysis.

The execution profile collected in the previous step is the input for the cluster analysis. The profile information of each test case is the object of clustering. The total number of objects is equal to the number of test cases contained in the test case set T. We use Weka as the clustering tool and n-dimensional Eudlidean distance as the distance function. Given the profile of 2 test cases t: <e1, e2, ..., en> and t′: <e1′, e2′, ..., en′>, the distance of these 2 test cases is:

$\mathrm{<span\; class="notranslate">\; D.</span>}<span\; class="notranslate">\; (</span>\mathrm{<span\; class="notranslate">\; t</span>}<span\; class="notranslate">\; ,</span>{\mathrm{<span\; class="notranslate">\; t</span>}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span><span\; class="notranslate">\; =</span>\sqrt{{\mathrm{<span\; class="notranslate">\; \&Sigma;</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}<span\; class="notranslate">\; =</span>\mathrm{<span\; class="notranslate">\; 1</span>}}^{\mathrm{<span\; class="notranslate">\; no</span>}}{<span\; class="notranslate">\; (</span>{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}<span\; class="notranslate">\; -</span>{{\mathrm{<span\; class="notranslate">\; e</span>}}_{\mathrm{<span\; class="notranslate">\; i</span>}}}^{<span\; class="notranslate">\; \&prime;</span>}<span\; class="notranslate">\; )</span>}^{\mathrm{<span\; class="notranslate">\; 2</span>}}}<span\; class="notranslate">\; .</span>$

We use simple K-means as the clustering algorithm. Simple K-means uses the number of clusters as an argument. In the present invention, the size of this parameter is set according to the size of the test case set T. Let CN represent the number of clusters, CN=|T|*p, |T| is the size of test case set T, and 0<p<1. Since this embodiment selects the K-means clustering method, which uses the number of clusters as a parameter, the length of the code should meet certain conditions, so that the execution profiles of all test cases meet a certain degree of difference, otherwise it will not reach Set the number of clusters.

3. Identification and processing of occasionally correct test cases.

Identify test cases that are occasionally correct: After performing clustering on the profile information, the test cases are divided into several clusters, and those "passed" test cases are identified in a cluster with the "failed" test cases For the test cases that are occasionally correct, and add them to the set Ticc, the set Ticc is the set of identified test cases that are occasionally correct;

Handling of occasional correct test cases: choose one of the following two ways to process the recognition results in the set Ticc:

1) filtering strategy, the test cases in Ticc are deleted from the test case set T;

2) Relabeling strategy, changing the execution result judgment label of the test case in Ticc from "pass" to "fail";

After the above-mentioned processing is performed on the test cases identified as accidental correctness, an optimized test case set T' is obtained.

The above identification and processing can be implemented using a program written in Java. The program takes the clustering results of the second step and the execution results of each test case as input, selects and processes the test cases described above for identifying and processing contingency correct test cases and Handle occasional correct test cases.

4. Wrong positioning.

Use Tarantula as a coverage-based error localization method. It is the first system to apply spectrum-based ranking methods to the field of software diagnostics. Its input is a reduced and optimized test case set, and the output is a list of suspicious statements arranged in descending order of suspiciousness. Programmers can check related statements one by one against the list until an erroneous statement is found.

Fig. 2-4 is the experimental effect diagram of the present invention on the Siemens program.

What Fig. 2 represented is the accuracy of the correct test case of identifying contingency of the present invention, and p represents the ratio of the number of clusters, and Fig. 2 (a)-(e) represents p=1%, 2%, 4%, 6% respectively ,8%. "FN" and "FP" stand for "" "false negative" and "false positive" respectively. The calculation formula of "False negative" is: |Tcc-Ticc|/|Tcc|, this metric evaluates the ability of the present invention to identify test cases with contingency correctness, that is, whether the present invention can identify contingency as much as possible correct test case. The smaller this metric, the better. The calculation formula of "false positive" is: |(Tp-Tcc)∩Ticc|/|Tp-Tcc|, this metric evaluates the correctness of the present invention to identify test cases that are contingency correct. That is to say, whether the present invention can mistake other test cases as accidental correct test cases as little as possible. The abscissa represents the range where "FN" and "FP" are located, and the ordinate represents the proportion of program versions that fall in the corresponding range.

What Fig. 3 represented is to apply filter strategy, the present invention is to the improving effect of the method for error localization based on coverage, p represents the ratio of clustering number, and Fig. 3 (a)-(e) represent respectively p=1%, 2 %, 4%, 6%, 8%.

What Fig. 4 represents is to apply relabeling strategy, the present invention is to the improving effect of the method for the error localization based on coverage, p represents the ratio of clustering number, Fig. 4 (a)-(e) represent p=1% respectively, 2%, 4%, 6%, 8%.

Fig. 3 and Fig. 4 have represented the statistical result of T-score reduction for each target program with the box plot, T-scorereduction △ TS=TS-TS', wherein, TS and TS ' have represented respectively in applying the present invention T-score before and after. T-score is an effective metric for evaluating error localization methods. It represents the proportion of code that needs to be checked in order to find errors. The smaller this metric, the better. The meaning of the T-score expressed by the formula is as follows:

TS=|Vexamined|/|V|*100%

Among them, |V| represents the total number of executable statements of the target program, and |Vexamined| represents the number of lines of code that the programmer needs to check in order to find the wrong statement. Therefore, the larger the value of T-score reduction, the better the improvement of error localization.

The upper and lower lines of the boxplots in Figures 3 and 4 represent the third quartile and the first quartile, respectively. The horizontal line in the middle of the boxplot represents the median, while the black dot represents the mean. The points above and below the box represent the maximum and minimum values, respectively.

For reference, Figures 3 and 4 also show the effect of CBFL using the respective strategies ideally. The so-called ideal situation is the situation where it is assumed that all the occasional correct test cases are selected, that is, the situation where "false negative" and "false positive" are both 0%.

Claims (1)

1. the test suite optimization method of the location of mistake technology based on coverage, it is characterized in that for given test use cases T, by cluster, therefrom identify the correct test case of contingency, the correct test case of described contingency refers to that mistake statement is performed, the test case of " passing through " but execution result is still, the test case that the contingency identifying is correct is processed, and the test use cases being optimized is for the location of mistake based on coverage:

1) on the target program of needs test, move test use cases T, in operation test case, collect the execution profile information of test case, after having moved test use cases T, according to Output rusults, the execution result of judging each test case as " pass through " or " by ";

2) the execution profile information obtaining is carried out to cluster;

3) the correct test case of identification contingency: execution profile information is carried out after cluster, test case is divided in several classes bunch, by the test case of those and " not by " be gathered in a class bunch " by " test case be designated the test case that contingency is correct, and they are joined in set Ticc, set Ticc is the set of the test case that identified contingency is correct;

4) process the correct test case of contingency: a kind of recognition result of processing in set Ticc in optional following two kinds of modes:

41) filtering policy is deleted the test case in Ticc from test use cases T;

42) relabel strategy, by the execution result of the test case in Ticc judge label from " by " make " not passing through " into;

Be designated test case that contingency is correct after above-mentioned processing, the test use cases T ' being optimized;

5) by using Tarantula system as the location of mistake based on coverage, by input, through the test use cases T ' optimization, the list of the suspicious statement that output is arranged from big to small according to suspicious degree, checks correlative one by one according to this list, until find wrong statement.

Images