CN110688309A - General software test case sequence quantitative evaluation method based on requirements - Google Patents
General software test case sequence quantitative evaluation method based on requirements Download PDFInfo
- Publication number
- CN110688309A CN110688309A CN201910889705.6A CN201910889705A CN110688309A CN 110688309 A CN110688309 A CN 110688309A CN 201910889705 A CN201910889705 A CN 201910889705A CN 110688309 A CN110688309 A CN 110688309A
- Authority
- CN
- China
- Prior art keywords
- test
- sequence
- test case
- cost
- cases
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/34—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment
- G06F11/3409—Recording or statistical evaluation of computer activity, e.g. of down time, of input/output operation ; Recording or statistical evaluation of user activity, e.g. usability assessment for performance assessment
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- General Engineering & Computer Science (AREA)
- Computer Hardware Design (AREA)
- Quality & Reliability (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention provides a general software test case sequence evaluation method, and belongs to the technical field of software testing. The method of the invention comprises the following steps: s1, calculating the comprehensive benefits of the test points and the comprehensive cost of the test cases based on the attributes of the test points and the test cases of the software to be tested and the corresponding relation between the test points and the test cases; s2, calculating the average test yield of the test case sequence by utilizing the comprehensive benefits of the test points, the comprehensive cost of the test cases and the sequence of the test cases in the test case sequence; and S3, evaluating different test case sequences by using the average test yield. The method can be suitable for different test priority criteria or sorting requirements, gives quantitative evaluation results of the test case sequences, and accurately reflects the advantages and disadvantages of the different test case sequences.
Description
Technical Field
The invention belongs to the technical field of software testing, and relates to a general software test case sequence quantitative evaluation method based on requirements.
Background
Software testing is a key link and an important means for guaranteeing software quality. As software scale and complexity increase, the overhead of software testing also increases, in which case it is desirable to order the test cases so that the cases with higher priority can be executed as early as possible. The test case prioritization Technology (TCP) reorders the test cases under the guidance of a specific ordering criterion according to a preselected test target, improves the test efficiency by optimizing the execution order of the test cases, and is a research hotspot in the field of software testing.
Elbaum et al (Elbaum S, Malishevsky AG, Rothermel G.priority Testing casting for regression Testing. in: Proc. of the int. Symp. on Software Testing and analysis. ACM Press, 2000.102-112.) gave a general description of the TCP problem in 2000. Rothermel et al (Rothermel G, Untech RH, Chu C, Harrol MJ. priority test cases for regression testing. IEEE Trans. on Software Engineering, 2001, 27 (10): 929-948.) confirmed the effectiveness of the priority technique in improving error detection rate through a series of targeted experimental studies. The TCP technology is divided into 3 types of code-based, model-based and demand-based TCP and the like by Chenxiang and the like (Chenxiang, Chengyong, Juxialin, Zhaoqing. regression test, test case priority ranking technical statement evaluation, software academy, 2013, 24 (8): 1695-1712), and the current research mainly focuses on the code-based TCP technology, has a lot of research achievements from different angles such as a greedy method, a machine learning method, a fusion expert knowledge method and the like, and is relatively sufficient in research. The research results of the demand-based TCP technology are still relatively few. The dynamic adjustment algorithm of the test case priority based on the requirements is provided based on the design information of the test case by the aid of the design information of the test case, such as the wave bending, the Nie-Changhai, the Xubao. The TCP technique based on the Total policy and the Additional policy considering the test requirement priority and the test case execution overhead was proposed by Zhang XF, Nie CH, Xu BW, Qu B.test case priority based on changing the requirements and test cases costs in: Proc.of the int' l Conf.on Quality software IEEE Press, 2007.15-24.
To measure the effectiveness of the TCP technique, the test case ordering result needs to be evaluated. The advantage of TCP is that it is able to reach the test target faster than random sequential testing. Elbaum et al (Elbaum S, Malishevsky A, Rothermel G.priority testing cases for regression testing. in: Proc. of the int' l Symp. on software testing and analysis. ACM Press, 2000.102-112.) use the relationship between the number of test cases used and the number of errors detected to quantify the merits of test case sequences, and give APFD (amplitude performance of fault detection) evaluation indexes. Since the defect detection information of the test case cannot be known before the test case is completely executed, the APFD has obvious defects. Li Zheng et al (Li Z, Harman M, hierarchy RM. Searchalgorithms for regression test case priority. IEEE Trans. on software engineering, 2007, 33 (4): 225-.
In a requirement-based test (or black box test or functional test), the tester bases on the specification of software requirements. Firstly, converting software requirements into test requirements, then refining and decomposing the test requirements into test points, and finally designing test cases aiming at the test points to form a test case set. For this reason, Zhang Wei Xiang et al (Zhang Wei Xiang, Weibo, Du Hui Sen, a test case priority ordering method based on genetic algorithm. Small-sized microcomputer system. 2015, 36 (9): 1998-. Zhang Wei Xiang, et al (Zhang Wei Xiang, Qiyuhua, Li De Zhi, priority ordering of test case based on discrete particle swarm optimization, computer application 2017, 37 (1): 108-.
Disclosure of Invention
The invention aims to provide a general software test case sequence evaluation method which can be suitable for different test priority criteria or sorting requirements, gives a quantitative evaluation result of a test case sequence and accurately reflects the quality of different test case sequences.
In order to achieve the above object, the present invention provides a general software test case sequence evaluation method, which comprises the following steps:
s1, calculating the comprehensive benefits of the test points and the comprehensive cost of the test cases based on the attributes of the test points and the test cases of the software to be tested and the corresponding relation between the test points and the test cases;
s2, calculating the average test yield of the test case sequence by using the comprehensive benefits of the test points, the comprehensive cost of the test cases and the sequence of the test cases in the test case sequence;
and S3, evaluating different test case sequences by utilizing the test average yield.
According to an aspect of the invention, the test average rate of return, eAPWC, is:
wherein, Wiλ (W) is the ith test point piThe comprehensive profit of (1) represents the test profit type factor W to the ith test point piThe combined effect of (a); cjμ (C) is the jth test case tjThe comprehensive cost of (a) represents the test cost type factor C to the jth test case tjThe combined effect of (a); lambda is a weight function of the test profit type factor, mu is a weight function of the test cost type factor;
TTithe test case showing the first test point to be covered isThe sequence of the test cases; n is the number of test cases, and m is the number of test points.
According to one aspect of the present invention, when the comprehensive gains of all the test points are the same and the comprehensive cost of the test case is the same, formula (5) degenerates to an evaluation index APTC based on test point coverage:
according to one aspect of the invention, the test revenue type factors include a demand priority, a demand importance level, a demand implementation complexity, demand change information, a defect occurrence probability, and an influence level; the test cost type factors include test case time cost, labor cost, tool and environment cost, and other execution overhead.
According to one aspect of the invention, the weight function λ of the test revenue type factor is: w → R, → representing a mapping from the set W to the real number set R; the weight function μ of the test cost type factor: c → R, → represents a mapping relationship from the set C to the real number set R; r represents a real number set.
According to an aspect of the invention, the weight function λ of the test benefit type factor, the weight function μ of the test cost type factor, is a linear function, or is a non-linear function.
According to one aspect of the invention, the value range of the test average yield is 0-100%, and the higher the value is, the better the evaluation result is.
The invention has the beneficial effects that:
1) according to the method, the average test yield is used as an evaluation index, and based on the test points and the test case attributes, a relatively accurate quantitative evaluation result can be obtained, and the quality of the test case sequence can be objectively evaluated.
2) The invention is suitable for all software tests (or black box tests or functional tests) based on requirements, and has wide application range and good universality.
3) The invention has good formulation form, easily-compared result quantification and simple and convenient operation.
Drawings
FIG. 1 schematically shows a flow diagram according to an embodiment of the invention;
FIG. 2 schematically represents eAPWC values corresponding to different test case sequences, in accordance with an embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the following drawings and examples, and it should be noted that the following detailed description is only for illustrative purposes and should not be construed as limiting the scope of the present invention.
As shown in fig. 1, according to an embodiment of the present invention, a method for quantitatively evaluating a sequence of a general-purpose software test case based on requirements includes the following steps:
s1, calculating the comprehensive benefits of the test points and the comprehensive cost of the test cases based on the attributes of the test points and the test cases of the software to be tested and the corresponding relation between the test points and the test cases;
s2, calculating the average test yield of the test case sequence by utilizing the comprehensive benefits of the test points, the comprehensive cost of the test cases and the sequence of the test cases in the test case sequence;
and S3, evaluating different test case sequences by using the average test yield.
The invention adopts the average yield rate eAPWC (enhanced average probability of win-costcoverage) as an evaluation index, which can be expressed in a formula way as follows:
wherein:
Wiλ (W) is the ith test point piThe comprehensive profit of (1) represents the test profit type factor W to the ith test point piThe common test yield type factors comprise the priority of the demand, the importance degree of the demand and the actual demandComplexity, requirement change information, defect occurrence probability, influence degree and the like; cjμ (C) is the jth test case tjThe comprehensive cost of (a) represents the test cost type factor C to the jth test case tjThe common test cost type factors include test case time cost, labor cost, tool and environment cost and other execution overhead; lambda and mu are weight functions of the test profit type factor and the test cost type factor respectively.
TTiRepresenting the sequence of the first test case which can cover the ith test point in the test case sequence; n is the number of test cases; and m is the number of the test points.
T＝{t1，t2，…，tnDenotes a set of test cases, tjE T (j is 1, 2, …, n) represents any test case; p ═ P1，p2，…，pmDenotes a set of test points, piE P (i ═ 1, 2, …, m) represents any test point; test yield form factor W ═ W1，w2，…，wl}，wiE W (i ═ 1, 2, …, l) represents any test yield type factor; test cost factor C ═ C1，c2，…，ck}，ciE C (i ═ 1, 2, …, k) represents any test cost type factor. Testing the weight function lambda of the profit type factor: w → R, → represents a mapping (correspondence) from the set W to the real number set R; testing the weight function mu of the cost type factor: c → R, → represents a mapping relationship (correspondence) from the set C to the real number set R; r represents a real number set. The weight function of the test profit type factor and the weight function of the test cost type factor can be linear functions or nonlinear functions.
When the comprehensive income of all test points is the same and the comprehensive cost of the test cases is the same, the formula (5) degenerates into an evaluation index APTC based on the test point coverage:
the value range of the average yield of the test is 0-100%, and the higher the value is, the better the evaluation result is.
An example is given below in the context of a triangle classification program. The triangle classification program determines which triangle will be formed by using the 3 numbers as three sides by analyzing the values of the 3 input variables (x, y, z) and their interrelations.
There were 7 test points as shown in table 1 by test requirement analysis. The test yield type factor considers 3 aspects of the importance degree of the test point, the realization complexity of the corresponding function, the defect occurrence probability and the like, and the value of the test point is determined according to the priori knowledge.
Through the black box test method, 6 test cases are designed, and the relationship between the test cases and the test points is shown in table 2. The test cost type factor considers 2 aspects of time cost, environment cost and the like of test case execution, and the value of the test case is determined according to the priori knowledge.
Table 1 test points of triangle classification program
TABLE 2 correspondence between test points and test cases for triangle classification procedures
Without loss of generality, let the weight function λ: w → R, μ: c → R are all arithmetic mean functions, i.e.: for test point pi，
For test case tj，
Then it is possible to obtain, by simple calculation,
W2＝2，
W5＝3，W6＝3，
C1＝1，C2＝1，
C6＝2。
taking the test sequence A-B-C-D-E-F as an example, the evaluation index eAPWC value is calculated as follows:
the same calculation yields: eAPWC(F-E-D-C-B-A)＝0.6837，eAPWC(D-C-F-E-A-B)＝0.7009，eAPWC(D-F-C-B-A-E)＝0.7719。
As can be seen from the calculation results, the eAPWC value of the test sequence D-F-C-B-A-E is the largest in 4 test sequences such as A-B-C-D-E-F, F-E-D-C-B-A, D-C-F-E-A-B, D-F-C-B-A-E. This is because, in the event of comparable test case costs, the sequence first covers the most profitable test point 3 with test case DAnd the next largest test point 5 (W)53) and then the test case F is used to cover the test point 6 with the next highest yield (W)63) and larger test points
And the test case E which is relatively useless at the moment is put to be executed at last, which accords with the principle that a good test case is executed first.
Fig. 2 shows a relationship diagram between the test case execution ratio (considering cost) and the covered test points (considering yield) in different execution orders, and the ratio of the shaded area below the broken line to the entire area is the value of eAPWC.
The invention better solves the problem of test case priority, is suitable for all software tests (or black box tests or functional tests) based on requirements and has good universality.
The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the invention, and any modifications, equivalents, improvements and the like that fall within the spirit and principle of the present invention are intended to be included therein.
Claims (7)
1. A quantitative evaluation method for a general software test case sequence based on requirements is characterized by comprising the following steps:
s1, calculating the comprehensive benefits of the test points and the comprehensive cost of the test cases based on the attributes of the test points and the test cases of the software to be tested and the corresponding relation between the test points and the test cases;
s2, calculating the average test yield of the test case sequence by using the comprehensive benefits of the test points, the comprehensive cost of the test cases and the sequence of the test cases in the test case sequence;
and S3, evaluating different test case sequences by utilizing the test average yield.
2. The method for quantitatively evaluating the sequence of the universal software test case according to claim 1, wherein the test average yield eAPWC is:
wherein, Wiλ (W) is the ith test point piThe comprehensive profit of (1) represents the test profit type factor W to the ith test point piThe combined effect of (a); cjμ (C) is the jth test case tjThe comprehensive cost of (a) represents the test cost type factor C to the jth test case tjThe combined effect of (a); lambda is a weight function of the test profit type factor, mu is a weight function of the test cost type factor;
TTirepresenting the sequence of the first test case which can cover the ith test point in the test case sequence; n is the number of test cases, and m is the number of test points.
3. The method according to claim 2, wherein when the comprehensive gains of all the test points are the same and the comprehensive costs of the test cases are the same, formula (5) degenerates to an evaluation index APTC based on test point coverage:
4. the method for quantitatively evaluating the sequence of the universal software test case according to claim 2, wherein the test profit type factors include a demand priority, a demand importance degree, a demand implementation complexity, demand change information, a defect occurrence probability and an influence degree; the test cost type factors include test case time cost, labor cost, tool and environment cost, and other execution overhead.
5. The method for quantitatively evaluating the sequence of the universal software test case according to claim 2, wherein the weight function λ of the test profit type factor is: w → R, → representing a mapping from the set W to the real number set R; the weight function μ of the test cost type factor: c → R, → represents a mapping relationship from the set C to the real number set R; r represents a real number set.
6. The method for quantitatively evaluating a sequence of test cases of general-purpose software according to claim 2 or 5, wherein the weight function λ of the test benefit type factor and the weight function μ of the test cost type factor are linear functions or non-linear functions.
7. The method for quantitatively evaluating the sequences of the universal software test cases according to any one of claims 1 to 5, wherein the test average yield ranges from 0% to 100%, and a higher value indicates a better evaluation result.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910889705.6A CN110688309A (en) | 2019-09-20 | 2019-09-20 | General software test case sequence quantitative evaluation method based on requirements |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910889705.6A CN110688309A (en) | 2019-09-20 | 2019-09-20 | General software test case sequence quantitative evaluation method based on requirements |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110688309A true CN110688309A (en) | 2020-01-14 |
Family
ID=69109705
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910889705.6A Pending CN110688309A (en) | 2019-09-20 | 2019-09-20 | General software test case sequence quantitative evaluation method based on requirements |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110688309A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113177002A (en) * | 2021-05-24 | 2021-07-27 | 中国工商银行股份有限公司 | Test design method and device based on test points, electronic equipment and medium |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105446885A (en) * | 2015-12-28 | 2016-03-30 | 西南大学 | Regression testing case priority ranking technology based on needs |
| US20160162392A1 (en) * | 2014-12-09 | 2016-06-09 | Ziheng Hu | Adaptive Framework Automatically Prioritizing Software Test Cases |
-
2019
- 2019-09-20 CN CN201910889705.6A patent/CN110688309A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20160162392A1 (en) * | 2014-12-09 | 2016-06-09 | Ziheng Hu | Adaptive Framework Automatically Prioritizing Software Test Cases |
| CN105446885A (en) * | 2015-12-28 | 2016-03-30 | 西南大学 | Regression testing case priority ranking technology based on needs |
Non-Patent Citations (1)
| Title |
|---|
| 张卫祥等: "基于离散粒子群算法的测试用例优先排序", 《计算机应用》 * |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113177002A (en) * | 2021-05-24 | 2021-07-27 | 中国工商银行股份有限公司 | Test design method and device based on test points, electronic equipment and medium |
| CN113177002B (en) * | 2021-05-24 | 2024-03-19 | 中国工商银行股份有限公司 | Test design method and device based on test points, electronic equipment and medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Baresel et al. | Fitness function design to improve evolutionary structural testing | |
| US8751436B2 (en) | Analyzing data quality | |
| US5655074A (en) | Method and system for conducting statistical quality analysis of a complex system | |
| Labro et al. | A simulation analysis of interactions among errors in costing systems | |
| CN108241574A (en) | A kind of method and system analyzed based on test and management tool QC software test defect | |
| CN105975797A (en) | Product early-fault root cause recognition method based on fuzzy data processing | |
| CN115600891A (en) | Big data analysis method and system applied to production monitoring of water-based acrylic resin | |
| CN109214625A (en) | A kind of oil tank evaluation method for failure and device | |
| CN116506200A (en) | Cloud security service implementation system and method | |
| Rentala et al. | POD evaluation: the key performance indicator for NDE 4.0 | |
| CN116028887A (en) | Analysis method of continuous industrial production data | |
| CN111177640A (en) | Data center operation and maintenance work performance evaluation system | |
| CN110688309A (en) | General software test case sequence quantitative evaluation method based on requirements | |
| CN114800486A (en) | Industrial robot fault diagnosis method and system based on statistical characteristics | |
| JP3703064B2 (en) | Software quality evaluation apparatus and quality evaluation method | |
| CN108399284A (en) | It is a kind of about subtracted based on deviation big data Trading Model analysis and restorative procedure | |
| Tiwari et al. | Functionality based code smell detection and severity classification | |
| CN110766248A (en) | Workshop human factor reliability evaluation method based on SHEL and interval intuition fuzzy evaluation | |
| Peters et al. | Application of the Choquet integral in software cost estimation | |
| Galyean | Orthogonal PSF taxonomy for human reliability analysis | |
| Klyatis et al. | Multi-variate Weibull model for predicting system-reliability, from testing results of the components | |
| CN109284320A (en) | Automatic returning diagnostic method in big data platform | |
| Molawade et al. | Software reliability prediction using NHPP and Least median of squares (LMS) for parameters estimation | |
| CN114819371B (en) | Tax data-based method and system for constructing yield and sewage discharge prediction model | |
| CN114818990B (en) | Method and system for grading quality of maintenance effect of aero-engine |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| WD01 | Invention patent application deemed withdrawn after publication | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20200114 |