CN111427802B - Test method and system for carrying out test case priority sequencing by utilizing ensemble learning - Google Patents
Test method and system for carrying out test case priority sequencing by utilizing ensemble learning Download PDFInfo
- Publication number
- CN111427802B CN111427802B CN202010432137.XA CN202010432137A CN111427802B CN 111427802 B CN111427802 B CN 111427802B CN 202010432137 A CN202010432137 A CN 202010432137A CN 111427802 B CN111427802 B CN 111427802B
- Authority
- CN
- China
- Prior art keywords
- test
- model
- data
- test case
- execution
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3684—Test management for test design, e.g. generating new test cases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3688—Test management for test execution, e.g. scheduling of test suites
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/36—Preventing errors by testing or debugging software
- G06F11/3668—Software testing
- G06F11/3672—Test management
- G06F11/3692—Test management for test results analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N20/00—Machine learning
- G06N20/20—Ensemble learning
Abstract
The invention belongs to the technical field of software testing, and particularly relates to a testing method and a testing system for carrying out priority sequencing on test cases by utilizing ensemble learning, which comprises the following steps: establishing a regression prediction model of the test case execution error rate by using an ensemble learning algorithm according to the attribute information and the historical execution data of the test case, wherein the regression prediction model is used for predicting the possible error probability of each test case in the test process to be executed; predicting the error rate of each test case based on an error rate prediction model, carrying out priority ordering on each test case according to the sequence of the predicted error rate from large to small, and executing the test cases according to the priority order; in the testing process, once the test case fails to be executed, the test case is immediately submitted to a developer for analysis and repair. The invention shortens the time from the test of the tester to the repair of the developer and improves the test efficiency.
Description
Technical Field
The invention belongs to the technical field of software testing, and particularly relates to a testing method and a testing system for carrying out priority sequencing on test cases by utilizing ensemble learning.
Background
With the increase of the scale of software products and the increase of the updating iteration speed, the frequency of test execution is higher and higher. During the development of software, versions are updated and continuously integrated every day. Before and after the code is submitted, the tester needs to perform the test, and the cost is undoubtedly huge. In this case, the testing process not only consumes a lot of cost, but also leads the testing speed to be behind the product updating iteration speed, and the software quality cannot be guaranteed.
Therefore, researchers have proposed a Test Case Priority (TCP) technique starting from optimizing the execution order of Test cases. The test case prioritization technique desirably prioritizes test cases so that code failures are detected earlier. In a scenario where the regression testing time is too long due to the excessive number of test cases or the testing time is limited and the complete test case cannot be executed, the optimization of the test cases will play a great role.
However, the existing TCP technologies are proposed for Java or C language, and rely on the tool for identifying the code coverage information. As more languages are used in software development for enterprises, information related to the implementation of TCP techniques is in many cases difficult or costly to obtain. It remains a problem how to implement TCP techniques with common, ubiquitous data in enterprises. Meanwhile, the application of the TCP technology by enterprises at present mainly focuses on solving the problem that the iteration speed of a product is influenced by overlong test time by selecting a small number of effective test cases in a test case set to execute. However, in some software enterprises, due to the complex architecture and the large software scale, selecting a part of test cases for execution may cause some errors to be difficult to find, which affects the reliability of the product. This can have a significant impact on businesses that have high product quality and safety requirements. The method of selecting a portion of use case execution via TCP techniques is therefore not applicable in many enterprises.
Disclosure of Invention
The invention aims to: aiming at the defects of the existing method, the test method and the test system for sequencing the priority of the test cases by utilizing the ensemble learning can improve the fault detection rate and shorten the time from the test of a tester to the repair of a developer.
In order to achieve the purpose, the technical scheme of the invention is as follows: the test method for carrying out test case priority sequencing by using ensemble learning is provided, and comprises the following steps:
s1: according to the attribute information and the historical execution data of the test cases, establishing a regression prediction model of the execution error rate of the test cases by using an integrated learning algorithm, and predicting the possible error probability of each test case in the test process to be executed;
s2: predicting the error rate of each test case based on an error rate prediction model, carrying out priority ordering on each test case according to the sequence of the predicted error rate from large to small, and executing the test cases according to the priority order;
s3: in the testing process, once the test case fails to be executed, analysis and repair are immediately carried out.
Preferably, before establishing the regression prediction model for the test case execution error rate, the method further includes the following steps:
obtaining attribute information of all test cases to be sequenced, including but not limited to: test case name, ID, creation time;
obtaining historical execution information of all test cases to be sequenced, including but not limited to: executing date, executing duration, executing result of this time, executing result of last time and historical executing error rate;
the calculation method of the historical execution error rate comprises the following steps: the number of times of executing errors of the test case (within a certain time), and the total number of times of executing the test case (within the certain time);
and combining the attribute information of the test case and the historical data into the combination of the attribute information of the test case and the execution information of a certain test.
Preferably, before establishing the regression prediction model for the test case execution error rate, the method further comprises the following steps:
processing missing or abnormal data in the test case information, and converting the data information into a format which can be identified by a model;
processing missing or abnormal data in test case information includes: processing the missing value and the abnormal value, and completing the attribute information of the test case in a mode of manual completion or mean value, similar value and majority value when the attribute information of the test case is missing or abnormal; when the execution data of the test case is missing or abnormal, the test case needs to be executed again to obtain the execution data such as the execution time length and the like;
and (3) processing outdated data: and marking the test cases which are not executed within a certain time, and deleting the test cases from the test case set after the test cases are examined so as to ensure the timeliness of the current test case set.
Preferably, the specific step of establishing a regression prediction model for the test case execution error rate by using the ensemble learning algorithm includes:
s1-1: setting parameters of an ensemble learning model;
s1-2: selecting various characteristics of the data, selecting the characteristics with high error rate correlation coefficient, and using the characteristics as the data characteristics of model training;
s1-3: the data set is divided into a training data set and a testing data set, the training data set is used for training the model, and the testing data set is used for evaluating the model;
s1-4: and taking the current execution result of the test case in certain execution as a label, and taking other characteristics except the current execution result as input data for training to obtain a prediction model of which the prediction result is the execution error rate of each test case in next execution.
Preferably, the specific method for executing the test case according to the ordered priority sequence in the test process includes:
preferentially executing the test cases with high predicted error rate, wherein the probability of finding code faults in the test cases with high error rate is higher, so that the code faults can be found as early as possible;
and the found code faults are immediately repaired, and the time from testing to repairing is shortened.
The invention also provides a test system for carrying out test case priority sequencing by utilizing ensemble learning, which comprises a data processing module, a model establishing module, a prediction sequencing module and an execution test module;
the data processing module is used for processing null values and abnormal values of the data and converting the data into a format which can be identified by a model;
the model building module builds a test case error rate prediction model based on the attribute information and the historical execution data of the test cases by using an integrated learning algorithm, and the model is used for predicting the possible error probability of each test case in the next test process;
the prediction sorting module predicts the error rates of all the test cases to be sorted by using the established error rate prediction model and carries out priority sorting on the test cases according to the sequence of the error rates from large to small;
the execution test module is used for generating a test script with a corresponding execution sequence according to the priority sequence of the test case to assist a tester in testing.
Preferably, the content of the data processing performed by the data processing module includes:
processing abnormal values in the test case data, including null values and repeated values;
processing the characteristic value in the test case data, wherein the characteristic value is converted into a digital value through one-hot coding, characteristic selection is carried out through a correlation coefficient method, and the like;
converting a data format, namely converting the data into a form which is easy to read based on a regression model of ensemble learning;
and segmenting the data into training data and test data according to a certain proportion, wherein the training data is used for training the model, and the test data is used for evaluating the model.
Preferably, the specific process of establishing the error rate prediction model by the model establishing module is setting model parameters, reading training data, training a model, reading test data, evaluating the model and adjusting parameters;
setting model parameters refers to setting necessary parameters to enable the prediction result of the model to be closer to a true value;
reading training data refers to reading a divided training data set for training a model in the next step;
the training model is input into a training data set, an error rate prediction model is trained by utilizing an integrated learning algorithm, the error rate of next execution of the test case is predicted, and the model is stored in a specified path;
the evaluation model is used for predicting the currently established model after reading the test data set and comparing the model with the original label to evaluate whether the current model reaches an expected target;
the adjusting parameters refer to adjusting the set model parameters according to the test results.
Preferably, the specific process of the prediction ranking performed by the prediction ranking module is as follows: reading data information of test cases to be sequenced, predicting and sequencing according to a prediction result;
the prediction means that after the information of the test cases to be sequenced is read, prediction is carried out on the current model to obtain the prediction error rate of each test case;
the sequencing according to the prediction result means that the results predicted according to the model are arranged in a sequence from high to low, and then the sequence is converted into the priority sequencing of the test cases according to the corresponding relation between the test cases and the prediction value, namely the test cases with high prediction error rate have higher priority.
Preferably, the specific process of the execution test module performing the execution test is as follows:
reading the latest priority sequencing result, packaging the test case execution command, the test case name and the information including the log path and the log format into a test script file named by the timestamp according to the sequence, and automatically downloading.
The invention has the beneficial effects that: and establishing an error rate prediction model through attribute data, historical execution data, historical error rate and other information of the test cases, and predicting the error rate of each test case in the test process to be executed. The method comprises the steps of analyzing the characteristics of test cases with high failure rate, predicting the error rate of each test case in next execution, and recommending a user to preferentially execute the test cases with high predicted error rate. Once errors are found in the testing process, the errors are immediately handed to developers for analysis and repair, and the testing process is continued. By using the available data, the large-scale test cases are analyzed and prioritized in a short time, so that the time of the whole test-repair process is shortened, and the test efficiency is improved.
Drawings
Fig. 1 is a flowchart of a test method for prioritizing test cases using ensemble learning according to a first embodiment of the present invention.
Fig. 2 is a schematic diagram of an error rate prediction model building process according to a first embodiment of the present invention.
Fig. 3 is a schematic diagram of a test case data information merging format in an embodiment of the present invention.
Fig. 4 is a schematic diagram of a sequence of partial test cases sorted by prediction error rate in the first embodiment of the present invention.
Fig. 5 is a flowchart of a test method for performing data feedback and using ensemble learning to prioritize test cases according to a second embodiment of the present invention.
Fig. 6 is data information of the test case T in the last model training process in the second embodiment of the present invention.
Fig. 7 is data information used for the model training of this time after the test case T undergoes data feedback in the second embodiment of the present invention.
Fig. 8 is a flowchart of a hierarchical test method for prioritizing test cases using ensemble learning according to a third embodiment of the present invention.
Fig. 9 is a schematic structural diagram of a test system for prioritizing test cases by utilizing ensemble learning according to a fourth embodiment of the present invention.
Fig. 10 is a schematic diagram of data converted into LIBSVM format in the fourth embodiment of the present invention.
Fig. 11 is a flowchart of the operation of the model building module in the fourth embodiment of the present invention.
Detailed Description
The present invention will be described in further detail with reference to the accompanying drawings and examples.
The invention provides a test method for carrying out test case priority sequencing by utilizing ensemble learning, which comprises the following steps:
s1: according to the attribute information and the historical execution data of the test cases, establishing a regression prediction model of the execution error rate of the test cases by using an integrated learning algorithm, and predicting the possible error probability of each test case in the test process to be executed;
s2: predicting the error rate of each test case based on an error rate prediction model, carrying out priority ordering on each test case according to the sequence of the predicted error rate from large to small, and executing the test cases according to the priority order;
s3: in the testing process, once the test case fails to be executed, analysis and repair are immediately carried out.
Preferably, before establishing the regression prediction model for the test case execution error rate, the method further includes the following steps:
obtaining attribute information of all test cases to be sequenced, including but not limited to: test case name, ID, creation time;
obtaining historical execution information of all test cases to be sequenced, including but not limited to: executing date, executing duration, executing result of this time, executing result of last time and historical executing error rate;
the calculation method of the historical execution error rate comprises the following steps: the number of times of executing errors of the test case (within a certain time), and the total number of times of executing the test case (within the certain time);
and combining the attribute information of the test case and the historical data into the combination of the attribute information of the test case and the execution information of a certain test.
Before the regression prediction model of the error rate is executed on the test case, the method also comprises the following steps:
processing missing or abnormal data in the test case information, and converting the data information into a format which can be identified by a model;
processing missing or abnormal data in test case information includes: processing the missing value and the abnormal value, and completing the attribute information of the test case in a mode of manual completion or mean value, similar value and majority value when the attribute information of the test case is missing or abnormal; when the execution data of the test case is missing or abnormal, the test case needs to be executed again to obtain the execution data such as the execution time length and the like;
and (3) processing outdated data: and marking the test cases which are not executed within a certain time, and deleting the test cases from the test case set after the test cases are examined so as to ensure the timeliness of the current test case set.
The specific steps of establishing a regression prediction model for the test case execution error rate by using an ensemble learning algorithm comprise:
s1-1: setting parameters of an ensemble learning model;
s1-2: selecting various characteristics of the data, selecting the characteristics with high error rate correlation coefficient, and using the characteristics as the data characteristics of model training;
s1-3: the data set is divided into a training data set and a testing data set, the training data set is used for training the model, and the testing data set is used for evaluating the model;
s1-4: and taking the current execution result of the test case in certain execution as a label, and taking other characteristics except the current execution result as input data for training to obtain a prediction model of which the prediction result is the execution error rate of each test case in next execution.
The specific method for executing the test case according to the arranged priority sequence in the test process comprises the following steps:
preferentially executing the test cases with high predicted error rate, wherein the probability of finding code faults in the test cases with high error rate is higher, so that the code faults can be found as early as possible;
and the found code faults are immediately repaired, and the time from testing to repairing is shortened.
The invention also provides a test system for carrying out test case priority sequencing by utilizing ensemble learning, which comprises a data processing module, a model establishing module, a prediction sequencing module and an execution test module;
the data processing module is used for processing null values and abnormal values of the data and converting the data into a format which can be identified by a model;
the model building module builds a test case error rate prediction model based on the attribute information and the historical execution data of the test cases by using an integrated learning algorithm, and the model is used for predicting the possible error probability of each test case in the next test process;
the prediction sorting module predicts the error rates of all the test cases to be sorted by using the established error rate prediction model and carries out priority sorting on the test cases according to the sequence of the error rates from large to small;
the execution test module is used for generating a test script with a corresponding execution sequence according to the priority sequence of the test case to assist a tester in testing.
The content of the data processing module for data processing comprises the following steps:
processing abnormal values in the test case data, including null values and repeated values;
processing the characteristic value in the test case data, wherein the characteristic value is converted into a digital value through one-hot coding, characteristic selection is carried out through a correlation coefficient method, and the like;
converting a data format, namely converting the data into a form which is easy to read based on a regression model of ensemble learning;
and segmenting the data into training data and test data according to a certain proportion, wherein the training data is used for training the model, and the test data is used for evaluating the model.
The specific process of the model establishing module for establishing the error rate prediction model comprises the steps of setting model parameters, reading training data, training a model, reading test data, evaluating the model and adjusting parameters;
setting model parameters refers to setting necessary parameters to enable the prediction result of the model to be closer to a true value;
reading training data refers to reading a divided training data set for training a model in the next step;
the training model is input into a training data set, an error rate prediction model is trained by utilizing an integrated learning algorithm, the error rate of next execution of the test case is predicted, and the model is stored in a specified path;
the evaluation model is used for predicting the currently established model after reading the test data set and comparing the model with the original label to evaluate whether the current model reaches an expected target;
the adjusting parameters refer to adjusting the set model parameters according to the test results.
The specific process of the prediction sorting module for performing prediction sorting is as follows: reading data information of test cases to be sequenced, predicting and sequencing according to a prediction result;
the prediction means that after the information of the test cases to be sequenced is read, prediction is carried out on the current model to obtain the prediction error rate of each test case;
the sequencing according to the prediction result means that the results predicted according to the model are arranged in a sequence from high to low, and then the sequence is converted into the priority sequencing of the test cases according to the corresponding relation between the test cases and the prediction value, namely the test cases with high prediction error rate have higher priority.
The specific process of the execution test module for executing the execution test is as follows:
reading the latest priority sequencing result, packaging the test case execution command, the test case name and the information including the log path and the log format into a test script file named by the timestamp according to the sequence, and automatically downloading.
For ease of understanding, the main inventive concepts of the embodiments of the present invention are briefly described.
Example one
Fig. 1 is a flowchart of a test method for test case priority ranking based on ensemble learning in an embodiment one, and the present scheme is applied to a scenario where too many test cases result in too long test time affecting product iteration speed or test time is limited and a complete test case cannot be executed, and specifically includes the following steps:
and step 110, establishing a regression prediction model of the error rate of the test case by using an ensemble learning algorithm according to the attribute information and the historical execution data of the test case, wherein the model is used for predicting the possible error probability of each test case in the test process to be executed.
When a model is built, data information of a test case is collected first, then is processed, and is converted into data that can be identified by the model, and an error rate prediction model is built, where fig. 2 is a building process of the error rate prediction model in this embodiment, and the specific steps are as follows:
and step 111, collecting effective information of the test cases, and combining the effective information into attribute information of the test cases and a form of executing data at a certain time.
And when the test case information is collected, effective information is mined from the database, the test log, the integration log and the test case code or script to be used as training data of the model.
In this embodiment, a test script comprises two parts, namely annotation information and script code. Wherein, the name, the level, the belonged type and the creation time of the test case are indicated in the annotation information. One test log records the result of one test, including the time of the test, the state of the product during the test, the output information of all executed test cases, the execution result, the error reporting information and the like;
the information collected in this embodiment is: the method comprises the following steps of using case name (name), using case type (type), creating time (create time), executing time (test time), executing result (result), last executing result (last result), executing duration (duration), total executing times and the like (execution times), and total failure times (failed times).
Firstly, collecting attribute information of a test case from a test case script, and crawling keywords 'type', 'name' and the like through the script to obtain information: use case name (name), use case type (type), creation time (create time). Collecting the execution data of the test case from the latest test log, and crawling the keywords "result", "duration" and the like through the script, wherein the obtained information comprises the following information: execution time (test time), result of this execution (result), duration of execution (duration), and the like.
And crawling all test logs in three months, counting the execution conditions of all test cases, and calculating the total execution times (execution times) and the total failure times (failed times) of all the test cases. Specifically, taking a certain test case as an example, if the name of the test case appears in a certain test log, it represents that the test case is executed in the test, and the total execution times (execution times) of the test case is increased by one, and if not, the total execution times is unchanged. Under the condition that the test case is executed, if the execution result of the test case in the test log is 'FAIL', the test case is represented to FAIL to be executed in the test, and the total failure times (failed times) of the test case is increased by one; if the execution result is PASS, the case is successfully executed, and the data is not changed.
The execution result of the test case in the last test is extracted from the last test log, and the result is used as the value of the last execution result (last result) of the test case.
Specifically, the last execution result (last result) indicates whether a certain use case was successfully or unsuccessfully executed in the last test process, and is an important characteristic of the training data. If a test case in the last test is successfully executed, i.e. no code fault is found, the value of this feature is "PASS", and if it FAILs, it is "FAIL". The execution result (result) is whether the execution of the test case in the test process of the log record is successful or failed.
The historical execution error rate (history failure) is also an important feature, and refers to the probability of the execution failure of a certain test case within a certain time range, and can reflect the situation of the execution failure of the test case within a period of time. The method aims to recommend the test cases with high error rate to be preferentially executed by the user. The historical error rate is calculated in the present embodiment using the ratio of the total number of failures or the like (executed times) to the total number of executions (failed times).
Selecting the latest execution data, associating the attribute information of the test case with the current execution information through the name of the test case, and combining the attribute information of the test case and the execution data of a certain time, as shown in fig. 3.
And step 112, processing abnormal values in the test case data, and converting the data format into a format which can be identified by the model.
Processing deletions and outliers. The attribute information of the missing test cases, namely the information such as the grade, the creation time and the like, is obtained by consulting the tester. The unavailable test cases need to be associated with the same period and the same type, and are completed by means of mean values, similarity values or most worthwhile modes. For the condition that the test case execution data is missing or abnormal, the test case needs to be re-executed to obtain the execution data such as the execution time length.
And eliminating the overtime data. And marking the unexecuted test cases within three months, and deleting the test cases from the test case set after manual examination so as to ensure the timeliness of the current test case set.
In this embodiment, the test case information further includes a plurality of category features, which are all represented by character strings, and the character strings in the data cannot be recognized when the model is trained. It is necessary to convert the character string into numbers by means of a one-hot encoding. For example, there are four types (type) features, and the four types are respectively represented by four numbers from 1 to 4; there are two types of execution results: success and failure, setting the result value to 0 if failure, and to 1 if success.
Optionally, after the data format processing is completed, the features of the current test case are selected, and features with high error rate correlation coefficients are selected by using methods such as a correlation coefficient method and the like, so as to be used by a model and an algorithm.
And 113, setting model parameters, and constructing a test case error rate prediction model by using the existing data.
Firstly, segmenting data into training data and testing data, wherein the training data is used for training a model, and the testing data is used for evaluating the model.
In this embodiment, a gradient lifting tree algorithm in ensemble learning is used to research use case characteristics with a high error rate. The parameters that may need to be set are therefore: the iteration depth of the algorithm, the loss function, the maximum feature quantity, the maximum depth of the regression tree, the maximum leaf node number and the like.
Specifically, considering that the feature number of the sample is not large, it is not necessary to limit, and the maximum feature number and the maximum leaf node number may be obtained by using default parameter values. Since the present model is a predictive model, the most common mean square error function in the regression problem is chosen as the loss function. And setting the maximum iteration times and the maximum depth of the regression tree according to the data condition.
Inputting training data and training the model. And using the data executed last time as training data, using the execution result of the test case at this time as a label, and using other characteristics as training data input. In this embodiment, the model establishment steps are as follows:
where T is the maximum number of iterations of the algorithm, m is the number of samples (i.e., the number of test cases used for training), L (y, f (x)) is the loss function, f (y, f (x)), (m) is the number of samples used for training, andt(x) And obtaining a strong learner for the t round training.
First of all, a weak learner f is initialized which minimizes the loss function L0。
Secondly, the weak learner is iteratively trained, with T being 1,2.. T:
(1) for each test case sample, respectively calculating the negative gradient rti：
(2) Using negative gradient rtiFitting a regression tree to obtain the t-th regression tree, wherein the corresponding leaf node region is Rtj，j＝1,2...J；
(3) Calculating the best fit value ctj：
(4) Adding the weak learner into the trained model to obtain a new strong learner:
after the cycle is completed, the final strong learner, i.e., the error rate prediction model in the present embodiment, is obtainedT(x)。
The result of applying the data to the predictive model is a probability that represents the error rate of the test case during the next test. And testing in the trained model by using the test data, outputting a prediction result and an actual value of the test data and an MSE (mean Squared error) value of the model, and evaluating the model. The MSE value refers to the mean square error of the model and is used for evaluating the prediction accuracy of the model. And adjusting the model parameters according to the test result until the expected standard is reached.
And 120, predicting the error rate of each test case based on the error rate prediction model, performing priority ordering on each test case according to the sequence of the predicted error rate from large to small, and executing the test cases according to the priority order.
Importing the attribute information of the test case to be executed and the historical execution data into the error rate prediction model established in the step 113, and predicting to obtain a probability value. The test cases are arranged according to the sequence of the predicted values from large to small, and fig. 4 is a part of the sequence of the arranged test cases, and the sequence is the sequence of the test cases to be executed.
During testing, the test cases are executed according to the sequence.
Specifically, in the process of executing the test cases according to the sorted sequence, if an error is found when the nth test case TN is executed, the tester submits the error report information of the TN to the developer for analysis and repair, and then continues to execute other test cases. And when the test case is executed to the Nth (N' > N) test case, the developer finishes repairing the error found by the TN and continues to wait for the submission of the code fault information of the subsequent tester. The repair of the fault discovered by the TN so far takes no additional time, and the repair process is performed in parallel with the test.
According to the technical scheme of the embodiment of the invention, the attribute information and the historical execution data of the test case are collected and processed and analyzed, so that the characteristics of the test case with high error rate are discovered. And establishing a test case error rate prediction model by using an ensemble learning algorithm, and predicting the probability of possible errors in the next test process. And according to the prediction result, carrying out priority sequencing on the test cases according to the sequence of the prediction error rate from large to small, and executing the test cases according to the sequence. In the testing process, the test cases with high error probability are preferentially executed, and the error finding efficiency is obviously improved compared with the original execution sequence. The errors discovered first are submitted to the developer for modification without having to wait for the test to be completed before submitting the errors. The time of the whole testing and repairing process is obviously shortened, and even the state that all code faults are repaired when the testing is finished can be achieved, so that the testing speed is accelerated.
Example two
Fig. 5 is a flowchart of performing data feedback according to a second embodiment of the present invention, and this embodiment adds a step of performing feedback after execution on the basis of the above embodiment. A test method for performing test case priority ranking by ensemble learning and a data feedback method thereof according to a second embodiment of the present invention are described below with reference to fig. 5, including the following steps:
The historical execution data of the test case used in the model building is the data generated in the last test process, so the historical execution data used in the model building in the last test needs to be updated and replaced by the execution result. That is, the latest execution result needs to be fed back to the modeling data to maintain the timeliness of the test case execution information, so that the model is continuously changed to meet the current test situation.
The attribute information of the test case can not be changed under the condition of not modifying the test case and can not be changed along with the progress of the test.
Specifically, when data feedback is performed, the case name (name), the case type (type), and the creation time (create time) of the test case are not changed. Updating the execution duration (duration) of the test case, adding one to the total execution times (executed times), and determining whether the total failure times (failed times) changes according to the latest execution result. Meanwhile, the value of the feature of the "last execution result" needs to be set as the value of the "current execution result", and the value of the "current execution result" is set according to the latest execution result. Accordingly, the 'historical execution error rate' is updated along with the change of the execution times and the failure times of the test cases.
Taking test case T as an example, it failed to execute in the last test process. Fig. 6 is data information of the test case T during the last test for model training, and fig. 7 is data used for the current model training after the test case T is updated by data feedback after the last test is completed, where the "last execution result" is PASS and the "current execution result" is FAIL.
And step 220, establishing a regression prediction model of the test case execution error rate by using an ensemble learning algorithm according to the attribute information and the historical execution data of the test case, wherein the model is used for predicting the possible error probability of each test case in the test process to be executed.
And establishing an error rate prediction model according to the fed-back data. And training by taking the execution result as a label and other characteristics as training data. When the model is trained, the characteristics of the test case with high failure rate are analyzed, the characteristics of the test case failing in the last execution process are given higher weight, and the obtained prediction error rate is correspondingly increased in the prediction. According to the thought, the probability that the test case which fails to be executed fails in the next execution process is higher.
And step 230, predicting the error rate of each test case based on the error rate prediction model, performing priority ordering on each test case according to the sequence of the predicted error rate from large to small, and executing the test cases according to the priority order.
And 240, in the test process, once the test case fails to be executed, immediately submitting the test case to a developer for analysis and repair.
Because the test case failing in the last execution process obtains a higher prediction error rate in the prediction, the position of the test case in the test case priority sequence is more advanced when the test case priority sequence is carried out. During the test process, the test case is executed earlier, namely, the user is recommended to execute the test case with high prediction error rate preferentially. The errors are discovered earlier, meaning that developers can deal with and repair earlier, and the time from the code fault being discovered to being repaired is greatly shortened.
During the test, the steps 210, 220, 230 and 240 are executed in a loop, the steps are executed once per test, the latest execution result is fed back to the data, and accordingly an error rate prediction model is built and prediction sequencing is performed.
According to the technical scheme of the embodiment of the invention, after the test process is executed according to the optimized result at a certain time, the executed result is updated in the corresponding test case information, so that a feedback basis is provided for the establishment and prediction of the next model. Through the process of data feedback, the model is continuously adjusted to be more consistent with the current test situation. Even if a certain case continuously predicts the low error rate, as long as the case has errors in certain execution, the result of the case in the subsequent prediction is correspondingly improved by feeding back data in time, and the condition that certain cases are always in the low prediction error rate and are not regarded as important is avoided.
EXAMPLE III
Fig. 8 is a flowchart of a hierarchical test method for prioritizing test cases using ensemble learning according to a third embodiment of the present invention. The present embodiment adds the grades 1-3 of the test cases on the basis of the above embodiment, that is, each test case has its own grade, and this grade represents the importance degree of the tested content. The level 1 represents that the test case is a basic test case, and the tested content is the basic and core functions of the software product, and has the greatest influence on the user. Rank 2. And the level 3 represents that the content range of the test case test is very small, the detail function is emphasized, the influence on the user is minimum, and the importance degree is minimum. During testing, the failure rate of the test case with high priority is ensured to be as low as possible, and the influence degree on the user is minimum.
A description is given below, with reference to fig. 8, of a layered test method for performing test case prioritization by ensemble learning according to a third embodiment of the present invention, where the method includes the following steps:
and 311, collecting the grade information of the test cases, and dividing the test cases into three sets according to the grade.
Specifically, in this embodiment, the case class is marked in the test case script. Crawling the 'level' keyword through a crawler to obtain the level (level) of each test case, and dividing the test cases into three sets according to the level: a level 1 set, a level 2 set, and a level 3 set. The level (level) is not trained as a feature of the test case, but is used only for grouping.
And step 312, collecting other attribute information of the test case and historical execution data for processing, and establishing an error rate prediction model by using an ensemble learning algorithm.
And 320, predicting the error rate of each test case based on the error rate prediction model, carrying out priority ordering on each test case according to the sequence of the predicted error rate from large to small, and executing the test cases according to the priority order.
And predicting the error rate of each test case in the group according to the groups, and sequencing the test cases in the group according to the sequence of the predicted values from large to small. When the test cases are executed, the test cases are executed according to the sequence that the level grouping is from small to large, and the error rate in the group is from large to small, so that the test case with the highest level and the highest error probability is found is ensured to be executed firstly, and the test case with the lowest level and the lowest error probability is found is executed finally.
Specifically, the level 1 group is taken as an example. If the nth test case TN of the level 1 group fails to execute, this is a test case that is critical to the user experience. And the tester delivers the error information and the positioning thereof to the developer for analysis and repair, and then continues to execute other cases. The errors found by the TN have been repaired when the test cases to the next two groups are executed. The process realizes the test and repair, when the test is carried out to the second half, the probability of error finding of the test case is very low, and the error information submitted to developers is very little. It is even possible to reach the situation where all code failures have been successfully repaired at the end of the test.
According to the technical scheme of the embodiment of the invention, the attribute information and the historical execution data of the test case are collected, and the integrated learning algorithm is utilized to establish the test case error rate prediction model. According to the error rate obtained by model prediction, the test cases are subjected to priority sequencing according to the sequence from large to small, and the test cases are executed according to the sequence, so that the fault discovery speed is increased, and the fault repair time is shortened. Under the condition of definite classification of the test cases, the method can optimize the test cases in a classification manner, and ensures that important test cases are executed preferentially so as to improve the test efficiency. The method is still applicable when the test case has additional requirements or attributes.
Example four
Fig. 9 is a schematic structural diagram of a device for prioritizing test cases based on ensemble learning according to a fourth embodiment of the present invention, where the device for prioritizing test cases includes: a data processing module 410, a model building module 420, a prediction ranking module 430, and an execution testing module 440.
The data processing module 410 is configured to process data of a test case, process an abnormal value in the data, and convert the data into a format that can be recognized by a model, and includes:
the abnormal value processing unit is responsible for processing null values, abnormal values, repeated values and the like in the test case information;
the characteristic processing unit is responsible for converting the character string into numbers in a form of unique hot coding and performing optional functions such as characteristic selection and the like;
and the data format conversion unit is responsible for converting the data into a corresponding format according to the external packet for realizing the algorithm. In this embodiment, implementing the ensemble learning algorithm using Spark MLlib requires converting the data into LIBSVM format, as shown in fig. 10. And the LIBSVM file after the format conversion is completed is stored under the specified path. Spark MLlib is an open source machine learning library of the large-scale data computing engine Spark for providing a machine learning algorithm interface.
The model building module 420 uses an ensemble learning algorithm to build an error rate prediction model based on the data information of the test cases, and predicts the error rate of each test case in the next execution process, and the working flow is shown in fig. 11, and it includes:
the data reading unit is responsible for reading the data file, segmenting the data file into training data and test data and using the training data and the test data for a subsequent modeling process;
and the model establishing unit is responsible for establishing the error rate prediction model. Specifically, an external machine learning package Spark MLlib is introduced, and a machine learning interface of the external machine learning package Spark MLlib is called to realize an ensemble learning algorithm. Reading training data through a data reading unit, establishing an error rate prediction model by using an ensemble learning algorithm, and storing the error rate prediction model under a path specified by a system;
and the model evaluation unit is responsible for evaluating the established model by using the test data and judging whether the performance of the model reaches an expected result. Specifically, reading the established prediction model under the specified path, reading the training data through the data reading unit, and importing the training data into the model for prediction. And comparing the original label with the prediction result, and outputting a model MSE value.
The prediction sorting module 430 predicts the error rate of each test case based on the established model, and arranges the test cases according to the order of the error rate from large to small, which comprises:
the prediction unit reads the error rate prediction model under the specified path, inputs the data of the test case to be executed and predicts the error rate of the test case;
and the sequencing unit is used for sequencing the test cases according to the sequence of the prediction error rate from high to low and outputting a sequencing result.
The execution test module 440 is configured to generate a test script according to the sorted test case sequence, without manually arranging execution.
Specifically, the test script encapsulates an execution command of the test case, an execution sequence of the test case, a path of the test log, content and output information of the test log, and the like. When the test script is generated, the file is named by the timestamp and is automatically downloaded after the content is written.
It is to be noted that the foregoing is only illustrative of the preferred embodiments of the present invention and the technical principles employed. It will be understood by those skilled in the art that the present invention is not limited to the particular embodiments described herein, but is capable of various obvious changes, rearrangements and substitutions as will now become apparent to those skilled in the art without departing from the scope of the invention. The invention is thus not limited to the embodiments described above, but may include many other equivalent embodiments without departing from the inventive concept, the scope of which is to be determined from the scope of the appended claims.
Claims (9)
1. A test method for carrying out test case priority sequencing by ensemble learning is characterized by comprising the following steps:
s1: according to the attribute information and the historical execution data of the test cases, establishing a regression prediction model for the execution error rate of the test cases by using an integrated learning algorithm, and predicting the error rate of each test case in the test process to be executed;
s2: predicting the error rate of each test case based on a regression prediction model of the error rate, carrying out priority ordering on each test case according to the sequence of the predicted error rate from large to small, and executing the test cases according to the priority order;
s3: in the test process, once the test case fails to be executed, the test case is immediately analyzed and repaired;
the specific steps of establishing a regression prediction model for the test case execution error rate by using an ensemble learning algorithm comprise:
s1-1: setting parameters of an ensemble learning model;
s1-2: selecting various characteristics of the data, selecting the characteristics with high error rate correlation coefficient, and using the characteristics as the data characteristics of model training;
s1-3: the data set is divided into a training data set and a testing data set, the training data set is used for training the model, and the testing data set is used for evaluating the model;
s1-4: taking the current execution result of the test case in certain execution as a label, taking other characteristics except the current execution result as input data for training, and obtaining a regression prediction model of which the prediction result is the execution error rate of each test case in next execution;
the model is a prediction model, so the most common mean square error function in the regression problem is selected as a loss function, and the maximum iteration times and the maximum depth of a regression tree are set according to the data condition;
inputting training data, training a model, using the data executed last time as the training data, using the execution result of the test case this time as a label, and using other characteristics as the training data input, wherein the model establishment steps are as follows:
where T is the maximum number of iterations of the algorithm, m is the number of samples, i.e., the number of test cases used for training, L (y, f (x)) is a loss function, f (y, f (x)) is a function of the losst(x) A strong learner obtained for the t-th round of training;
first of all, a weak learner f is initialized which minimizes the loss function L0；
Secondly, the weak learner is iteratively trained, with T being 1,2.. T:
(1) for each test case sample, respectively calculating the negative gradient rti：
(2) Using negative gradient rtiFitting a regression tree to obtain the t regression tree, wherein the corresponding leaf node region is Rtj，j＝1,2...J；
(3) Calculating the best fit value ctj：
(4) Adding the weak learner into the trained model to obtain a new strong learner:
after the cycle is completed, the final strong learner, i.e., the error rate prediction model in the present embodiment, is obtainedT(x)。
2. The method according to claim 1, further comprising the following steps before establishing the regression prediction model for the test case execution error rate:
obtaining attribute information of all test cases to be sequenced, including but not limited to: test case name, ID, creation time;
obtaining historical execution data of all test cases to be sequenced, including but not limited to: executing date, executing duration, executing result of this time, executing result of last time and historical executing error rate;
the calculation method of the historical execution error rate comprises the following steps: the number of times of executing errors of the test case in a certain time \ the total number of times of executing the test case in a certain time;
and combining the attribute information of the test case and the historical execution data into the combination of the attribute information of the test case and the execution information of a certain test.
3. The method of claim 1, wherein before the step of building a regression prediction model for the test case execution error rate, the step of:
processing missing or abnormal data in the test case information, and converting the data information into a format which can be identified by a model;
processing missing or abnormal data in test case information includes: processing the missing value and the abnormal value, and completing the attribute information of the test case in a mode of manual completion or mean value, similar value and majority value when the attribute information of the test case is missing or abnormal; when the execution data of the test case is missing or abnormal, the test case needs to be executed again to obtain the execution data such as the execution time length and the like;
and (3) processing outdated data: and marking the test cases which are not executed within a certain time, and deleting the test cases from the test case set after the test cases are examined so as to ensure the timeliness of the current test case set.
4. The method for testing the priority ranking of the test cases by using ensemble learning according to claim 1, wherein the specific method for executing the test cases according to the ranked priority sequence in the testing process comprises:
preferentially executing the test cases with high predicted error rate, wherein the probability of finding code faults in the test cases with high error rate is higher, so that the code faults can be found as early as possible;
and the found code faults are immediately repaired, and the time from testing to repairing is shortened.
5. A test system for carrying out test case priority sequencing by utilizing ensemble learning is characterized by comprising a data processing module, a model establishing module, a prediction sequencing module and an execution test module;
the data processing module is used for processing null values and abnormal values of the data and converting the data into a format which can be identified by a model;
the model establishing module establishes a regression prediction model of the error rate of the test cases by utilizing an integrated learning algorithm based on the attribute information and the historical execution data of the test cases, and the model is used for predicting the error rate of each test case in the next test process;
the prediction sorting module predicts the error rates of all test cases to be sorted by using the established regression prediction model of the error rates and sorts the test cases according to the sequence of the error rates from large to small;
the execution test module is used for generating a test script with a corresponding execution sequence according to the priority sequence of the test case to assist a tester in testing.
6. The system of claim 5, wherein the data processing module performs data processing comprising:
processing abnormal values in the test case data, including null values and repeated values;
processing the characteristic value in the test case data, including converting the characteristic value into a digital value through one-hot coding and performing characteristic selection through a correlation coefficient method;
converting a data format, namely converting the data into a form which is easy to read based on a regression prediction model of the error rate of ensemble learning;
and segmenting the data into training data and test data according to a certain proportion, wherein the training data is used for training the model, and the test data is used for evaluating the model.
7. The system of claim 5, wherein the model building module builds the regression prediction model of error rate by setting model parameters, reading training data, training a model, reading test data, evaluating a model, and adjusting parameters;
setting model parameters refers to setting necessary parameters to enable the prediction result of the model to be closer to a true value;
reading training data refers to reading a divided training data set for training a model in the next step;
the training model is input into a training data set, an error rate prediction model is trained by utilizing an integrated learning algorithm, the error rate of next execution of the test case is predicted, and the model is stored in a specified path;
the evaluation model is used for predicting the currently established model after reading the test data set and comparing the model with the original label to evaluate whether the current model reaches an expected target;
the adjusting parameters refer to adjusting the set model parameters according to the test results.
8. The system for testing case prioritization using ensemble learning according to claim 5, wherein the specific process of the prediction ranking module for performing prediction ranking is as follows: reading data information of test cases to be sequenced, predicting and sequencing according to a prediction result;
the prediction means that after the information of the test cases to be sequenced is read, prediction is carried out on the current model to obtain the prediction error rate of each test case;
the sequencing according to the prediction result means that the results predicted according to the model are arranged in a sequence from high to low, and then the sequence is converted into the priority sequencing of the test cases according to the corresponding relation between the test cases and the prediction value, namely the test cases with high prediction error rate have higher priority.
9. The system for testing the priority ranking of the test cases by using ensemble learning according to claim 5, wherein the specific process of the execution testing module for executing the execution testing is as follows:
reading the latest priority sequencing result, packaging the test case execution command, the test case name and the information including the log path and the log format into a test script file named by the timestamp according to the sequence, and automatically downloading.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010432137.XA CN111427802B (en) | 2020-06-09 | 2020-06-09 | Test method and system for carrying out test case priority sequencing by utilizing ensemble learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010432137.XA CN111427802B (en) | 2020-06-09 | 2020-06-09 | Test method and system for carrying out test case priority sequencing by utilizing ensemble learning |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111427802A CN111427802A (en) | 2020-07-17 |
| CN111427802B true CN111427802B (en) | 2021-06-22 |
Family
ID=71551185
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010432137.XA Active CN111427802B (en) | 2020-06-09 | 2020-06-09 | Test method and system for carrying out test case priority sequencing by utilizing ensemble learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111427802B (en) |
Families Citing this family (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111813443B (en) * | 2020-07-28 | 2023-07-18 | 南京大学 | Method and tool for automatically filling code sample by using Java FX |
| CN112650685B (en) * | 2020-12-29 | 2023-09-22 | 抖音视界有限公司 | Automatic test method, device, electronic equipment and computer storage medium |
| CN112732565B (en) * | 2020-12-31 | 2023-07-18 | 星环信息科技(上海)股份有限公司 | Evaluation method for continuous integration of software, computer equipment and medium |
| CN113342651B (en) * | 2021-06-01 | 2023-11-03 | 南京大学 | Recovery method for testing fuzzy association relation between case defects and cases |
| CN113590472B (en) * | 2021-07-06 | 2023-03-14 | 四川大学 | Test case priority ranking method in regression test |
| CN113672506B (en) * | 2021-08-06 | 2023-06-13 | 中国科学院软件研究所 | Dynamic proportion test case sorting and selecting method and system based on machine learning |
| CN115248783B (en) * | 2022-09-26 | 2022-12-23 | 江西萤火虫微电子科技有限公司 | Software testing method, system, readable storage medium and computer equipment |
| CN117435516B (en) * | 2023-12-21 | 2024-02-27 | 江西财经大学 | Test case priority ordering method and system |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108681505A (en) * | 2018-04-13 | 2018-10-19 | 广东睿江云计算股份有限公司 | A kind of Test Case Prioritization method and apparatus based on decision tree |
| CN109976990A (en) * | 2017-12-27 | 2019-07-05 | 航天信息股份有限公司 | It is a kind of for confirming the method and system of software test case priority |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US7359820B1 (en) * | 2007-01-03 | 2008-04-15 | International Business Machines Corporation | In-cycle system test adaptation |
| CN109634833A (en) * | 2017-10-09 | 2019-04-16 | 北京京东尚科信息技术有限公司 | A kind of Software Defects Predict Methods and device |
| CN110987439B (en) * | 2019-12-05 | 2022-03-22 | 超越科技股份有限公司 | Aeroengine fault prediction method based on Logitics regression and Xgboost model |
-
2020
- 2020-06-09 CN CN202010432137.XA patent/CN111427802B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN109976990A (en) * | 2017-12-27 | 2019-07-05 | 航天信息股份有限公司 | It is a kind of for confirming the method and system of software test case priority |
| CN108681505A (en) * | 2018-04-13 | 2018-10-19 | 广东睿江云计算股份有限公司 | A kind of Test Case Prioritization method and apparatus based on decision tree |
Non-Patent Citations (1)
| Title |
|---|
| "System-level test case prioritization using machine learning";Remo Lachmann等;《IEEE》;20161220;正文摘要及第三节 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111427802A (en) | 2020-07-17 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111427802B (en) | Test method and system for carrying out test case priority sequencing by utilizing ensemble learning | |
| US11675691B2 (en) | System and method for performing automated API tests | |
| CN111459799B (en) | Software defect detection model establishing and detecting method and system based on Github | |
| Hayes et al. | Advancing candidate link generation for requirements tracing: The study of methods | |
| Xia et al. | Collective personalized change classification with multiobjective search | |
| US20210374040A1 (en) | Auto Test Generator | |
| Ray et al. | The uniqueness of changes: Characteristics and applications | |
| CN108763091B (en) | Method, device and system for regression testing | |
| CN107844414A (en) | A kind of spanned item mesh based on defect report analysis, parallelization defect positioning method | |
| CN108763064B (en) | Code test generation method and device based on black box function and machine learning | |
| Li et al. | A scenario-based approach to predicting software defects using compressed C4. 5 model | |
| CN113221960A (en) | Construction method and collection method of high-quality vulnerability data collection model | |
| CN110597718A (en) | Automatic test implementation method and system based on AI | |
| CN112685320B (en) | Software defect repairing method and device based on multiple candidate programs | |
| CN110232130A (en) | Metadata management pedigree generation method, device, computer equipment and storage medium | |
| Sarkar et al. | A method for detecting and measuring architectural layering violations in source code | |
| JP3195031B2 (en) | Test specification generation method, semiconductor device inspection apparatus, and semiconductor device inspection method | |
| CN115221045A (en) | Multi-target software defect prediction method based on multi-task and multi-view learning | |
| Zheng et al. | A method of optimizing multi-locators based on machine learning | |
| CN113505283A (en) | Test data screening method and system | |
| Cao et al. | Automatic Repair of Java Programs Weighted Fusion Similarity via Genetic Programming | |
| CN114791886B (en) | Software problem tracking method and system | |
| Akinsola et al. | Qualitative comparative analysis of software integration testing techniques | |
| CN116882212B (en) | Error reporting and tracing method, device and equipment of non-causal equation of whole vehicle part simulation | |
| Yadav et al. | Hybrid model for software fault prediction |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |