WO2021143175A1 - 测试用例筛选方法、装置及介质 - Google Patents

测试用例筛选方法、装置及介质 Download PDF

Info

Publication number
WO2021143175A1
WO2021143175A1 PCT/CN2020/115580 CN2020115580W WO2021143175A1 WO 2021143175 A1 WO2021143175 A1 WO 2021143175A1 CN 2020115580 W CN2020115580 W CN 2020115580W WO 2021143175 A1 WO2021143175 A1 WO 2021143175A1
Authority
WO
WIPO (PCT)
Prior art keywords
test case
execution
test
case
classifier
Prior art date
Application number
PCT/CN2020/115580
Other languages
English (en)
French (fr)
Inventor
何悦之
赵勇
郝月婵
黄戌
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2021143175A1 publication Critical patent/WO2021143175A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3684—Test management for test design, e.g. generating new test cases
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3676—Test management for coverage analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00—Error detection; Error correction; Monitoring
    • G06F11/36—Preventing errors by testing or debugging software
    • G06F11/3668—Software testing
    • G06F11/3672—Test management
    • G06F11/3688—Test management for test execution, e.g. scheduling of test suites

Definitions

  • This application relates to the technical field of software testing, and specifically relates to a method for screening test cases.
  • this application also relates to a test case screening device and a computer-readable storage medium.
  • Regression test is a testing process that retests the software after modifying the original code of the software. Regression testing can confirm whether the original function of the software is damaged, that is, whether new errors are introduced after the original code is modified, or other problems such as errors in the original code are caused, that is, whether there are defects in the modified software. If there are defects, then developers need to modify the software. In the software development life cycle (System Development Life Cycle, SDLC), each version of the software needs to be fully tested many times before it can be released. Therefore, regression testing is an ongoing task.
  • SDLC System Development Life Cycle
  • test case refers to a set of data provided to the system under test (ie, software) for testing, which generally includes input data, execution conditions, and expected results. During testing, if a test case fails to execute, it may mean the discovery of a defect. To test a software, it is usually necessary to design many test cases to form a test case library. In the life cycle of software development, the test cases in the test case library may be continuously updated to cope with the regression test of the modified software.
  • test cases In regression testing, in order to meet different requirements, testers usually develop different regression testing strategies. Choosing an effective regression testing strategy can accurately screen out suitable test cases from the test case library, thereby improving the efficiency and effectiveness of regression testing. Screening test cases generally includes three types of methods:
  • test cases in the test case library can ensure the lowest risk of defect omissions, but requires high test resources such as time and equipment, resulting in low test efficiency for regression testing.
  • test cases are screened out from the test case library. Specifically, in an implementation, it is first necessary to conduct a coverage analysis on all test cases to determine the mapping relationship between each code line and the test case, and then according to the modified code line and the mapping relationship, from the test case library Screen out suitable test cases.
  • This method can improve the test efficiency to a certain extent, but the modified code does not necessarily cause defects, and the unmodified code does not necessarily have defects. It is possible that the unmodified code may be related to the modified code. And produce defects. In other words, the mapping relationship between the lines of code and the test cases obtained by the coverage analysis actually does not accurately reflect the risk of the test case (that is, the probability of the test case execution failure). Therefore, using this method cannot guarantee the effectiveness of regression testing well.
  • test cases Based on the risk of test cases, select some test cases from the test case library. Specifically, in an implementation manner, the execution data during the execution of the use cases in the previous test is first analyzed to determine the defects found in each test case and the severity level of the defects. Then, according to the number of defects and the severity level of the defects, the risks of these test cases are evaluated, and high-risk use cases are screened out for this regression test.
  • This application provides a test case screening method that combines two types: a screening method based on the risk of the test case and a screening method based on the modified code in the software, which can more accurately screen high-risk use cases, and then Improve the effectiveness of the test.
  • the present application provides a test case screening method, including: obtaining the predicted execution failure rate of at least one first test case; obtaining the correlation degree of at least one second test case; and predicting based on the above at least one first test case The execution failure rate, and the correlation degree of the at least one second test case described above, select high-risk use cases from at least one candidate test case.
  • the aforementioned predicted execution failure rate is obtained by predicting at least one first test case by a use case classifier
  • the aforementioned use case classifier is a classifier obtained by training using historical execution data
  • the aforementioned historical execution data includes executing at least one first test case.
  • At least one execution record generated; the above-mentioned second test case is the test case corresponding to the modified software code, and the above-mentioned correlation degree is used to characterize the degree of correlation between the second test case and the corresponding modified software code; the above-mentioned at least A test case to be selected is a union of at least one first test case and at least one second test case.
  • historical execution data is used as training samples, machine learning technology is used to train a use case classifier, and the use case classifier is used to predict at least one first test case, and at least one first test case can be obtained.
  • the predicted execution failure rate of the test case may specifically be: acquiring historical execution data; training the classifier using historical execution data to obtain a use case classifier; inputting at least one first test case into the use case classifier to obtain the predicted execution failure rate of each of the at least one first test case .
  • the execution record includes a test case identification, an execution result, execution time, and execution duration corresponding to the test case identification.
  • the execution time of the test case may be related to the execution result. Therefore, the test case identification, execution time, and execution result of the test case can be used to train the classifier model to obtain the use case classifier, so that the use case classifier can be used to predict the subsequent performance of the test case The predicted execution failure rate during execution.
  • the execution time is used as the basis for the order of input training samples to ensure the training effect.
  • the execution result can be used to clean the historical execution data.
  • the implementation may specifically be: cleaning the historical execution data according to the execution result in the execution record to obtain the cleaned data; and inputting the execution record in the cleaned data into the classifier to obtain the use case classifier. Cleaning the historical execution data according to the execution results can prevent the execution records with unclear execution results from interfering with training and affecting the prediction accuracy of the classifier.
  • the execution result and the failure reason in the execution record can be used to clean the historical execution data.
  • the implementation manner may specifically be: cleaning the historical execution data according to the execution result and the failure reason to obtain the cleaned data.
  • Using one or more of the failure reasons as a filtering condition can filter out execution records that are not associated with software defects, and avoid these execution records from interfering with training and affecting the prediction accuracy of the classifier.
  • the execution result and the number of executions can be used to clean the historical execution data.
  • the implementation may specifically be: determining the number of execution records corresponding to each first test case in the historical execution data as the number of executions corresponding to each first test case; cleaning the historical execution data according to the execution result and the number of executions , Get the data after cleaning.
  • the execution record corresponding to the test case with too few execution times has relatively low confidence. Therefore, filtering out these execution records can prevent these execution records from interfering with training and affecting the prediction accuracy of the classifier.
  • the execution record in the cleaned data is input into the classifier according to the execution time in the execution record to obtain the use case classifier.
  • this implementation method can ensure the training effect and improve the prediction accuracy of the use case classifier.
  • the relevance of the second test case can be determined through a question sheet or a demand sheet.
  • the implementation method may specifically be: obtaining the demand list/problem list corresponding to the modified software code; acquiring at least one second test case corresponding to the demand list/problem list; and assigning the demand list/problem corresponding to each second test case
  • the number of questionnaires is determined as the relevance of the second test case.
  • the correlation degree of the second test case obtained by adopting this implementation manner can not only reflect the corresponding relationship between the second test case and the modified software code, but also express the closeness of the correlation between the two. This is related to the risk of test cases and whether the test cases can cover test requirements. Based on this, the introduction of the correlation degree for screening high-risk use cases is conducive to more accurate screening of high-risk use cases, and is conducive to improving the effectiveness of subsequent tests.
  • the screening of high-risk use cases can have multiple implementation forms, which can be determined according to different business requirements.
  • the predicted execution failure rate and relevance can be used as a sorting basis to sort all test cases to be selected, and then select high-risk test cases based on the sorting results.
  • the implementation method may specifically be: sorting the above at least one candidate test case based on the predicted execution failure rate and the degree of relevance; selecting a preset number of predicted execution failure rates or high risks with the highest degree of relevance from the ranking results Example.
  • high-risk use cases can be screened out more accurately, which helps to improve the effectiveness of the test.
  • this implementation does not require setting a weight value and is easier to use.
  • the risk corresponding to each test case to be selected can be evaluated according to the weight of the predicted execution failure rate and the degree of relevance, so as to screen out high-risk use cases.
  • the implementation method can specifically be: determine the weight of the predicted execution failure rate and the weight of the correlation degree; use the weight of the predicted execution failure rate and the weight of the correlation degree to calculate the risk evaluation value corresponding to each test case to be selected; and the risk evaluation value Test cases to be selected that are higher than the preset first threshold are determined to be high-risk use cases.
  • the present application provides a test case screening device, including an acquisition module and a processing module; the acquisition module is used to acquire the predicted execution failure rate of at least one first test case; and to acquire the association of at least one second test case Degree; wherein the predicted execution failure rate is obtained by predicting the at least one first test case by the use case classifier, the use case classifier is a classifier obtained by training using historical execution data, and the historical execution data includes executing the at least one first test case At least one execution record is generated; the second test case is the test case corresponding to the modified software code, and the correlation degree is used to characterize the degree of correlation between the second test case and the corresponding modified software code; the above processing module According to the predicted execution failure rate of the at least one first test case, and the correlation degree of the at least one second test case, select high-risk use cases from at least one test case to be selected; among them, at least one test case to be selected It is the union of at least one first test case and at least one second test case.
  • the present application provides a test case screening device, including: at least one memory and at least one processor; at least one memory stores one or more instructions, when one or more instructions are executed by at least one processor , So that the device implements part or all of the steps of any method in the first aspect.
  • the present application provides a computer-readable storage medium.
  • the computer storage medium includes instructions that, when run on a computer, cause the computer to execute part or all of the steps of any method of the first aspect.
  • test case screening device of the second aspect and the third aspect, and the computer-readable storage medium of the fourth aspect are all used to execute the corresponding method provided above, and the beneficial effects that can be achieved can be Refer to the beneficial effects of the corresponding methods above, which will not be repeated here.
  • FIG. 1 is a schematic diagram of screening high-risk use cases in the embodiments of this application.
  • FIG. 2 is a flowchart of one implementation of the test case screening method according to an embodiment of the application.
  • FIG. 3 is a flowchart of one of the implementation manners of the step S200 in the test case screening method according to the embodiment of the application.
  • FIG. 4 is a flowchart of one of the implementation manners of the step S300 in the test case screening method according to the embodiment of the application.
  • FIG. 5 is a flowchart of the second implementation manner of step S300 in the test case screening method according to an embodiment of the application.
  • FIG. 6 is a schematic structural diagram of an implementation manner of the screening device in an embodiment of the application.
  • FIG. 7 is a schematic structural diagram of another implementation manner of the screening device in an embodiment of the application.
  • the risk of a test case refers to the failure rate of the test case during execution.
  • the high risk of a test case means that the test case has a high failure rate during execution, for example, the failure rate is higher than a certain threshold.
  • Test cases with such characteristics are also referred to as high-risk use cases in this application;
  • Ground, the low risk of a test case means that the failure rate of the test case during execution is low, for example, the failure rate is lower than a certain threshold.
  • the value of the above threshold may be arbitrary, and this application does not limit the specific value of the threshold.
  • Defects generally refer to errors in the software code. Sometimes they are also referred to as "errors", "BUG”, etc.
  • errors By executing test cases, you can find some or all of the defects in a software. Whether you can find as many defects as possible, whether you can find as many defects as possible in a limited time, and whether you can find as many relatively serious defects as possible in a limited time depends largely on the test Design and screening of use cases.
  • Test requirements generally refer to the content that needs to be tested, for example, testing whether a certain function is reflected in the software, etc.
  • test validity refers to the proportion of test cases that can find software defects or cover requirements in a test in all test cases executed in the test.
  • test cases need to be executed in the process of testing the software.
  • a test case may be executed one or more times. Each time a test case is executed, a corresponding execution record is generated. The collection of these execution records is called historical execution data.
  • the execution record records the execution of the corresponding test case at a certain time.
  • the execution record may include information such as test case identification, execution result, execution time, and execution duration.
  • the historical execution data may include all execution records generated by the test cases executed in the previous N-1 tests.
  • the historical execution data can include not only the execution records generated by the test cases executed in the previous N-1 tests, but also the execution records generated by the test cases executed in the Nth test.
  • the execution record can include the execution record generated by the execution of the test case in the previous N tests.
  • Figure 1 is a schematic diagram of screening high-risk use cases.
  • the first test case comes from the test case library, including use case 1 to use case 5 and other test cases;
  • the second test case also comes from the test case library, which is the test case corresponding to the modified code, including use case 1 to Use case 3 and other test cases.
  • the screening method of the embodiments of the present application uses historical execution data generated by the software during testing as machine learning training samples, and uses machine learning technology to predict the risk of test cases (that is, predict the execution failure rate), and on the other hand obtain tests
  • the correlation between the use case and the modified software code, combined with the two aspects of information, can more accurately screen out high-risk use cases.
  • Using these high-risk use cases for regression testing can detect defects in the modified software as soon as possible, and promote defect discovery; at the same time, test cases that can cover test requirements can be executed as early as possible, and test cases that cover requirements can be moved forward. In turn, the effectiveness of regression testing is improved.
  • the screening method of the present application can be applied in the regression testing process of all software products, and is particularly suitable for being applied in the regression testing scenarios of software products with faster version iterations, shorter test cycles, or fewer test resources.
  • FIG. 2 is a flowchart of one implementation of the test case screening method according to an embodiment of the application.
  • the test case screening method may include the following steps S100 to S300.
  • the first test case in the embodiment of the present application may come from a test case library.
  • the aforementioned at least one first test case refers to a collection of one or more first test cases.
  • all test cases in the test case library may be used as the first test case.
  • the predicted execution failure rate of the first test case can be predicted by the use case classifier. Before the use case classifier makes predictions, it needs to use the execution records generated when the first test cases were executed before for training.
  • the function describing the classifier model is unchanged, and some parameters in the function can be adjusted.
  • Training can be understood as a process of adjusting some parameters in the classifier model.
  • the above-mentioned classifier model may use an existing machine learning model, or may also use other improved machine learning models, which is not limited in this application.
  • Training the classifier model requires the use of training samples.
  • the form of training samples is "training input data, labeling data”.
  • the "training input data” is input into the classifier model, and the classifier model calculates the "training input data” to obtain the "prediction data”.
  • the classifier model adjusts the values of the parameters in the classifier model according to the difference between the “predicted data” and the “labeled data” so that when the same “training input data” is input into the classifier model, the classification
  • the engine model can output "predicted data” that is closer to the "labeled data”.
  • the parameters in the classifier model have undergone a series of adjustments accordingly. At this time, these parameters in the current classifier model can be determined. In this way, the classifier model is trained, that is, a trained use case classifier is obtained.
  • the training sample may use the execution record in the historical execution data, which contains the execution record generated by the execution of the first test case in the previous test. It should be noted that the training samples may also include execution records generated by other test cases being executed, which is not limited in this application.
  • the execution record can include information such as test case identification, execution result, execution time, and execution duration.
  • test case identifier is used to uniquely identify a test case, so that different test cases can be distinguished through the test case identifier.
  • Execution results can generally include three categories: success, failure, and ambiguity. For a test case, if the actual result obtained after the test case is executed is consistent with the preset expected result, the corresponding execution result is successful; if the actual result is inconsistent with the expected result, the corresponding execution The result is failure.
  • the execution result may also be unclear, which may be caused by a variety of different reasons, such as "not completed", "blocking caused by the execution environment” and so on.
  • the execution time may include the start time point and the end time point of the execution of the test case.
  • the execution time refers to the time it takes to execute the test case.
  • the execution duration may be determined by the difference between the aforementioned end time point and the start time point.
  • the execution record may also include the execution environment.
  • Execution environment refers to the general term for computer hardware, software, network equipment, etc. necessary to execute test cases during software testing.
  • the execution environment may include an operating system, hardware configuration information, and so on.
  • Table 1 shows examples of several execution records included in the historical execution data.
  • the form of training samples is "training input data, labeling data”.
  • information such as the test case identification and execution duration can be used as the "training input data” in the training sample, and the execution result can be used as the "labeled data” in the training sample.
  • execution time as one of the "training input data” mainly considers that the execution time of the test case may be related to the execution result. For example, if the same test case is executed in different tests, if the execution time is very close, or even the same, the execution results are likely to be the same.
  • the execution records in the historical execution data can be input into the classifier model for training according to the execution time in the execution records.
  • the execution record used by the training case classifier must at least include the test case identification, execution time, and execution time.
  • the execution time may include the start time point and the end time point of the execution of the test case.
  • any one of the time points may be used to input the execution record.
  • the training samples are input in the order of execution time, which can be in the order of execution time from near to far, or from far to near, which is not limited in this application.
  • the execution records as training samples can be recorded in the order of their corresponding end time points, in the order from the nearest to the current time point, that is, the latest time point to the farthest time
  • the order of the points is input into the classifier one by one.
  • the execution environment when the execution environment is included in the execution record, the execution environment may also be used as one of the "training input data" in the training sample.
  • the execution environment of the test case may also be related to the execution result. For example, if the same test case is executed in different tests, if the execution environment is the same and the execution time is very close, the execution results are likely to be the same. Adding the execution environment to the "training input data" can further improve the training effect of the classifier and improve the prediction accuracy of the classifier.
  • data cleaning may be performed on the historical execution data first, so as to filter out some of the execution records that are likely to interfere with the training, and obtain the cleaned data. Then the execution records in the cleaned data are input into the classifier for training, thereby improving the training effect, and then improving the prediction accuracy of the classifier.
  • the acquired historical execution data may be cleaned according to the execution result in the execution record to obtain the cleaned data.
  • the execution results include "success", “failure”, and "unclear” three categories, as shown in the example in Table 1. Filter out the execution records whose execution result is "unclear”, and only keep the execution records whose execution result is "success” or "failure”, that is, filter out Z6 of the six execution records from Z1 to Z6, and keep Z1 to Z5. Five execution records are used as training samples for training the classifier. Cleaning the historical execution data according to the execution results can prevent the execution records with unclear execution results from interfering with training and affecting the prediction accuracy of the classifier.
  • the historical execution data can be cleaned according to the failure reason in the execution record to obtain the cleaned data.
  • the execution record can include the reason for the failure.
  • the corresponding failure reason is used to describe the reason for the execution failure of the test case corresponding to the execution record, such as "the execution environment caused "Failure”, “Failure caused by non-product problems”, etc.
  • the execution result in a certain execution record is "success”
  • the corresponding failure reason can be empty.
  • the historical execution data is cleaned according to the number of executions to obtain the cleaned data. For example, if the number of executions of a test case is too few, and is less than a certain number of thresholds, during data cleaning, all execution records corresponding to the test case can be filtered out. This is mainly because the execution records corresponding to the test cases with too few execution times have relatively low confidence. Filtering them can prevent these execution records from interfering with training and affecting the training effect of the classifier.
  • the use case classifier For the trained use case classifier, input at least one first test case that needs to be screened this time into the use case classifier, and the use case classifier will predict the failure probability of each first test case when it is executed next time, that is Predict the execution failure rate.
  • the first test case includes use case 1, use case 2, and use case 3, and inputting them into the use case classifier can obtain the predicted execution failure rate corresponding to each first test case, as shown in Table 2 below.
  • the use case classifier may also output the predicted execution result of each first test case.
  • a certain threshold for example, 50%
  • the predicted execution result is "success”
  • the predicted execution result is "failure”
  • First test case identification Forecast execution failure rate Predict execution result Use case 1 30% success Use case 2 7% success Use case 3 45% success Use case 4 90% fail Use case 5 30% success ... ... ...
  • the computer can execute these high-risk use cases to test the software and generate corresponding execution records. These newly generated execution records can be updated to the historical execution data, and then used to train the use case classifier, so that the newly trained use case classifier can be used to predict the predicted execution failure of the test case to be screened the next time the test case is screened. Rate.
  • the second test case in the embodiment of the present application is a test case corresponding to the modified code, and it may also come from a test case library.
  • the aforementioned at least one second test case refers to a collection of one or more second test cases.
  • set of second test cases may be completely the same, partly the same, or completely different from the set of first test cases, which is not limited in this application.
  • all test cases corresponding to the modified code in the test case library may be used as second test cases.
  • the set of the second test case is a subset of the set of the first test case.
  • the degree of relevance is used to characterize the degree of relevance between the second test case and the modified software code.
  • the greater the value of the degree of association the closer the degree of association between the two.
  • the correlation degree is used to characterize in the embodiment of the present application.
  • the correlation degree not only reflects the corresponding relationship between the second test case and the modified software code, but also shows the closeness of the correlation between the two. This is related to the risk of test cases and whether the test cases can cover test requirements. Therefore, the introduction of relevance for screening high-risk use cases is conducive to more accurate screening of high-risk use cases and is conducive to improving the effectiveness of subsequent tests.
  • the correlation degree corresponding to the test case can be determined through the demand list/problem list.
  • FIG. 3 is a flowchart of one of the implementation manners of the step S200 in the test case screening method according to the embodiment of the application. This implementation may include the following steps S201 to S203.
  • the demand sheet records the technical requirements of the software designers to the developers.
  • the problem list records the problems found by the software users in the process of using the software.
  • the modified code is recorded in the modification record, including the location and content of the modified code, and the modification record will also record the demand order or problem ticket corresponding to the modified code. Based on this, the demand list/problem list corresponding to the software code that has been modified this time can be obtained from the modification record.
  • a software may be modified many times, and each modification may require one or more regression tests.
  • the modified software code in this embodiment only refers to the modified software code this time, rather than all the modified software codes since the software was developed.
  • the modified code can be identified in a variety of ways, such as the position number of the code, or the module number of the code.
  • the demand order can be uniquely identified in a variety of ways, such as the demand order number.
  • the problem ticket can be uniquely identified in a variety of ways, such as the problem ticket number.
  • the mapping relationship between the demand order/problem order and the modified software code can be represented by the mapping relationship between the demand order number/problem order number and the identification of the software code, such as Table 3 shows.
  • S202 Acquire at least one second test case corresponding to the demand list/problem list.
  • Software testers generally look for corresponding test cases in the test case library for the demand list or problem list, or design new test cases corresponding to it, or modify the original test cases in the test case library to make the modified test cases Correspond to the demand list or the problem list to construct the mapping relationship between the demand list/problem list and the test case. Based on this, it is possible to obtain the demand list/problem list involved in this modification (that is, the demand list/problem list corresponding to the modified software code) and the mapping relationship between all the corresponding second test cases, thereby determining The second test case corresponding to the modified software code.
  • demand orders can be uniquely identified in a variety of ways, such as demand ticket numbers, and problem tickets can also be uniquely identified in many ways, such as problem ticket numbers, and test cases can also pass
  • the test case ID is used to uniquely identify the test case, such as the test case number.
  • the mapping relationship between the demand order/problem order and the test case can be represented by the mapping relationship between the demand order number/problem order number and the test case identifier, as shown in Table 4. .
  • S203 Determine the number of demand orders/problem orders corresponding to each second test case as the degree of relevance of the second test case.
  • use case 1 corresponds to 3 demand order numbers/problem order numbers, and its relevance is 3.
  • the process of determining the relevance of other use cases is similar.
  • S300 According to the predicted execution failure rate of the at least one first test case and the correlation degree of the at least one second test case, select high-risk use cases from at least one test case to be selected.
  • the aforementioned at least one test case to be selected refers to the union of the at least one first test case and the at least one second test case.
  • the set of second test cases may be completely the same, partly the same, or completely different from the set of first test cases described above. Based on this, when screening, the same test cases of the two can be merged, and the union of the two can be taken to form a complete set of test cases to be selected. Then select high-risk use cases from the set of test cases to be selected.
  • the predicted execution failure rate and relevance can be used as a sorting basis to sort all test cases to be selected, and then select high-risk use cases based on the sorting results.
  • this implementation is easier to use because there is no need to set a weight value.
  • FIG. 4 is a flowchart of one of the implementation manners of the steps of S300 in the test case screening method according to the embodiment of the application.
  • This implementation may include the following steps S311 to S312.
  • S311 Sort the at least one test case to be selected based on the predicted execution failure rate and the degree of association.
  • the above-mentioned sorting basis can also be understood as the keywords according to the sorting.
  • the predicted execution failure rate and relevance of the test cases to be selected can be used as a sorting basis.
  • the sorting can be performed in ascending order or descending order of predicted execution failure rate or degree of relevance, which is not limited in this application.
  • the priority of different sorting basis may be different, and the priority of the predicted execution failure rate may be higher than the priority of the relevance degree, or may be lower than the priority of the relevance degree, which is not limited in this application.
  • different sorting methods can be selected, and different sorting based on priority.
  • S312 Filter out a preset number of high-risk use cases with the predicted execution failure rate or the highest correlation degree from the sorting result.
  • the above-mentioned preset quantity may be manually preset, or it may be set by a computer according to other information. This application does not limit the preset method of the preset quantity and the specific value of the preset quantity.
  • the preset number of candidate test cases with the highest predicted execution failure rate can be filtered from the ranking results; when the priority of the correlation degree is higher than the predicted execution For the failure rate, a preset number of candidate test cases with the highest degree of relevance can be filtered from the sorting results.
  • the high-risk use cases selected are: use case 4, use case 3, and use case 1.
  • the risk corresponding to each test case to be selected can be evaluated according to the weight of the predicted execution failure rate and the degree of relevance, so as to screen high-risk use cases.
  • FIG. 5 is a flowchart of the second implementation manner of the step S300 in the test case screening method according to the embodiment of the application. This implementation may include the following steps S321 to S323.
  • S321 Determine the weight of the predicted execution failure rate and the weight of the degree of association.
  • the weight of the predicted execution failure rate and the weight of the correlation degree can be determined according to different application scenarios. In different application scenarios, determining the weight of the predicted execution failure rate and the weight of the correlation degree as different values will help to more accurately screen out high-risk use cases, and thus help improve the effectiveness of the test.
  • the user can manually set the value of the weight of the two, so that the computer directly determines the weight of the two according to the value set by the user.
  • S322 Use the weight of the predicted execution failure rate and the weight of the correlation degree to calculate a risk assessment value corresponding to each candidate test case.
  • the corresponding risk assessment value can be determined by the weight of the predicted execution failure rate and the correlation degree of the test case to be selected Add and get. which is,
  • Ri represents the risk assessment value of the i-th candidate test case in the at least one candidate test case
  • Fi represents the predicted execution failure rate of the i-th candidate test case in the at least one candidate test case
  • Ci represents the degree of relevance of the i-th candidate test case in the at least one candidate test case.
  • S323 Determine the candidate test case whose risk assessment value is higher than the preset first threshold value as a high-risk use case.
  • the above-mentioned preset first threshold may be manually preset, or it may be set by a computer according to other information.
  • the preset method of the preset first threshold in this application, and the specific selection of the preset first threshold The values are not limited.
  • the high-risk use cases selected from the candidate test cases shown in Table 7 are: Use Case 1, Use Case 3, Use Case 4 .
  • the predicted execution failure rate of the first test case or the correlation degree of the second test case can also be used alone to screen high-risk use cases.
  • the first test case whose predicted execution failure rate exceeds a preset second threshold may be determined as a high-risk use case.
  • a second test case with a correlation degree higher than a preset third threshold may be determined as a high-risk use case.
  • steps S100 to S300 only describe the steps of one test case screening.
  • multiple tests may be required, and the above screening method can be re-executed for each screening test case.
  • the training samples (ie historical execution data) of the training case classifier will be updated, the relevance of test cases may also be updated, and the test cases in the test case library may also be updated. Therefore, the results obtained in different screenings There may also be differences in the screening results.
  • the method in the embodiment of this application combines two types: a screening method based on the risk of a test case and a screening method based on a modified code in the software.
  • machine learning technology is used to predict the risk of a test case
  • the predicted execution failure rate is related to the risk of the test case
  • the correlation degree is also related to the risk of the test case
  • the prior art screening method based on the mapping relationship between each code line and the test case requires software source code as input information, which is a white-box testing method.
  • This method needs to disclose the source code of the software, which poses risks to the information security of the software.
  • the implementation of this method requires coverage analysis of all test cases, there is a problem of high implementation costs.
  • the method of assessing the risk of a test case based on the defects found in each test case and the severity of the defect in the prior art is also a white-box testing method, which also has the above-mentioned problems.
  • the method in the embodiment of the present application is a black box testing method, which does not need to use the source code of the product as input, which is beneficial to guarantee the information security of the software.
  • this method does not need to perform coverage analysis on all test cases, does not need to use the defects found in the execution of each test case, and the severity of the defects, and the use case classifier used does not need to use the test case when training.
  • only the historical execution data of the test case can be used, which greatly reduces the implementation cost.
  • the information of the product code or the content of the test case is required, which takes a long time.
  • the time required to train the use case classifier is relatively short. Based on this, the predicted execution failure rate of the first test case obtained in the embodiment of the present application has better real-time performance.
  • the correlation degree of the second test case obtained by using the method of the embodiment of the present application is also better in real time.
  • the historical execution data and the correlation degree in the embodiments of the present application can be updated in real time, which makes the predicted execution failure rate of the first test case and the correlation degree of the second test case
  • these two parameters can be updated in minutes, which in turn also improves the timeliness of implementing the screening method in the embodiments of the present application.
  • the foregoing embodiment introduces the screening method provided in this application.
  • the screening method in this application can be implemented not only in the form of hardware or computer software, but also in the form of a combination of hardware and computer software.
  • the form in which a certain function is implemented depends on the specific application and design constraints of the technical solution.
  • FIG. 6 shows a schematic structural diagram of an implementation manner of the screening device in the present application.
  • the screening device 400 includes an acquisition module 401 and a processing module 402; among them,
  • the obtaining module 401 is configured to obtain the predicted execution failure rate of at least one first test case; and obtain the correlation degree of at least one second test case; wherein the predicted execution failure rate is determined by the use case classifier for the at least one execution failure rate.
  • the first test case is predicted
  • the use case classifier is a classifier trained using historical execution data
  • the historical execution data includes at least one execution record generated by executing the at least one first test case
  • the first The second test case is a test case corresponding to the modified software code
  • the correlation degree is used to characterize the degree of correlation between the second test case and the corresponding modified software code.
  • the processing module 402 is configured to screen out high-risk use cases from at least one test case to be selected according to the predicted execution failure rate of the at least one first test case and the correlation degree of the at least one second test case; where , The at least one test case to be selected is a union of the at least one first test case and the at least one second test case.
  • the obtaining module 401 is further configured to obtain the historical execution data.
  • the processing module 402 is further configured to: use the historical execution data to train a classifier to obtain a use case classifier; input the at least one first test case into the use case classifier to obtain each of the at least one first test case The predicted execution failure rate of.
  • the execution record includes a test case identification, an execution result, execution time, and execution duration corresponding to the test case identification.
  • the processing module 402 is further configured to: clean the historical execution data according to the execution result in the execution record to obtain the cleaned data; and input the execution record in the cleaned data to the classifier , Get the use case classifier.
  • the execution record further includes a failure reason; the processing module 402 is further configured to clean the historical execution data according to the execution result and the failure reason to obtain cleaned data.
  • the processing module 402 is further configured to: determine the number of execution records corresponding to each of the first test cases in the historical execution data as the respective execution records of each of the first test cases. Times; and, cleaning the historical execution data according to the execution result and the execution times to obtain the cleaned data.
  • processing module 402 is further configured to: input the execution record in the cleaned data into the classifier according to the execution time in the execution record to obtain the use case classifier.
  • the acquiring module 401 is further configured to: acquire a demand ticket/problem ticket corresponding to the modified software code; and acquire at least one second test case corresponding to the demand ticket/problem ticket.
  • the processing module 402 is further configured to determine the number of demand orders/problem orders corresponding to each second test case as the degree of relevance of the second test case.
  • the processing module 402 is further configured to: sort the at least one test case to be selected based on the predicted execution failure rate and the degree of association; and filter a preset number of test cases from the sorting result The predicted execution failure rate or the high-risk use case with the highest correlation degree.
  • the processing module 402 is further configured to: determine the weight of the predicted execution failure rate and the weight of the correlation degree; use the weight of the predicted execution failure rate and the weight of the correlation degree to calculate each The risk assessment value corresponding to the test case to be selected; and the test case for which the risk assessment value is higher than the preset first threshold is determined as a high-risk use case.
  • the division of the functional modules of the screening device in this embodiment is illustrative, and is only a logical function division, and another division method may be used in actual implementation.
  • the function of the acquisition module 401 may be implemented by a bus interface or a transceiver
  • the function of the processing module 402 may be implemented by a processor.
  • each independently operating functional entity can be integrated in a hardware entity, such as a server, a chip, and so on.
  • Figure 7 shows a schematic structural diagram of another implementation of the screening device in the present application. Fig. 7 introduces the screening device of the present application from the perspective of an independent functional entity.
  • the screening device 500 may include at least one memory 501 and at least one processor 502.
  • the at least one memory 501 stores one or more computer programs or instructions.
  • the memory 501 may include a volatile memory (volatile memory), such as random access memory (random access memory, RAM); and may also include a non-volatile memory (non-volatile memory), such as flash memory (flash memory), Hard disk drive (HDD) or solid-state drive (SSD); storage 501 may also include a combination of the foregoing types of storage.
  • the memory 501 may store computer executable programs or instructions.
  • the processor 502 executes instructions stored in the memory 501, so as to realize the function of the screening device or data processing.
  • the processor 502 executes a program or instruction stored in the memory 501, so that the screening device 500 implements part or all of the steps of any one of the foregoing screening methods.
  • the function of the processing module 402 in the aforementioned screening device 400 may be implemented by the processor 502.
  • the aforementioned processor 502 may include one or more processing units.
  • the processor 502 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), and image signal processing.
  • the different processing units may be independent devices or integrated in one or more processors.
  • the processor 502 may further include a hardware chip.
  • the above-mentioned hardware chip may be an application-specific integrated circuit (ASIC), a programmable logic device (PLD) or a combination thereof.
  • the aforementioned PLD may be a complex programmable logic device (CPLD), a field-programmable gate array (FPGA), a general array logic (generic array logic, GAL), or any combination thereof.
  • the aforementioned screening device may further include a transceiver 503, and the transceiver 503 may be used to receive data external to the screening device. For example, when the predicted execution failure rate of the first test case or the correlation degree of the second test case is stored outside the screening device, these data may be received through the transceiver 503. Exemplarily, the function of the acquisition module 401 in the aforementioned screening device 400 may be implemented by the transceiver 503.
  • This embodiment also provides a terminal device, including a memory, a processor, and a computer program stored in the memory and running on the processor.
  • the terminal device is caused to execute the foregoing Part or all of the steps of any screening method.
  • This embodiment also provides a chip, which is connected to a memory or includes a memory, and is used to read and execute a computer program stored in the memory.
  • the chip executes any one of the foregoing. Part or all of the steps of a screening method.
  • This embodiment also provides a computer-readable storage medium.
  • the computer-readable storage medium is used to store a computer program or instruction. When the computer program or instruction runs on a computer, the computer executes part or all of the steps of any one of the aforementioned screening methods.
  • the readable storage medium here can be a magnetic disk, an optical disc, a DVD, a USB, a read only memory (ROM) or a random access memory (RAM), etc.
  • the application does not limit the specific storage medium form.
  • the methods of the foregoing embodiments may be implemented in whole or in part by software, hardware, firmware, or any combination thereof.
  • software When implemented by software, it can be implemented in the form of a computer program product in whole or in part.
  • the computer program product includes one or more computer instructions.
  • the computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices.
  • the computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website, computer, server, or data center.
  • the computer-readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server or a data center integrated with one or more available media.
  • the usable medium may be a magnetic medium (such as a floppy disk, a hard disk, and a magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).
  • the execution order of each step should be determined by its function and internal logic, and the size of the sequence number of each step does not mean the order of the execution order, and does not limit the implementation process of the embodiment.
  • the “plurality” in this specification refers to two or more.
  • words such as “first” and “second” are used to distinguish the same items or similar items that have substantially the same function and effect.
  • the words “first”, “second” and the like do not limit the quantity and order of execution, and the words “first” and “second” do not limit the difference.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Quality & Reliability (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Debugging And Monitoring (AREA)

Abstract

一种测试用例筛选方法，该方法可以被应用于软件测试领域。该方法包括：将测试用例的历史执行数据用作训练样本，利用机器学习技术训练得到用例分类器，并利用该用例分类器预测得到至少一个第一测试用例的预测执行失败率；获取至少一个与被修改的软件代码对应的第二测试用例的关联度；然后，根据至少一个第一测试用例的预测执行失败率，以及至少一个第二测试用例的关联度，从至少一个待选测试用例中筛选出高风险用例；其中，至少一个待选测试用例为前述第一测试用例和第二测试用例的并集。采用上述技术方案可以从待选测试用例中更精准地筛选出高风险用例，从而促使缺陷发现或覆盖需求的测试用例前移，进而提高测试的有效性。

Description

测试用例筛选方法、装置及介质 技术领域
本申请涉及软件测试技术领域，具体涉及一种测试用例筛选方法。此外，本申请还涉及一种测试用例筛选装置，以及一种计算机可读存储介质。
背景技术
回归测试(regression test)是在修改了软件的原有代码之后，重新对软件进行测试的一种测试流程。通过回归测试可以确认软件的原有功能是否被破坏，即确认修改了原有代码之后是否引入新的错误，或者导致其他原有代码产生错误等问题，即判断修改后的软件中是否存在缺陷。如果存在缺陷，那么开发人员就需要对软件再进行修改。在软件开发生命周期(System Development Life Cycle，SDLC)中，软件的各个版本都需要经过多次充分的测试才能够发布。因此，回归测试是一项需要持续进行的任务。
测试用例(test case)是指为了进行测试而向被测试系统(即软件)提供的一组数据，一般包括输入数据、执行条件以及预期结果。在测试时，如果一个测试用例执行失败，可能意味着缺陷的发现。为测试一个软件，通常需要设计很多测试用例，形成测试用例库。在软件开发的生命周期中，测试用例库中的测试用例可能会被不断更新，以便应对经过修改之后的软件的回归测试。
在回归测试中，为满足不同的需求，测试人员通常会制定不同的回归测试策略。选择有效的回归测试策略，可以精准地从测试用例库中筛选出合适的测试用例，从而提升回归测试的测试效率和有效性。筛选测试用例一般包括三种类型的方法：
(1)选取测试用例库中全部的测试用例。采用该类型的方法能够保证缺陷遗漏风险最低，但是对时间和设备等测试资源的需求较高，导致回归测试的测试效率较低。
(2)基于软件中被修改的代码，从测试用例库中筛选出部分测试用例。具体地，在一种实现方式中，首先需要对所有测试用例进行覆盖分析，确定每个代码行与测试用例之间的映射关系，然后才能根据被修改的代码行以及映射关系，从测试用例库中筛选出合适的测试用例。
该方法一定程度上可以提高测试效率，但被修改的代码并不一定会导致缺陷，未被修改的代码也不一定不存在缺陷，有可能未被修改的代码会由于与被修改的代码存在关联而产生缺陷。也就是说，覆盖分析所得到的代码行与测试用例之间的映射关系，实际上并不能准确地反映出测试用例的风险(即测试用例执行失败的概率高低)。因此，采用该方法无法很好地保证回归测试的有效性。
(3)基于测试用例的风险，从测试用例库中筛选出部分测试用例。具体地，在一种实现方式中，首先对此前测试中执行用例时的执行数据进行分析，确定每个测试用例所发现的缺陷，以及缺陷的严重等级。然后根据缺陷的个数和缺陷的严重等级，来 评估这些测试用例的风险，从中筛选出高风险用例，用于本次回归测试。
采用该方法一定程度上可以筛选出高风险用例，提高回归测试的有效性。但是，发明人经过分析认为，该方法采用执行测试用例所发现的缺陷和缺陷的严重等级来衡量测试用例的风险，并仅据此来进行筛选，筛选的精准程度仍然有待提高。
发明内容
本申请提供一种测试用例筛选方法，该方法结合了基于测试用例的风险的筛选方法，以及基于软件中被修改的代码的筛选方法这两种类型，可以更加精准地筛选出高风险用例，进而提高测试的有效性。
第一方面，本申请提供一种测试用例筛选方法，包括：获取至少一个第一测试用例的预测执行失败率；获取至少一个第二测试用例的关联度；根据上述至少一个第一测试用例的预测执行失败率，以及上述至少一个第二测试用例的关联度，从至少一个待选测试用例中筛选出高风险用例。其中，上述预测执行失败率由用例分类器对至少一个第一测试用例进行预测得到，上述用例分类器为利用历史执行数据训练得到的分类器，上述历史执行数据包括执行至少一个第一测试用例而生成的至少一条执行记录；上述第二测试用例为被修改的软件代码所对应的测试用例，上述关联度用于表征第二测试用例与对应的被修改的软件代码之间的关联程度；上述至少一个待选测试用例为至少一个第一测试用例和至少一个第二测试用例的并集。
采用本实现方式，一方面利用机器学习技术来预测测试用例的风险，另一方面获取测试用例与软件代码之间的关联度，结合这两方面的信息，可以更加精准地筛选出高风险用例，促使缺陷发现前移，促使覆盖需求的测试用例前移，进而提高测试的有效性。此外，采用该方法还有利于保障软件的信息安全，降低测试用例筛选的实施成本，提升筛选的时效性。
在一种可能的实现方式中，将历史执行数据用作训练样本，利用机器学习技术训练得到用例分类器，并利用该用例分类器对至少一个第一测试用例进行预测，可以得到至少一个第一测试用例的预测执行失败率。该实现方式具体可以为：获取历史执行数据；利用历史执行数据训练分类器，得到用例分类器；将至少一个第一测试用例输入用例分类器，得到至少一个第一测试用例各自的预测执行失败率。采用本实现方式，无需使用产品的源码作为输入，有利于保障软件的信息安全。并且，由于在训练的时候无需使用测试用例的具体内容，仅仅使用测试用例的历史执行数据即可，从而大大降低了实施成本。
在一种可能的实现方式中，执行记录包括测试用例标识，与测试用例标识对应的执行结果、执行时间和执行时长。测试用例的执行时长可能与执行结果相关，因此可以利用测试用例的测试用例标识、执行时间和执行结果来训练得到分类器模型，得到用例分类器，以便利用该用例分类器来预测测试用例后续被执行时的预测执行失败率。执行时间则被应用于作为输入训练样本的顺序的依据，以保证训练效果。
在训练之前，可以先对历史执行数据进行数据清洗，以过滤掉其中一些容易对训练产生干扰的执行记录，得到清洗后的数据。再将清洗后的数据中的执行记录，输入到分类器中进行训练。数据清洗可以有多种实现方式，例如可以利用执行结果、失败 原因和执行次数中的一种或者多种来实现。
在一种可能的实现方式中，可以利用执行结果来清洗历史执行数据。该实现方式具体可以为：根据执行记录中的执行结果清洗历史执行数据，得到清洗后数据；将清洗后数据中的执行记录输入至分类器中，得到用例分类器。根据执行结果来清洗历史执行数据，可以避免执行结果不明确的执行记录对训练产生干扰，影响分类器的预测准确率。
在另一种可能的实现方式中，可以利用执行结果和执行记录中的失败原因来清洗历史执行数据。该实现方式具体可以为：根据执行结果和失败原因清洗历史执行数据，得到清洗后数据。将失败原因中的一个或者多个原因作为过滤的条件，可以过滤掉与软件缺陷没有关联的执行记录，避免这些执行记录对训练产生干扰，影响分类器的预测准确率。
在又一种可能的实现方式中，可以利用执行结果和执行次数来清洗历史执行数据。该实现方式具体可以为：将历史执行数据中每一个第一测试用例各自对应的执行记录的数量，确定为每一个第一测试用例各自对应的执行次数；根据执行结果和执行次数清洗历史执行数据，得到清洗后数据。执行次数过少的测试用例对应的执行记录，其置信度相对较低。因此，将这些执行记录过滤掉，可以避免这些执行记录对训练产生干扰，影响分类器的预测准确率。
在一种可能的实现方式中，将清洗后数据中的执行记录，按照执行记录中的执行时间输入至分类器中，得到用例分类器。与不按照执行时间的先后顺序输入训练样本的方式相比，采用本实现方式可以保证训练效果，提升用例分类器的预测准确率。
在一种可能的实现方式中，第二测试用例的关联度可以通过问题单或者需求单来确定。该实现方式具体可以为：获取与被修改的软件代码对应的需求单/问题单；获取与需求单/问题单对应的至少一个第二测试用例；将每一个第二测试用例对应的需求单/问题单的数量，确定为第二测试用例的关联度。采用本实现方式所得到的第二测试用例的关联度，不但能够体现出第二测试用例与被修改的软件代码之间的对应关系，也能够表示出二者之间的关联紧密程度。这与测试用例的风险相关，也与测试用例是否能够覆盖测试需求相关。基于此，将该关联度引入进来用于筛选高风险用例，有利于更加精准地筛选出高风险用例，有利于提高后续测试的有效性。
筛选高风险用例可以有多种实现形式，可以根据不同的业务需求来确定。
在一种可能的实现方式中，可以以预测执行失败率和关联度作为排序依据，对所有的待选测试用例进行排序，然后根据排序结果，筛选出高风险用例。该实现方式具体可以为：以预测执行失败率和关联度为排序依据，对上述至少一个待选测试用例排序；从排序结果中筛选出预设数量个预测执行失败率或者关联度最高的高风险用例。采用该实现方式，可以较精准地筛选出高风险用例，有助于提高测试的有效性。并且，与后续的另一种实现方式相比，该实现方式不需要设定权重值，更加容易被使用。
在另一种可能的实现方式中，可以根据预测执行失败率和关联度的权重，来评估每个待选测试用例对应的风险，从而筛选出高风险用例。该实现方式具体可以为：确定预测执行失败率的权重和关联度的权重；利用预测执行失败率的权重和关联度的权重，计算每一个待选测试用例对应的风险评估值；将风险评估值高于预设的第一阈值 的待选测试用例，确定为高风险用例。采用该实现方式，可以较精准地筛选出高风险用例，有助于提高测试的有效性。
第二方面，本申请提供一种测试用例筛选装置，包括获取模块和处理模块；上述获取模块用于获取至少一个第一测试用例的预测执行失败率；以及，获取至少一个第二测试用例的关联度；其中，预测执行失败率由用例分类器对上述至少一个第一测试用例进行预测得到，用例分类器为利用历史执行数据训练得到的分类器，历史执行数据包括执行上述至少一个第一测试用例而生成的至少一条执行记录；第二测试用例为被修改的软件代码所对应的测试用例，关联度用于表征第二测试用例与对应的被修改的软件代码之间的关联程度；上述处理模块用于根据上述至少一个第一测试用例的预测执行失败率，以及上述至少一个第二测试用例的关联度，从至少一个待选测试用例中筛选出高风险用例；其中，至少一个待选测试用例为至少一个第一测试用例和至少一个第二测试用例的并集。
第三方面，本申请提供一种测试用例筛选装置，包括：至少一个存储器，以及至少一个处理器；至少一个存储器存储有一个或多个指令，当一个或多个指令被至少一个处理器执行时，使得该装置实现第一方面的任一种方法的部分或全部步骤。
第四方面，本申请提供一种计算机可读存储介质，该计算机存储介质中包括指令，当指令在计算机上运行时，使得该计算机执行第一方面的任一种方法的部分或全部步骤。
可以理解地，上述第二方面、第三方面的测试用例筛选装置，以及第四方面的计算机可读存储介质，均用于执行上文所提供的对应的方法，其所能达到的有益效果可参考上文对应方法的有益效果，此处不再赘述。
附图说明
为了更清楚地说明本申请的技术方案，下面将对实施例中的附图作简单地介绍。
图1为本申请实施例中筛选出高风险用例的一个示意图。
图2为本申请实施例的测试用例筛选方法的实现方式之一的流程图。
图3为本申请实施例的测试用例筛选方法中，S200的步骤的实现方式之一的流程图。
图4为本申请实施例的测试用例筛选方法中，S300的步骤的实现方式之一的流程图。
图5为本申请实施例的测试用例筛选方法中，S300的步骤的实现方式之二的流程图。
图6为本申请实施例中筛选装置的一种实现方式的结构示意图。
图7为本申请实施例中筛选装置的另一种实现方式的结构示意图。
具体实施方式
为便于理解本申请的技术方案，以下将对本申请涉及到的几个技术术语作简单的介绍。
测试用例的风险，指的是测试用例在执行时的失败率。一个测试用例的风险高，指的是该测试用例在执行时的失败率较高，例如失败率高于某一阈值，具有这样特点的测试用例在本申请中也被称为高风险用例；相应地，一个测试用例的风险低，指的是该测试用例在执行时的失败率较低，例如失败率低于某一阈值。上述阈值的取值可以是任意的，本申请对于阈值的具体数值不作限定。
缺陷，一般是指存在于软件代码中的错误，有时也会被称为“错误”、“BUG”等。通过执行测试用例，可以发现一个软件中的部分或者全部缺陷。能否尽可能多地发现缺陷，能否在有限的时间能尽可能多地发现缺陷，以及，能否在有限的时间内尽可能多地发现相对严重的缺陷，这很大程度上取决于测试用例的设计和筛选。
测试需求一般是指需要测试的内容，例如，测试某个功能是否在软件中有所体现等。
测试的有效性，指的是在一次测试中，能够发现软件缺陷或者覆盖需求的测试用例，在该次测试所执行的全部测试用例中所占的比例。
对于一个软件而言，在测试该软件的过程中需要执行测试用例。一个测试用例可能被执行一次或者多次。一个测试用例每被执行一次，就会生成一条对应的执行记录。这些执行记录的集合，称之为历史执行数据。
执行记录记载了与之对应的测试用例某一次被执行的情况。一般地，执行记录可以包括测试用例标识、执行结果、执行时间，以及执行时长等信息。
需要说明的是，随着测试的进行，测试用例库中的测试用例被执行的次数也随之增加，相应地，一个软件的历史执行数据中所包含的执行记录也会不断增加。例如，在为第N次测试筛选测试用例时，历史执行数据中可以包括前N-1次测试中执行测试用例所生成的全部执行记录。而在为第N+1次测试筛选测试用例时，历史执行数据中不但可以包括前N-1次测试中执行测试用例所生成的执行记录，也可以包括第N次测试中执行测试用例所生成的执行记录，即可以包括前N次测试中执行测试用例所生成的执行记录。
本申请结合了基于测试用例的风险的筛选方法，以及基于软件中被修改的代码的筛选方法这两种类型，提出一种新的测试用例筛选方法。图1为筛选出高风险用例的一个示意图。示例性地，第一测试用例来自测试用例库，包括了用例1至用例5等测试用例；第二测试用例也来自测试用例库，是与被修改的代码对应的测试用例，包括了用例1至用例3等测试用例。以第一测试用例和第二测试用例的并集为筛选的基础(即待选测试用例)，根据第一测试用例的预测执行失败率和第二测试用例的关联度，从待选测试用例中筛选出高风险用例，例如用例1、用例3和用例4等，以用于后续的软件测试。
本申请实施例的筛选方法，一方面利用软件在测试时生成的历史执行数据作为机器学习的训练样本，利用机器学习技术来预测测试用例的风险(即预测执行失败率)，另一方面获取测试用例与被修改的软件代码之间的关联度，结合这两方面的信息，从而更加精准地筛选出高风险用例。将这些高风险用例用于回归测试，可以尽早发现修改后的软件中的缺陷，促使缺陷发现前移；同时，也可以尽早执行能够覆盖测试需求 的测试用例，促使覆盖需求的测试用例前移，进而提升回归测试的有效性。
本申请的筛选方法可以被应用在所有的软件产品的回归测试过程中，尤其适合被应用在版本迭代较快、测试周期较短或者测试资源较少的软件产品的回归测试场景中。
以下将结合图1至图2来对本申请实施例中的方法作进一步介绍。图2为本申请实施例的测试用例筛选方法的实现方式之一的流程图。该测试用例筛选方法，可以包括以下S100至S300步骤。
S100：获取至少一个第一测试用例的预测执行失败率。
本申请实施例中的第一测试用例可以来自测试用例库。上述的至少一个第一测试用例，指的是一个或者多个第一测试用例的集合。在一种实现方式中，可以将测试用例库中所有的测试用例，都作为第一测试用例。
第一测试用例的预测执行失败率，可以通过用例分类器来预测得到。用例分类器在预测之前，需要先利用这些第一测试用例此前被执行时所生成的执行记录来进行训练。
为了便于理解，以下将先介绍用例分类器的训练过程，然后介绍利用其来预测第一测试用例的预测执行失败率的过程。
对于一个特定的分类器模型而言，描述该分类器模型的函数是不变的，而函数中的一些参数是可以调整的。训练，可以理解为是对该分类器模型中的一些参数进行调整的过程。
在本申请实施例中，上述分类器模型可以采用现有的机器学习模型，也可以采用其他被改进后的机器学习模型，本申请对此不作限定。
训练分类器模型需要使用训练样本。在有监督训练的一种实现方式中，训练样本的形式为“训练输入数据，标注数据”。将“训练输入数据”输入到分类器模型中，分类器模型对该“训练输入数据”进行计算，得到“预测数据”。然后，分类器模型根据“预测数据”与“标注数据”之间的差异，调整分类器模型中的参数的取值，以便以后将相同的“训练输入数据”输入到分类器模型中时，分类器模型可以输出与“标注数据”更接近的“预测数据”。使用大量训练样本进行训练之后，分类器模型中的参数相应地经过了一系列调整，此时可以将当前分类器模型中的这些参数确定下来。这样，该分类器模型就被训练好了，即得到了训练好的用例分类器。
在本申请实施例中，训练样本可以采用历史执行数据中的执行记录，其中包含了此前的测试中第一测试用例被执行而生成的执行记录。需要说明的是，训练样本也可能包含其他测试用例被执行而生成的执行记录，本申请对此不作限定。
执行记录可以包括测试用例标识、执行结果、执行时间，以及执行时长等信息。
其中，测试用例标识用于唯一标识一个测试用例，以便通过测试用例标识就可以将不同的测试用例区分开。
执行结果一般可以包括三类：成功，失败，以及不明确。对于一个测试用例而言，如果该测试用例被执行完毕之后所得到的实际结果与预设的预期结果一致，则其对应的执行结果为成功；如果实际结果与预期结果不一致，则其对应的执行结果为失败。执行结果还可能存在不明确的情况，这可能由多种不同的原因造成，例如“没有执行完毕”、“执行环境导致的阻塞”等。
执行时间可以包括该测试用例执行的开始时间点和结束时间点。
执行时长指的是执行该测试用例所耗费的时间。在一种实现方式中，执行时长可以由前述的结束时间点与开始时间点的差值来确定。
可选地，执行记录还可以包括执行环境。执行环境是指在软件测试过程中执行测试用例时所必需的计算机硬件、软件、网络设备等的总称。例如，执行环境可以包括操作系统、硬件配置信息等。
表1示出了历史执行数据中包括的几个执行记录的示例。
表1历史执行数据局部示例
Figure PCTCN2020115580-appb-000001
如前所述，训练样本的形式为“训练输入数据，标注数据”。在本申请实施例中，可以将测试用例标识和执行时长等信息，作为训练样本中的“训练输入数据”，将执行结果作为训练样本中的“标注数据”。其中，采用执行时间作为“训练输入数据”之一，主要是考虑到测试用例的执行时长可能与执行结果相关。例如，在不同次的测试中执行同一个测试用例，如果其执行时长非常接近，甚至相同，则其执行结果很可能相同。
在训练分类器模型的一种实现方式中，可以按照执行记录中的执行时间，将历史执行数据中的执行记录输入到分类器模型中，进行训练。此时，训练用例分类器所使用的执行记录中，至少需要包括测试用例标识、执行时长和执行时间。
可选地，执行时间可以包括测试用例执行的开始时间点和结束时间点，在本申请实施例中可以统一以其中任一个时间点为准，来输入执行记录。可选地，按照执行时 间的先后顺序输入训练样本，可以按照执行时间由近至远的顺序，也可以按照由远至近的顺序，本申请对此不作限定。
例如，在一个示例中，可以将作为训练样本的执行记录，按照各自对应的结束时间点的先后顺序，按照距离当前时间点由近至远的顺序，即最晚的时间点至最远的时间点的顺序，逐条输入至分类器中。
可选地，当执行记录中包括执行环境时，也可以将执行环境作为训练样本中的“训练输入数据”之一。这主要是考虑到测试用例的执行环境可能也与执行结果相关。例如，在不同次的测试中执行同一个测试用例，如果其执行环境相同，并且其执行时长非常接近，则其执行结果很可能相同。将执行环境增加到“训练输入数据”中，可以进一步提升分类器的训练效果，提升分类器的预测准确率。
在训练阶段，首先需要获取历史执行数据。然后，利用历史执行数据训练分类器，得到用例分类器。即，以历史执行数据中的执行记录为训练样本，将其输入一个预先设计好的分类器模型进行训练，以得到用例分类器。
可选地，在训练的一种实现方式中，可以先对历史执行数据进行数据清洗，以过滤掉其中一些容易对训练产生干扰的执行记录，得到清洗后的数据。再将清洗后的数据中的执行记录，输入到分类器中进行训练，从而提升训练效果，进而提升分类器的预测准确率。
在数据清洗的一种实现方式中，可以根据执行记录中的执行结果清洗获取到的历史执行数据，得到清洗后数据。例如，执行结果包括“成功”，“失败”，以及“不明确”三类，如表1的例子所示。将执行结果为“不明确”的执行记录过滤掉，只保留执行结果为“成功”或“失败”的执行记录，即将Z1至Z6这6条执行记录中的Z6过滤掉，保留Z1至Z5这5条执行记录，将其作为训练分类器的训练样本。根据执行结果来清洗历史执行数据，可以避免执行结果不明确的执行记录对训练产生干扰，影响分类器的预测准确率。
在数据清洗的另一种实现方式中，当执行记录包括失败原因时，可以根据执行记录中的失败原因清洗历史执行数据，得到清洗后数据。
执行记录可以包括失败原因，当某一条执行记录中的执行结果为“失败”时，其对应的失败原因用于描述该条执行记录对应的测试用例执行失败的原因，例如“执行环境的原因导致的失败”、“非产品问题导致的失败”等。当某一条执行记录中的执行结果为“成功”时，其对应的失败原因可以为空。
除了软件中的缺陷可能会导致执行失败以外，其他一些与软件缺陷无关的因素也可能会导致执行失败，例如前述的“非产品问题导致的失败”，更具体地例如“网络阻塞导致的失败”等。如果将这些执行结果为“失败”，但与软件缺陷没有关联的执行记录也作为训练样本，可能会对训练产生干扰，影响分类器的预测准确率。为此，在清洗历史执行数据时，可以将失败原因中的一个或者多个原因作为过滤的条件，以此来过滤掉这些与软件缺陷没有关联的执行记录。
在数据清洗的又一种实现方式中，还可以统计历史执行数据中每一个测试用例对应的执行记录的数量，将其记录为测试用例对应的执行次数。然后，根据执行次数清洗历史执行数据，得到清洗后数据。例如，如果一个测试用例的执行次数过少，小于 一定的次数阈值，则在数据清洗的时候，可以将该测试用例对应的执行记录都过滤掉。这主要是由于这些执行次数过少的测试用例对应的执行记录，其置信度相对较低，将其过滤可以避免这些执行记录对训练产生干扰，影响分类器的训练效果。
需要说明的是，上述清洗数据的实现方式，以及上述训练分类器的实现方式，均可以相互结合。
对于训练好的用例分类器，将本次需要筛选的至少一个第一测试用例输入到该用例分类器中，用例分类器会预测每一个第一测试用例在下一次被执行时对应的失败概率，即预测执行失败率。
例如，第一测试用例包括用例1、用例2和用例3等，将其输入用例分类器，可以得到每一个第一测试用例对应的预测执行失败率，如下表2所示。可选地，用例分类器还可以输出每一个第一测试用例的预测执行结果。当预测执行失败率低于一定阈值(例如50％)时，预测执行结果为“成功”，而当预测执行失败率高于一定阈值时，预测执行结果为“失败”，如下表2所示。
表2用例分类器输出结果示例
第一测试用例标识 预测执行失败率 预测执行结果
用例1 30％ 成功
用例2 7％ 成功
用例3 45％ 成功
用例4 90％ 失败
用例5 30％ 成功
…… …… ……
需要说明的是，当后续筛选出高风险用例之后，计算机可以执行这些高风险用例来测试软件，并生成对应的执行记录。这些新生成的执行记录，可以被更新到历史执行数据中，再用于训练用例分类器，以便在下一次筛选测试用例的时候使用新训练的用例分类器来预测待筛选的测试用例的预测执行失败率。
S200：获取至少一个第二测试用例的关联度。
本申请实施例中的第二测试用例是被修改的代码所对应的测试用例，其也可以来自测试用例库。上述的至少一个第二测试用例，指的是一个或者多个第二测试用例的集合。
需要说明的是，第二测试用例的集合，可以和前述第一测试用例的集合完全相同、部分相同，或者完全不相同，本申请对此不作限定。
在一种实现方式中，可以将测试用例库中所有的被修改的代码对应的测试用例，都作为第二测试用例。当将测试用例库中所有的测试用例都作为第一测试用例时，第二测试用例的集合是第一测试用例的集合的一个子集。
关联度用于表征第二测试用例与被修改的软件代码之间的关联程度。在一种表征方式中，关联度的取值越大，则表示二者的关联程度越紧密。
在现有技术中，一般需要对测试用例库中的所有的测试用例进行覆盖分析，来确定每个代码行与测试用例之间的映射关系。然后，通过本次被修改的代码以及前述的映射关系，来确定出本次回归测试所需要采用的，与被修改的代码具有映射关系的测试用例。在该方法中，通过覆盖分析仅仅得到了每个代码行与测试用例之间的映射关系，一方面并不能准确地反映出测试用例的风险，另一方面也不能体现出代码行与测试用例之间的关联程度是否紧密。
为此，在本申请实施例中采用关联度来表征。关联度在体现出第二测试用例与被修改的软件代码之间的对应关系的同时，也能够表示出二者之间的关联紧密程度。这与测试用例的风险相关，也与测试用例是否能够覆盖测试需求相关。因此，将关联度引入进来用于筛选高风险用例，有利于更加精准地筛选出高风险用例，有利于提高后续测试的有效性。
在获取第二测试用例对应的关联度的一种实现方式中，可以通过需求单/问题单来确定测试用例对应的关联度。请参见图3，图3为本申请实施例的测试用例筛选方法中，S200的步骤的实现方式之一的流程图。该实现方式可以包括以下S201至S203的步骤。
S201：获取与被修改的软件代码对应的需求单/问题单。
需求单记录了软件的设计人员对开发人员所提出的技术上的需求。问题单记录了软件使用人员在使用软件过程中所发现的问题。
开发人员会根据需求单所提出的需求或者问题单所提出的问题来修改软件的代码。在修改之后，开发人员一般会填写修改记录。修改记录中记录了被修改的代码，包括被修改的代码的位置和内容等，修改记录中还会记录与被修改的代码对应的需求单或者问题单。基于此，从修改记录中可以获取到本次被修改的软件代码对应的需求单/问题单。
需要说明的是，在软件开发生命周期中，一个软件可能被修改很多次，每一次修改都可能需要进行一次或者多次回归测试。本实施例中的被修改的软件代码，仅仅是指本次被修改的软件代码，而不是指软件被开发以来所有的被修改的软件代码。
还需要说明的是，被修改的代码可以通过多种方式来标识，例如代码的位置编号，或者代码的模块编号等。需求单可以通过多种方式来唯一标识，例如需求单号等。类似地，问题单可以通过多种方式来唯一标识，例如问题单号等。基于此，在本申请实施例中，需求单/问题单与被修改的软件代码之间的映射关系，可以通过需求单号/问题单号与软件代码的标识之间的映射关系来表示，如表3所示。
表3被修改的软件代码与需求单/问题单之间的映射关系示例
Figure PCTCN2020115580-appb-000002
Figure PCTCN2020115580-appb-000003
S202：获取与所述需求单/问题单对应的至少一个第二测试用例。
软件测试人员一般会针对需求单或者问题单去测试用例库中寻找对应的测试用例，或者设计与之对应的新的测试用例，或者修改测试用例库中原本的测试用例，使修改后的测试用例与需求单或者问题单对应，从而构建需求单/问题单与测试用例之间的映射关系。基于此，从中可以获取到本次修改所涉及的需求单/问题单(即被修改的软件代码对应的需求单/问题单)以及其对应的全部第二测试用例之间的映射关系，从而确定被修改的软件代码所对应的第二测试用例。
需要说明的是，如前所述，需求单可以通过多种方式来唯一标识，例如需求单号等，问题单也可以通过多种方式来唯一标识，例如问题单号等，测试用例也可以通过测试用例标识来唯一标识，例如测试用例编号等。基于此，在本申请实施例中，需求单/问题单与测试用例之间的映射关系，可以通过需求单号/问题单号与测试用例标识之间的映射关系来表示，如表4所示。
表4需求单/问题单与测试用例之间的映射关系示例
Figure PCTCN2020115580-appb-000004
S203：将每一个所述第二测试用例对应的需求单/问题单的数量，确定为所述第二测试用例的关联度。
转换需求单/问题单与第二测试用例之间映射关系的表示方式，将每一个需求单/问题单会对应一个或者多个第二测试用例的映射关系，转换为每一个第二测试用例对应一个或者多个需求单/问题单。然后，统计一个第二测试用例对应的需求单/问题单的数量，将该数量确定为该第二测试用例的关联度。
例如，在表5所示的例子中，用例1对应3个需求单号/问题单号，则其关联度为3。其他用例的关联度的确定过程与此类似。
表5关联度示例
Figure PCTCN2020115580-appb-000005
需要说明的是，随着软件开发人员提交更多新的修改记录，代码和需求单/问题单的对应关系可能会发生变化。类似地，随着软件测试人员针对需求单/问题单而更新测试用例库，需求单/问题单与测试用例之间的映射关系也可能发生变化。因此，在不同次的筛选中所获取到的关联度可能存在差异，进而导致筛选结果也可能存在差异。
S300：根据所述至少一个第一测试用例的预测执行失败率，以及所述至少一个第二测试用例的关联度，从至少一个待选测试用例中筛选出高风险用例。
上述的至少一个待选测试用例，是指所述至少一个第一测试用例和所述至少一个第二测试用例的并集。如S100的步骤中所述，第二测试用例的集合，可以和前述第一测试用例的集合完全相同、部分相同，或者完全不相同。基于此，在筛选的时候，可以将二者中相同的测试用例合并，取二者的并集，形成一个完整的待选测试用例的集合。然后从该待选测试用例的集合去筛选出高风险用例。
在筛选高风险用例的一种实现方式中，可以以预测执行失败率和关联度作为排序依据，对所有的待选测试用例进行排序，然后根据排序结果，筛选出高风险用例。该实现方式与后续的另一种实现方式相比，由于不需要设定权重值，故而更加容易被使用。
请参见图4，图4为本申请实施例的测试用例筛选方法中，S300的步骤的实现方式之一的流程图。该实现方式可以包括以下S311至S312的步骤。
S311：以所述预测执行失败率和所述关联度为排序依据，对所述至少一个待选测试用例排序。
上述的排序依据，也可以理解为是排序时所依照的关键词。待选测试用例的预测执行失败率、关联度均可以被作为一个排序依据。
需要说明的是，在排序时，可以按照预测执行失败率或关联度的升序进行排序，也可以按照降序进行排序，本申请对此不作限定。在排序时，不同排序依据的优先级可以不同，预测执行失败率的优先级可以高于关联度的优先级，也可以低于关联度的优先级，本申请对此也不作限定。根据应用场景的不同，可以选择不同的排序方式， 以及不同的排序依据的优先级。
例如，沿用表2的预测执行失败率和表5的关联度的例子，假设排序时预测执行失败率的优先级高于关联度的优先级，按照预测执行失败率和关联度的降序排序，则排序结果如表6所示。
表6排序结果示例
Figure PCTCN2020115580-appb-000006
S312：从排序结果中筛选出预设数量个所述预测执行失败率或者所述关联度最高的高风险用例。
上述预设数量可以是人工预设的，也可以是计算机根据其他信息来设定的，本申请对于预设数量的预设方式，以及预设数量的具体取值均不作限定。
在排序时，当预测执行失败率的优先级高于关联度时，可以从排序结果中筛选出预设数量个预测执行失败率最高的待选测试用例；当关联度的优先级高于预测执行失败率时，可以从排序结果中筛选出预设数量个关联度最高的待选测试用例。
例如，沿用表6的例子，假设将排序最靠前的3个待选测试用例确定为高风险用例，则筛选出的高风险用例为：用例4，用例3，用例1。
在筛选高风险用例的另一种实现方式中，可以根据预测执行失败率和关联度的权重，来评估每个待选测试用例对应的风险，从而筛选出高风险用例。
请参见图5，图5为本申请实施例的测试用例筛选方法中，S300的步骤的实现方式之二的流程图。该实现方式可以包括以下S321至S323的步骤。
S321：确定所述预测执行失败率的权重和所述关联度的权重。
上述预测执行失败率的权重和关联度的权重可以根据不同的应用场景来确定。在不同的应用场景中，将预测执行失败率的权重和关联度的权重确定为不同的取值，有助于更加精准地筛选出高风险用例，进而有助于提高测试的有效性。
预测执行失败率的权重w1和关联度的权重w2均可以取[0％,100％]的区间内的任意数值，并且w1+w2＝100％。
在一种实现方式中，用户可以手动设置二者的权重的取值，从而使计算机直接根据用户设置的取值来确定二者的权重。
S322：利用所述预测执行失败率的权重和所述关联度的权重，计算每一个所述待 选测试用例对应的风险评估值。
在计算待选测试用例的风险评估值的一种实现方式中，对于任一个待选测试用例而言，其对应的风险评估值可以由该待选测试用例的预测执行失败率和关联度的权重加和得到。即，
Ri＝Fi×w1+Ci×w2；其中，
Ri表示所述至少一个待选测试用例中的第i个待选测试用例的风险评估值；
Fi表示所述至少一个待选测试用例中的第i个待选测试用例的预测执行失败率；
Ci表示所述至少一个待选测试用例中的第i个待选测试用例的关联度。
需要说明的是，对于任一个待选测试用例而言，如果未获取到该待选测试用例的预测执行失败率Fi，则取Fi＝0，以此计算该待选测试用例的Ri。类似地，对于任一个待选测试用例而言，如果未获取到该待选测试用例的关联度Ci，则取Ci＝0，以此计算该待选测试用例的Ri。
例如，沿用表2的预测执行失败率和表5的关联度的例子，假设w1取90％，w2取10％，则计算出的各个待选测试用例的风险评估值如表7所示。
表7风险评估值示例
待选测试用例标识 计算过程 风险评估值
用例1 R1＝30％×90％+3×10％ 0.57
用例2 R2＝7％×90％+1×10％ 0.163
用例3 R3＝45％×90％+2×10％ 0.605
用例4 R4＝90％×90％+0×10％ 0.81
用例5 R5＝30％×90％+0×10％ 0.27
…… …… ……
S323：将风险评估值高于预设的第一阈值的所述待选测试用例，确定为高风险用例。
上述预设的第一阈值可以是人工预设的，也可以是计算机根据其他信息来设定的，本申请对于预设的第一阈值的预设方式，以及预设的第一阈值的具体取值均不作限定。
例如，沿用以表7的风险评估值的例子，假设预设第一阈值为0.55，则从表7所示的待选测试用例中筛选出的高风险用例为：用例1，用例3，用例4。
需要说明的是，当应用于某些特定应用场景中时，也可以单独利用第一测试用例的预测执行失败率或者第二测试用例的关联度来筛选高风险用例。比如，在一个示例中，可以将预测执行失败率超过预设的第二阈值的第一测试用例，确定为高风险用例。又比如，在另一个示例中，可以将关联度高于预设的第三阈值的第二测试用例，确定为高风险用例。
还需要说明的是，以上S100至S300的步骤仅仅描述了一次测试用例筛选的步骤。在软件的回归测试过程中，可能需要进行多次测试，每一次筛选测试用例都可以重新 执行上述的筛选方法。由于训练用例分类器的训练样本(即历史执行数据)会被更新，测试用例的关联度也可能被更新，测试用例库中的测试用例也可能被更新，因此，在不同次的筛选中所得到的筛选结果也可能存在差异。
本申请实施例中的方法结合了基于测试用例的风险的筛选方法，以及基于软件中被修改的代码的筛选方法这两种类型，一方面利用机器学习技术来预测测试用例的风险，另一方面获取测试用例与软件代码之间的关联度。预测执行失败率与测试用例的风险相关，关联度也与测试用例的风险相关，同时还与测试用例的需求覆盖相关。因此，结合这两方面的信息，可以更加精准地筛选出高风险用例，促使缺陷发现前移，促使覆盖需求的测试用例前移，进而提高回归测试的有效性。
现有技术中的基于每个代码行与测试用例之间的映射关系来筛选的方法，需要软件源代码作为输入信息，是一种白盒测试方法。该方法需要公开软件的源代码，对软件的信息安全造成风险。并且，由于实现该方法需要对所有测试用例进行覆盖分析，故而存在实施成本较高的问题。现有技术中的基于每个测试用例所发现的缺陷，以及缺陷的严重等级来评估测试用例的风险的方法，也是一种白盒测试方法，同样存在上述的问题。
而本申请实施例中的方法，是一种黑盒测试方法，无需使用产品的源码作为输入，有利于保障软件的信息安全。同时，该方法无需对所有测试用例进行覆盖分析，无需使用到执行每个测试用例所发现的缺陷，以及缺陷的严重等级，所使用到的用例分类器在训练的时候，也无需使用测试用例的具体内容，仅仅使用测试用例的历史执行数据即可，从而大大降低了实施成本。
此外，在现有技术中，无论是做覆盖分析，还是采用测试用例的内容来训练用例分类器，都需要产品代码的信息或者测试用例的内容，耗费较长时间。而在本申请实施例中，由于无需使用测试用例的具体内容，当获取到被更新的历史执行数据之后，用其来训练用例分类器所需的时间较短。基于此，本申请实施例所获取的第一测试用例的预测执行失败率实时性更好。同时，采用本申请实施例的方法获取的第二测试用例的关联度实时性也较好。通常获得覆盖分析的结果所需要的时间较长，而本申请实施例中的历史执行数据和关联度都可以实时更新，这使得第一测试用例的预测执行失败率和第二测试用例的关联度可这两个参数可以实现分钟级更新，进而也提升了实现本申请实施例中的筛选方法的时效性。
前述实施例对本申请提供的筛选方法进行了介绍。本领域技术人员应该很容易意识到，结合前述实施例中描述的步骤，本申请中的筛选方法不仅能够以硬件或计算机软件的形式来实现，还可以以硬件和计算机软件结合的形式来实现。某个功能究竟以何种形式来实现，取决于技术方案的特定应用和设计约束条件。
本实施例提供一种测试用例的筛选装置。在筛选装置的一种实现方式中，可以对该装置进行功能模块的划分。在采用对应各个功能划分各个功能模块的情况下，图6示出了本申请中筛选装置的一种实现方式的结构示意图。该筛选装置400包括获取模块401和处理模块402；其中，
所述获取模块401用于获取至少一个第一测试用例的预测执行失败率；以及，获 取至少一个第二测试用例的关联度；其中，所述预测执行失败率由用例分类器对所述至少一个第一测试用例进行预测得到，所述用例分类器为利用历史执行数据训练得到的分类器，所述历史执行数据包括执行所述至少一个第一测试用例而生成的至少一条执行记录；所述第二测试用例为被修改的软件代码所对应的测试用例，所述关联度用于表征所述第二测试用例与对应的所述被修改的软件代码之间的关联程度。
所述处理模块402用于根据所述至少一个第一测试用例的预测执行失败率，以及所述至少一个第二测试用例的关联度，从至少一个待选测试用例中筛选出高风险用例；其中，所述至少一个待选测试用例为所述至少一个第一测试用例和所述至少一个第二测试用例的并集。可选地，所述获取模块401还用于：获取所述历史执行数据。
所述处理模块402还用于：利用所述历史执行数据训练分类器，得到用例分类器；将所述至少一个第一测试用例输入所述用例分类器，得到所述至少一个第一测试用例各自的所述预测执行失败率。
可选地，所述执行记录包括测试用例标识，与所述测试用例标识对应的执行结果、执行时间和执行时长。
可选地，所述处理模块402还用于：根据所述执行记录中的执行结果清洗所述历史执行数据，得到清洗后数据；将所述清洗后数据中的执行记录输入至所述分类器中，得到所述用例分类器。
可选地，所述执行记录还包括失败原因；所述处理模块402还用于：根据所述执行结果和所述失败原因清洗所述历史执行数据，得到清洗后数据。
可选地，所述处理模块402还用于：将所述历史执行数据中每一个所述第一测试用例各自对应的执行记录的数量，确定为每一个所述第一测试用例各自对应的执行次数；以及，根据所述执行结果和所述执行次数清洗所述历史执行数据，得到清洗后数据。
可选地，所述处理模块402还用于：将所述清洗后数据中的执行记录，按照所述执行记录中的执行时间输入至所述分类器中，得到所述用例分类器。
可选地，所述获取模块401还用于：获取与被修改的软件代码对应的需求单/问题单；获取与所述需求单/问题单对应的至少一个第二测试用例。
所述处理模块402还用于：将每一个所述第二测试用例对应的需求单/问题单的数量，确定为所述第二测试用例的关联度。
可选地，所述处理模块402还用于：以所述预测执行失败率和所述关联度为排序依据，对所述至少一个待选测试用例排序；从排序结果中筛选出预设数量个所述预测执行失败率或者所述关联度最高的高风险用例。
可选地，所述处理模块402还用于：确定所述预测执行失败率的权重和所述关联度的权重；利用所述预测执行失败率的权重和所述关联度的权重，计算每一个所述待选测试用例对应的风险评估值；以及，将风险评估值高于预设的第一阈值的所述待选测试用例，确定为高风险用例。
可以理解的是，本实施例中对筛选装置的功能模块的划分是示意性的，仅仅为一种逻辑功能划分，在实际实现时可以采用另外的划分方式。例如，获取模块401的功能可以由总线接口或者收发器实现，处理模块402的功能可以由处理器实现。
在筛选装置的另一种实现方式中，各独立运行的功能实体可以集成在一个硬件实体中，例如服务器、芯片等。图7示出了本申请中筛选装置的另一种实现方式的结构示意图。图7从独立功能实体的角度对本申请的筛选装置进行介绍。该筛选装置500可以包括至少一个存储器501，以及至少一个处理器502。
所述至少一个存储器501存储有一个或多个计算机程序或指令。存储器501可以包括易失性存储器(volatile memory)，例如随机存取内存(random access memory，RAM)；还可以包括非易失性存储器(non-volatile memory)，例如快闪存储器(flash memory)，硬盘(hard disk drive，HDD)或固态硬盘(solid-state drive，SSD)；存储器501还可以包括上述种类的存储器的组合。所述存储器501中可以存储计算机可执行程序或指令。
处理器502通过执行存储在存储器501中的指令，从而可以实现筛选装置的功能或者数据处理。例如，处理器502通过执行存储器501中存储的程序或指令，使得所述筛选装置500实现前述任一种筛选方法的部分或者全部步骤。示例性地，前述筛选装置400中的处理模块402的功能，可以由处理器502实现。
上述处理器502可以包括一个或多个处理单元，例如：处理器502可以包括应用处理器(application processor，AP)，调制解调处理器，图形处理器(graphics processing unit，GPU)，图像信号处理器(image signal processor，ISP)，控制器，视频编解码器，数字信号处理器(digital signal processor，DSP)，基带处理器，和/或神经网络处理器(neural-network processing unit，NPU)等。其中，不同的处理单元可以是独立的器件，也可以集成在一个或多个处理器中。所述处理器502还可以进一步包括硬件芯片。上述硬件芯片可以是专用集成电路(application-specific integrated circuit，ASIC)，可编程逻辑器件(programmable logic device，PLD)或其组合。上述PLD可以是复杂可编程逻辑器件(complex programmable logic device，CPLD)，现场可编程逻辑门阵列(field-programmable gate array，FPGA)，通用阵列逻辑(generic array logic,简称GAL)或其任意组合。
上述筛选装置还可以包括收发器503，收发器503可以用于接收筛选装置外部的数据。例如，当第一测试用例的预测执行失败率或者第二测试用例的关联度存储在筛选装置外部时，可以通过收发器503来接收这些数据。示例性地，前述筛选装置400中的获取模块401的功能，可以由收发器503实现。
本实施例还提供一种终端设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，当所述计算机程序在终端设备上运行时，使得所述终端设备执行前述任一种筛选方法的部分或全部步骤。
本实施例还提供一种芯片，与存储器相连或者包括存储器，用于读取并执行所述存储器中存储的计算机程序，当所述计算机程序在芯片上运行时，使得所述芯片执行前述任一种筛选方法的部分或全部步骤。
本实施例还提供一种计算机可读存储介质。该计算机可读存储介质用于存储计算机程序或指令，当所述计算机程序或指令在计算机上运行时，使得所述计算机执行前述任一种筛选方法的部分或全部步骤。
这里的可读存储介质可为磁碟、光盘、DVD、USB、只读存储记忆体(ROM)或 随机存储记忆体(RAM)等，本申请对具体的存储介质形式不作限定。
上述实施例的方法可以全部或部分地通过软件、硬件、固件或者其任意组合来实现。当使用软件实现时，可以全部或部分地以计算机程序产品的形式实现。所述计算机程序产品包括一个或多个计算机指令。在计算机上加载和执行所述计算机程序指令时，全部或部分地产生按照本申请所述的流程或功能。所述计算机可以是通用计算机、专用计算机、计算机网络、或者其他可编程装置。所述计算机指令可以存储在计算机可读存储介质中，或者从一个计算机可读存储介质向另一个计算机可读存储介质传输，例如，所述计算机指令可以从一个网站站点、计算机、服务器或数据中心通过有线(例如同轴电缆、光纤、数字用户线(DSL))或无线(例如红外、无线、微波)方式向另一个网站站点、计算机、服务器或数据中心进行传输。所述计算机可读存储介质可以是计算机能够存取的任何可用介质或者是包含一个或多个可用介质集成的服务器、数据中心等数据存储设备。所述可用介质可以是磁性介质(例如软盘、硬盘、磁带)、光介质(例如DVD)或者半导体介质(例如固态硬盘(solid state disk，SSD))等。
应理解，在本申请的各种实施例中，各步骤的执行顺序应以其功能和内在逻辑确定，各步骤序号的大小并不意味着执行顺序的先后，不对实施例的实施过程构成限定。
除非另外说明，本说明书中的“多个”，指的是两个或者两个以上。在本申请的实施例中，采用了“第一”、“第二”等字样对功能和作用基本相同的相同项或相似项进行区分。本领域技术人员可以理解，“第一”、“第二”等字样并不对数量和执行次序构成限定，并且“第一”、“第二”等字样也并不限定一定不同。
还应理解，本说明书中各个实施例中的实现方式，只要逻辑上不矛盾，均可以相互结合。各个实施例之间相同相似的部分互相参见即可。尤其，对于筛选装置、计算机可读存储介质和计算机程序产品的实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例中的说明即可。以上的实施方式并不构成对本申请保护范围的限定。

Claims (16)

  1. 一种测试用例筛选方法，其特征在于，包括：
    获取至少一个第一测试用例的预测执行失败率；其中，所述预测执行失败率由用例分类器对所述至少一个第一测试用例进行预测得到，所述用例分类器为利用历史执行数据训练得到的分类器，所述历史执行数据包括执行所述至少一个第一测试用例而生成的至少一条执行记录；
    获取至少一个第二测试用例的关联度；其中，所述第二测试用例为被修改的软件代码所对应的测试用例，所述关联度用于表征所述第二测试用例与对应的所述被修改的软件代码之间的关联程度；
    根据所述至少一个第一测试用例的预测执行失败率，以及所述至少一个第二测试用例的关联度，从至少一个待选测试用例中筛选出高风险用例；其中，所述至少一个待选测试用例为所述至少一个第一测试用例和所述至少一个第二测试用例的并集。
  2. 根据权利要求1所述的测试用例筛选方法，其特征在于，获取至少一个第一测试用例的预测执行失败率，包括：
    获取所述历史执行数据；
    利用所述历史执行数据训练分类器，得到用例分类器；
    将所述至少一个第一测试用例输入所述用例分类器，得到所述至少一个第一测试用例各自的所述预测执行失败率。
  3. 根据权利要求1或2所述的测试用例筛选方法，其特征在于，所述执行记录包括测试用例标识，与所述测试用例标识对应的执行结果、执行时间和执行时长。
  4. 根据权利要求2所述的测试用例筛选方法，其特征在于，利用所述历史执行数据训练分类器，得到用例分类器，包括：
    根据所述执行记录中的执行结果清洗所述历史执行数据，得到清洗后数据；
    将所述清洗后数据中的执行记录输入至所述分类器中，得到所述用例分类器。
  5. 根据权利要求4所述的测试用例筛选方法，其特征在于，所述执行记录还包括失败原因；
    根据所述执行记录中的执行结果清洗所述历史执行数据，得到清洗后数据，包括：
    根据所述执行结果和所述失败原因清洗所述历史执行数据，得到清洗后数据。
  6. 根据权利要求4所述的测试用例筛选方法，其特征在于，所述方法还包括：
    将所述历史执行数据中每一个所述第一测试用例各自对应的执行记录的数量，确定为每一个所述第一测试用例各自对应的执行次数；
    根据所述执行记录中的执行结果清洗所述历史执行数据，得到清洗后数据，包括：
    根据所述执行结果和所述执行次数清洗所述历史执行数据，得到清洗后数据。
  7. 根据权利要求4所述的测试用例筛选方法，其特征在于，将所述清洗后数据中的执行记录输入至所述分类器中，得到所述用例分类器，包括:
    将所述清洗后数据中的执行记录，按照所述执行记录中的执行时间输入至所述分类器中，得到所述用例分类器。
  8. 根据权利要求1所述的测试用例筛选方法，其特征在于，获取至少一个第二测试用例的关联度，包括：
    获取与被修改的软件代码对应的需求单/问题单；
    获取与所述需求单/问题单对应的至少一个第二测试用例；
    将每一个所述第二测试用例对应的需求单/问题单的数量，确定为所述第二测试用例的关联度。
  9. 根据权利要求1所述的测试用例筛选方法，其特征在于，根据所述至少一个第一测试用例的预测执行失败率，以及所述至少一个第二测试用例的关联度，从至少一个待选测试用例中筛选出高风险用例，包括：
    以所述预测执行失败率和所述关联度为排序依据，对所述至少一个待选测试用例排序；
    从排序结果中筛选出预设数量个所述预测执行失败率或者所述关联度最高的高风险用例。
  10. 根据权利要求1所述的测试用例筛选方法，其特征在于，根据所述至少一个第一测试用例的预测执行失败率，以及所述至少一个第二测试用例的关联度，从至少一个待选测试用例中筛选出高风险用例，包括：
    确定所述预测执行失败率的权重和所述关联度的权重；
    利用所述预测执行失败率的权重和所述关联度的权重，计算每一个所述待选测试用例对应的风险评估值；
    将风险评估值高于预设的第一阈值的所述待选测试用例，确定为高风险用例。
  11. 一种测试用例筛选装置，其特征在于，包括获取模块和处理模块；
    所述获取模块用于获取至少一个第一测试用例的预测执行失败率；以及，获取至少一个第二测试用例的关联度；其中，所述预测执行失败率由用例分类器对所述至少一个第一测试用例进行预测得到，所述用例分类器为利用历史执行数据训练得到的分类器，所述历史执行数据包括执行所述至少一个第一测试用例而生成的至少一条执行记录；所述第二测试用例为被修改的软件代码所对应的测试用例，所述关联度用于表征所述第二测试用例与对应的所述被修改的软件代码之间的关联程度；
    所述处理模块用于根据所述至少一个第一测试用例的预测执行失败率，以及所述至少一个第二测试用例的关联度，从至少一个待选测试用例中筛选出高风险用例；其中，所述至少一个待选测试用例为所述至少一个第一测试用例和所述至少一个第二测试用例的并集。
  12. 一种测试用例筛选装置，其特征在于，包括：至少一个存储器，以及至少一个处理器；所述至少一个存储器存储有一个或多个指令，当所述一个或多个所述指令被所述至少一个处理器执行时，使得所述装置实现权利要求1-10任一项所述的方法。
  13. 一种计算机可读存储介质，其特征在于，所述计算机存储介质中包括指令，当所述指令在计算机上运行时，使得所述计算机执行如权利要求1-10任一项所述的方法。
  14. 一种计算机程序产品，其特征在于，当其在计算机上运行时，使得计算机执行如权利要求1-10中任一项所述的方法。
  15. 一种终端设备，其特征在于，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行所述程序时，实现如权利要求1-10中任 一项所述的方法。
  16. 一种芯片，其特征在于，与存储器相连或者包括存储器，用于读取并执行所述存储器中存储的计算机程序，以实现如权利要求1-10中任一项所述的方法。
PCT/CN2020/115580 2020-01-14 2020-09-16 测试用例筛选方法、装置及介质 WO2021143175A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202010037435.9A CN111274126B (zh) 2020-01-14 2020-01-14 测试用例筛选方法、装置及介质
CN202010037435.9 2020-01-14

Publications (1)

Publication Number Publication Date
WO2021143175A1 true WO2021143175A1 (zh) 2021-07-22

Family

ID=71001849

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/115580 WO2021143175A1 (zh) 2020-01-14 2020-09-16 测试用例筛选方法、装置及介质

Country Status (2)

Country Link
CN (1) CN111274126B (zh)
WO (1) WO2021143175A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113886256A (zh) * 2021-10-14 2022-01-04 广州九尾信息科技有限公司 一种软件质量监控训练模型的方法
CN114564388A (zh) * 2022-02-23 2022-05-31 中国农业银行股份有限公司 一种程序测试方法、装置、电子设备及存储介质

Families Citing this family (13)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111274126B (zh) * 2020-01-14 2022-10-04 华为云计算技术有限公司 测试用例筛选方法、装置及介质
CN111897722B (zh) * 2020-07-15 2022-11-22 重庆紫光华山智安科技有限公司 自动化测试用例处理方法、装置、服务器及存储介质
WO2022061779A1 (zh) * 2020-09-25 2022-03-31 西门子股份公司 基于变更相关性分析的测试用例选择方法及装置
CN112363928B (zh) * 2020-11-10 2023-08-22 网易（杭州）网络有限公司 测试用例的处理方法、装置、处理器及电子装置
CN113064824B (zh) * 2021-03-31 2022-12-02 重庆紫光华山智安科技有限公司 结果分析方法和装置、电子设备及存储介质
CN112988606B (zh) * 2021-05-11 2021-07-23 成都爱瑞无线科技有限公司 测试用例生成方法、装置、设备及存储介质
CN113296836B (zh) * 2021-06-08 2022-07-22 北京百度网讯科技有限公司 训练模型的方法、测试方法、装置、电子设备及存储介质
CN113434436B (zh) * 2021-07-30 2022-12-16 拉扎斯网络科技（上海）有限公司 测试用例的生成方法、装置、电子设备及存储介质
CN113672506B (zh) * 2021-08-06 2023-06-13 中国科学院软件研究所 基于机器学习的动态比例测试用例排序选择方法及系统
CN113609023A (zh) * 2021-08-16 2021-11-05 未鲲(上海)科技服务有限公司 精准测试方法、装置、设备及存储介质
CN113868122B (zh) * 2021-09-10 2023-11-14 苏州浪潮智能科技有限公司 一种用于回归测试的测试用例筛选方法、系统及装置
CN114168472A (zh) * 2021-12-09 2022-03-11 惠州Tcl移动通信有限公司 用例执行方法、装置、电子设备及计算机可读存储介质
CN118642935A (zh) * 2023-03-06 2024-09-13 华为云计算技术有限公司 选取测试用例的方法、装置、设备和存储介质

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103560829A (zh) * 2013-10-31 2014-02-05 上海斐讯数据通信技术有限公司 一种uni 端口的自动化测试方法及系统
US20160378647A1 (en) * 2014-07-30 2016-12-29 Hitachi, Ltd. Development supporting system
CN108255653A (zh) * 2018-01-02 2018-07-06 深圳壹账通智能科技有限公司 一种产品的测试方法及其终端
CN110147321A (zh) * 2019-04-19 2019-08-20 北京航空航天大学 一种基于软件网络的缺陷高风险模块的识别方法
CN110489321A (zh) * 2019-07-08 2019-11-22 平安科技（深圳）有限公司 测试用例筛选方法、装置、计算机设备和存储介质
CN111274126A (zh) * 2020-01-14 2020-06-12 华为技术有限公司 测试用例筛选方法、装置及介质

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20190294531A1 (en) * 2018-03-26 2019-09-26 Ca, Inc. Automated software deployment and testing based on code modification and test failure correlation

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103560829A (zh) * 2013-10-31 2014-02-05 上海斐讯数据通信技术有限公司 一种uni 端口的自动化测试方法及系统
US20160378647A1 (en) * 2014-07-30 2016-12-29 Hitachi, Ltd. Development supporting system
CN108255653A (zh) * 2018-01-02 2018-07-06 深圳壹账通智能科技有限公司 一种产品的测试方法及其终端
CN110147321A (zh) * 2019-04-19 2019-08-20 北京航空航天大学 一种基于软件网络的缺陷高风险模块的识别方法
CN110489321A (zh) * 2019-07-08 2019-11-22 平安科技（深圳）有限公司 测试用例筛选方法、装置、计算机设备和存储介质
CN111274126A (zh) * 2020-01-14 2020-06-12 华为技术有限公司 测试用例筛选方法、装置及介质

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113886256A (zh) * 2021-10-14 2022-01-04 广州九尾信息科技有限公司 一种软件质量监控训练模型的方法
CN113886256B (zh) * 2021-10-14 2023-10-10 广州九尾信息科技有限公司 一种软件质量监控训练模型的方法
CN114564388A (zh) * 2022-02-23 2022-05-31 中国农业银行股份有限公司 一种程序测试方法、装置、电子设备及存储介质

Also Published As

Publication number Publication date
CN111274126B (zh) 2022-10-04
CN111274126A (zh) 2020-06-12

Similar Documents

Publication Publication Date Title
WO2021143175A1 (zh) 测试用例筛选方法、装置及介质
Lu et al. Making disk failure predictions {SMARTer}!
US20180082215A1 (en) Information processing apparatus and information processing method
US10216558B1 (en) Predicting drive failures
US8707268B2 (en) Testing operations of software
CN107391369B (zh) 一种基于数据筛选和数据过采样的跨项目缺陷预测方法
CN110502374A (zh) 识别自动测试时设备故障的根本原因的流量捕获调试工具
TW201732789A (zh) 磁片的故障預測方法和裝置
US9959197B2 (en) Automated bug detection with virtual machine forking
CN112214369A (zh) 基于模型融合的硬盘故障预测模型建立方法及其应用
WO2021243508A1 (en) Scanning for information according to scan objectives
CN113010389A (zh) 一种训练方法、故障预测方法、相关装置及设备
CN107168995A (zh) 一种数据处理方法及服务器
US20220138032A1 (en) Analysis of deep-level cause of fault of storage management
CN116401232B (zh) 数据库参数配置优化方法、装置、电子设备及存储介质
CN115964211A (zh) 一种根因定位方法、装置、设备和可读介质
US20050262399A1 (en) Aggregating and prioritizing failure signatures by a parsing program
JP7421136B2 (ja) 機械学習プログラム、機械学習方法および機械学習装置
CN106096635B (zh) 基于阈值操作的代价敏感神经网络的警告分类方法
CN116701222A (zh) 基于特征加权迁移学习的跨项目软件缺陷预测方法及系统
US12066910B2 (en) Reinforcement learning based group testing
CN106055483B (zh) 基于欠采样操作的代价敏感神经网络的警告分类方法
CN112380127B (zh) 测试用例回归方法、装置、设备和存储介质
WO2021074995A1 (ja) 閾値取得装置、その方法、およびプログラム
CN106095671B (zh) 基于过采样操作的代价敏感神经网络的警告分类方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20914213

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20914213

Country of ref document: EP

Kind code of ref document: A1