DE102019210562A1 - Method and device for testing software - Google Patents

Abstract

A method (10) for testing software, characterized by the following features: - random input data (22) for the software are obtained (11) from a test case collection (21), - test cases are selected by a cluster analysis (12) of the input data (22) (23), - in the selected test cases, observations (29) are made with regard to a runtime behavior of the software (13), - a regression model of the runtime behavior is derived from the test cases and respective observations (29) (14) and - based on the regression model, further Test cases created (15) and executed (24).

Description

The present invention relates to a method for testing software. The present invention also relates to a corresponding device, a corresponding computer program and a corresponding storage medium.

State of the art

An automated process for software tests, in which the program to be tested is repeatedly loaded with random data at one or more input interfaces, is technically referred to as fuzzing in software technology. With such random data, so-called crashes or other exceptional situations that do not occur with other test methods are usually to be generated during operation of the program.

Fuzzing is usually used in state-of-the-art software development projects as part of a black box test to check new software for susceptibility to errors.

EP3468131A1 For example, discloses a safety test system for a control device for a vehicle network, which comprises a safety test unit with a test signal transmission part that sends random test signals that include an invalid signal for diagnosing the safety of a control device for a vehicle network, and a diagnostic part that the invalid signal on the basis a process result specified in the control device using the test signals, and a function evaluation unit having a process information acquisition part that acquires information on the process result in the control device, an anomaly detection part that recognizes anomalies in the process result in the control device based on the test signal, and an anomaly Information output part that finally outputs information about the detected anomaly.

Also known are so-called gray box fuzzers such as “American Fuzzy Lop” (AFL), which support programs such. B. to minimize the test case collection, the so-called corpus, include. By instrumenting the source code of the program to be tested during translation, such a fuzzer can later evaluate which blocks of the software are run through with a certain test stimulus in order to achieve a comparatively high code coverage after a relatively short time.

Disclosure of the invention

The invention provides a method for checking software, a corresponding device, a corresponding computer program and a corresponding storage medium according to the independent claims.

This solution is based on the knowledge that AFL generates new fuzzy inputs from its current corpus by proceeding according to certain strategies. Modified test cases that have caused new state transitions within the program are added to the input queue and used as a starting point for future fuzzing rounds. Conventional fuzzers such as AFL are dependent on the execution of many test cases (sometimes hundreds or thousands per second), which is not feasible for slow software - especially software systems that require some form of data transmission. It is currently up to the developer of the fuzz wrapper to find a fast enough solution to this problem.

Against this background, a main idea of the proposed approach is to subject the input queue or the corpus to a clustering analysis before the actual test execution in order to select an optimal input from each cluster and in this way to accelerate the fuzz testing or a test metric (e.g. coverage) to maximize. The clustering process is learned through machine learning approaches.

One advantage of this solution is that based on existing test cases, e.g. B. from previous fuzzing runs, clusters can be found in such a way that only a few tests from each cluster have to be carried out in the next fuzzing round in order to potentially deepen the following findings. If the inputs are selected from different clusters, then these inputs should have a greater distance from one another than to inputs from the same cluster. In this way, the fuzzer can maximize the test metric, such as coverage, by executing only a few test cases from a large corpus.

This idea is particularly impressive with slow-running software, as clustering can take place not only before the first test run, but in every fuzzing round.

The measures listed in the dependent claims enable advantageous developments and improvements of the basic idea specified in the independent claim. So can derive a regression model be provided from the executed test cases by a Gaussian process regression, since Gaussian processes (GPs) learn continuous functions, model uncertainties and prior knowledge such as domain knowledge in different ways - e.g. B. in the form of a parametric mean or a covariance function - can integrate. In addition, GPs are more independent of model assumptions than parametric methods.

Figure list

• 1 das Flussdiagramm eines Verfahrens gemäß einer ersten Ausführungsform.
• 2 das Verfahren eines Fuzzing-Ansatzes, bei dem der Korpus vor der eigentlichen Testfallausführung geclustert wird.
• 3 das Verfahren eines Fuzzing-Ansatzes, bei dem nur der Ausgangskorpus in Cluster eingeteilt wird. Danach wird das Fuzzing auf herkömmliche Weise durchgeführt.
• 4 schematisch eine Arbeitsstation gemäß einer zweiten Ausführungsform der Erfindung.

Exemplary embodiments of the invention are shown in the drawings and explained in more detail in the description below. It shows:

• 1 the flow chart of a method according to a first embodiment.
• 2 the method of a fuzzing approach, in which the corpus is clustered before the actual test case execution.
• 3 the process of a fuzzing approach, in which only the initial body is divided into clusters. Then fuzzing is carried out in the conventional manner.
• 4th schematically a work station according to a second embodiment of the invention.

Embodiments of the invention

1 illustrates the basic steps of the proposed method ( 10 ) for testing software available at least as a binary program ( 16 ).

A first step (process 11 ) of the procedure consists in _taking a first sentence D _raw = {x _i | i = 1 ... n} of test cases using a fuzzing tool. This initially corresponds to the conventional procedure in the context of a typical fuzzing analysis. In order to learn a helpful model within a broader spectrum of procedural variants, it is advisable to generate a large number of test cases. It should be noted that these test cases do not contain any additional information from the fuzzing tool such as B. contain the code coverage because the test cases have not yet passed and have been executed in the software. These test cases are used as the basis for creating the model in the following step (process 12 ) is used.

After the described data generation ( 11 ) it is necessary to cluster unattended based on the test case set Draw in order to avoid running test cases that are similar and therefore likely to lead to similar test results. Since the properties of the generated test cases are unknown, the clustering algorithm used for this should be able to find clusters of any shape without knowing the total number of clusters. Therefore, a hierarchical density-based spatial clustering of applications with noise (HDBSCAN) is recommended, which allows non-convex clusters of different densities to be learned without prior knowledge of the number of total clusters. HDBSCAN carries out the well-known DBSCAN on the in process 11 generated data over different ε values and integrates the result in order to find a clustering that offers the best possible stability over ε.

Since HDBSCAN uses a hierarchical tree for cluster search, it is possible to determine the strengths of the cluster membership. This, as well as the structure of the tree itself, can be used to examine n test cases per cluster, on this basis to select those data sets that both have a high cluster membership and fall into different branches of the tree, and thus ultimately a high variance within the respective one To achieve clusters. In this way, a subset D _clust = (x̂ _i | i = 1 ... n} of test cases can be defined.

The subset D _cluster of test cases defined in this way is transferred as input to the fuzzing tool, which then executes the software to be tested (process 13th ). For each test run, the fuzzing tool generates additional information, which is referred to below as "observations", such as code coverage, information about software crashes and other relevant information. These observations y can be combined with the test cases x̂ used in the next step (process 14th ) can be used for supervised learning of the model. The data obtained are stored in a data set D _obs = {(x̂ _i , y _i ) | i = 1 ... n}, which consists of the test cases x̂ and the observations y obtained through their respective evaluation.

A regression model is generated from the data set D _obs (process 14th ), which predicts the observation y _i for a given test case x̂ _i . Based on this model, it is possible to use only promising example sentences of D _obs for further testing of the software. Promising in this context means that this data guarantees the greatest possible code coverage and has a high probability of leading to a software crash. It is therefore important how certain a prediction is in order to decide whether it makes sense to use the test case in question. In order to learn a basic model, it is advisable to determine the parameters of a Gaussian process using the data D _obs from process 13th to investigate, by maximizing the logarithmic plausibility function (log likelihood): $${\text{argmax}}_{\theta }\text{log}\left(p\left(\text{y}|\text{X},\mathrm{\theta }\right)\right)$$

When using the trained model, active learning can optionally be used to determine which test cases should be checked on a random basis in order to improve the model online.

As mentioned above, the generated regression model is used to select the most promising test cases from the cluster data D _obs . This selection of test cases could then be used - in terms of time and computing power - more complex checks for runtime errors that require instrumentation of the source code. Therefore, only test cases that penetrate the software to the maximum are executed.

2 illustrates the approach, the test cases before the actual test execution ( 24 ) in the input queue ( 22nd ), the body, to cluster ( 12 ). Nevertheless, it can be provided that only the initial body ( 21st ) to cluster ( 12 ) without departing from the scope of the invention. 3 exemplifies such an alternative embodiment.

This method ( 10 ) can be in software or hardware or in a mixed form of software and hardware, for example in a workstation 40 ) be implemented as shown in the schematic representation of the 4th clarified.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• EP 3468131 A1 [0004]

Claims (10)

Method (10) for testing software, characterized by the following features: - input data (22) for the software are obtained (11) from a test case collection (21), - test cases are selected by a cluster analysis (12) of the input data (22) ( 23), - observations (29) are made by the software in the selected test cases (13), - a regression model with regard to the selected observations is derived from the test cases and respective observations (29) and - further test cases are created using the regression model (15) and executed (24). Method (10) according to Claim 1 , characterized by the following features: - the derivation (14) of the regression model takes place in a monitored learning process and - the learning process is set up to maximize a dependent variable of the regression model (30). Method (10) according to Claim 2 , characterized by the following feature: - the variable is a probability of occurrence of an exceptional situation. Method (10) according to Claim 3 , characterized by the following features: - the exceptional situation is a crash (27) of the software and - if the crash (27) does not occur in a test case, it is discarded (28). Method (10) according to one of the Claims 1 to 4th , characterized by the following feature: - before the test cases are executed (24), a binary program (16) of the software is instrumented (25). Method (10) according to one of the Claims 1 to 5 , characterized by the following feature: the cluster analysis (12) is a hierarchical density-based spatial cluster analysis with noise. Method (10) according to one of the Claims 1 to 6th , characterized by the following feature: the derivation (14) of the regression model takes place by means of a Gaussian process regression. Computer program which is set up, the method (10) according to one of the Claims 1 to 7th execute. Machine-readable storage medium on which the computer program is based Claim 8 is stored. Device (40) which is set up, the method (10) according to one of the Claims 1 to 7th execute.

Images