DE102022202542A1 - Method for testing a computer program - Google Patents

Abstract

According to various embodiments, a method for testing a computer program is described, comprising executing the computer program for each input data set for the computer program of a plurality of input data sets and in each case determining a coverage of the computer program that was achieved when executing the computer program for the respective input data set, generating, for each input data set, a training data element containing the coverage determined for the input data set as training input and the input data set as target output, training a machine learning model using supervised learning using the generated training data elements, the machine learning model having an encoder followed by a decoder, wherein the The encoder is set up to generate representations in a representation space from covers and the decoder is set up to generate input data sets for the computer program from representations supplied to it from the representation space, selecting representations from the representation space, generating additional input data sets by supplying the selected representations to the Decoder and test the computer program by running the computer program on the additional input data sets.

Description

State of the art

The present disclosure relates to methods for testing computer programs.

An essential part of developing software applications is testing. In particular, errors that cause an application to fail should be identified and corrected. An example of software testing is the dynamic software testing method fuzzing.

Fuzzing enables efficient testing with good test coverage, but its use is difficult for stateful computer programs, i.e. computer programs that store an internal memory state, especially for the reason that it is typically not known which states a given computer program can assume.

It is therefore desirable to have approaches that make it possible to test the behavior of a computer program in different states (e.g. using fuzzing) or to research the states of the computer program if these are not explicitly known.

Disclosure of the invention

According to various embodiments, a method for testing a computer program is provided, comprising executing the computer program for each input data set for the computer program of a plurality of input data sets and in each case determining a coverage of the computer program that was achieved when executing the computer program for the respective input data set, generating, for each input data set, a training data element containing the coverage determined for the input data set as training input and the input data set as target output, training a machine learning model using supervised learning using the generated training data elements, the machine learning model having an encoder followed by a decoder, wherein the The encoder is set up to generate representations in a representation space from covers and the decoder is set up to generate input data sets for the computer program from representations supplied to it from the representation space, selecting representations from the representation space, generating additional input data sets by supplying the selected representations to the Decoder and test the computer program by executing the computer program on the additional input data sets generated.

The method described above enables the state dependence of a computer program to be captured from data observed during execution of the computer program and the use of this knowledge to test the computer program without prior knowledge of the states of the computer program.

It creates new test cases based on the intrinsic generative capability of the ML model (i.e. machine learning model) without the need for user-developed test case mutation strategies.

Various exemplary embodiments are given below.

Embodiment 1 is a method for testing a computer program as described above.

Embodiment 2 is the method according to embodiment 1, where the representations are randomly selected from the representation space.

In other words, representations are sampled from the representation space. Since it can be expected after training that different representations correspond to different states of the program, states that have not yet been reached during testing can be achieved with a high degree of probability with the additional input data sets.

Embodiment 3 is the method according to embodiment 1 or 2, with representations other than those onto which the encoder maps the coverages of the training data elements being selected from the representation space.

For example, the known representations, i.e. those onto which the encoder maps the coverages of the training data elements, can be randomly disturbed using noise. This means that previously unexplored (untested) states of the computer program are most likely reached and tested by the additional input data sets generated by the decoder from the selected representations.

Embodiment 4 is the method according to one of embodiments 1 to 3, where the machine learning model is a variational autoencoder.

The training of a variational autoencoder (VAE) causes the encoder to map onto the representation space in such a way that the representation space is largely structured in a meaningful way in the sense that representations selected from these parts can be interpreted in a meaningful way by the decoder Input data sets are mapped. This allows useful input data sets to be generated for testing the computer program.

Embodiment 5 is the method according to one of embodiments 1 to 4, wherein each input data set for the computer program has a sequence of input data elements for the computer program and executing the computer program for the input data set comprises executing the computer program, the input data elements sequentially according to the sequence of the computer program be supplied.

In this way, a certain state can be achieved with early elements of the sequence, which is explored in more detail by the further elements of the sequence.

Embodiment 6 is the method according to any one of embodiments 1 to 5, wherein the representation space is multidimensional, wherein the machine learning model is trained to learn representations of covers in disentangled form, and wherein selecting the representations from the representation space by setting means setting Components of the representations take place in selected dimensions.

This allows states (or at least areas of states) of the computer program to be achieved in a targeted manner to a certain extent.

Embodiment 7 is a software test system that is set up to carry out a method according to one of embodiments 1 to 6.

Embodiment 8 is a computer program with instructions that, when executed by a processor, cause the processor to perform a method according to any of Embodiments 1 to 6.

Embodiment 9 is a computer-readable medium that stores instructions that, when executed by a processor, cause the processor to perform a method according to any of Embodiments 1 to 6.

• 1 zeigt einen Computer für die Entwicklung und/oder das Testen von Softwareanwendungen.
• 2 zeigt eine Anordnung zum Testen eines Fuzz-Targets mittels eines Fuzzers.
• 3 zeigt ein ML-Modell gemäß einer Ausführungsform.
• 4 zeigt ein Ablaufdiagramm, das ein Verfahren zum Testen eines Computerprogramms gemäß einer Ausführungsform darstellt.

In the drawings, similar reference numbers generally refer to the same parts throughout the several views. The drawings are not necessarily to scale, emphasis instead being placed generally on illustrating the principles of the invention. In the following description, various aspects will be described with reference to the following drawings.

• 1 shows a computer for developing and/or testing software applications.
• 2 shows an arrangement for testing a fuzz target using a fuzzer.
• 3 shows an ML model according to an embodiment.
• 4 shows a flowchart illustrating a method for testing a computer program according to an embodiment.

The following detailed description refers to the accompanying drawings, which show by way of explanation specific details and aspects of this disclosure in which the invention may be practiced. Other aspects may be used and structural, logical and electrical changes may be made without departing from the scope of the invention. The various aspects of this disclosure are not necessarily mutually exclusive, as some aspects of this disclosure may be combined with one or more other aspects of this disclosure to form new aspects.

Various examples are described in more detail below.

1 shows a computer 100 for developing and/or testing software applications.

The computer 100 has a CPU (Central Processing Unit) 101 and a main memory (RAM) 102. The main memory 102 is used to load program code, for example from a hard drive 103, and the CPU 101 executes the program code.

This example assumes that a user intends to use computer 100 to develop and/or test a software application.

To do this, the user runs a software development environment 104 on the CPU 101.

The software development environment 104 allows the user to create an application (ie, software) 105 for various devices 106, i.e. target hardware, such as embedded systems for controlling robotic devices, including robotic arms and autonomous vehicles, or even for mobile (communications) devices , develop and test. For this purpose, the CPU 101 can run an emulator as part of the software development environment 104 to simulate the behavior of the respective device 106 for which an application is or was developed. If it is only used to test software from another source, the software development environment 104 can also be viewed or designed as a software testing environment.

The user can distribute the finished application to corresponding devices 106 via a communication network 107. Instead of via a communication network 107, this can also be done in another way, for example using a USB stick.

However, before this happens, the user should test the application 105 to avoid distributing an improperly functioning application to the devices 106. This can also be the case if the user did not write the application 105 himself on the computer 100. In particular, the case may arise that the user does not have the source code of the application, but only its executable code (i.e. the binary program).

One test method is so-called fuzzing. Fuzzing or fuzz testing is an automated software testing procedure in which invalid, unexpected or random data is fed as input to a computer program under test. The program is then monitored for exceptions such as crashes, missing failed built-in code assertions, or potential memory leaks.

Typically, fuzzers (i.e. test programs that use fuzzing) are used to test programs that process structured input. This structure is e.g. B. specified in a file format or a file format or protocol and distinguishes between valid and invalid inputs. An effective fuzzer produces semi-valid inputs that are "valid enough" not to be directly rejected by the program under test's input parser, but "invalid enough" to reveal unexpected behaviors and edge cases that are not properly handled in the program under test become.

• • Fuzzing oder Fuzz-Testing ist der automatisierte Test-Prozess, zufällig generierte Eingaben an ein Zielprogramm (zu testendes Programm) zu senden und seine Reaktion zu beobachten.
• • Ein Fuzzer oder eine Fuzzing-Engine ist ein Programm, das automatisch Eingaben generiert. Es ist also nicht mit der zu testenden Software verknüpft (z.B. durch Instrumentierung). Der Fuzzer hat jedoch die Fähigkeit, Code zu instrumentieren, Testfälle zu erzeugen und zu testende Programme auszuführen. Bekannte Beispiele sind afl und libfuzzer.
• • Ein Fuzz-Target ist ein Softwareprogramm oder eine Funktion, die durch Fuzzing getestet werden soll. Ein Hauptmerkmal eines Fuzz-Targets sollte sein, dass es potenziell nicht vertrauenswürdige Eingaben annimmt, die vom Fuzzer während des während des Fuzzing-Prozesses erzeugt wird.
• • Ein Fuzz-Test ist die kombinierte Version eines Fuzzers und eines Fuzz-Targets. Ein Fuzz-Target kann dann instrumentierter Code sein, bei dem ein Fuzzer mit seinen Eingaben verknüpft ist (d.h. diese liefert). Ein Fuzz-Test ist ausführbar. Ein Fuzzer kann auch mehrere Fuzz-Tests starten, beobachten und stoppen, d.h. mehrere Testdurchläufe durchführen (normalerweise Hunderte oder Tausende pro Sekunde), jeder mit einer etwas anderen Eingabe, die vom Fuzzer erzeugt wird.
• • Ein Testfall ist eine bestimmte Eingabe und ein bestimmter Testdurchlauf aus einem Fuzz-Test. Normalerweise werden für die Reproduzierbarkeit interessante Läufe (Finden von neuen Codepfaden oder Abstürzen) gespeichert. Auf diese Weise kann ein bestimmter Testfall mit der entsprechenden Eingabe auch auf einem Fuzz-Target ausgeführt werden, das nicht mit einem Fuzzer verbunden ist, z.B. die Release-Version eines Programms.
• • Abdeckungsgesteuertes Fuzzing (engl. coverage-guided fuzzing) verwendet Code-Abdeckungsinformationen als Feedback während des Fuzzings, um zu erkennen, ob eine Eingabe die Ausführung neuer Code-Pfade oder Blöcke verursacht hat.
• • Generator-basiertes Fuzzing (engl. generation-based fuzzing) verwendet vorheriges Wissen über das Zielprogramm (Fuzz-Target), um Testeingaben zu erstellen. Ein Beispiel für ein solches Vorwissen ist eine Grammatik, die der Eingabespezifikation des Fuzz-Targets entspricht, d.h. die Eingabe-Grammatik des Fuzz-Targets (d.h. des zu testenden Programms).
• • Statische Instrumentierung ist das Einfügen von Anweisungen in ein (zu testendes) Programm, um Feedback über die Ausführung zu erhalten. Sie wird meist durch den Compiler realisiert und kann zum Beispiel die erreichten Codeblöcke während der Ausführung angeben.
• • Dynamische Instrumentierung ist die Steuerung der Ausführung eines (zu testenden) Programms während der Laufzeit, um Feedback aus der Ausführung zu generieren. Sie wird meist durch Betriebssystem-Systemfunktionalitäten oder durch die Verwendung von Emulatoren realisiert.
• • Ein Debugger ist eine Vorrichtung oder ein Programm, das ein Zielgerät oder Zielprogramm steuern kann und Funktionen bereitstellen kann, z.B. zum Abrufen von Register- oder Speicherwerten und zum Pausieren und Ausführen es Zielprogramms in Einzelschritten.
• • Ein Breakpoint wird über einen Debugger auf eine Anweisung des Zielprogramms oder Geräts gesetzt, um die Ausführung bei Erreichen zu stoppen und den steuernden Prozess darüber zu informieren.
• • Ein Daten-Watchpoint wird über einen Debugger auf eine Speicheradresse eines Zielprogramms oder Zielgeräts gesetzt, um die Ausführung anzuhalten, wenn auf die Speicheradresse zugegriffen wird, und den steuernden Prozess darüber zu informieren, indem ein Interrupt ausgelöst wird.

The following describes terminology used in the context of fuzzing:

• • Fuzzing or fuzz testing is the automated testing process of sending randomly generated inputs to a target program (program under test) and observing its response.
• • A fuzzer or fuzzing engine is a program that automatically generates input. So it is not linked to the software under test (e.g. through instrumentation). However, the fuzzer has the ability to instrument code, generate test cases, and execute programs under test. Well-known examples are afl and libfuzzer.
• • A fuzz target is a software program or function to be tested through fuzzing. A key feature of a fuzz target should be that it accepts potentially untrusted input generated by the fuzzer during the fuzzing process.
• • A fuzz test is the combined version of a fuzzer and a fuzz target. A fuzz target can then be instrumented code, where a fuzzer is associated with (i.e. provides) its inputs. A fuzz test can be carried out. A fuzzer can also start, observe, and stop multiple fuzz tests, i.e., perform multiple test runs (typically hundreds or thousands per second), each with a slightly different input produced by the fuzzer.
• • A test case is a specific input and test run from a fuzz test. Typically, runs that are interesting for reproducibility (finding new code paths or crashes) are saved. In this way, a specific test case with the corresponding input can also be executed on a fuzz target that is not connected to a fuzzer, e.g. the release version of a program.
• • Coverage-guided fuzzing uses code coverage information as feedback during fuzzing to detect whether an input has caused the execution of new code paths or blocks.
• • Generator-based fuzzing uses prior knowledge about the target program (fuzz target) to create test inputs. An example of such prior knowledge is a grammar that conforms to the input specification of the fuzz target, i.e., the input grammar of the fuzz target (i.e., the program under test).
• • Static instrumentation is the insertion of instructions into a program (under test) to obtain feedback on execution. It is usually implemented by the compiler and can, for example, specify the code blocks reached during execution.
• • Dynamic instrumentation is the control of the execution of a program (under test) during runtime in order to generate feedback from the execution. It is usually implemented through operating system functionalities or through the use of emulators.
• • A debugger is a device or program that can control a target device or target program and can provide functions such as retrieving register or memory values and pausing and executing the target program in individual steps.
• • A breakpoint is set via a debugger to an instruction of the target program or device in order to stop execution when reached and to inform the controlling process of this.
• • A data watchpoint is set to a memory address of a target program or device via a debugger to halt execution when the memory address is accessed and to inform the controlling process by triggering an interrupt.

When a debugger is connected to an embedded device, instruction breakpoints can be used to pause execution at a desired code location, and data watchpoints can be used to pause execution when a specific memory location is accessed. However, the number of breakpoints and watchpoints is typically limited and depends on the system used, for example the maximum number for a typical microcontroller is four breakpoints and two data watchpoints.

A computer program 105 may be stateful or stateless, depending on whether or not it stores internal memory state.

A stateful program accepts well-defined sequences of inputs that affect the internal state of the program. Examples of stateful programs include server programs, communication protocol implementations, web services, and the software for most embedded devices.

• • Zustandsbehaftete Programme nehmen als Eingaben Sequenzen von Nachrichten auf. Standard-Fuzzer sind nicht in der Lage, diese zu erzeugen oder zu analysieren. Aus der Sicht des Fuzzers zeigt das Programm ein nichtdeterministisches Verhalten, da seine Reaktion von früheren Eingaben abhängt, die es gesehen hat.
• • Die Menge aller möglichen Zustände eines zustandsabhängigen Programms ist in der Regel unbekannt.
• • Wenn das Programm eine Implementierung eines Protokolls mit einer formalen Spezifikation ist, kann eine Liste aller Zustände aus dieser Spezifikation abgeleitet werden. Jedoch tritt häufig der Fall auf, dass ein Programm nicht das gesamte Protokoll implementiert oder es implementiert das Protokoll nicht exakt, wodurch die formale Spezifikation für das Fuzzing weniger nützlich wird. Außerdem könnten die durch Fuzzing zu entdeckenden Schwachstellen gerade in den Diskrepanzen zwischen der Spezifikation und der Implementierung des Protokolls liegen.

There are several difficulties in applying fuzzing approaches such as black box fuzzing or coverage-based greybox fuzzing to stateful software:

• • Stateful programs take sequences of messages as input. Standard fuzzers are unable to generate or analyze these. From the fuzzer's perspective, the program exhibits nondeterministic behavior because its response depends on previous inputs that it has seen.
• • The set of all possible states of a state-dependent program is usually unknown.
• • If the program is an implementation of a protocol with a formal specification, a list of all states can be derived from that specification. However, it is often the case that a program does not implement the entire protocol or does not implement the protocol exactly, making the formal specification less useful for fuzzing. In addition, the vulnerabilities that can be discovered through fuzzing could lie precisely in the discrepancies between the specification and the implementation of the protocol.

According to various embodiments, a test input (and associated test cases) is generated, which makes it possible to explore the states of a program without requiring explicit knowledge of the program states.

2 shows an arrangement for testing a fuzz target 202 using a fuzzer 201.

The fuzzer 201 generates an input 203 for the fuzz target 202, and the fuzz target 202 provides feedback 204 to the fuzzer 201 in response to the input. In this way, a fuzzing feedback loop is formed.

The fuzzer 201 and the fuzz target 202 may run on the same computer (e.g., both on the computer 100) or run on different computers (e.g., fuzzer on the computer 100, fuzz target on a target device 106). In this example, the fuzzer 201 is running on a test computer 205 (which may or may not also be running the fuzz target 202). The test computer 205 forms a test system, possibly with another computer or device on which the fuzz target 202 is executed.

In the following it is assumed that the fuzz target 202 is stateful and for this test cases (with respective input 203) should be generated.

According to various exemplary embodiments, the test computer 205 implements a generative machine learning (ML) model 206, in particular the decoder of an ML model (e.g. the decoder of an autoencoder).

The test computer 205 trains the ML model so that it implicitly learns the (hidden) states of the target program (i.e. the fuzz target) 202. A generative ML model allows new inputs 203 to be generated with which specific program states can be achieved for which the ML model has learned representations.

3 shows an ML model 300 according to one embodiment.

In this example, the ML Model 300 has the classic architecture of an autoencoder. Other architectures can also be used, such as a UNet architecture or a sequence-to-sequence architecture.

First, the test computer 205 trains the ML model 300 using supervised learning. For this purpose, training data is generated by executing the target program 202 for various inputs. The training data has a variety of training data elements, each consisting of a training input for the ML model and an associated (ground truth) label (i.e. a target output). Training the ML model 300 can be done using standard methods for supervised learning, i.e. standard optimization methods for this purpose such as stochastic gradient descent.

After training, the test computer 205 can use the ML model 300 to generate new test inputs (and thus test cases), which the test computer 205 feeds to the target program 202 and for which it observes the reaction of the target program 202 in the form of feedback 204. Depending on the feedback 204, the test computer 205 determines whether the target program 202 is functioning as intended, i.e. without errors.

The training data is generated by the test computer 205 executing the target program 202 for various inputs and observing (code) coverage of the target program 202 for each input.

Each training data element of the training data is thus a pair of coverage and (program) input to the computer program (also referred to as a (program) input data set) with which this coverage is achieved. The coverage is the training input for the ML model and the program input is the target output of the ML model (in other words: the label for the training input).

Coverage can be expressed as the value of a coverage metric, such as that used in fuzzing. However, it can also explicitly state (e.g. in the form of a bitmap) which parts of the program (i.e. code parts, such as functions or methods) were executed when the program 202 was executed with the respective input data set as input for the program 202.

So the coverage for an input data set can be expressed in different forms. Depending on the case, it explicitly indicates which parts of the program were reached by the input data set (i.e. were executed when the program 202 was executed with the input data set as program input) or indicates which part of the program was reached by the input data set. Covers are typically used in greybox fuzzing.

A (program) input data set may contain a sequence of input data elements. For each test case, i.e. each input data set, the input data elements are then fed sequentially (according to the sequence) to the program 202. The coverage is determined and saved for each test case. This coverage is viewed as a representation of the state of the program 202 achieved with this input data set and is thus recorded.

The state that is achieved with an input data set, which is a sequence of input data elements, can in particular depend to a large extent on the first input data elements of the sequence. In other words: The first input data elements roughly define the state of the program and the further input data elements then lead to substates of this state or lead to the execution of certain parts of the program in this state.

After executing program 202 on an input data set, test computer 205 resets program 202.

As mentioned above, each training data item (i, c) consists of the coverage i observed for a (program) input data set, as a training ML input data item for the ML model, and the input data set c as a label (target output). The ML model has an encoder 302, labeled e, which is a ML input data element 301 maps to a representation labeled z. The representation for an ML input data item 301 can be viewed as a compressed version of the ML input data item 301. The representation is an element (e.g. vector) in a representation space 303 (which can be seen as a latent space).

A decoder 304, designated d, maps an element of the representation space 303 supplied to it onto a (program) input data set 305 for the program 202. During training, encoders 302 and decoders 304 (which are, for example, neural networks) are trained in such a way that the decoder 304 generates, if possible, the input data set 305 for each training data element, which belongs to the training data element, from whose coverage 301 the encoder generates the representation from which the Decoder 304 generates the input data set 305.

An ML model from the sequence-to-sequence family can be used, i.e. a model that learns a mapping between sequences of inputs and outputs.

Since the elements of the representation space 303 are intended to be used to generate (by the decoder 304) input data sets for the target program 202, according to various embodiments, new input data sets are generated in a stochastic manner. This can be done, for example, using a variational autoencoder (VAE). A VAE does not learn a fixed representation for an ML input (here a cover), but rather maps each ML input to a probability distribution. To generate an input data set 305 for the target program 202, a value and an element from the representation space is then drawn (sampled) from such a probability distribution. The probability distribution to which a coverage is mapped implicitly represents the state of the program in which that coverage is achieved (where this mapping is specifically observed for coverage and input data set from a training data item).

A VAE provides good generative capabilities on large parts of the representation space 303 (those parts reached by coverages from the training data). To generate a new (not previously used in the test of the program 202) input data set for the program for a new test case, the test computer 205 can (after training) simply sample from the representation space 303 (e.g. using random noise) and the sample using of the decoder to an input data set for the program.

The test computer 205 may also select a coverage that has not yet been achieved to generate a new input data set for the program (for testing the program). For example, in the case where the coverage is specified in a form that specifies which parts of code have been reached, the test computer 205 may select a coverage in which parts of code that have not yet been reached are indicated. The test computer 205 can then map the selected coverage onto a representation using the encoder 302 and map the representation onto an input data set using the decoder 304 with which (ideally) this coverage is achieved (e.g. new code parts, i.e. not yet achieved in the test, are achieved can be).

The ML model 300 can be designed and trained to learn representations of covers in a distentangled form. The training takes place in such a way that the ML model is stimulated to generate different dimensions of the representation space (i.e. different components of vectors from the representation space) for different properties of the input data sets it generates. This can be achieved, for example, by adding a corresponding regularization term to the loss function for training the autoencoder (e.g. variational autoencoder). Various well-known approaches can be used to disentangle a VAE.

Once the test computer 201 has trained the ML model 300 in this way (with a disentanglement of the representation space), it can generate new new input data sets for the program (for testing the program) by varying certain components of the representations (e.g. relying on areas, which have not yet occurred for the input data sets so far in training) and maps them to new input data sets using the encoder 305. In this way, the test computer 201 can specifically reach new states of the target program 202 during testing and test the behavior of the target program 202 for these states. This provides more precise control over the exploration of the state space and allows the test computer 201 to generate input data sets more often for states that are of higher interest than for others.

In summary, according to various embodiments, a method is provided as in 4 shown.

4 shows a flowchart illustrating a method for testing a computer program according to an embodiment.

In 401, the computer program is executed for each input data set for the computer program of a plurality of input data sets and a coverage of the computer program is determined in each case, which was achieved when executing the computer program for the respective input data set.

In 402, a training data element is created for each input data set, which contains the coverage determined for the input data set as the training input and the input data set as the target output.

In 403, a machine learning model is trained using supervised learning using the generated training data elements, the machine learning model comprising an encoder followed by a decoder, the encoder being configured to generate representations in a representation space from covers and the decoder being configured from it supplied representations from the representation space to generate input data sets for the computer program.

In 404, representations are selected from the representation space.

At 405, additional input data sets are generated by feeding the selected representations to the decoder.

At 406, the computer program is tested by executing the computer program on the additional input data sets generated.

Determining the coverage may be determining the value of a coverage metric and/or a data structure (in particular a bitmap) that indicates the coverage.

The procedure of 4 can be carried out by one or more computers with one or more data processing units. The term “data processing unit” can be understood as any type of entity that enables the processing of data or signals. For example, the data or signals may be treated according to at least one (ie, one or more than one) specific function performed by the data processing unit. A data processing unit may be an analog circuit, a digital circuit, a logic circuit, a microprocessor, a microcontroller, a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), a programmable gate array integrated circuit (FPGA), or any combination include it or be formed from it. Any other way to implement the respective functions described in more detail herein may also be understood as a data processing unit or logic circuitry. One or more of the method steps described in detail here can be carried out (e.g. implemented) by a data processing unit through one or more special functions that are carried out by the data processing unit.

The approach of 4 is used to test a program, for example control software for a robot device. The term “robotic device” can be understood as referring to any technical system, such as: B. a computer-controlled machine, a vehicle, a household appliance, a power tool, a manufacturing machine, a personal assistant or an access control system. The control software can also be used for data processing systems such as a navigation device. Errors found during testing can be corrected and, depending on the type of software, the reliability and/or safety of the respective robot device can be increased.

Although specific embodiments have been shown and described herein, it will be recognized by those skilled in the art that the specific embodiments shown and described may be substituted for a variety of alternative and/or equivalent implementations without departing from the scope of the present invention. This application is intended to cover any adaptations or variations of the specific embodiments discussed herein. Therefore, it is intended that this invention be limited only by the claims and the equivalents thereof.

Claims (9)

Method for testing a computer program, comprising: executing the computer program for each input data set for the computer program of a plurality of input data sets and in each case determining a coverage of the computer program that was achieved when executing the computer program for the respective input data set; Generating, for each input data set, a training data element containing the coverage determined for the input data set as training input and the input data set as target output; Training a machine learning model using supervised learning using the generated training data elements, the machine learning model comprising an encoder followed by a decoder, the encoder being set up is to generate representations in a representation space from covers and the decoder is set up to generate input data sets for the computer program from representations supplied to it from the representation space; selecting representations from the representation space; generating additional input data sets by feeding the selected representations to the decoder; and testing the computer program by executing the computer program on the additional input data sets generated. Procedure according to Claim 1 , where the representations are randomly selected from the representation space. Procedure according to Claim 1 or 2 , where representations other than those to which the encoder maps the coverages of the training data elements are selected from the representation space. Procedure according to one of the Claims 1 until 3 , where the machine learning model is a variational autoencoder. Procedure according to one of the Claims 1 until 4 , wherein each input data set for the computer program has a sequence of input data elements for the computer program and executing the computer program for the input data set comprises executing the computer program, wherein the input data elements are sequentially supplied to the computer program according to the sequence. Procedure according to one of the Claims 1 until 5 , wherein the representation space is multidimensional, wherein the machine learning model is trained to learn representations of covers in disentangled form, and wherein selecting the representations from the representation space is done by setting components of the representations in selected dimensions. Software testing system that is set up to implement a method according to one of the Claims 1 until 6 to carry out. Computer program containing instructions that, when executed by a processor, cause the processor to carry out a method according to one of the Claims 1 until 6 carries out. A computer-readable medium that stores instructions that, when executed by a processor, cause the processor to perform a method according to one of the Claims 1 until 6 carries out.

Images