WO2023144158A1

WO2023144158A1 - Test assembly and method for testing a test object, and computer program for carrying out the method

Info

Publication number: WO2023144158A1
Application number: PCT/EP2023/051720
Authority: WO
Inventors: Anne Borcherding; Steffen Pfrang; Christian Haas
Original assignee: Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V.
Priority date: 2022-01-27
Filing date: 2023-01-24
Publication date: 2023-08-03
Also published as: DE102022200928A1

Abstract

Embodiments of the invention relate to a test assembly (101) for testing a test object (104, 410), for example for testing a technical device or an industrial device or a component of a device or a component having a communication interface, for example a wired or wireless network interface and/or a web interface. The test assembly (101) is designed to provide a model (107) on the basis of an interaction with the test object (104) in order to generate one or more test cases (102, 420, 440) using the model (107) and in order to improve the model (107) of the test object (104, 410) on the basis of real test results (109) based on the test cases (102, 420, 440) that have been generated.

Description

A test arrangement and a method for testing a test object, and a computer program for carrying out the method

Description

technical field

The invention relates to a test arrangement for testing a test object. The invention can be used, for example, to examine test objects or technical devices for weak points. These include industrial automation components, Internet of Things devices (Internet of Things devices, loT devices, lloT devices) and medical devices. The invention relates to model-based security testing for technical devices.

Technical problem

Technical devices fulfill tasks. Attacks can prevent or prevent the technical devices from performing their tasks. Attacks, such as virtual attacks or attacks via the network/internet, exploit weaknesses in the technical device. In order to prevent the attacks, the vulnerabilities can be discovered and eliminated. Security tests are used to discover vulnerabilities. A test environment or a "test environment" (TE) tests the technical device automatically as a device under test or test object or "subject under test" (SUT).

Technical devices are rarely fully tested for vulnerabilities. The reason for this is, on the one hand, the complexity of the technical devices and, on the other hand, the fact that technical devices are made up of different components. These components can come from different suppliers and not all details can always be viewed.

If the technical devices have been tested at all, this is done on the basis of experience from previous tests. These tests are either done manually by humans, for example by so-called "penetration testers" or penetration testers or by means of test environments.

Manual tests mean a high level of personnel effort, especially if the tests are to be carried out regularly. For this reason, in practice, they are rarely performed continuously on all devices under test or subjects under test (SUT).

For example, the test cases for automated tests can be generated based on experiences from previous tests and grammars. However, apart from random input data, these are static and, in particular, do not take into account the internal functions of the SlIT. It was recognized that the failures of the SLIT are only presented as the result of the investigation, but are not included in the creation of new, specialized test cases.

It was found that the internal functioning of the SUT can be described using a model. This model includes all of the functionality of the SUT, not just the parts required to perform the SUT's tasks. It can therefore be used to construct test cases.

Previous approaches for model-based testing of technical devices require a complete model of the SUT. This model can be derived from the design documents, for example. For the reasons mentioned above, this approach is not readily possible with composite technical devices. Examples of these approaches are the work of Bringmann et al. [1] and the work of Bauer et al. [2],

A manual construction of the model is very complex and often not possible. Manual construction of a model also means a high level of personnel expenditure and a complete model can rarely be specified. This is particularly the case when individual parts of the system have been purchased from third parties.

Approaches to a test environment for automated testing of technical devices have been recorded in various patents. Some embodiments relate to the general generation of test cases based on a grammar and to the Monitor a square wave as the output of the component under test. It can thus be checked whether the device being examined has assumed an error state or whether a general state change has taken place.

US 8,132,053 B2 and US 8,006,136 B2 relate to testing a system with software and/or hardware components. A test environment or a test framework is presented in the exemplary embodiments, which generates test cases for a system to be tested using a grammar. The test framework checks the system with the test case and receives an answer to the test case from the system to be tested. The inputs and outputs are then compared to the expected results to determine if the system under test is functioning correctly.

US 8,433,542 B2 and US 9,026,394 B2 also relate to testing a system with software and/or hardware components. The test framework of the above patents generates test cases for a system under test based on a grammar. The test framework checks the system with the test case and receives an answer to the test case from the system to be tested. The inputs and outputs are then compared to the expected results. The data can then be interpreted in the grammar system and/or used as input to an error isolation system. In this way it can be checked whether the device being examined has assumed an error status or whether a general status change has been carried out.

The various components, such as test case generation and/or monitoring, are also summarized, for example, in the joint patent US Pat. No. 9,400,725 B2. US 9,400,725 B2 generally relates to automated testing of a system with software and/or hardware components. The test framework generates test cases for a system to be tested based on a grammar. The test framework checks the system with the test case and receives an answer to the test case from the system under test. The inputs and outputs are then compared to the expected results to determine if the system under test is functioning correctly. In particular, the system under test can be analyzed to determine whether it is capable of properly processing control instructions and input signals and/or producing expected output control signals and additional control/feedback information. The data can then be interpreted in the grammar system and/or used as input to an error isolation system to detect anomalies in the system under test. The object of the present invention is to create a concept for testing test objects that provides an improved compromise between test effort and test scope or test reliability.

This object is achieved by a test arrangement or test environment (TE) according to claim 1, by a method according to claim 43, or by a computer program according to claim 44.

Summary of the Invention

Embodiments of the invention provide a test arrangement for testing a test object, such as. B. for testing a technical device or an industrial device or a component of a device or a component with a communication interface, for example with a wired or wireless network interface and / or with a web interface.

The test arrangement is designed to create a model based on an interaction with the test object, to generate one or more test cases using the model, and to improve the model of the test object based on real test results based on the generated test cases.

The test device (TE), for example, based on its interaction with the test object (SUT) actively and automates a model of the SUT. The model is then used by TE, for example, to specifically generate new test cases. The generated test cases or tests are executed by the TE against the SUT, for example. During and after the test, the TE checks whether the SUT fulfills or can fulfill its task. For example, she monitors the SUT, checks its outputs and also uses the model of the SUT. The SUT model is continuously updated during testing. A special technical feature is, for example, that the TE automatically creates a model from the interaction with a non-transparent SUT and uses this automatically for the adaptive test case generation.

For example, the TE creates an initial model of the SUT using interactions with the SUT. For example, TE creates test cases based on the initial model. The TE For example, runs the test cases against the SUT to generate real test results. For example, the model is improved based on the test results. For example, new test cases are continuously created using the new model and the model of the SLIT is continuously updated, for example.

For example, the effect of an automated derivation of a model of the SLIT is that knowledge about the SUT is derived that was not previously explicitly available. With this additional knowledge, TE can, for example, generate new test cases more efficiently and in a more targeted manner. This means that the entire test of the SUT can be carried out more efficiently and in a more targeted manner. In addition, this knowledge can be used, for example, to decide whether the behavior of the SUT shows an anomaly. This can be an indication of a vulnerability in the SUT.

The most important advantage is, for example, that the automatically generated model can be used to test automation components more efficiently and in a more targeted manner. The test arrangement thus creates more and more test cases that test the test object for weak points as comprehensively as possible.

In exemplary embodiments, the test arrangement is designed to receive one, several or no response given to a generated test case from a test object whose behavior is unknown and/or opaque via one or more communication paths.

The test arrangement is designed to detect all possible responses from the test object. By receiving as much of the test object's responses as possible, the test object's model based on the test results becomes more accurate, even though the test object is unknown or opaque as to its behavior.

In exemplary embodiments, the test arrangement is designed to check the functionality of the test object or the functionality of one or more functions of the test object using one or more monitors and/or monitoring modules.

The test object can react to a test case or to a query with a change in behavior in addition to or as an alternative to an answer. The test arrangement can monitor this change in behavior by means of one or more monitors and/or monitoring modules, for example. In exemplary embodiments, the test arrangement is designed to evaluate the responses received and/or the results of the monitors and to generate test results on the basis of this evaluation.

All changes in behavior and/or responses of the test object given to a test case are summarized under a test result, for example. This can simplify subsequent processing of the test results.

In exemplary embodiments, the test arrangement is designed to evaluate the quality of the test case based on the test result and/or based on a probability of a malfunction of the test object tested with the test case.

For example, the test setup simulates virtual attacks against the technical device or the test object in order to discover weak points in the device. The more vulnerabilities a test case of the test arrangement discovers, or the higher the probability that a test case discovers a vulnerability, the better a test case is or the better a test case is rated. The quality of a test case can be measured, for example, based on the number of unexpected error messages and/or malfunctions. On the basis of the evaluation, for example (e.g. preferred) those test cases (e.g. for an actual application on the test object) can be selected which have a high probability of revealing weak points in the test object, which improves the test efficiency.

In exemplary embodiments, the test arrangement is designed to predict or estimate the test results of the test cases using the model, or e.g. B. to approximate.

The test arrangement can use the model to predict the test results of a test case and thus the quality of the test case without executing the test case against the SUT. This means that test time can be saved, for example, by selecting the test cases that are actually executed appropriately.

In exemplary embodiments, the test arrangement is designed to selectively select the test cases in which a malfunction of the test object or a malfunction of one or more functions of the test object has been predicted, and to test the test object with the selected test cases in order to generate test cases that are as produce many malfunctions of the test object as a real test result.

The test arrangement can use the model to predict the test results of a test case and thus the quality of the test case. As a result, the test arrangement should not include all test cases run against the SUT. With a reduction in the number of test cases executed against the SUT, resources such as B. Time saved.

In exemplary embodiments, the test arrangement is designed to determine whether a real test result is judged to be abnormal using the model.

The test setup can use the model to predict the test results of a test case. The test results of the executed test cases are compared with the expected results. If there is a discrepancy, the test result is considered abnormal. Anomalous test results indicate vulnerabilities. Thus, for example, test results judged to be abnormal can be reported or output selectively or preferentially (for example marked) to a user of the test arrangement.

In exemplary embodiments, the test arrangement is designed to assess the real test result as abnormal if the real test result indicates a malfunction of the test object or a malfunction of one or more functions of the test object, or if the real test result deviates from the test result predicted using the model.

The test setup can use the model to predict the test results of a test case. The test results of the executed test cases are compared with the expected results. If there is a discrepancy between a real test result and an expected test result, or if the test object malfunctions, the test result is judged to be abnormal. Anomalous test results indicate weaknesses in the model and/or the test object and are particularly valuable in error analysis, for example.

In embodiments, the model of the SLIT includes a state machine.

Handling a state machine is easy, since the number of states of a state machine is often relatively small or often lies within a framework that can be processed easily, the states are independent of one another, and the transition between the states follows a logical rule. For example, it is assumed that the test object is in a state until a state-changing request is executed.

In exemplary embodiments, a state of the state machine comprises at least one test case that leads to the state and at least one test result resulting therefrom. In exemplary embodiments, the test arrangement is designed, for example, to use a hash function to calculate a value for a state of the test object and to identify the state with the calculated value.

For example, the test arrangement is designed to add a new state to the state machine that forms the model of the test object if this state does not yet exist.

A hash function is an injective mapping that maps, for example, an input of different lengths to a target set, or hash value, of a fixed length. For example, the length of the hash value remains fixed regardless of the length of the input values and/or the length of the test results. A hash value is well suited for use as an identification number, identifying, for example, a state in a space-efficient yet typically unique form.

In exemplary embodiments, the test arrangement is designed to bring the test object into a specified state and to test the test object in the specified state for one or more different test cases.

For example, the test arrangement is designed to provoke and/or identify one or more status changes based on the specified status. In this way, the model of the SUT is supplemented by the newly identified status changes and/or by the newly identified statuses.

In exemplary embodiments, the test arrangement is designed to identify state-changing test cases that lead to a state change of the test object and to update the model based on this.

With a continuous update of the model, the test setup creates a positive feedback loop. An improved model increases the number of state-changing test cases. An increase in state-changing test cases leads to a better SUT model.

In embodiments, the test arrangement is designed to identify the state-changing test cases using a heuristic. The heuristic is designed to take into account a frequency of execution of a test case and/or a frequency of identifying a/the test case as a state-changing test case and/or a category or a type of/the test case. With the help of a heuristic, the test arrangement can efficiently identify new state-changing test cases. The statistics about and/or experiences with previous test cases can lead to the discovery or identification of new state-changing test cases.

In embodiments, the testing arrangement is designed to assess whether a condition is sufficiently explored. The test arrangement is designed to generate new test cases that give rise to this state when the state is not sufficiently explored, or to generate new test cases with a large difference from the already known test cases when the state is sufficiently explored. In this case, the difference is preferably large enough, for example, to provoke and/or identify one or more status changes.

This enables the test arrangement to systematically explore each state sufficiently. If a state is tested with enough test cases, the test arrangement will, for example, provoke a state change and create new test cases for the new state.

In exemplary embodiments, the test arrangement is designed to model a decision on whether to further investigate known states or to discover new states as a multi-armed bandit problem and to model the multi-armed bandit problem using a reinforcement learning method , such as B. the upper confidence limit method or the Upper Confidence Bound (UCB) method to solve to make the decision.

It has been recognized that a multi-armed bandit problem can well describe the dilemma between further investigating known states and searching for new states. An advantage of the multiple-armed bandit problem is that this problem can be solved mathematically using different methods.

In exemplary embodiments, the test arrangement is designed to repeatedly test the test object with a test case in order to determine whether the test result of the test case changes over time and to identify a state change based thereon.

The test arrangement can thus test whether the states are really independent of one another and/or whether the states are time-dependent. In exemplary embodiments, the test arrangement is designed to identify test cases with which the test object was tested between the test case and the repeated test case as potentially state-changing test cases.

For example, the test arrangement assumes that the states are dependent on each other and designates all executed test cases between the test case and the repeated test case as potentially state-changing test cases. The potentially state-changing test cases can, for example, be further examined by the test arrangement.

In embodiments, the test arrangement is designed to generate invalid and/or unexpected and/or random inputs when generating the test cases.

The test arrangement is thus designed, for example, to discover weak points in the test object using generated test cases. For example, the test arrangement is successful if it finds test cases and/or inputs that the constructor or designer of the test object did not think of, such as: B. invalid and/or unexpected and/or random inputs.

In exemplary embodiments, the test arrangement is designed to use a mutation algorithm when generating the test cases, which algorithm generates new test cases on the basis of old test cases. Thus, for example, test cases can be developed successively or step by step, for example by only changing part of a test case in the event of a mutation.

In exemplary embodiments, the test arrangement is designed to use one or more machine learning algorithms when there is a mutation of the test cases.

In exemplary embodiments, the test arrangement is designed to generate a plurality of intermediate test cases, e.g. B. by means of a mutation. The test arrangement is designed to assign a fitness value to each of the intermediate test cases, to select the intermediate test cases depending on the respective assigned fitness value, and to continue using the selected intermediate test cases as new test cases.

For example, e.g. B. the intermediate test cases with the highest fitness values or z.

B. the intermediate test cases with a fitness value above a predefined threshold are selected and reused as new test cases. Thus, for example, test cases can be successively changed (e.g. only partially or in individual values or subsets of values) or further developed, and on the other hand test cases can be selected for the actual test execution that are particularly capable of a malfunction due to their assigned fitness value of the test object.

In exemplary embodiments, the test arrangement is designed to assign a fitness value to each of the intermediate test cases based on a probability of a malfunction of the test object tested with the intermediate test case.

The test arrangement is designed to discover weak points in the test object using selected intermediate test cases. The intermediate test cases are evaluated based on their (e.g. expected) ability to cause the test object to malfunction (which is described, for example, by the fitness value). In this way, as many vulnerabilities as possible are discovered.

In embodiments, the test arrangement is configured to use one or more machine learning algorithms in determining the fitness value.

In embodiments, the test arrangement is configured to use one or more machine learning algorithms, such as support vector machine algorithms, or support vector machine algorithms and/or random tree algorithms, or random tree algorithms and/or neural network algorithms.

The use of well-known algorithms simplifies the use of the machine learning concept. Known algorithms are easy to implement.

In exemplary embodiments, the test arrangement is designed to retrain the machine learning algorithm or machine learning algorithms using the test results of the test cases that have been executed.

A continuously learning algorithm improves itself over time and always stays up to date. Improvements in the model can thus be exploited, for example to generate new test cases and/or to evaluate test results. In exemplary embodiments, the test arrangement is designed to execute a sequence of test cases and to use their test results as an observation sequence for training a hidden Markov model (Hidden Markov Model), and to determine states of a state machine with the Hidden Markov Model (HMM) based on an observation sequence to estimate or approximate.

It was recognized that the HMM is well suited, for example, for modeling states and transitions with different probabilities between the states, for example in the case of a test object with unknown behavior.

In exemplary embodiments, the test arrangement is designed to bring the test object into the individual states of the Hidden Markov model and to generate several test cases per state, which bring the test object into the other states of the Hidden Markov model, starting from the current state, in order to obtain a good achieve path coverage of the Hidden Markov Model.

Good path coverage helps the test setup to understand the internal workings of the test object and thereby generate better test cases. Errors and thus potential points of attack can occur with every state transition, which can be identified in this way.

In exemplary embodiments, the test arrangement is designed to calculate the most probable execution path of a test case in the Hidden Markov Model using a Viterbi algorithm, and to calculate the path coverage of the test cases based thereon.

The Viterbi algorithm can be used in an HMM to calculate the path that was most likely taken through the model when a test case was executed. This path approximates the execution path of the SLIT and is used to quantify the behavior of the SLIT, which in turn is used to generate new test cases. The path coverage can also be used, for example, as a criterion or as a measure of the quality of the test or test case or for the test coverage.

In exemplary embodiments, the test arrangement is designed to continue to refine the model and/or the generated test cases. The test arrangement thus creates, for example, a positive feedback loop in which the new test cases improve the model and new test cases are in turn generated on the basis of the improved model.

The test arrangement can test different test objects, such as web applications, network protocols or network interfaces. These concrete examples are explained below.

In embodiments, the test arrangement is designed to generate one or more test cases for a web application and to receive and/or evaluate the response of the web application.

The test arrangement can be designed to generate the test case or test cases in the form of entries in one or more input fields of the web application and/or activation of one or more buttons of the web application and/or activation of one or more links of the web application.

The test arrangement can be designed to receive and/or evaluate the response of the web application. The web application's response may be in the form of activating or deactivating one or more input fields of the web application and/or activating or deactivating one or more buttons of the web application and/or activating or deactivating one or more links of the web application and/or adding or removing from one or more input fields and/or one or more buttons and/or one or more links, and/or changing one or more texts or numbers.

In other embodiments, the test arrangement is configured to generate one or more test cases for a device under test using a network protocol and to receive and/or evaluate the response of the device under test.

The test arrangement may be arranged to generate the test case or test cases in the form of one or more inputs in a sequence of one or more network packets.

The test arrangement can be designed to receive and/or evaluate the response of the test object in the form of a sequence of one or more network packets and/or in the form of information regarding a code coverage of the input. Code coverage is, for example, the ratio of statements actually made in a test to statements that are theoretically possible, or the proportion of software blocks that have been run through (and thus tested).

The test arrangement can be designed to set up a state machine based on the test cases and code coverage based thereon, to calculate a value for a state of the test object using a hash function, and to identify the state with the calculated value.

The test arrangement can be designed to use a sequence of communication between the test arrangement and the test object as an observation sequence for training a hidden Markov model (Hidden Markov model), and based on the observation sequence the code coverage with the hidden Estimate or approximate Markov model.

The test arrangement can be designed to bring the test object into the individual states of the hidden Markov model as the initial state and to generate several test cases per state, which bring the test object into the other states of the hidden Markov model, starting from the current state, in order to achieve good path coverage of the Hidden Markov Model.

The test arrangement can be designed to calculate the most probable execution path of a test case in the Hidden Markov Model using a Viterbi algorithm and to calculate the path coverage of the test cases based thereon.

In further embodiments, the test arrangement is designed to generate test cases in the form of inputs of a network packet or a sequence of network packets for a network interface of a test object, and to evaluate the operability of the test object or the operability of one or more functions of the test object using monitors.

Further exemplary embodiments according to the present invention create corresponding methods or computer programs.

Character brief description Exemplary embodiments according to the present invention are explained in more detail below with reference to the enclosed figures.

1 shows a schematic representation of a sequence of an exemplary embodiment of a test arrangement;

2 shows a flow chart of an embodiment of a positive feedback loop;

3 shows a flow diagram of an embodiment of a network protocol with access to information regarding code coverage;

4 shows a schematic representation of an embodiment of an evolutionary algorithm.

Detailed description of the exemplary embodiments according to the figures

Exemplary embodiments are described in more detail below with reference to the figures. In addition, method steps that relate to a specific feature of a device are interchangeable with that same feature of the device, and vice versa.

FIG. 1 shows a schematic representation of a sequence 100 of an exemplary embodiment of a test arrangement (TE) 101. The TE 101 is linked to a test object (SUT) 104 via one or more communication paths 103. The TE 101 has one or more monitors 106 or monitoring modules 106 and is additionally linked to a model 107 of the device under test 104 .

The model 107 of the SLIT 104 is or was created, for example, by the TE 101 on the basis of an interaction 108 or an initial interaction 108 with the test object 104 .

The TE 101 is designed to create one or more test cases 102 based on the model 104 and to receive one, multiple or no response 105 given to the test case 102 created. The TE 101 is also designed to check the functionality of one or more functions of the SLIT using one or more monitors 106 . The responses 105 received and/or the results of the monitors 106 are evaluated in order to create test results 109 . Based on the test results 109, the TE 101 improves the model 107 of the SLIT 104. The newly generated test cases 102 are thereby improved and the results 109 given thereon are in turn the model 107 refined.

In other words, FIG. 1 shows the schematic sequence of an exemplary embodiment. The test arrangement (TE) 101 generates one or more new test cases 102 and sends them to the test object (SUT) 104 via a communication path 103. The SUT 104 then reacts with one or more or no response 105 via one or more communication paths 103. The TE 101 evaluates the responses received 105 and the results of the monitors 106. The model 107 of the SLIT 104 is derived from the test cases 102 sent, the responses 105 received and the results of the monitors 106 and, for example, is constantly updated. Based on the (possibly improved) model 107 and the evaluation of the received responses 105 and/or the results of the monitors 106, the TE 101 generates one or more new test cases 102 and sends these in turn to the SUT 104.

The test arrangement 101 uses the test cases 102 to simulate virtual attacks in order to discover weaknesses in the test object 104 and also logs the test cases 102 that led to an anomaly in the behavior of the SUT 104 . The decision as to whether an anomaly is present is made on the basis of model 107, for example. By using the model 107, the security tests (security tests) can be carried out in full.

In addition, the model 107 is created and updated automatically without the need for manual intervention or having to analyze the internal functionality of the SUT 104 . As a result, security tests (security tests) can be carried out with less effort. This enables higher test coverage to discover or eliminate multiple vulnerabilities and thus increase resilience to virtual attacks.

The test arrangement creates a positive feedback loop in which the new test cases 102 improve the model 107 and use the improved model 107 to in turn generate new improved test cases 102 . This feedback loop will be explained further in FIG. FIG. 2 shows a flow chart of an exemplary embodiment of a positive feedback loop 200, which can be used, for example, optionally in test arrangements according to exemplary embodiments of the present invention. For example, the feedback loop 200 has five steps.

The first step 210 involves executing a query or test case, such as B. the test case 102 in Figure 1 on. The first step 210 is linked to the second step 220 and the fifth step 250, for example.

The second step 220 includes a summary of the responses, such as B. the answer 105 in Figure 1, or the test results. The second step 220 is linked to the first step 210 and to the third step 230, for example.

The third step 230 includes identification of state changing requests or test cases. The third step 230 is linked to the second step 220 and to the fourth step 240, for example.

The fourth step 240 comprises a derivation or an improvement of the state machine. The fourth step 240 is linked to the third step 230 and to the fifth step 250, for example.

The fifth step 250 involves selection of the next query or test case, such as e.g. B. the test case 102 in Figure 1 on. The fifth step 250 is linked to the fourth step 240 and to the first step 210 .

The TE in the first step 210 executes a test case or query. The execution 210 of the test case provokes a test result, which is or was generated or summarized in the second step 220 from one, several or no answer and/or a change in behavior. The state-changing test cases are identified in the third step 230 based on the provoked test results. The model of the SLIT or the state machine is updated or derived in the fourth step 240 using the identified state-changing test cases. In the fifth step 250, new test cases are generated using the test cases that have already been executed and using the updated model. The newly created test cases are again executed in the first step 210 and thus the feedback loop 200 starts over. Until a meaningful set of queries or test cases has been reached, the TE cannot yet make a well-founded statement about the behavior of the test object and the three steps on the right-hand side of the illustration have no effect. This case is represented by the dashed arrow.

Concrete implementations

The concrete implementation of the solution mentioned above can be done in different ways. Various exemplary implementations are presented below. This is not an exhaustive list and, in particular, combinations of implementations are possible. The diversity of the examples illustrates that the invention can be implemented in many ways.

One implementation of the invention is so-called “fuzzing” (robustness testing) of a web application based on a state machine, or includes fuzzing. In this case, the SUT corresponds to a web application that is made available on a technical device or test object, the model is formed by a state machine and the test strategy is based on "fuzzing". "Fuzzing" in this case means the use of many different values and the reaction of the SUT to these possibly unexpected values. In the specific case of a web application, this can be, for example, the entry of special characters or program code, such as e.g. B. Java Script code in a text field. In the case of this implementation, the SUT is regarded as a so-called "black box" (i.e., for example, as a closed system neglecting the internal structure), so no information about the internals of the SUT is known. In this implementation, a test case consists of an HTTP request, for example.

The specific procedure is illustrated with reference to FIG. The cycle begins in the illustration at the top left. First, for example, the initial request is sent to the web application, so the home page of the web application is called, for example. Until a meaningful number of requests have been made, no well-founded statement can be made about the behavior of the web application. For this reason, the three steps on the right-hand side of the illustration initially have no effect. This case is represented by the dashed arrow. Based on the queried home page, the next queries are now derived. For example, newly found hyperlinks are selected or buttons are pressed. The respective answers are summarized, for example, in so-called answer clusters. For example, similar answers are grouped together in a cluster or group. This is the case, for example, for websites that only differ in terms of the current time, but are otherwise identical in terms of content.

During the investigation, for example, the same inquiries are specifically asked more frequently. It is also possible to determine whether the web application's response to the same request changes over time. If, for example, a response to a request differs from the previous response to the same request, it is assumed, for example, that an internal state transition of the web application took place between these two requests.

This becomes clear, for example, in an online shop. If I have not yet placed an item in my shopping cart and still click on "My shopping cart", I reach a website that shows my empty shopping cart. However, if I put an item in my shopping cart and thus change the internal state of the web application, I get a different response to the same request for my shopping cart. For example, it could now be possible to press the button or button "Checkout", which was previously not possible with an empty shopping cart.

All queries that lie between the two queries under consideration, e.g. B. between the two calls from the "My shopping cart" page, can be a trigger for the status change or identified as a trigger for a status change. These status-changing requests are identified, for example, on the basis of a heuristic. These heuristics include, for example, how often the request has already been identified as state-changing in the past, how often the request has already been executed, and the type of response or message, such as e.g. GET, POST.

Based on the information obtained in this way, the current state machine is created or improved. It is assumed that the web application is in a state until a state-changing request is executed.

Now the cycle continues. More and more possible queries are selected and the state machine is constantly being refined. Are they through the website? accessible hyperlinks, input fields and buttons are exhausted, the robustness test algorithm or fuzzing algorithm takes over the selection of the next request. For this purpose, for example, available URL parameters and input values are now tried out a large selection of values. For example, the reaction of the web application is compared with the state machine. For example, if the web application behaves differently than the model, this is evaluated as an anomaly that can indicate a vulnerability.

Another implementation of the invention is also based on a state machine, but aims to implement a network protocol. In this implementation, a test case consists of a sequence of one or more network packets. In addition, this implementation assumes access to information regarding the code coverage of a particular input. In concrete terms, this means that after entering a specific sequence of network packets, the TE or the algorithm receives information about which code blocks the SUT ran through in response to this sequence of network packets. This information is used to evaluate the quality of a test case. For example, a test case that triggers many new places in the code is rated with a high quality. The course of this example is illustrated in FIG.

FIG. 3 shows a flow diagram 300 of an implementation of a network protocol with access to information regarding code coverage. The flow chart 300 has a loop with, for example, four steps.

The first step 310 includes, for example, providing or executing a sequence of network packets or a test case, such as e.g. B. the test case 102 in Figure 1 on. The first step 310 is linked to the second step 320 and the fourth step 340 .

The second step 320 includes, for example, a calculation of the behavior of the SUT or a code coverage analysis. The second step 320 is linked to the first step 310 and to the third step 330, for example.

The third step 330 includes, for example, a check and, if necessary, an update of the state machine. The third step 330 is linked to the second step 320 and to the fourth step 340, for example. The fourth step 340 includes, for example, selection of the next sequence of network packets. The fourth step 340 is linked to the third step 330 and to the first step 310, for example.

In the first step 310, the TE executes a test case. In this implementation, the test case consists of a sequence of one or more network packets. After inputting a certain sequence of network packets (e.g. into the SUT), the TE gets information (e.g. from the SUT) about which code blocks the SUT passed in response to the sequence of network packets. Based on this information or this code coverage, the behavior of the SUT is analyzed or calculated in the second step 320, for example. In the third step 330, for example, this analysis leads to a check and, if necessary, updating of the state machine of the SUT model. Based on the updated model, the next sequence of network packets is generated or selected in the fourth step 340, for example, which in turn is executed in the first step.

The concrete procedure of the example is shown in FIG. The cycle also begins here at the top left. In this case, the initial sequence of network packets is specified by a start value or a so-called "seed". This can be, for example, a single valid network packet, a sequence of network packets, or an empty set. So, for example, this beginning is sent to the SUT and the code coverage information is preserved. Based on this code coverage, a value for the behavior of the SUT is calculated using a hash function, for example. A state machine, for example, is now set up on the basis of this behavior. For example, it is assumed that the system is in a certain state until behavior that contradicts this is observed. This is based on an L* algorithm, for example.

The selection of the next sequence is based on a so-called seed selection algorithm. This selection also includes, for example, a mutation algorithm that generates and/or executes new test cases based on the old test cases. In addition, the selection of the next test case includes the state machine, for example.

For example, if a state has not yet been adequately researched, new sequences are generated that evoke precisely this state in order to learn how the SUT behaves in this state. If all currently known states have been sufficiently examined, new test cases with greater differences to already known test cases are generated, for example. These should help to achieve new states.

Alternatively, the decision between exploitation and exploration, i. H. whether known states should be further investigated or whether new states should be found, are modeled as a multi-armed bandit problem. For example, established reinforcement learning methods such as the Upper Confidence Bound (UCB) algorithm can be used to make this decision.

During testing, the SUT is constantly monitored. If, for example, a program crash is detected during the tests, this information is saved together with the previously sent test case.

Figure 3 may also illustrate another implementation of the invention in which code coverage is illustrated based on a Hidden Markov Model (HMM).

For example, the latter implementation assumed that access to code coverage information was possible. In this way, the behavior of the SUT can be quantified and used to build a state machine. An alternative to this is the implementation described below. For example, the objective, the SUT, the algorithm for creating the state machine, and the mutation and selection of the next sequence of network packets remain the same. The calculation of the behavior of the SUT is different. It is now assumed that the code coverage information is no longer directly available. This information is therefore also approximated using a model. For example, a hidden Markov model (Hidden Markov model (HMM)) is used as the model type for this purpose.

At the beginning of the test process, for example, the first test cases are carried out without using a state machine. For this purpose, for example, the starting value described above or the seed described above and, for example, the available mutations are used. The resulting network traffic is used, for example, to train the HMM. For example, a sequence of network work packets of communication between the tester and the SUT are considered as an observation sequence for the HMM. A sequence of network packets designates or describes, for example, the communication between a TCP connection establishment and a TCP connection termination. Each network packet is viewed as an individual observation, for example, which in some cases necessitates the use of a multivariate HMM.

A good choice for the number of internally used states can be made, for example, by repeatedly training the HMMs with a different number of nodes and a subsequent quality assessment, e.g. B. can be made via the Bayesian information criterion. The trained HMM now represents, for example, a model of the internal processes of the SUT and can therefore be used, for example, as follows to approximate the code coverage. For example, in an HMM, the most likely path taken by the model in the last execution can be computed (or estimated) by the Viterbi algorithm. For example, this path approximates the execution path of the SUT. This allows this information to be used, for example, as the code coverage information was used in the previous implementation. For example, it is again used to quantify the behavior of the SUT, which in turn is used to create or update the state machine.

The selection of the next sequence of network packets is also based on the so-called seed selection algorithm. This selection also includes, for example, a mutation algorithm that generates and/or executes new test cases based on the old test cases. This algorithm is explained below.

Evolutionary test case determination according to Figure 4

FIG. 4 shows a schematic representation of an embodiment of an evolutionary algorithm 400, which can be used, for example, alone or in connection with the embodiments described herein.

A population of test cases 420 of algorithm 400 is associated with SUT 410, for example. The algorithm includes, for example, a machine learning algorithm 460 and a loop from the population of test cases 420, a mutation 430, offspring test cases or "offsprings" 440 and a selection 450. In other words, a population of the test cases can be generated in the loop, for example, and a mutation can then take place, so offspring test cases can be obtained and a selection or selection can then take place.

For example, the machine learning algorithm 460 is associated with the population of test cases 420, mutation 430, and selection 450.

The population of test cases 420 are run against the SUT 410, for example, to collect test results, for example using monitors. The machine learning algorithm 460, for example, is trained on the basis of the test cases 420 and the test results.

For example, by mutation 430, the machine learning algorithm 460 will use the population of test cases 420 to create new test cases, called “offsprings” 440. For example, each test case is scored by the machine learning algorithm 460 using a fitness score. For example, based on the fitness value, a new population of test cases is selected 450, which in turn is run against the SUT 410 to collect test results.

In other words, the evolutionary algorithm is used, for example, to create new test cases. For this purpose, a starting value (seed) is again assumed, which represents the first population of the evaluation. New individuals are then created from the individuals in this population by mutation. These are called "offspring" or offspring. A fitness value is then assigned to each of these individuals in the “offspring” (e.g. to the offspring) using a fitness function. Subsequently, for example, those individuals with the highest fitness values are included in the new generation of the population. In this method, for example, each individual corresponds to a test case for the test method. The new generation of test cases is executed against the SUT, for example, and the reaction is observed, for example, by monitors. This implementation of the invention consists, for example, of a combination of an evolutionary algorithm and a machine learning algorithm (machine learning (ML) algorithm). For example, a support vector machine (SVM), a random tree (RT) or a neural network (NN) can be used as the ML algorithm.

In this implementation z. B. the network interface of an automation component can be tested. For example, the test cases consist of a network packet (or several network packets). In principle, it is also possible, for example, to extend the approach in such a way that a test case consists of a sequence of network packets. The model of the automation component is represented by the respective ML algorithm, for example, and models the behavior of the automation component in response to a test case, for example. The behavior is described, for example, by the failed monitors. In the case of an automation component, these monitors consist, for example, of checking whether the automation component is still responding to ping requests and/or whether the web server of the automation component is still reachable.

The selected ML algorithm can be used, for example, in the above-mentioned evolutionary algorithm for determining the fitness value. For example, the ML algorithm is trained in such a way that it approximates a function that maps a test case to the outputs of the monitors. For example, it estimates which monitors are likely to fail in a given test case.

A possible result of the estimation could be that the automation component still reacts to ping after the test case has been executed, but the web server is no longer accessible. This estimate can be used, for example, to determine the quality, i.e. the fitness, of a test case or an "individual". The more monitors fail due to the test case (or the more failures are determined to be probable), the higher the quality. In this way, a prediction can be made as to whether (or with what expected probability) the test case will lead to a failure of the monitors. This makes sense, for example, in the case of automation components, since the execution of a test case takes a relatively long time. The estimation of the ML algorithm for the probable reaction of the automation component can, for example, be calculated much more quickly. For example, the ML algorithm is first trained on the reaction of the SUT to the test cases, which are reflected in the initial values or in the "Seed" are located. It is then, for example, continuously retrained on the test cases that have actually been executed. Here, for example, the necessary evaluations or markings or “labels” for the test cases are also available through the subsequently measured behavior of the monitors.

In addition, the ML algorithm also affects the mutation of the test cases, for example. What this effect actually looks like depends, for example, on the selected ML algorithm. With an NN, for example, the gradient can be calculated that points in the direction of a (local) maximum of the approximated function. This local maximum then corresponds, for example, to a test case that will affect as many monitors as possible. By adapting the test cases according to this gradient, new test cases can be generated, for example, which have a high probability of causing many monitors to fail. It is similar with the SVM, here too, for example, vectors can be calculated that point to the test cases that cause a large number of monitors to fail.

Gradients or vectors cannot be calculated with an RT, but it can still be used for mutation. By tracing the path that a test case has taken through the tree, conclusions can be drawn about necessary changes in a test case, for example. For example, it can be checked which decisions in the RT led to the estimation that the test case will not cause many monitors to fail. These decisions are based, for example, on threshold values of certain properties of the test case. Now, for example, new test cases can be created that come above or below this threshold. These test cases then have a higher probability of causing a large number of monitors to fail, for example.

As shown in FIG. 4, the entire process begins with a starting value or “seed”, which is used, for example, as an initial population. The test cases of this population are executed against the SUT and the reactions of the monitors, for example, are recorded. This is how the algorithm receives the ratings or markings or “labels” for the test cases in the population. The combination of test cases and associated labels are now used, for example, to carry out initial training of the ML algorithm. This is then used, for example, to carry out the mutation and the selection. For example, a new population is created that in turn executed against the SUT. For example, the reactions of the monitors are recorded again and the ML algorithm is retrained, for example, with the data set newly generated in this way. A robustness test process or fuzzing process is established in which the test case generation is supported by mutation, for example, and the selection of the test cases generated is supported by an ML algorithm. For example, this ML algorithm represents a model of the SUT by approximating a function that describes the behavior of the SUT in response to a test case.

Alternate Embodiments

The embodiments described above are merely illustrative of the principles of the present test arrangement. It is understood that modifications and variations of the arrangements and details described herein will occur to those skilled in the art. Therefore, it is intended that the test assembly be limited only by the scope of the following claims and not by the specific details presented in the description and explanation of the exemplary embodiments herein.

Although some aspects have been described in the context of a device, it is understood that these aspects also represent a description of the corresponding method, so that a block or a component of a device is also to be understood as a corresponding method step or as a feature of a method step. Similarly, aspects described in connection with or as a method step also constitute a description of a corresponding block or detail or feature of a corresponding device. Some or all of the method steps may be performed by hardware apparatus (or using a hardware Apparatus) are executed. In some embodiments, some or more of the essential process steps can be performed by such an apparatus.

Depending on particular implementation requirements, embodiments of the invention may be implemented in hardware or in software. Implementation can be performed using a digital storage medium such as a floppy disk, DVD, Blu-ray Disc, CD, ROM, PROM, EPROM, EEPROM or FLASH memory, hard disk or other magnetic or optical memory can be carried out on which electronically readable control signals are stored, which can interact with a programmable computer system in such a way or interact that the respective method is carried out. Therefore, the digital storage medium can be computer-readable.

Thus, some embodiments according to the invention comprise a data carrier having electronically readable control signals capable of interacting with a programmable computer system in such a way that one of the methods described herein is carried out.

In general, embodiments of the present invention can be implemented as a computer program product with a program code, wherein the program code is operative to perform one of the methods when the computer program product runs on a computer.

The program code can also be stored on a machine-readable carrier, for example.

Other exemplary embodiments include the computer program for performing one of the methods described herein, the computer program being stored on a machine-readable carrier.

In other words, an exemplary embodiment of the method according to the invention is therefore a computer program that has a program code for performing one of the methods described herein when the computer program runs on a computer.

A further exemplary embodiment of the method according to the invention is therefore a data carrier (or a digital storage medium or a computer-readable medium) on which the computer program for carrying out one of the methods described herein is recorded.

A further exemplary embodiment of the method according to the invention is therefore a data stream or a sequence of signals which represents the computer program for carrying out one of the methods described herein. The data stream or the sequence of signals can be configured in this way, for example be to be transferred over a data communication link, for example over the Internet.

Another embodiment includes a processing device, such as a computer or programmable logic device, configured or adapted to perform any of the methods described herein.

Another embodiment includes a computer on which the computer program for performing one of the methods described herein is installed.

A further exemplary embodiment according to the invention comprises a device or a system which is designed to transmit a computer program for carrying out at least one of the methods described herein to a recipient. The transmission can take place electronically or optically, for example. For example, the recipient may be a computer, mobile device, storage device, or similar device. The device or the system can, for example, comprise a file server for transmission of the computer program to the recipient.

In some embodiments, a programmable logic device (e.g., a field programmable gate array, an FPGA) may be used to perform some or all of the functionality of the methods described herein. In some embodiments, a field programmable gate array may cooperate with a microprocessor to perform any of the methods described herein. In general, in some embodiments, the methods are performed on the part of any hardware device. This can be hardware that can be used universally, such as a computer processor (CPU), or hardware that is specific to the method, such as an ASIC.

The embodiments described above are merely illustrative of the principles of the present invention. It is understood that modifications and variations to the arrangements and details described herein will occur to those skilled in the art. Therefore, it is intended that the invention be limited only by the scope of the following claims and not by the specific details presented in the description and explanation of the embodiments herein. literature

[1] Bringmann, Eckard, and Andreas Kramer. "Model-based testing of automotive systems." 2008 1st international conference on software testing, verification, and validation. IEEE, 2008.

[2] Bauer, Thomas, et al. "Combining combinatorial and model-based test approaches for highly configurable safety-critical systems." Model-based Testing in Practice (2009): 9.

Claims

Claims A test arrangement (101) for testing (210, 310) a test object (104, 410), wherein the test arrangement (101) is designed to create a model (107 ) to create (240, 330) to generate (250, 340, 430) one or more test cases (102, 420, 440) using the model (107) to generate the model (107) of the test object (104, 410 ) based on real test results (109) based on the generated test cases (102, 420, 440). The test arrangement (101) according to claim 1, wherein the test arrangement (101) is designed to receive one, several or no response given to a generated test case (102, 420, 440) from a test object which is unknown and/or opaque with regard to its behavior ( 104, 410) via one or more communication paths (103). The test arrangement (101) according to claim 1 or 2, wherein the test arrangement (101) is designed to check the functionality of the test object (101) or the functionality of one or more functions of the test object (101) by means of one or more monitors (106) and/or Check monitoring modules (106). The test arrangement (101) according to claim 2 or 3, wherein the test arrangement (101) is designed to evaluate the received responses (105) and/or the results of the monitors (106) and to generate (220 , 320). The test arrangement (101) according to any one of claims 1 to 4, wherein the test arrangement (101) is designed to the quality of the test case (102, 420, 440) based on the test result (109) and / or based on a probability of a malfunction with to evaluate the test object (104, 410) tested in the test case (102, 420, 440). 6. The test arrangement (101) according to any one of claims 1 to 5, wherein the test arrangement (101) is designed to predict (320) the test results (109) of the test cases (102, 420, 440) using the model (107) or appreciate.

7. The test arrangement (101) according to claim 6, wherein the test arrangement (101) is designed to test cases (102, 420, 440) in which a malfunction of the test object (104, 410) or a malfunction of one or more functions of the Test object (104, 410) was predicted (320) to select and to test the test object (104, 410) with the selected test cases (102, 420, 440) to generate test cases (102, 420, 440) that generate as many malfunctions of the test object (104, 410) as a real test result (109).

The test arrangement (101) according to any one of claims 1 to 7, wherein the test arrangement (101) is adapted to determine using the model (107) whether a real test result (109) judged as abnormal (230, 320) becomes.

The test arrangement (101) according to claim 8, wherein the test arrangement (101) is designed to judge the real test result (102, 420, 440) as abnormal (230, 320) when the real test result (109) indicates a malfunction of the test object (104, 410) or indicates a malfunction of one or more functions of the test object (104, 410), or if the real test result (109) differs (230) from the test result (109) predicted using the model (107).

10. The test arrangement (101) according to any one of claims 1 to 9, wherein the model (107) comprises a state machine (107).

11. The test arrangement (101) according to claim 10, in which a state of the state machine (107) comprises at least one test case (102, 420, 440) and at least one test result (109) resulting therefrom. The test arrangement (101) according to claim 10 or 11, wherein the test arrangement (101) is designed to use a hash function to calculate a value for a state of the test object (104, 410) and to the state with the calculated value identify. The test arrangement (101) according to any one of claims 1 to 12, wherein the test arrangement (101) is designed to bring the test object (104, 410) into a predetermined state and to bring the test object (104, 410) in the predetermined state to test on one or more different test cases (102, 420, 440). The test arrangement (101) according to any one of claims 1 to 13, wherein the test arrangement (101) is designed to identify (230 ) and to update (240, 330) the model (107) based thereon. The test arrangement (101) according to any one of claims 1 to 14, wherein the test arrangement (101) is arranged to identify the state-changing test cases (102, 420, 440) using a heuristic (230), the heuristic being arranged to a frequency of execution of a test case (102, 420, 440) and/or a frequency of identification (230) of a/the test case (102, 420, 440) as a state-changing test case (102, 420, 440) and/or a category or a type of test case (102, 420, 440) to consider. The test arrangement (101) according to any one of claims 1 to 15, wherein the test arrangement (101) is designed to assess whether a condition is sufficiently explored and to create new test cases (102, 420, 440) that cause this condition. to generate when the state is not sufficiently explored, or to generate new test cases (102, 420, 440) with a big difference to the already known test cases (102, 420, 440) when the state is sufficiently explored. The test arrangement (101) according to any one of claims 1 to 16, wherein the test arrangement (101) is arranged to make a decision as to whether known conditions should be further examined or whether new conditions should be discovered as a multi-armed bandit problem to model, and to solve the multiple-armed bandit problem using a reinforcement learning method to make the decision. The test arrangement (101) according to any one of claims 1 to 17, wherein the test arrangement (101) is designed to repeatedly test the test object (104, 410) with a test case (102, 420, 440) to determine whether the test result (109) of the test case (102, 420, 440) changes over time and to identify a state change based thereon. The test arrangement (101) according to claim 18, wherein the test arrangement (101) is designed to test cases (102, 420, 440) with which the test object (104, 410) between the test case (102, 420, 440) and the repeated test case (102, 420, 440) was tested to identify (230) as potentially state-changing test cases (102, 420, 440). 20. The test arrangement (101) according to any one of claims 1 to 19, wherein the test arrangement (101) is designed to generate invalid and/or unexpected and/or random inputs when generating the test cases (102, 420, 440) ( 250, 340).

21. The test arrangement (101) according to any one of claims 1 to 20, wherein the test arrangement (101) is designed to use a mutation algorithm (430) when generating (250, 340) the test cases (102, 420, 440), which creates new test cases (102, 420, 440) based on old test cases (102, 420, 440).

22. The test arrangement (101) according to claim 21, wherein the test arrangement (101) is designed to use one or more machine learning algorithms (460) in the event of a mutation (430) of the test cases (102, 420, 440).

23. The test arrangement (101) according to any one of claims 1 to 22, wherein the test arrangement (101) is designed to start from one or more predefined and/or random test cases (102, 420, 440) a plurality of intermediate test cases (440) generate in order to assign a fitness value to each of the intermediate test cases (440), to select (450) the intermediate test cases (440) depending on the respective assigned fitness value, and to continue using the selected intermediate test cases (440) as new test cases (102, 420, 440). .

24. The test arrangement (101) according to claim 23, wherein the test arrangement (101) is designed to assign a fitness value to the intermediate test cases (440) based on a probability of a malfunction of the test object (102) tested with the intermediate test case (440).

The test arrangement (101) according to claim 23 or 24, wherein the test arrangement (101) is configured to use one or more machine learning algorithms (460) in a determination of the fitness value. The test arrangement (101) according to any one of claims 22 to 25, wherein the test arrangement (101) is designed to implement one or more machine learning algorithms (460), such as support vector machine algorithms and/or random tree algorithms and/or neural network algorithms , to use. The test arrangement (101) according to one of claims 22 to 26, wherein the test arrangement (101) is designed to use the algorithm (460) of machine learning or the algorithms (460) of machine learning based on the test results (109) of the executed test cases ( 102, 420, 440) to retrain. The test arrangement (101) according to any one of claims 1 to 27, which is designed to execute a sequence of test cases (102, 420, 440) and to use their test results (109) as an observation sequence for training a Hidden Markov Model, and to estimate (320) or to approximate (320) states of a state machine (107) with the Hidden Markov Model (107) based on an observation sequence. The test arrangement (101) according to claim 28, which is designed to bring the test object (104, 410) into the individual states of the Hidden Markov Model (107) and to generate a plurality of test cases (102, 420, 440) per state, which, starting from the current state, bring the test object (102) into the other states of the Hidden Markov Model (107) in order to achieve good path coverage of the Hidden Markov Model (107). The test arrangement (101) according to claim 29, which is designed to calculate the most probable execution path of a test case in the Hidden Markov Model (107) by a Viterbi algorithm and based thereon to calculate the path coverage of the test cases (102, 420, 440) to calculate. The test arrangement (101) according to any one of claims 1 to 30, wherein the test arrangement (101) is designed to continuously refine the model (107) and/or the generated test cases (102, 420, 440). The test arrangement (101) according to any one of claims 1 to 31, which is designed to generate one or more test cases (102, 420, 440) for a web application, and to receive and/or evaluate the response (105) of the web application (220, 320). The test arrangement (101) according to claim 32, which is designed to the test case (102, 420, 440) or the test cases (102, 420, 440) in the form of

Entries in one or more input fields of the web application and/or

activation of one or more buttons of the web application and/or

generate activation of one or more links of the web application. The test arrangement (101) according to claim 32 or 33, which is designed to the response (105) of the web application in the form of

Activation or deactivation of one or more input fields of the web application and/or

Activation or deactivation of one or more buttons of the web application and/or

Activation or deactivation of one or more links of the web application and/or

Addition or removal of one or more input fields and/or one or more buttons and/or one or more links, and/or Change of one or more texts or numbers to be received and/or evaluated (220, 320).

35. The test arrangement (101) according to any one of claims 1 to 31, which is designed to generate one or more test cases (102, 420, 440) for a test object (104, 410) that uses a network protocol, and to to receive and/or evaluate (220, 320) the response (105) of the test object (104, 410).

36. The test arrangement (101) according to claim 35, which is arranged for the test case (102, 420, 440) or the test cases (102, 420, 440) in the form of one or more inputs in a sequence of one or more network packets to generate (250, 340, 430).

37. The test arrangement (101) according to claim 35 or 36, which is designed to receive the response (105) of the test object (104, 410) in the form of a sequence of one or more network packets and/or in the form of information regarding a code coverage to receive and/or evaluate (220, 320) the input.

38. The test arrangement (101) according to claim 37, which is designed to build a state machine (107) based on the test cases (102, 420, 440) and code coverage based thereon, in order to use a hash function to generate a value for a state of the to calculate the test object (104, 410) and to identify the condition with the calculated value.

39. The test arrangement (101) according to claim 37 or 38, which is designed to record a sequence of communication between the test arrangement (101) and the test object (104, 410) as an observation sequence for training a hidden Markov model (107). use, and to estimate or approximate the code coverage with the Hidden Markov Model (107) based on the observation sequence. The test arrangement (101) according to claim 39, which is designed to bring the test object (104, 410) into the individual states of the Hidden Markov Model (107) as the initial state and to create a plurality of test cases (102, 420, 440) for each state generate which, starting from the current state, bring the test object (104, 410) into the other states of the Hidden Markov Model (107) in order to achieve good path coverage of the Hidden Markov Model (107). The test arrangement (101) according to claim 40, which is designed to calculate the most probable execution path of a test case in the Hidden Markov Model (107) by a Viterbi algorithm and to calculate the path coverage of the test cases based thereon. The test arrangement (101) according to any one of claims 1 to 31, which is designed to generate test cases (102, 420, 440) in the form of inputs of a network packet or a sequence of network packets for a network interface of a test object (104, 410), and to evaluate the functionality of the test object (104, 410) or the functionality of one or more functions of the test object (104, 410) by means of monitors (106). in a method for testing a test object (104, 410), comprising the steps of:

Creating a model (107) based on an interaction with the test object (104, 410),

generating one or more test cases (102, 420, 440) using the model (107),

Improving the model (107) of the test object using real test results (109) based on the generated test cases (102, 420, 440).

44. A computer program for performing the method according to claim 43, when the computer program is executed on a computer.