DE102021204707A1 - Procedure for testing a facility - Google Patents

Abstract

Method for testing a device (108), in which test data are entered into the device (108) and the state of the device (108) is then monitored in order to identify at least one weak point in the device (108), wherein when selecting the Test data side channel information (112) are taken into account.

Description

The invention relates to a method for testing a device, in particular a technical one, an arrangement for carrying out the method, and a computer program and a machine-readable storage medium.

State of the art

Various methods and techniques are known for testing software and products or devices in which software plays an important role. A common technique for automated testing of software is referred to as fuzzing testing. With this technique, the program to be tested is repeatedly fed with data at one or more input interfaces. This is incorrect or incorrectly formed, unexpected or random data in order to find weaknesses in the software. The output is generally not checked or verified, only the general state of the target. A software freeze or crash is considered an indication of the presence of a vulnerability.

This is based on the consideration that such data creates situations in the operation of the software or program that cannot be achieved with other test methods. This means that programs can crash unintentionally if the data is implausible. In this way, security gaps in the program that would otherwise remain hidden can be detected.

In a so-called white box approach, internal information such as the source code or test access, typically diagnostic or test access for error detection and error elimination, is available, while no internal information is provided in a black box approach.

Various tools are available for white box source code fuzzing that use a debugger to monitor code execution during the fuzzing process. Feedback from the debugger is used to direct the generated input to the untapped core areas, optimizing code coverage. However, these techniques are not available without access to the source code.

In black box fuzzing testing, where the source code is not available, there is typically no feedback to indicate whether a given input executes deeper parts of the software or whether it was discarded early in processing. It is only when an error is detected that a problem is known to have been triggered.

Black box tests are used to detect errors in the specification. However, white box tests are used to detect faults in certain components or even in fault-triggering components themselves.

Black box tests are regularly more complex to carry out than white box tests, but typically enable better verification of the overall system.

The main disadvantage of black box fuzzing testing is that without access to the internal state of the device or software, it is not possible to properly test the coverage and progress of the testing process in general and a concrete input in particular. Only binary return is available, which only determines whether the target application crashed or not. No additional information is provided.

Even if this data is available after running the test case, this return is not sufficient to recognize the effectiveness of an input and to make appropriate modifications to it during the test. Therefore, in this approach, the input is manipulated randomly, without any intelligence and without any feedback on the testing progress on the parts of the tested application.

White box based software approaches monitor the internal state of the target application during testing and collect useful data such as code coverage to evaluate the test cases. This information is then considered when additional inputs are generated, so that more extensive code is covered or specific functions are hit more often. The disadvantage of this approach is that the software runs in a sterile, simulated environment rather than on the actual hardware environment.

If few problems are detected, it is not possible to optimize input generation for improved efficiency since there is no feedback from the fuzzing process.

White box fuzzing testing with instrumented source code or a debugger can be used to evaluate the input in terms of the executed code to find input that provides greater code coverage. An instrumented source code is a Source code with additional commands added for testing purposes. The input generator can favor an input that is more likely to uncover a vulnerability by extending code coverage or triggering more complex flows.

Disclosure of Invention

Against this background, a method having the features of claim 1, an arrangement according to claim 10, a computer program according to claim 11 and a machine-readable storage medium according to claim 12 are presented. Embodiments emerge from the dependent claims and from the description.

The method presented is used to test a device, in particular a technical device. This facility may include both software and hardware. In one embodiment, the device is provided by software. In this case, the method presented is used to test software.

It is provided that test data is entered into the device and the condition of the device is then monitored in order to detect at least one weak point or error in the device, with side channel information being taken into account when selecting the test data. Side channel information is information that is obtained in a so-called side channel attack or a side channel attack. This means in particular information that is not obtained directly through the application at its outputs, but indirectly through the operation of the device as a side effect. For example, the energy consumption or the reaction or response time of the device is side channel information. This is discussed further below.

In one embodiment, the device is tested using black box fuzzing testing. Black box fuzzing testing is a method for testing software, in which tests are developed based on the specification or requirement. The tests are thus developed without knowledge of the inner workings or implementation of the system to be tested. Black box tests are used to detect errors in the specification. White box tests are preferably used to detect errors in certain components or even in error-triggering components themselves.

Black box tests are regularly more complex to carry out than white box tests, but typically enable better verification of the overall system.

In the realm of black box fuzzing testing, where there is no explicit internal information, side channel information can be used as an indication of the amount of internal processing induced by the input.

Side-channel attacks, also known as side-channel attacks, are cryptanalytic methods that exploit the physical implementation of a system, e.g. a cryptosystem, in a device or software. Typically, it is not the cryptographic method itself that is attacked, but only a specific implementation.

In so-called side-channel attacks, non-functional information such as timing, energy consumption or electromagnetic radiation is used to obtain indirect information about internal processing. For example, so-called timing attacks use the response time to identify a valid input, typically an existing user ID on a login page, based on the response time. An invalid user ID can be discarded faster without checking the password than an existing user ID with an invalid password, with the verification of the password taking additional time.

The principle of a side-channel attack is based on observing the attacked device or the attacked software when executing processes or algorithms and finding correlations between the observed data and a key used. This characteristic information regarding the correlation can be obtained by analyzing the running time of the algorithm, the energy consumption or the electromagnetic radiation.

For example, in fuzz testing with an input where a response is expected, the timing of that response or the power consumption of the device provides indirect information about the amount of processing the target system is performing.

The level or amount of processing(s), in turn, can be used as an indication of how much source code is involved in the processing. This measure is an indication of how useful the input is in capturing potential errors.

For example, input that is discarded at the entry validation stage would require far less processing and is correspondingly less useful in finding vulnerabilities than input that passes the entry validation stage and enters the functional processing portion of the software.

Based on the usefulness of each input, it is possible to tune the input generation logic to find more vulnerabilities in the same amount of time.

Random data, unexpected data and/or incorrectly formed data can be entered as test data.

Furthermore, side channel information selected from a set comprising: energy consumption, electromagnetic radiation, timing information can be used.

In one embodiment of the method, test data are repeatedly generated in several passes in a first step, in a second step the test data are entered into the device and then both the state of the device is monitored and side-channel information is used to determine an effect of the first test data to determine the facility, with the effect determined being taken into account when generating the test data in the next pass.

The course of the procedure in the case of repeated execution in rounds can be interrupted as soon as the monitoring of the condition of the facility indicates a weak point in the facility.

For example, machine learning or other artificial intelligence techniques may be used to favor an input that is expected to generate more hits.

It should be noted that when developing software, white box fuzzing testing is also used, especially during the active software development phase. Black box fuzzing testing is typically used during the final verification phase, both for hardware and software, once the hardware and/or software is integrated into the final product and debugger access is difficult. With the method presented, the effectiveness of the final verification in terms of security and robustness can be improved in particular.

The arrangement presented is set up to carry out the method presented here and can be implemented in hardware and/or software. A corresponding computer program, which is also the subject of the application, is used to carry out the method and can be stored on a machine-readable storage medium.

Further advantages and refinements of the invention result from the description and the accompanying drawings.

It goes without saying that the features mentioned above and those still to be explained below can be used not only in the combination specified in each case, but also in other combinations or on their own, without departing from the scope of the present invention.

character list

• 1 shows a sequence of a known method in a flow chart.
• 2 shows a sequence of a further known method in a flow chart.
• 3 shows an embodiment of the presented method in a diagram.
• 4 shows a schematic representation of an arrangement for carrying out the method.

Embodiments of the invention

The invention is shown schematically on the basis of embodiments in the drawings and is described in detail below with reference to the drawings.

In 1 a method according to the prior art is shown in a flow chart. In this case, the procedure is white box fuzzing testing with feedback from a debugger.

The method starts with a first step 10, with a second step 12 the software to be tested is initialized. In a further step 14 intelligent test scenarios are then developed. In a step 16, incorrectly formed data or messages are then sent to the device 18 to be tested (DUT: device under test), which in this case includes software 20 to be tested. In addition or as an alternative, random data or messages can be sent. The software sends data or information to a debugger 22. In a next step 24, it is monitored whether the software crashes or freezes. Finally, in a step 26, it is checked whether an error was found. If so, execution ends. If this is not the case (arrow 28), the system jumps back to step 12.

In 2 is shown in a flowchart a sequence of black box fuzzing testing without feedback. The test method starts in a first step 50 . In a next step 52, an initialization takes place. A random test scenario is then generated in a step 54 . In a step 56, incorrectly formed messages or data are then sent to a DUT 58 with software 60 to be tested. The representation uses the dashed lines to make it clear that here in comparison to 1 there is no debugger.

In a next step 64 it is monitored whether the software crashes or freezes. Finally, in a step 66, it is checked whether an error was found. If so, execution ends. If this is not the case (arrow 68), there is a return to step 52.

In 3 a possible sequence of the method presented, namely black box fuzzing testing with feedback from a side-channel attack, is shown in a flow chart.

In a first step 100, the test method starts. In a next step 102, an initialization takes place. A random test scenario is then generated in a step 104 . In a step 106, incorrectly formed messages or data are then sent to a device 108 to be tested with software 110 to be tested. The software 110 sends data from a side channel attack, the so-called side channel information 112, to step 104, which is considered here. Thus, a feedback is given.

In a next step 114 it is monitored whether the software 110 crashes or freezes. Finally, in a step 116, it is checked whether an error was found. If so, execution ends. If this is not the case (arrow 118), the system jumps back to step 102.

4 shows a schematic representation of an arrangement for carrying out the method, which is denoted overall by the reference numeral 150. A device 152 is checked with this arrangement 150, for which purpose test data 154 are generated by the arrangement, with which the device 152 is fed. The arrangement 150 then monitors the condition of the device 152 in order to identify possible weak points or faults. When generating the test data 154, side channel information 156, which is also obtained by the arrangement 150, is also taken into account. In principle, a further arrangement for carrying out side-channel attacks and for obtaining the side-channel information can also be provided.

Claims (12)

Method for testing a device (108, 152), in which test data (154) are entered into the device (108, 152) and the state of the device (108, 152) is then monitored in order to identify at least one weak point in the device (108 , 152), side channel information (112, 156) being taken into account when selecting the test data (154). procedure after claim 1 , in which the device (108, 152) is tested as part of black box fuzzing testing. procedure after claim 1 or 2 , in which random data, unexpected data and/or incorrectly formed data are entered as test data (154). Procedure according to one of Claims 1 until 3 , wherein the device (108, 152) comprises software (110). Procedure according to one of Claims 1 until 4 using side channel information (112, 156) selected from a set comprising: energy consumption, electromagnetic radiation, timing information. Procedure according to one of Claims 1 until 5 , in which repeatedly in several passes - in a first step test data (154) are generated, - in a second step the test data (154) are entered into the device (108, 152) and then both the state of the device (108, 152 ) monitored as well as side channel information (112, 156) are used to determine an effect of the first test data (154) on the device (108, 152), the determined effect in the generation of the test data (154) in the next pass is taken into account. procedure after claim 6 , in which the process is aborted as soon as monitoring the state of the device (108, 152) indicates a vulnerability in the device (108, 152). Procedure according to one of Claims 1 until 7 , which uses artificial intelligence techniques. procedure after claim 8 , which uses machine learning. Arrangement for testing a control unit for performing a method according to one of Claims 1 until 9 is set up. Computer program with program code means that is set up to implement a method according to one of Claims 1 until 9 execute if the computer program on a computing unit, in particular a computing unit in an arrangement (150) according to claim 10 , is performed. Machine-readable storage medium with a computer program stored on it claim 11 .

Images