WO2023066663A1

WO2023066663A1 - Method for testing a device, and test apparatus

Info

Publication number: WO2023066663A1
Application number: PCT/EP2022/077655
Authority: WO
Inventors: Christoph Fischer; Merita Haklaj; Klaus Lukas; Reinhard Riedmüller
Original assignee: Siemens Aktiengesellschaft
Priority date: 2021-10-18
Filing date: 2022-10-05
Publication date: 2023-04-27

Abstract

In the method for testing IT security of a device (DUT), the device (DUT) is firstly subjected to at least one first IT security test (STES) and at least one datum of at least one test result (TRES) of the at least one first IT security test (STES) is gathered, the datum characterising a further hitherto unconsidered device property, and an assignment rule is applied which assigns to the at least one datum (DAT) a test method for at least one further IT security test of the device (DUT), which test subsequently subjects the at least one further device property (GSPEZ, USPEZ) of the device (DUT) to a test (DTES). The relevant information on specific test results or protocols, which is yielded by the computer program product and is used for the development of further tests (DTES) of such a method, forms and expands the assignment database and/or the assignment rule. The test apparatus for testing IT security of a device comprises at least one such computer program product.

Description

Procedure for testing a device and test setup

The invention relates to a method for testing a device and a test device.

The aim of security tests in the area of IT security, i. H . especially in information security and/or functional security, is normally determined manually by an auditor. D. H . , the tester makes a selection of specific test tools, test cases and their parameterization and compiles the test tools and test cases into a holistic test strategy . Particularly in the area of devices in the form of industrial automation devices, such as field devices in automation networks, such security tests depend heavily on the respective device type and the environmental requirements for such a device type. Testers usually know the device under test and define test environments for these devices specifically for these devices, in particular with a protocol-specific selection of test cases.

Testers are often unaware of the devices and their areas of application and, as a result, the environmental requirements of the device environment, or the devices first have to be found and classified using an inventory process that is laborious.

So far, test strategies have primarily been known in which the requirements are selected by the tester through suitable manual tool selection and the parameterization of the test cases.

It is therefore the object of the invention to specify an improved method for testing a device which, in particular, can be implemented more reliably and/or less expensively. Furthermore, it is an object of the invention to specify a test device by means of which the aforesaid improved method can be carried out.

This object of the invention is achieved with a method having the features specified in claim 1, with a computer program product having the features specified in claim 14 and with a test device having the features specified in claim 15. Preferred developments of the invention are specified in the associated dependent claims, the following description and the drawing.

In the method according to the invention for testing an IT security of a device, the device is first subjected to at least one first IT security test and at least one date of at least one test result of the at least one first IT security test is used in the method, with the date being another so far characterizes a device property that is not taken into account in the method for testing according to the invention, and an allocation rule is used in the method, which assigns a test method for at least one further IT security test of the device to the at least one datum, which subsequently subjects the at least one further device property of the device to a test .

The method according to the invention can be used to efficiently generate tests for testing the IT security of the device dynamically. Using the method according to the invention, it is possible to use data from test results in order to automate a test method, d. H . to develop a test strategy . In the method according to the invention, such data are advantageously used which are specific to certain device properties of the technical device, so that test methods can be obtained which cover applications of the technical device associated with the device properties. The device property is preferably a technical function of the technical device, d. H . the device property is not just limited to keeping the date available. The at least one item of data within the meaning of the present invention is therefore suitably not the device property alone.

Ideally, the method is a computer-implemented method, i . H . the steps of this method are performed using a computer.

The at least one datum expediently comprises at least one configuration datum and/or one program library datum or at least one function parameter for the technical function. In particular, the configuration data can be part of the configuration of the technical device for a technical function of the device or information that the technical device is configured for this technical function.

The assignment rule can be explicitly in the form of a multi-stage assignment rule: the assignment rule can first determine the device property from the date that characterizes a device property of the technical device and then from the device property by means of a subsequent step a test method, in particular Test cases and/or test tools , identify .

Advantageously, two or more pieces of data which characterize at least one device property can also be used in the method according to the invention. In this way, data in combination with one another can determine specific test cases and test tools. Configuration data for a number of device functions can thus be present, with firewalls of the devices or an environment of the devices that are present at the same time preventing certain device functions. In this case, about no test cases or test tools for the prevented Device functions are provided. In this way, in the method according to the invention, multiple data in combination with one another enable a resource-efficient selection of test methods.

In the method according to the invention, the at least one piece of data expediently includes at least one item of protocol information, characterizing an ability of the device to carry out a function, in particular data processing, by means of a protocol to which the protocol information relates. In the method according to the invention, the at least one device property expediently includes at least one capability of the device for data processing using the protocol or data transmission using the protocol.

In the method according to the invention, the at least one protocol of the protocol information is suitably a communication protocol. In a preferred development of the method according to the invention, enabling the device for data processing is or includes enabling the device for communication.

In the method, the at least one device property is preferably a design of the device to meet an environmental requirement.

In a preferred development of the method according to the invention, the method is repeated, with the at least one further IT security test taking the place of the at least one first IT security test. In this way, the method according to the invention is continued further and more and more device-specifically.

In the method according to the invention, scoring is suitably carried out as a function of a frequency and/or a criticality of test results of the IT security tests for the respective device property. In the method according to the invention, the assignment specification is preferably formed by means of an assignment database, in which one or more test methods are assigned to a plurality of data. The assignment database can be stored in the form of an assignment matrix in that one or more data is assigned to one or more test tools and/or test cases.

In the method according to an advantageous development of the invention, the assignment rule is formed by means of a neural network. In this further development of the invention, the allocation rule is therefore not explicitly in the form of an algorithm that can be briefly specified, but rather implicitly in the form of a suitably trained neural network. In this case, in particular specific data which characterize a device property can index specific test cases and/or test tools individually or in combination with one another. Such an assignment of test cases and/or test tools to individual or multiple data from test results can be obtained, for example, by an appropriately trained neural network, the neural network using previously generated training data, which data from test results and test methods then recommended by one or more human test engineers include, trained is . D. H . The neural network is trained by means of training to receive test results or data from test results as input data and to output test methods, including test tools and/or test cases and/or their parameterization as output data.

In the method according to the invention, the test method is preferably formed with at least one test tool and/or at least one test case. In particular, test tools in combination with test cases represent focused test strategies that produce selective test results. Precisely such selective test results regularly contain information on specific device properties of the device. Such specific device properties allow a Continuation of the method according to the invention with tests tailored to the respective device. Optionally and also advantageously, the test method also includes parameterization of the test case.

In the method according to the invention, in a preferred development of the invention, depending on the at least one test result, in particular the at least one date of the at least one test result, and/or the first and/or further IT security tests, at least one test method of preferably several test methods, in particular at least one Test tool and/or at least one test case, excluding .

In the case of the method according to the invention, the assignment rule is suitably changed as a function of the first and/or the at least one further IT security test.

The computer program product according to the invention is designed to carry out a method as described above and preferably has the assignment database and/or the assignment rule.

Alternatively or additionally, the relevant information produced by the computer program product of specific test results or logs forms and/or expands the assignment database and/or the assignment rule, used to define further tests of a method according to the invention as described above.

The test device according to the invention for testing an IT security of a device comprises at least one computer program product as described above.

The invention is explained in more detail below with reference to an exemplary embodiment shown in the drawing. The only drawing figure 1 shows an automation network with a test device TUNG according to the invention for executing a method according to the invention for testing an IT security of a device of the automation network.

The one in Fig. 1 illustrated method according to the invention is used for computer-implemented, automatic testing of IT security of devices DUT an automation network. For example, these devices can be DUT manufacturing devices and/or control devices and/or I IoT devices (IIoT = "Industrial Internet of Things") of the automation network and include actuators or sensors of the automation network. For example, the devices DUT are field devices of the automation network.

The method according to the invention is based on an automated and intelligent selection GEN of test strategies and then performed tests of the device DUT and includes an iterative execution DTES of tests and a respective subsequent intelligent automatic selection GEN of new tests based on previous tests, namely test results TRES previous ones Tests , derived device properties GESPEZ . The new tests include test procedures derived from device properties GESPEZ and environmental requirements USPEZ. These test procedures are aimed at testing these device characteristics GESPEZ and environmental requirements USPEZ. For this purpose, suitable test cases and test tools are assigned to these device properties GESPEZ and environmental requirements USPEZ by means of a selection GEN. These new test methods, including test cases and test tools, selected by means of the automatic selection GEN, are carried out below. Insofar as tests, test strategies and test cases are mentioned in the present description, tests of IT security and test strategies and test cases for testing the IT security of the device DUT are meant in each case. First of all, initial test cases are selected in the method. This selection of initial test cases takes place via a definition of device-specific criteria, which takes place in a first iteration from a device database GDB. Already known protocols of the device DUT and available interfaces of the device DUT are stored in this device database GDB.

If, on the other hand, the devices DUT are not initially known, a number of standard tests STES and inventory algorithms are used first in order to determine more precise information about the devices DUT. The standard tests and inventory algorithms are provided by a test generation module DYNTG, which has a large number of standard tests STES and inventory algorithms ready for this purpose. Environmental requirements USPEZ relevant for the operation of the device DUT in the automation network, for example a necessary Internet facing or a necessary access to web-based services, are also initially defined and through generic test cases, here due to the OT environment of automation systems (OT = "Operational Technology" ) supplemented by industry-specific test cases .

As a result of the various device properties GESPEZ and environmental requirements USPEZ of the device DUT, the test facility TUNG makes a selection GEN of a test strategy comprising device-specific test tools and device-specific test cases and their optimal parameterization based on the determined data from test results TRES of the tests previously undertaken by the test facility TUNG. For example, the presence of a firewall forms a device property GESPEZ, which implies that testing must be less intensive, which is reflected in the present case in a lower intensity of fuzzing and web testing. Instead, device properties GESPEZ newly found by the test results TRES are used for subsequent tests, in that these newly found device properties GESPEZ be subjected to subsequent tests selected with the GEN selection.

To do this, these newly found device properties GESPEZ are added to the device database GDB. Similarly, environmental requirements USPEZ are added to an environmental requirements database UANF.

The intelligent selection GEN of the test strategy, i. H . the selection GEN of the tests by means of test tools and test cases and their parameterization takes place taking into account individual and/or combined test results TRES as follows:

From data DAT of the test results TRES received so far, new device properties GESPEZ that have not previously been found directly can be inferred: For example, from the date of the test result TRES, according to which the device DUT is enabled for HTTPS connections, it can be derived that the device DUT the Device property GESPEZ has to be able to undertake web communication. This device property GESPEZ is now in turn used to provide a test of IT security of this device property GESPEZ training the device DUT for secure web communication and consequently provide specific test cases for the device property GESPEZ that test the IT security of web communication.

The assignment of data DAT of the test results TRES to device properties GESPEZ also takes place via the device database GDB, which for this purpose also includes assignments that assigns the data DAT of the test results TRES to the device properties GESPEZ in the form of an assignment matrix.

Analogous to this, new environmental requirements UANF of the device can be derived from found device properties GESPEZ and/or from data DAT of the residual results TRES: For example, based on data DAT in test results TRES, a Availability of the MODBUS protocol on the device DUT are identified and from this availability of the MODBUS protocol relevant environmental requirements USPEZ to the MODBUS communication are determined from this availability of the MODBUS protocol by loading them from an environmental requirements database UANF using the data DAT. This environmental requirements database UANF contains assignments of data DAT of the test results TRES, which characterize environmental requirements USPEZ, to environmental requirements USPEZ, for example by means of an assignment matrix.

In the method according to the invention, new test cases are detailed and TLS protocols found in the test results TRES or a determined fuzzing depth are used for new, additional test cases. This can be repeated iteratively.

The found device properties GESPEZ can be used to optimize the test strategy, d. H . of testing procedures to undertake . In addition, it is possible not to pursue certain test cases or "branches" in a branched test tree based on the GESPEZ device properties, since these are not necessary or useful device-specific. For example, certain test cases can be excluded via a whitelisting process For example, due to a lack of all protocols required for a specific communication module, such as a lack of communication protocols, it can be concluded that this specific communication module is not present or set up in the device Device DUT is omitted In this way, resource optimization of the test device TUMG, by means of which the method is carried out, can be supported in the method.

In addition to the iterative optimization of the test strategy, the following advantageous extensions of the method can also be made in other exemplary embodiments: The method described above is also used, for example, to determine a device risk and at least one risk indicator is determined using a scoring method.

The method described above is also used in the exemplary embodiment shown for adaptive test generation: Based on a criticality and a found device property, new test cases or parameterizations can be generated dynamically, for example new input fields for XSS tests or an intensification of fuzzing tests in order to to optimize the test strategy for specific devices and environments. Such a generation of new test cases and parameterizations is also carried out using the test generation module DYNTG, which defines an intensity of fuzzing tests or a test depth of the fuzzing tests.

The method described above can also be used for differential testing compared to tests that have already been carried out: On the one hand, further tests can be optimized in such a way that tests that have already been carried out are not carried out again with identical parameterization, but are carried out with a new, different parameterization .

Furthermore, test results TRES already obtained in previous tests can be used in order to use further test cases or test tools which, for example, already occurred in combination in the past. Consequently, test cases and/or test tools that have often been combined with one another in the past will also be used in combination with one another in the future.

The method described above can also be used for indicator-driven testing from test results TRES: In this way, depending on test results TRES within a category of test tools, further test tools are used Category used or less frequently used test tools of this category . If, for example, more test tools from a certain category of test tools find weak points or incorrect behavior of the device, more test tools from this category will be used in the future and these test tools will be parameterized all the more precisely. In particular, an indicator can be assigned to categories of test tools, for example depending on a number of test tools that find vulnerabilities within this category or depending on a number and/or a criticality of vulnerabilities that are found with test tools in this category. Such an indicator consequently indicates a susceptibility to attacks within this category or a specific code quality of a software of the device DUT within this category. Depending on this indicator, a number of test tools in this category and/or a degree of detail in parameterization of test tools and/or a test depth in subsequent tests of the device DUT can then be provided. For example, if a comparatively large number of errors are found during fuzzing with, for example, two test tools, more test tools available or all available test tools are used for fuzzing with increased or full test depth.

The method described above can be expanded in such a way that after all available tests have been carried out, information is given as to which test types/tools/methods etc. , which are currently not yet used by the test facility TUNG, are available and could be used in subsequent tests.

The device DUT is operated in such a way that the device DUT is tested using the previously described method for testing IT security of a device DUT. If the method shows that the device does not meet IT security requirements, the operation of the DUT device is terminated or a measure that increases the IT security of the DUT device is taken, in particular a software update for the device and/or terminating operation of the device or placing the device DUT in a safe mode of operation. In the method, the further test cases and/or test tools determined are used in a further exemplary embodiment for testing the device DUT.

The method according to the invention presented above is carried out using a test device TUNG according to the invention, which is set up as a computer with a corresponding computer program product. The computer carries out the method according to the invention by means of the computer program product.

Claims

patent claims

1. Method for testing IT security of a device (DUT), in which the device (DUT) is first subjected to at least one first IT security test (STES) and in which at least one date of at least one test result (TRES) of the at least one first IT security tests (STES) is used, with the date marking another previously unconsidered device property, and in which an assignment rule is used that assigns a test procedure for at least one further IT security test of the device (DUT) to the at least one date (DAT). , which subsequently subjects the at least one further device property (GSPEZ, USPEZ) of the device (DUT) to a test (DTES).

2. The method as claimed in the preceding claim, in which the at least one piece of data (DAT) is or comprises at least one item of protocol information, indicating that the device is capable of a protocol.

3. The method as claimed in one of the preceding claims, in which the at least one device property comprises at least one capability of the device (DUT) for data processing using the protocol.

4. The method according to any one of the preceding claims, wherein the at least one protocol is a communication protocol.

5. The method as claimed in one of the preceding claims, in which the enabling of the device (DUT) for data processing is or comprises an enabling of the device (DUT) for communication.

6. The method according to any one of the preceding claims, wherein the at least one device property Training the device (DUT) to meet an environmental requirement. Method according to one of the preceding claims, in which the method is repeated, the at least one further IT security test taking the place of the at least one first IT security test (STES). Method according to one of the preceding claims, in which scoring is carried out depending on a frequency and/or a criticality of test results (TRES) of the IT security tests of the respective device property (GESPEZ, USPEZ). Method according to one of the preceding claims, in which the assignment specification (GDB) is formed by means of an assignment database, in which one or more test methods are respectively assigned to a plurality of data. Method according to one of the preceding claims, in which the assignment rule (GDB) is formed by means of a neural network. Method according to one of the preceding claims, in which the test method is formed with at least one test tool and/or at least one test case. Method according to one of the preceding claims, in which a test method, in particular at least a test tool and/or at least one test case, is excluded. 16. The method as claimed in one of the preceding claims, in which the allocation rule (GDB) is changed as a function of the first and/or the at least one further IT security test (STES, DTES). Computer program product designed to execute a method according to one of the preceding claims, which has the assignment database and/or the assignment rule (GDB). Test device for testing an IT security of a device (DUT), comprising at least one computer program product according to the preceding claim.