DE102021109133A1 - Method and device for creating test cases for a test system - Google Patents

Abstract

Method (10) for creating test cases (3c) for a test system (1a), characterized by the following features: - Requirements (1) for the test system (1a) are based on a corpus (2a) in test scenarios according to a first learning algorithm (2). converted,- the test cases (3c) are created from the test scenarios according to a second learning algorithm (3) using predetermined boundary conditions (3a) in a predetermined test framework (3b) and- according to a third learning algorithm (4), characteristics of the test system (1a ) formed a first feature matrix (4a), which is included (11) in the body (2a).

Description

The present invention relates to a method for creating test cases for a test system. The present invention also relates to a corresponding device, a corresponding computer program and a storage medium.

State of the art

Embedded systems in particular are dependent on coherent input signals from sensors and in turn stimulate their environment through output signals to a wide variety of actuators. In the course of the verification and upstream development phases of such a system, its software (software in the loop, SiL), processor (processor in the loop, PiL) or entire hardware (hardware in the loop, HiL) are combined with a model in a control loop simulated in the environment. In automotive engineering, simulators for testing electronic control units based on this principle are sometimes referred to as component, module or integration test benches, depending on the test phase and object.

DE 10 2019 212458 discloses a computer-implemented method for testing a system for at least one requirement. The system includes in particular a computer program, hardware or an embedded system. The method can be used particularly advantageously if the tested system is a subsystem of an at least partially autonomous vehicle or an at least partially autonomous robot or that the tested system is an at least partially autonomous vehicle or an at least partially autonomous robot. The requirement includes in particular a functional or performance-related requirement or a requirement for the functional safety of the system during its (actual) execution. In the method presented, the request is received in machine-readable form. At least one first input variable is determined to test the system for the received request, and an execution of the system is simulated as a function of the first input variable determined. Furthermore, an output variable of the simulated system is determined and, depending on the output variable, it is determined whether the system meets the requirement. It is also checked whether the simulation meets a quality requirement. In this case, the check includes in particular a check as to whether the simulation leaves a permissible parameter range or violates predetermined boundary conditions or whether models or partial models required for the simulation are available. If the simulation meets the quality requirement and the system meets the requirement, the procedure presented checks whether sufficient test coverage has been achieved for the requirement. If the test coverage for the requirement is sufficient, the test for the requirement is completed.

Disclosure of Invention

The invention provides a method for creating test cases for a test system, a corresponding device, a corresponding computer program and a storage medium according to the independent claims.

An advantage of the solution presented below is that it takes into account the characteristics of the system under test in order to optimize the test execution.

Advantageous further developments and improvements of the basic idea specified in the independent claim are possible as a result of the measures listed in the dependent claims. Provision can thus be made for feature matrices to be formed from the features of the test system and test cases using relevant algorithms from the field of supervised learning, and for the test cases to be divided into classes on the basis of the latter. This enables an automatic classification of the test cases for the respective test system - such as a HiL, SiL, PiL system or a complete vehicle - taking into account functional and non-functional system requirements. This classification supports both a systematic evaluation of the test results and an improvement in the properties of the test system.

According to a further aspect, provision can be made for the classification to be used to select test cases for the test system. This improves the efficiency of a continuous test, since the test cases that are most meaningful for the respective test system can be assigned automatically.

According to a further aspect, it can be provided that the quality of the respective test system is measured on the basis of the test results. In this way, a statement about the suitability of different test systems can be made for each of the selected test cases, for example to the effect that a specific case should be tested better in a HiL than in a SiL environment.

Finally, it can be provided that the feature matrix is updated with regard to the quality of the features. The knowledge gained on the test system can thus flow into the learning algorithm for creating the test cases, which increases the test coverage on the respective test system.

character list

• 1 das Flussdiagramm eines Verfahrens gemäß einer ersten Ausführungsform.
• 2 die Bildung einer Merkmalsmatrix der Testfälle im Dateiformat der Python-Programmbibliothek „NumPy“.
• 3 die Klasseneinteilung eines Testfalles gemäß einem Algorithmus, welcher Informationen aus den Merkmalsmatrizen der Testfälle, des Testsystems und des letzten Aktualisierungsvorgangs heranzieht. Aus den Matrizen extrahierte Daten betreffen neben Prüfrahmen und Qualitätskennzahlen beispielsweise geforderte Merkmale, deren Verteilung über das Testsystem sowie Häufigkeit und Kontext ihrer Überprüfung.
• 4 schematisch eine Arbeitsstation gemäß einer zweiten Ausführungsform.

Embodiments of the invention are shown in the drawings and explained in more detail in the following description. It shows:

• 1 the flowchart of a method according to a first embodiment.
• 2 the creation of a characteristic matrix of the test cases in the file format of the Python program library "NumPy".
• 3 the classification of a test case according to an algorithm which uses information from the feature matrices of the test cases, the test system and the last update process. Data extracted from the matrices, in addition to test frameworks and quality indicators, concern, for example, required characteristics, their distribution across the test system, as well as the frequency and context of their verification.
• 4 schematically a workstation according to a second embodiment.

Embodiments of the invention

1 illustrates the course of a method (10) according to the invention, the elements of which will now be explained in detail.

The functional and non-functional requirements (1) of the system to be tested (1a) are taken into account. Relevant test systems have features such as specific sensors, actuators, control units or communication networks that are either physical or virtual.

A machine learning algorithm (2) uses natural language processing tools to convert the requirements (1) placed on the test system (1a) into test scenarios, using known methods for text preprocessing, tokenization and normalization. The generation of the test scenarios is based on a feature matrix (4a) of the test system (1a), the structure of which will be examined in more detail below.

The data corpus (2a) used for supervised learning consists of features of the test system (1a), which are used within the framework of the learning algorithm (2) to generate the test scenarios.

From the said test scenarios, the test cases (3c) are created according to a second learning algorithm (3) based on given boundary conditions (3a), whereby, depending on the area of application and test framework (3b), either common methods such as search-based testing or specially designed methods for regression from the area of machine learning (ML) are used.

The boundary conditions (3a) of the parameters for the automatic creation of the test cases (3c) are specified in order to ensure the controllability and validity of the test cases (3c) for the test system (1a).

Certain areas of application such as powertrain, brakes, multimedia or driver assistance require different test scenarios, methods, systems and test frames (3b) in functional (e.g. validation of vehicle dynamics control according to ISO 19365:2016) and non-functional (e.g. quality of service). of a communication network) respect.

The test cases (3c) created in this way form the output of the second learning algorithm (3), which is intended for execution in the test system (1a).

According to a third learning algorithm (4), a feature matrix (4a) is formed from the features of the test system (1a) described above. The algorithm automatically extracts these features from the architecture of the test system (1a).

The features of the test system (1a) recorded by the feature matrix (4a) are mapped to the quality (12) measured in a later step (8) and - both in terms of quality (12) and new features - continuously updated (13). The matrix (4a) is continuously adjusted and further developed. To put it simply, the data from earlier repetitions is used as a basis in order to expand the knowledge of the quality (12) of the test system (1a) using new insights from the results (7a) of the test case currently being evaluated. The feature matrix (4a) is in the form of a tensor for the ML algorithms, for example: category module feature feature feature vehicle vehicle battery APP Drivers door actuator/ sensor exhaust system boost pressure sensor EGR mass flow sensor Turbocharger speed sensor vECU function combustion system engine torque ignition control ignition knock control ... ... ... ... ...

According to a fourth learning algorithm (5), which is similar to the third learning algorithm (4), a second feature matrix (5a) is formed from the features of the test cases (3c).

This second feature matrix (5a) is then used for a comparison with the available features of the test system (1a) in the first feature matrix (4a). The knowledge of which features or functions of the test system (1a) are affected by the test case under consideration is used to calculate the first feature matrix (4a) with regard to the quality (12) of these features or functions of the test system measured in a later step (8). (1a) to be updated (see 2 ).

According to a fifth learning algorithm (6), the test cases (3c) are divided into classes as part of a comparison of the feature matrices and assigned to the test system (1a)--depending on the availability and quality (12) of its features--according to the class division. The first feature matrix (4a) maps the scope and quality (12) of the test system (1a) to its current state. The second feature matrix (5a), on the other hand, represents those areas and features of the test system (1a) that are addressed, activated or checked by the respective test (7).

Using an algorithm (6), which analyzes the state of the test system (1a) relevant to the present test (7), a statement can be made about the verifiability of the system (1a) under consideration and the test case created can be classified accordingly. Referring to 3 For example, the exemplary feature "boost pressure sensor" (15) may be certified as 80% meeting the quality requirements (16).

The test system (1a) is now automatically subjected to the test (7) in the assigned test cases (3c) in accordance with the class division made according to the fifth learning algorithm (6).

The results (7a) provided by the test (7) are evaluated with regard to the criteria "passed" and "failed" and a test report is created in this regard.

According to a sixth learning algorithm (8), the results (7a) of the quality measurement test (7) are compared with the measurements made in a reference system (e.g. using signal comparison algorithms according to ISO/TS 18571) and a report on the quality ( 12) of the test system (1a) created.

The reference system may be a vehicle, a HiL or SiL test bench. When updating (13) a feature of the test system (1a) in the first feature matrix (4a), the quality flows (12) with a weighting that corresponds to the number of test cases (3c) performed that cover the feature in question.

A cost function (8a) indicates the effort for executing a test case in a specific test system (1a), which is used as a weighting factor to determine the quality (12) of the test system (1a). On the one hand, the cost function (8a) can support the test strategy by assigning the tests (7) to the available test systems in an effort-conscious manner. On the other hand, knowledge of the quality (12) of the test system (1a) in connection with the cost function (8a) enables the development and test process to be designed more economically. The knowledge of possible quality losses in the targeted test system (1a) allows the development and test process to be controlled in a targeted manner and in this way to significantly increase test coverage as early as the development phase.

By measuring the quality (12) of the test system (1a), it is possible to name weaknesses in the test system (1a) in addition to the classification. In this way, any potential for optimization (8b) with regard to specific requirements (1) can be shown, which may wrongly be considered subordinate, but whose increase would significantly increase the quality (12) of the respective target system or the test system (1a) itself. This in turn would result in an increase in the quality of the actual product or the test system (1a), which again supports the test strategy. For this purpose, the quality-reducing features and the test framework (3b) linked to these features of all previous test runs are analyzed and irregularities are identified based on the analysis.

This method (10) can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a workstation (20), such as the schematic representation of FIG 4 clarified.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102019212458 [0003]

Claims (10)

Method (10) for creating test cases (3c) for a test system (1a), in particular an at least partially autonomous robot or a vehicle, characterized by the following features: - Requirements (1) for the test system (1a) are determined according to a first learning algorithm (2nd ) are converted into test scenarios using a corpus (2a), - the test cases (3c) are created from the test scenarios according to a second learning algorithm (3) using specified boundary conditions (3a) in a specified test framework (3b) and - from characteristics of the test system ( 1a) a first feature matrix (4a) is formed according to a third learning algorithm (4), which is included (11) in the corpus (2a). Method (10) according to claim 1 , characterized by the following features: - a second feature matrix (5a) is formed from features of the test cases (3c) according to a fourth learning algorithm (5) and - according to a fifth learning algorithm (6), a class classification of the test cases (3c) is made using the feature matrices ( 4a, 5a). Method (10) according to claim 2 , characterized by the following features: - the test system (1a) is subjected to a test (7) in the test cases (3c) according to the classification and - the results (7a) supplied by the test (7) are evaluated. Method (10) according to claim 3 , characterized by the following feature: - according to a sixth learning algorithm (8), a quality (12) of the test system (1a) is measured based on the results (7a) and - the first feature matrix (4a) is updated with regard to the quality (12) of the features (13). Method (10) according to claim 4 , characterized by at least one of the following features: - the updating (13) takes place using a cost function (8a) defined for the test cases (3c) and - the updating (13) takes place using an optimization potential (8b) related to the requirements (1) of the test system (1a). Method (10) according to any one of claims 3 until 5 , characterized by the following features: - the features are specified in a tabular document template (14) and - the second feature matrix (5a) is formed on the basis of the document template (14). Method (10) according to any one of claims 3 until 6 , characterized in that errors in the test system (1a) detected by the test (7) are improved automatically. Computer program which is set up, the method (10) according to one of Claims 1 until 7 to execute. Machine-readable storage medium on which the computer program claim 8 is saved. Device (30) which is set up, the method (10) according to one of Claims 1 until 7 to execute.

Images