DE102020206327A1 - Method and device for testing a technical system - Google Patents

Abstract

Method (10) for testing a technical system, characterized by the following features: - simulation results (11) are obtained in a simulation of the system, - measurement data (13) corresponding to the simulation results (11) are recorded in validation tests (12) on the system, - Using the simulation results (11) and measurement data (13), a classifier (16, 17) is formed (19) for several catalogs (14, 15) of test cases of the system, - the classifiers (16, 17) are used to generate further test cases classified (18) as to whether the simulation is reliable (20), unreliable (21) or not to be assessed (22) in the respective further test case, - those further test cases in which the simulation is unreliable (21) or not to be assessed (22 ) are recorded and - on the basis of an improvement potential (23) associated with the recorded further test cases (21, 22) for the classifiers (16, 17), candidates (24) for possible further validation attempts (26) prioritized (25).

Description

The present invention relates to a method for testing a technical system. The present invention also relates to a corresponding device, a corresponding computer program and a corresponding storage medium.

State of the art

In software engineering, the use of models to automate test activities and to generate test artifacts in the test process is summarized under the heading of “model-based testing” (MBT). For example, the generation of test cases from models that describe the target behavior of the system to be tested is well known.

Embedded systems in particular are dependent on coherent input signals from sensors and in turn stimulate their environment through output signals to a wide variety of actuators. In the course of the verification and upstream development phases of such a system, its model (model in the loop, MiL), software (software in the loop, SiL), processor (processor in the loop, PiL) or entire hardware (hardware in the loop, HiL) is simulated together with a model of the environment. In vehicle technology, simulators corresponding to this principle for testing electronic control devices are sometimes referred to as component, module or integration test benches, depending on the test phase and object.

DE10303489A1 discloses such a method for testing software of a control unit of a vehicle, a power tool or a robotics system, in which a test system at least partially simulates a controlled system by the control unit by generating output signals from the control unit and these output signals from the control unit to the first hardware Modules are transferred via a first connection and signals from second hardware modules are transferred as input signals to the control unit via a second connection, the output signals being provided as first control values in the software and additionally via a communication interface in real time based on the controlled system to the test system be transmitted.

Such simulations are widespread in various fields of technology and are used, for example, to convert embedded systems in power tools, engine control units for drive, steering and braking systems, camera systems, systems with components of artificial intelligence and machine learning, robotics systems or autonomous vehicles in early phases to check their development for suitability. Nevertheless, the results of state-of-the-art simulation models are only included in release decisions to a limited extent due to a lack of confidence in their reliability.

Disclosure of the invention

The invention provides a method for testing a technical system, a corresponding device, a corresponding computer program and a corresponding storage medium according to the independent claims.

The approach according to the invention is based on the knowledge that the quality of simulation models is decisive for a correct prediction of the test results that can be achieved with them. In the field of MBT, the sub-discipline of validation deals with the task of comparing real measurements with simulation results. For this purpose, various metrics, measures or other comparators are used that link signals with one another and that are collectively referred to below as signal metrics (SM). Examples of such signal metrics are metrics that compare size, phase shift, and correlations. Some signal metrics are defined by relevant standards, e.g. B. according to ISO 18571 .

In more general terms, uncertainty quantification techniques support the estimation of the simulation and model quality. The result of an assessment of the model quality using a signal metric or, more generally, using an uncertainty quantification method for a specific input X, which can be a parameter or a scenario, is hereinafter referred to as the simulation model error metric - in short: error metric - SMerrorX. For generalization (interpolation and extrapolation) of SMerrorX for inputs, parameters or scenarios X that have not been considered previously, machine learning models can be used, for example on the basis of so-called Gaussian processes.

During verification, the test item (system under test, SUT) is typically examined on the basis of a requirement, specification or performance indicator. It should be noted that Boolean requirements or specifications can often be converted into quantitative measurements using formalisms such as signal temporal logic (STL). Such formalisms can serve as the basis of quantitative semantics, which in this respect represent a generalization of the verification, rather than a positive one Value indicates fulfillment and a negative value indicates violation of a requirement. In the following, such requirements, specifications or performance measures are collectively referred to as “quantitative requirements” (QSpec).

Such quantitative requirements can either be checked using the real SUT or a model of the same - a “virtual SUT”, so to speak. For the purpose of this verification, catalogs are compiled with test cases that an SUT must satisfy in order to decide whether it has the desired performance and security properties. Such a test case can be parameterized and thus cover any number of individual tests.

Against this background, the proposed approach takes into account the need for reliable test results in order to guarantee the performance and safety properties of an SUT. Especially when performing tests based on a simulation of the system or a subcomponent - instead of tests on the real system - it is important to ensure that the simulation results are trustworthy.

A major challenge for model validation is to determine real measurements and stimuli for the corresponding simulation that provide the highest level of information about the accuracy of the model. For simulation-based tests, information is also required about how well the simulated system meets the requirements.

Using the error metrics outlined above, a classifier can be created that assesses the reliability of the simulation in each test case under consideration for a parameterized set of test cases - hereinafter referred to as the “test catalog” - and classifies them accordingly. Such a procedure is based on the idea of recording the test cases classified as unreliable and examining them in practice. After the tests have been performed on the system, the knowledge gained can be used in various ways, e.g. B. to train the classifier, to improve the parameterization or to optimize the model. A limitation of this approach arises from the fact that a classification must be made for each set of test cases and all non-reliable cases are considered as candidates for experimental measurements, unless they are used to improve the model, the parameterization or the metamodel .

Against this background, one advantage of the solution according to the invention lies in the development of the knowledge to be drawn from various sets of parameterized test cases. The number of test cases recommended for real tests is reduced in this way by a combination of the information provided by several classifiers. In addition, the proposed approach mainly takes into account test cases that are classified as neither reliable nor unreliable. In this way, the number of actual tests can be reduced without influencing the model, the model parameterization or the parameterization of any validation or metamodel.

Such tests can be used in a wide variety of fields. One should think, for example, of the functional safety of automated systems, such as those used to automate driving functions (automated driving).

The measures listed in the dependent claims enable advantageous developments and improvements of the basic idea specified in the independent claim. An automated, computer-implemented test environment can thus be provided in order to improve the quality of the tested hardware or software products largely automatically.

Figure list

• 1 eine Datenbank für Simulation und experimentelle Messungen.
• 2 eine Darstellung des vorgeschlagenen Algorithmus.
• 3 die Berechnung einer erwarteten Verbesserung im Detail.
• 4 eine auf einer Klassifizierung mit hoher Genauigkeit basierende Variante.
• 5 eine Variante zur Berechnung der erwarteten Verbesserung.
• 6 schematisch eine Arbeitsstation.

Embodiments of the invention are shown in the drawings and explained in more detail in the description below. It shows:

• 1 a database for simulation and experimental measurements.
• 2 a representation of the proposed algorithm.
• 3 the calculation of an expected improvement in detail.
• 4th a variant based on a classification with high accuracy.
• 5 a variant for calculating the expected improvement.
• 6th schematically a workstation.

Embodiments of the invention

1 illustrates a database ( 27 ) with multiple catalogs ( 14th , 15th ) or parameterized sets of test cases of the system to be tested. The ones in each catalog ( 14th , 15th ) contained test cases are classified according to whether the simulation in the respective test case is reliable ( 20th ), unreliable ( 21 ) or - according to an optional embodiment - not to be assessed ( 22nd ) is.

It should be noted that this approach is independent of the classifier used for the classification, which can be formed in a variety of ways. In the application examples, however, reference is made to different implementation options.

The simulation results obtained in the simulation of the system ( 11 ) and in corresponding validation tests ( 12th ) measurement data recorded on the system ( 13th ) are stored in this common database ( 27 ) filed. In addition, the database contains ( 27 ) Candidates ( 24 ) both for possible validation experiments ( 26th ) as well as for further tests. All simulated and real tests are performed on the same product variant. The handling and versioning of variants of tests and simulation results ( 11 ) are not part of the present description.

As 2 clarified, is based on these simulation results ( 11 ) and measurement data ( 13th ) for each catalog ( 14th , 15th ) initially a classifier ( 16 , 17th ) educated ( 19th ). Through these classifiers ( 16 , 17th ) further test cases are classified accordingly ( 18th ), whether the simulation is reliable in the respective test case ( 20th ), unreliable ( 21 ) or not to be judged ( 22nd ) is. Those test cases in which the simulation is unreliable ( 21 ) or not to be judged ( 22nd ) are stored in the shared database ( 27 ) recorded.

By a pairwise comparison ( 28 ) of the classifiers ( 16 , 17th ) In the individually recorded test cases, the potential for improvement associated with the test case under consideration can now be identified ( 23 ) for the classifiers ( 16 , 17th ) to calculate. Based on this potential for improvement ( 23 ) become candidates for possible further validation attempts ( 12th ) prioritized ( 25th ). Finally, new measurements are carried out on the system and the classifiers ( 16 , 17th ) in the light of the new measurement data ( 13th ) at least partially trained ( 24 ). The steps mentioned can be repeated as required.

Depending on the properties of the classifiers ( 16 , 17th ) and the resulting uncertainties for the classification ( 18th ) said prioritization ( 25th ) can be done in different ways. According to 3 becomes one of the classifiers defined by a first classifier ( 16 ) classified test cases selected ( 29 ) and simulated. Based on the output ( 30th ) This simulation evaluates the QSpec requirement placed on the system for the test case ( 31 ) and a second classifier ( 17th ) trained ( 32 ).

For the original ( 17th ) and trained in this way ( 33 ) The classifier is then the uncertainty ( 37 , 38 ) when classifying a second test case corresponding to the test case. On this basis, the improvement potential ( 23 ) of the second classifier ( 17th ) finally by comparison ( 28 ) of uncertainty ( 37 ) of the original classifier ( 17th ) with that (38) of the advanced classifier ( 33 ) to calculate. As part of this procedure, instead of the original classifier ( 17th ) also use potential validation measurements as a benchmark.

In the in 4th illuminated variant, instead of considering individual test cases, the latter in groups ( 34 ), which are characterized by their information content in order to identify the potential for improvement ( 23 ) by the pairwise comparison ( 28 ) of the classifiers ( 16 , 17th ) for these groups ( 34 ) to calculate.

Instead of a pairwise comparison ( 28 ) individual test cases or groups ( 34 ) also likes that in 5 the illustrated procedure. Here the second classifier ( 17th ) based on the output ( 30th ) the simulation only hypothetically based on the candidate under consideration ( 24 ) trained ( 32 ) in order to identify its potential for improvement ( 23 ) from the increase ( 23 ) of the advanced training ( 32 ) gained knowledge ( 36 ) compared to that provided by the original second classifier ( 17th ) obtained knowledge gain ( 35 ) to derive. Only if such an increase ( 23 ) is actually to be expected, the test under consideration or the validation measurement is actually carried out on the system.

As already mentioned, the approach presented above does not depend on a specific implementation of the classifier ( 16 , 17th ) away. Below are some examples of how the possible improvement in the selection of new test points could be calculated.

For example, if the parameters of the classifier ( 16 , 17th ) - for example due to a limited amount of training data - are known with sufficient accuracy, there may be a number of test cases that do not prove to be reliable ( 20th ) or unreliable ( 21 ) classify ( 18th ) permit. In these cases, the reliability of the simulation cannot be clearly assessed ( 22nd - 3 and 5 ). Additional data is therefore required in order to avoid under-recording and to reduce the uncertainty ( 37 , 38 - 3 ) of the classifiers ( 16 , 17th ) to reduce. For this purpose, new real measurement data ( 13th ) are collected in order to use the classifiers ( 16 , 17th ) to further educate ( 32 ) and so to improve.

In this exemplary embodiment, the information content of the new measurements is calculated on the basis of conventional information measures. The prioritization ( 25th ) of test cases and Validation experiments are carried out by calculating the information content in pairs.

Another variant is advisable if at least one assessment criterion of the classifier ( 16 , 17th ) - for example the error metric SMerrorX - is based on a metamodel that is based on the simulation results ( 11 ) is assessed. For the re-evaluation of the classifier ( 16 , 17th ) new real measurement data ( 13th ), although this is only trained once at the beginning. It is assessed for which points in the parameter or state space additional information could be useful in order to reduce the uncertainty of the metamodel. This is done for all catalogs ( 14th , 15th ). The test cases are prioritized based on the respective information content ( 25th ) and those candidates ( 24 ) tested with the highest rating in reality. If some test catalogs ( 14th , 15th ) are classified on the basis of matching features - for example a uniform error metric based on the same signal - the classifiers ( 16 , 17th ) are assessed using a common metamodel. In this case, if the metamodel could be improved, the assessment is carried out in all test cases concerned.

This method ( 10 ) can be in software or hardware, for example, or in a mixed form of software and hardware, for example in a workstation ( 40 ) be implemented as shown in the schematic representation of the 6th made clear.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 10303489 A1 [0004]

Non-patent literature cited

• ISO 18571 [0007]

Claims (11)

Method (10) for testing a technical system, in particular an at least partially autonomous robot or vehicle, characterized by the following features: - in a simulation of the system, simulation results (11) are obtained, - in validation experiments (12) on the system, the simulation results (11) corresponding measurement data (13) are recorded, - based on the simulation results (11) and measurement data (13), a classifier (16, 17) is formed (19) for several catalogs (14, 15) of test cases of the system, - by the classifiers ( 16, 17), further test cases and potential validation measurements are classified (18) as to whether the simulation is reliable (20), unreliable (21) or not to be assessed (22) in the respective further test case, - those further test cases and Potential validation measurements in which the simulation is unreliable (21) or cannot be assessed (22) are recorded and attached d of an improvement potential (23) for the classifiers (16, 17) associated with the recorded further test cases (21, 22), candidates (24) for possible further validation attempts (26) are prioritized (25). Method (10) according to Claim 1 , characterized by the following features: - the simulation results (11) and measurement data (13) are stored in a common database (27) and - the candidates (24) as well as stimuli and parameters (28) of the corresponding simulation are added to the database (27) . Method (10) according to Claim 1 or 2 , characterized by the following feature: the improvement potential (23) is calculated by comparing (28) the classifiers (16, 17) in pairs in the individually recorded test cases. Method (10) according to Claim 1 or 2 , characterized by the following features: - a first test case is selected (29) from the test cases classified by a first (16) of the classifiers (16, 17), - an output (30) is generated by the simulation of the first test case, - based on of the output (30), a requirement placed on the system for the first test case is evaluated (31) and a second (17) of the classifiers (16, 17) is further developed (32), - for the original (17) and further developed (33) second classifier, an uncertainty in the classification (18) of a second test case corresponding to the first test case is assessed and - the improvement potential (23) of the second classifier (17) is determined by comparing the uncertainty (37) of the original second classifier (17) with the Uncertainty (38) of the advanced second classifier (33) is calculated. Method (10) according to Claim 1 or 2 , characterized by the following features: - the recorded further test cases (21, 22) are divided into groups (34) and - the improvement potential (23) is determined by a pairwise comparison (28) of the classifiers (16, 17) for the groups (34 ) calculated. Method (10) according to Claim 1 or 2 , characterized by the following features: - a test case is selected (29) from the test cases classified by a first (16) of the classifiers (16, 17), - an output (30) is generated by the simulation of the test case, - on the basis of the output (30), a requirement placed on the system for the test case is evaluated (31) and a second (17) of the classifiers (16, 17) is further developed (32), - for the original (17) and further developed (33) second classifier a gain in knowledge (35, 36) is determined in each case and - the potential for improvement (23) of the second classifier (17) is derived from the increase in the gain in knowledge (36) achieved by the advanced second classifier (33) compared to that achieved by the original second classifier (17) obtained knowledge gain (35) derived. Method (10) according to Claim 1 or 2 , characterized by the following features: - a possible measurement is selected from the possible validation measurements that have not yet been carried out, - an output (30) is generated by the simulation of the associated stimulus, - a request made to the system is made on the basis of the output (30) for a test case evaluated (31) and the associated classifiers (16, 17) advanced (32), - a gain in knowledge (35, 36) is determined for the original (17) and advanced (33) classifier and - the potential for improvement (23 ) of the classifier (17) is derived from the increase in the knowledge gain (36) achieved by the advanced classifier (33) compared to the knowledge gain (35) achieved by the original classifier (17). Method (10) according to one of the Claims 1 until 7th , characterized in that a automatic improvement of errors of the system recognized by the testing takes place. Computer program which is set up, the method (10) according to one of the Claims 1 until 8th to execute. Machine-readable storage medium on which the computer program is based Claim 9 is stored. Device (60) which is set up, the method (10) according to one of the Claims 1 until 8th to execute.

Images