DE102021109128A1 - Procedure for testing a product - Google Patents

Abstract

A computer-implemented method for testing a product, in particular software, hardware or a system comprising hardware and software, is presented, in which a simulation is carried out depending on input parameters for the product, with which a specific property of the product is tested. Depending on a comparison of a result of the simulation with a requirement for the specific property, a first classification is output for the result of the simulation. A second classification is determined depending on a comparison of reference data from an alternative test of the specific property of the product with the requirement for the specific property. An accuracy or robustness of the simulation is determined depending on the first classification and the second classification.

Description

The invention relates to a computer-implemented method for testing a product and a computer program set up for this purpose and a test environment set up for this purpose.

State of the art

From the DE 10 2020 205539 a method for testing a technical system, in particular an at least partially autonomous robot or vehicle, is known. A virtual test classifier is introduced, which combines the requirements of model validation and product testing.

Disclosure of Invention

A computer-implemented method for testing a product, in particular software, hardware or a system comprising hardware and software, e.g. an embedded system, is presented. In the method, a simulation is carried out depending on the input parameters for the product, with which a specific property of the product is tested. Depending on a comparison of a result of the simulation with a requirement for the specific property, a first classification is output for the result of the simulation. A second classification is determined depending on a comparison of reference data from an alternative test of the specific property of the product with the requirement for the specific property. Depending on the second classification, a classifier is trained using a classifier type that can output probabilities or confidences. Such classifiers can be implemented, for example, via Bayesian networks, Gaussian processes or committees of neural networks. Another implementation variant is the use of the Gauss Process Regression (GPR): Here, important variables are considered on which the product properties depend. A surrogate model is taught for these quantities with the help of GPR. For example, the test parameters or other characteristic variables can be used as inputs to the surrogate model. In this implementation, the local uncertainty of the surrogate model, which is calculated using the Gaussian process, is used as the probability. Alternatively, an exploration metric such as maximum probability of improvement (MPI), expected improvement (EI) or upper confidence bound (UCB) can also be used in this implementation. Other surrogate approaches such as chaos polynomial expansion can also be extended in such a way that a confidence band is output in addition to the value of the surrogate model. Depending on a result of the first classification, the classifier outputs a probability or a confidence that the first classification applies.

A computer-implemented method for testing a product, in particular software, hardware or a system comprising hardware and software, e.g. an embedded system, is also presented. In the method, a simulation is carried out depending on the input parameters for the product, with which a specific property of the product is tested. Depending on a comparison of a result of the simulation with a requirement for the specific property, a first classification is output for the result of the simulation. A second classification is determined depending on a comparison of reference data from an alternative test of the specific property of the product with the requirement for the specific property. Depending on the first classification and the second classification, a classifier is trained, which outputs a probability that the first classification and the second classification apply. For this purpose, a classifier type is used, which can output probabilities or confidences. Such classifiers can be implemented, for example, as described above.

In particularly preferred refinements of the methods presented, the specific classification is accepted or rejected depending on the output probability or confidence for the application of a specific classification, in particular depending on a comparison of the output probability or confidence with a predetermined threshold.

In a further preferred embodiment of the presented method, further reference data is requested depending on the probability or confidence that is output, or new reference data are generated automatically, in particular depending on a comparison of the probability or confidence with a predetermined threshold.

The proposed method can be used to assess whether there is a sufficient amount of reference data and the correct reference data for the specific application in order to assess the quality of the To be able to evaluate the simulation with sufficient certainty. A particularly automated iterative method for requesting further reference measurements can be implemented. These further measurements can then reduce the uncertainty of the classifier for the reference data.

The proposed methods can provide indicators of whether the simulation environment and models used are of sufficient quality to answer verification and validation issues at any point in the development process. These indicators can be part of a so-called validation of validation, i.e. they evaluate whether the simulation is suitable for validating the given question.

The proposed methods also enable an identification of areas in a predefined test space for which the results of the classifier have a low confidence.

The (technical) product can in particular be software that is to run on a computing unit in a technical system, for example a detection, monitoring or control function in an at least partially autonomous vehicle or robot. The (technical) product can also be hardware including software, for example a sensor, an actuator or a control unit of an at least partially autonomous robot or vehicle.

In particular, the classifications within the scope of the methods presented are not used to classify product properties or to identify defects in products through classification. Rather, it is classified whether and to what extent the evaluation of a product based on a simulation corresponds to the properties of the product in reality. It offers the possibility to determine the quality of a simulation in relation to given criteria. In addition, it offers the possibility of initiating automated simulations. Another possible application is to predict the quality of simulation results without having to carry out additional reference measurements.

In a preferred embodiment, the reference data come from a test on the real product or from a (particularly particularly accurate) reference simulation.

The first classification and the second classification can each include, for example, whether the product passed or failed the test or whether the test was not carried out because necessary conditions for it were not met. The first classification and the second classification can also each include whether the diagnostic or detection function comes to a false positive, a false negative, a true positive or a true negative result, in particular when simulating a detection or diagnostic function. In addition, the first and the second classification can each include whether or not a predefined, safety-relevant event has occurred, in particular in the case of a simulation of a safety-critical function. These classifications provide a stable and meaningful metric to determine the accuracy and robustness of the simulation.

• - Bewertung der Genauigkeit oder Robustheit einer Simulationsumgebung für den Test einer Diagnosefunktion oder einer Detektionsfunktion (beispielsweise Fahrassistenzfunktionen, sicherheitskritische Funktionen oder Diagnosefunktionen in einem Steuergerät eines Fahrzeugs),
• - Bewertung einer Simulationsumgebung im Kontext von SOTIF-Simulationen,
• - Bewertung der Genauigkeit oder Robustheit verwendeten Modelle in einer Hl-, MIL-, SIL-Umgebung für den Test einer Diagnosefunktion oder einer Detektionsfunktion (beispielsweise Fahrassistenzfunktionen, sicherheitskritische Funktionen oder Diagnosefunktionen in einem Steuergerät eines Fahrzeugs),
• - Einsparung realer Messungen durch Klassifikationsgestützte Bestimmung bzw. Prädiktion der Simulationsgenauigkeit oder -robustheit,
• - Klassifikation einer vorgegebenen Menge von Tests oder Szenarien, auf deren Grundlage bewertet wird, unter welchen Randbedingungen oder für welche Art von Eingangsgrößen die Simulation zu ähnlichen Ergebnissen führt wie die Realität und unter welchen Randbedingungen bzw. für welche Eingangsgrößen die verwendeten Simulationsmodelle zu ungenau sind,
• - Vergleich unterschiedlicher simulationsbasierter Testplattformen bzw. Testumgebungen auf Grundlage klassifikationsbasierter Metriken.

The method enables the following applications, among others, and offers the following advantages in particular:

• - Evaluation of the accuracy or robustness of a simulation environment for testing a diagnostic function or a detection function (e.g. driver assistance functions, safety-critical functions or diagnostic functions in a vehicle control unit),
• - Evaluation of a simulation environment in the context of SOTIF simulations,
• - Evaluation of the accuracy or robustness of models used in an Hl, MIL, SIL environment for testing a diagnostic function or a detection function (e.g. driver assistance functions, safety-critical functions or diagnostic functions in a vehicle control unit),
• - Saving of real measurements through classification-based determination or prediction of the simulation accuracy or robustness,
• - Classification of a specified set of tests or scenarios, on the basis of which it is evaluated under which boundary conditions or for which type of input variables the simulation leads to results similar to reality and under which boundary conditions or for which input variables the simulation models used are too imprecise,
• - Comparison of different simulation-based test platforms or test environments based on classification-based metrics.

• 1 eine beispielhafte Ausprägung eines computerimplementierten Verfahren zum Testen eines Produkts sowie
• 2 einen Ausschnitt aus 1 mit Details zu Block 130.

Embodiments of the invention are explained in more detail below with reference to the accompanying drawings. In the drawings show schematically:

• 1 an example of a computer-implemented method for testing a product, and
• 2 a snippet 1 with details on block 130.

Description of the exemplary embodiments

In software engineering, the use of models to automate test activities and to generate test artifacts in the test process is summarized under the generic term "model-based testing" (model-based testing, MBT). This can include, for example, the generation of test cases from models that describe the target behavior of the system to be tested.

Embedded systems in particular are dependent on coherent input signals from sensors and in turn stimulate their environment through output signals to a wide variety of actuators. In the course of the verification and upstream development phases of such a system, its model (model in the loop, MiL), software (software in the loop, SiL), processor (processor in the loop, PiL) or entire hardware (hardware in the loop, HiL) can be simulated together with a model of the environment. In automotive engineering, simulators for testing electronic control units based on this principle are sometimes referred to as component, module or integration test benches, depending on the test phase and object.

Such simulations are widespread in various fields of technology and are used, for example, to test embedded systems in power tools, engine control units for drive, steering and braking systems, camera systems, systems with artificial intelligence and machine learning components, robotic systems or autonomous vehicles in early phases to check their development for suitability.

A simulation model or the sub-models it contains are usually calibrated and validated with the aid of defined validation measurements. A function is usually tested on the basis of predefined scenarios or test catalogues. These can often be parameterized. The aim of a simulation is to save real measurements at any point in the development process. In order to compare the simulation results with reality and to be able to determine the quality of the simulation in relation to the specific application, real measurements or other reference data (e.g. from simulations with higher accuracy or other test platforms such as software in the loop (SIL), hardware in the Loop (HIL) or Vehicle in the Loop (VIL)).

So far, the results of such simulation models have usually only been included to a limited extent in release decisions due to a lack of confidence in their reliability.

In the context of product development, classification methods can be used, among other things, to develop diagnostic functions for control units and in the area of design of reliability. The focus is on recognizing typical error patterns based on measurement data. Classification methods and other machine learning methods are also used in a variety of ways in control unit functions for driver assistance functions.

The following describes examples of the use of classifiers or classification methods as part of a verification or validation of a product, for example for the validation of an embedded software function, in particular for driving functions of an at least partially automated vehicle or robot.

1 shows an example implementation of a computer-implemented method for testing a product. Different simulation models and different classifiers can be integrated into the validation using the framework described for this purpose.

Input variables 101 can be entered into an input model 102, which outputs output variables to a simulation-based test environment 110. Furthermore, other input variables 103, such as scenario lists, data from a test catalog or similar information, can be given directly to the simulation-based test environment 110.

The simulation-based test environment can access various models and tools and examine the product to be tested using a simulation. In particular, the test environment can use MiL, SiL, HiL or PiL methods.

Results of the test or tests are transferred from the simulation-based test environment 110 to a classification instance 111, which carries out a classification for them. A result of the classification is transferred from the classification instance 111 to block 130 .

Further information such as relevant signals of the simulation, reference data, model parameters or further input variables can be transferred to block 130 from block 104 .

Reference data from an alternative test are transferred from block 120 to a classification instance 121 . The alternative test can in particular be a test on the real product or it can be a particularly precise, more complex simulation than the simulation in block 110. Block 120 can in particular be a database with reference data. Alternatively, block 120 can also be a further test environment, for example also an apparatus for generating further measurements, in particular on the real product.

In the classification instances 111 or 121, the output variables or results of the simulation from block 110 or the output variables or reference data from block 120 are classified according to predetermined criteria. These criteria can be determined, for example, via an interface from a user input.

• - bei Software-, Hardware- oder Systemtests: Test bestanden / Test nicht bestanden oder Test/ freigegeben / Test nicht freigegeben
• - bei Diagnose- und Detektionsfunktionen: Falsch positiv / falsch negativ / wahr positiv / wahr negativ
• - im Kontext von Simulationen zur funktionalen Sicherheit, z.B. SOTIF-Simulationen (SOTIF = Safety Of The Intended Functionality): vordefiniertes Event eingetreten / nicht eingetreten

Possible classification criteria can be, for example:

• - for software, hardware or system tests: test passed / test failed or test/ approved / test not approved
• - for diagnostic and detection functions: false positive / false negative / true positive / true negative
• - in the context of simulations for functional safety, e.g. SOTIF simulations (SOTIF = Safety Of The Intended Functionality): predefined event occurred / not occurred

Depending on the application, further classification criteria can be defined. In this case, classification means in particular the assignment of the simulation results or reference data to one or more classes of a discrete number of classes.

Block 130 includes a classification-based evaluation of the accuracy or robustness of the simulation of block 110, the outputs 140 include a measure of the accuracy or robustness of the simulation of block 110. For example, the outputs 140 may include a robustness or accuracy metric such as a measure of information gain ( information gain) or characteristic variables, as known from machine learning, such as a confusion matrix.

In addition, a classifier can be trained in block 130, which makes it possible to carry out the accuracy or robustness of the simulation for further tests or scenarios without having to carry out corresponding new reference measurements.

2 shows a section 1 with details on block 130.

In 2 the connected blocks 110 and 120 and their inputs are not shown. The classification instances 111 and 121 and their inputs and outputs correspond to the descriptions 1 .

In addition to the classifications from blocks 110 and 120, block 130 can 1 also receive further input variables from block 104 as described above. In particular, block 130 includes an input entity 131 for receiving and processing input data.

Block 130 includes in 2 an optional block 132 for training a simulation quality classifier. For this purpose, this block 132 contains in particular an algorithm for comparing different possible classifiers and for selecting the classifier with the greatest accuracy (feature selection). This serves to optimize the result of the classification. Block 132 may receive information from block 131 to train the classifier.

Block 130 also includes a block 133 for model creation, in particular for creating a meta-model, with which an accuracy or robustness of the simulation can also be determined in areas for which no input data or no reference data are available. Block 133 can receive input data from block 131 for model creation.

Block 130 also includes a block 134 for calculating and outputting output variables 140, in particular including a measure of the accuracy or robustness of the simulation from block 110. Block 134 can receive input data from blocks 132 and 133 for the calculation.

For feature selection in block 132, model creation in block 133 and calculation of output variables 140 in block 134, block 130 has machine learning algorithms and classification algorithms or can access a corresponding library that includes these algorithms.

Two different variants are presented below for teaching the classifier.

In the first variant, an overall classifier, for example in the form of a confusion matrix, is determined in block 130 . This overall classifier includes e.g. n x n classes and possibly contains a probability for each of these classes. The result of this overall classification and the associated parameters and input variables are used for training. The training is carried out, for example, using established methods of supervised machine learning. A model is created through supervised learning, which establishes a connection between the parameters and input variables on the one hand and the known overall classification on the other. The user can optionally select for which elements of the n x n matrix the learned overall classifier should reproduce the results with the greatest accuracy. For example, if the user gives the diagonal elements of the n x n matrix a higher weight, this means that the classifier has a high level of accuracy for those simulations for which the same classification result was achieved in the simulation and in the reference measurement. To achieve this goal, a derived variable such as a weighted sum can be used instead of the n x n matrix for training the overall classifier. Another possible variant is to use methods for multi-objective optimization when training the classifier.

Once the classifier has been learned, only the results from the simulation and the parameters and input variables used for learning are used in its evaluation to determine the accuracy or robustness of the simulation. In order to determine the classification and the simulation quality for further scenarios or tests, reference measurements are no longer required.

In the second variant for teaching the classifier, the output of the classifier is determined for the reference data. The result of this classification and the associated parameters and input variables are used for training. The training can in turn be carried out according to established methods of supervised machine learning. Through the supervised learning, a model is created which establishes a connection between these parameters and input variables on the one hand and the known classification on the other hand.

In order to carry out the simulation quality for further tests or scenarios, additional simulations are carried out and evaluated with the existing methods. Instead of generating further reference data (e.g. through measurements), the learned classifier is now used for the reference data and evaluated with other input variables.

A classifier type is now to be used below, the result of which outputs information about a probability or confidence of the classification result. For example, Bayesian networks, classifiers based on Gaussian processes, committees of neural networks or other meta-models with uncertainty can be used as such classifiers. One implementation variant is the use of the Gaussian process regression (GPR): important variables are considered on which the product properties depend. A surrogate model is taught for these quantities with the help of GPR. For example, the test parameters or other characteristic variables can be used as inputs to the surrogate model.

Such an input variable could be a scalar resulting from the formalization of the requirement. In many cases, the requirement can be formalized in such a way that it is mapped to a specific size. For example, the requirement is met if and only if that particular quantity is greater than or equal to zero. In this example, a Gaussian process regression can be used to form a surrogate model for the specific quantity.

The surrogate model can depend on the test parameters or a number of other scalar variables as input variables. In this implementation, the local uncertainty of the surrogate model, which is calculated using the Gaussian process, is used as the probability. Alternatively, an exploration metric such as maximum probability of improvement (MPI), expected improvement (EI) or upper confidence bound (UCB) can also be used in this implementation. Other surrogate approaches such as chaos polynomial expansion can also be extended in such a way that a confidence band is output in addition to the value of the surrogate model.

The scatter of the classifiers or the surrogate models embedded in them is a measure of the confidence that can be placed in the results of the classifier. Confidence, which is expressed by probabilities, is to be distinguished from the spread of test results in the parameter space. The first-mentioned scatter can be interpreted as a probability in the sense of Bayes as a confidence measure, the scatter in the parameter space is described by frequentistic distributions.

When we speak of probabilities in the following, this refers to the confidence in the results of the classifier. For this purpose, the classifier is advantageously evaluated with data that are not contained in the training data set.

Such a classifier can be used to identify points in the test space for which the classification cannot be made unambiguously, i.e. a classification attribute cannot be assigned with a sufficiently high probability. In particular, these points can be proposed as candidates for new reference measurements.

1. 1. Training eines Klassifikators anhand der Klassifikation von Referenzdaten und ggf. Simulationsdaten.
2. 2. Auswertung des Klassifikators mit Hilfe von Parametern und / oder charakteristischen Größen aus der Simulation.
3. 3. Für jede Parameterkonfiguration wird ausgewertet, ob der eingelernte Klassifikator mit ausreichend hoher Wahrscheinlichkeit zu einem Ergebnis führt.
4. 4. Anforderung weiterer Referenzmessungen und erneutes Training des Klassifikators, bis der Klassifikator ausreichend eingelernt ist und im Rahmen der geforderten Wahrscheinlichkeiten ausreichend sichere Ergebnisse liefert.
5. 5. Optionale Bewertung des Testraums und Vergleich der Simulationsergebnisse mit den Ergebnissen, die aus Referenzdaten und Klassifikation abgeleitet wurden. Interpretation der Ergebnisse und Einbindung in den Validierungsprozess, den Produkt-Entwicklungsprozess oder die Weiterentwicklung des Simulationsmodells.

In order to be able to evaluate for which tests or scenarios a simulation is valid or for which tests or scenarios the simulation provides results of sufficient quality, a procedure comprising the following steps can be carried out:

1. 1. Training of a classifier based on the classification of reference data and, if necessary, simulation data.
2. 2. Evaluation of the classifier using parameters and/or characteristic values from the simulation.
3. 3. For each parameter configuration, it is evaluated whether the learned classifier leads to a result with a sufficiently high probability.
4. 4. Request for further reference measurements and renewed training of the classifier until the classifier has been sufficiently trained and delivers sufficiently reliable results within the framework of the required probabilities.
5. 5. Optional evaluation of the test room and comparison of the simulation results with the results derived from reference data and classification. Interpretation of the results and integration into the validation process, the product development process or the further development of the simulation model.

Two variants of methods using such a classifier type, the result of which outputs information about the probability of the classification result, are presented below.

In a first variant, the classifier for the reference data is trained using the existing data. For every point in the parameter space for which a reference measurement is available, the result is unambiguous, i.e. the probability is either 0 or 1.

The selection of which input variables or which classifier type is best suited for a specific application can be automated. In addition, priorities can also be specified. The efficiency of the classifier can be expressed using the usual quantities precision and recall. The training of the classifier depends on which type of misclassification is accepted or how the different types of misclassification are weighted. Some classifiers allow allows the user to weigh the different types of potential misclassification. This is often illustrated graphically with ROC curves (limit optimization curves). In a preferred embodiment of the described method, an interface can be used to select which type of potential incorrect classification is minimized. For example, if the method is used to assess the validity of tests, the user could specify that tests incorrectly rated as "pass" are weighted more heavily than tests incorrectly rated as "fail".

As an example, the following tables consider outputs from a classifier that evaluates whether a test passed, failed, or was not approved. The table above shows sample output after training for a data point included in the training dataset. The probabilities for a specific classification attribute are then either 0 or 1. If the learned classifier is later evaluated at a different point outside of the training data set, then probabilities between 0 and 1 result, see for example the table below. result probability Passed 0 Failed 1 Not released 0 result probability Passed 0.1 Failed 0.7 Not released 0.2

After training, the classifier can be used to evaluate scenarios or tests for which no reference data is available. If the probability for a class is above a previously defined confidence limit, the result is accepted. The classification result can then be compared with the classification result from the simulation and the overall result can be included in the evaluation of a simulation accuracy.

If the classifier does not lead to a clear result for the parameters used and other input variables (i.e. the probability for none of the possible attributes is greater than the limit value), this is an indicator that further reference data is required for the test in question or the specifically parameterized scenario will. In order to reduce the uncertainty, reference data can be requested for this point in the parameter space or for these input variables, or corresponding measurements or high-precision simulations can be carried out.

The following table shows a probability (Wk) for five points in the parameter space as the classification result: result WW 1 WW2 WW3 WW4 WW5 Passed 0.1 0.3 0.9 0.3 0.15 Failed 0.7 0.3 0.05 0.1 0.05 Not released 0.2 0.4 0.05 0.6 0.8

The classification result can now be evaluated by comparing it with a specified threshold. For example, a probability threshold for which a classification result is accepted can be set to >0.75. In this case, point 1 would be “failed”, point 3 “passed” and point 5 “not approved”. For points 2 and 4, the acceptance criterion is not met, so that the classification for these points is classified as uncertain.

Instead of teaching in the classifier for the reference data as in the first variant described, the classifier for the overall result can also be taught in in advance. In this case they have to Uncertainties do not have to be propagated through, but result directly from the intrinsic uncertainty of the classifier, learned through training, and the finite number of support points. In this case, too, the user has the option of prioritizing what type of misclassification should be avoided. In this case, the prioritization means for which type of events the assessment of the quality of the simulation should have a high degree of accuracy.

As an example, the following tables consider a classifier that evaluates whether a test passed, failed, or was not approved. Here, the probabilities for the entries in a matrix are learned. The top table shows the post-training status for a point from the training dataset. The probabilities are either 0 or 1. If the learned classifier is later evaluated at a different point, then probabilities between 0 and 1 result, as shown in the table below. simulation passed failed not released reference Passed 0 0 0 Failed 1 0 0 Not released 0 0 0 simulation passed failed not released reference Passed 0.1 0.05 0 Failed 0.5 0.05 0.05 Not released 0.2 0 0.05

The classification result can be evaluated as described above: If the probability for one of the attributes is above a predetermined limit, the classification result for the corresponding point in the parameter space is adopted; if this criterion is not met for any of the attributes, the point is classified as uncertain. In the example in the table below, the result would be classified as uncertain with a probability threshold >0.6, with a probability threshold >0.4 the result "failed in the reference - passed in the simulation" would be accepted.

Both variants shown make it possible to request new data in an iterative process. For each data point for which the classification is not clear, both a new reference (e.g. real measurement or simulation with high accuracy) and a new simulation are preferably requested.

Instead of directly requesting new reference data, a large number of simulations can also be carried out and the taught-in classifier for the reference data can be evaluated for each simulated parameter configuration. After completion of all simulations, an evaluation of the parameter space can then be made. Based on this evaluation, it can be estimated to what extent the result of the overall classifier and thus the evaluation of the reliability of the statistics can be improved by further reference measurements. The type of deviation can also be evaluated. If, for example, test cases in a certain region of the parameter space are not approved in the simulation, but are approved and passed in the reference data with a high degree of probability, it may be possible to dispense with carrying out further reference measurements. On the other hand, the points for which none of the classification results are given because all the probabilities are too low can be of particular interest.

In the following, the exemplary application of one of the methods presented for a function test for an adaptive cruise control (ACC) of a vehicle will be described. A typical scenario for testing an ACC is a so-called cut-in maneuver. In this case, the system to be tested is the ACC functionality of a vehicle under consideration. This is overtaken by another vehicle, which - if necessary at a short distance - cuts in in front of the car under consideration and reduces its speed. A requirement for the ACC for such a scenario is, that the vehicle under consideration must have restored a defined safety distance after a certain period of time. The distance when cutting in and the speeds of the vehicles can have different values. In this scenario, these play the role of input variables (101 in 1 ) for the entry model (102 in 1 ).

A test in the simulation environment (110 in 1 ) is performed and reference data is determined (from block 120 in 1 ). For each result of the simulation or the corresponding reference data, the classification instances (111 or 112 in 1 ) carried out a classification, in this case "test passed" or "test failed", depending on whether the vehicle in question has reached a prescribed safety distance after the prescribed time.

• Ein Eintrag in der Matrix entspricht der Anzahl der Tests, die sowohl in der Simulation als auch in der Referenzmessung bestanden wurden. Ein weiterer Eintrag ergibt sich aus der Zahl der Tests, die in der Simulation bestanden und in der Referenzmessung nicht bestanden waren, usw. In diesem einfachen Fall wird insbesondere über die Menge aller Tests summiert und ein Mittelwert gebildet.

To determine the accuracy or robustness of the simulation in the simulation environment (110 in 1 ) a confusion matrix can now be determined, for example, as follows (in block 130 in 1 ):

• An entry in the matrix corresponds to the number of tests that were passed both in the simulation and in the reference measurement. Another entry results from the number of tests that passed in the simulation and failed in the reference measurement, etc. In this simple case, the set of all tests is totaled and an average value is formed.

However, the user can optionally have the option of selecting via inputs which parameters are to be totaled. In addition, he can select, for example, that a confusion matrix should be created for each speed value. The parameters selected by the user can therefore be assigned to two different categories: those that are averaged and those for which the confusion matrix is formed separately.

The confusion matrix can also be a function in a multidimensional parameter space (in this case e.g. speed of the considered vehicle and distance when cutting in).

If no parameter is summed, one of the following classes results for each point in the multidimensional parameter space: "Test passed in simulation and passed in the reference", "Passed test in simulation and failed in reference", "Failed test in simulation and in passed the reference", "failed the test in simulation and failed the reference".

In the application example for the release of tests, the classification can be extended in the following ways: "Test passed", "Test failed", "Release condition for test not met". The number of entries in the confusion matrix and the combinatorics of the classes from the comparison of simulation and reference measurement then increase accordingly.

For example, a confusion matrix for a test scoring use case might look like this: simulation Passed Failed Not released reference Passed 0.1 0.2 0.1 Failed 0.15 0.4 0 Not released 0 0.05 0

For a driving maneuver that was not driven, a risk can be predicted solely on the basis of the initial vehicle speeds, the distance when cutting in and other system variables that the simulation will incorrectly predict the test result. For example, the classification methods described can provide a set of tests for which the simulation reports "passed" even though there is a high risk that the test has failed. The classifiers with confidence output now provide a probability that the assessment is wrong and that further reference measurements may be required to make the assessment of the comparison from simulations and reference data more robust. A prioritization for further reference measurements can thus be derived from the probabilities for the tests with ambiguous results.

As a further exemplary application, the accuracy or robustness of a simulation from a simulation environment for a diagnosis or detection function is to be determined.

In systems with embedded software, among others, diagnostic functions and detection functions are often used. Some of these functions are specifically designed to detect and report errors. Examples of this are the diagnostic functions for monitoring an exhaust aftertreatment system. If a conspicuous behavior occurs in relevant parameter ranges that indicates a malfunction, a status flag is set, for example, which is processed further. If such an event occurs repeatedly, an error is displayed to the driver and corresponding information is stored in the relevant error memory of the engine control unit.

Another example of the use of diagnostic or detection functions are safety-related functions for driver assistance systems such as AEB or ESP. Here, too, signals are checked for plausibility on the basis of specified criteria and, if certain events occur, suitable countermeasures are initiated (such as stopping safely, initiating emergency braking or other interventions in the vehicle's actuators).

• - Falsch positiv: Es ist kein Ereignis aufgetreten, der implementierte Algorithmus hat jedoch ein Ereignis gemeldet.
• - Falsch negativ: Es ist kein Ereignis aufgetreten, der implementierte Algorithmus hat ein Ereignis gemeldet.
• - Wahr positiv: Es ist ein Ereignis aufgetreten, der implementierte Algorithmus hat ein Ereignis gemeldet.
• - Wahr negativ: Es ist kein Ereignis aufgetreten, der implementierte Algorithmus hat kein Ereignis gemeldet.

In these examples and for many other applications of diagnostic and detection functions, the output of the relevant control unit function can be assigned to one of the following categories:

• - False positive: No event occurred, but the implemented algorithm reported an event.
• - False negative: No event occurred, the implemented algorithm reported an event.
• - True positive: An event occurred, the implemented algorithm reported an event.
• - True negative: no event occurred, the implemented algorithm did not report an event.

In order to save costs or to carry out tests in an early phase of development, such functions are increasingly being tested in simulation-based platforms or test environments, e.g. with SIL, MIL or HIL methods.

As a result of a determination of the accuracy or robustness of such a simulation in a test environment for a diagnosis or detection function, a metric, for example a confusion matrix, can be calculated from the comparison of classifications for the simulation results and for the corresponding reference data. In this case, too, the user can optionally select which parameters are to be totaled and which are not. If none of the parameters are averaged, each point in the parameter space can be assigned to one of the n classes (e.g. 16 classes in a 4x4 matrix for the above classifications), depending on what was observed in the simulation and in the reference measurement. When averaging over individual parameters, a trained classification model can specify a probability for each point and for each category (confusion matrix for each point, as in the previous case).

For example, a confusion matrix for the use case of evaluating the performance of a diagnostic or detection function could look like this: simulation Wrong pos. wrong neg true pos true neg reference False positive 0.1 0.2 0.05 0.05 False negative 0.15 0.3 0 0 True positive 0 0.05 0 0 true negative 0.05 0.05 0 0

The use and evaluation of the classification results usually depends heavily on the use case. If the primary aim is to minimize the risk of false negative events occurring, these results can be weighted accordingly. Points in the parameter space where the simulation gives a false negative, but the classifier of the reference data gives a different result (or vice versa reverse) could be prioritized higher and give reason to carry out further measurements. The confidence output of the classifier can then be used to identify possible candidates for further reference measurements. If it is not yet possible to make reliable statements about the accuracy of the simulation for certain cases, further reference measurements can be carried out in order to achieve a robust assessment of the simulation quality.

As a further exemplary application, simulations to ensure the functional safety of a product are to be considered, for example SOTIF simulations of driving functions of an at least partially automated vehicle.

For example, the scenario of a freeway exit can be considered as a simulation. Error models can be used to simulate incorrect lane detection, which can cause the vehicle to leave the lane. For each case that was observed in the real field, it is examined whether the vehicle left the lane at the exit or not as a result of the error. Now these scenarios are reproduced in the simulation with the help of error models and it is checked whether the vehicle has left the lane in the simulation or not. For each associated point in the parameter space of the error model, one of the classifications "Event occurred in simulation and real", "Event did not occur in simulation and did not occur in real", "Event occurred in simulation and did not occur in real", "Event in simulation not occurred and actually occurred". In this case, a classification using the parameters of the error model using a meta-model makes it possible to predict the prediction accuracy of the simulation even for simulations that are not actually observed. In this case, the confusion matrix and information-theoretic quantities derived from it provide an indication of the accuracy or robustness of the simulation model for this type of investigation.

In this case, too, depending on a confidence output from the classifier, suitable points for subsequent measurements or uncertain regions in the parameter space can be identified. Again it depends on the prioritization of the user how he evaluates and prioritizes deviations between the simulation and the classification derived from the reference data.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102020205539 [0002]

Claims (13)

Computer-implemented method for testing a product, in particular software, hardware or a system comprising hardware and software, characterized by the following steps: - depending on input parameters, a simulation is carried out for the product, with which a specific property of the product is tested, - depending on a comparison of a result of the simulation with a requirement for the specific property, a first classification is issued for the result of the simulation, - depending on a comparison of reference data from an alternative test of the specific property of the product with the requirement for the specific property, a second Classification determined, - depending on the second classification, a classifier is trained, - depending on a result of the first classification, the classifier outputs a probability or confidence for an application of the first classification. Computer-implemented method for testing a product, in particular software, hardware or a system comprising hardware and software, characterized by the following steps: - depending on input parameters, a simulation is carried out for the product, with which a specific property of the product is tested, - depending on a comparison of a result of the simulation with a requirement for the specific property, a first classification is issued for the result of the simulation, - depending on a comparison of reference data from an alternative test of the specific property of the product with the requirement for the specific property, a second Classification determined - depending on the first classification and the second classification, a classifier is trained, which outputs a probability or confidence for an application of the first classification and the second classification. Method according to one of the preceding claims, characterized in that the classifier uses a Bayesian network, a Gaussian process or a committee of neural networks or a surrogate model with an associated probability or confidence measure. Method according to one of the preceding claims, characterized in that depending on the output probability or confidence that a specific classification applies, the specific classification is adopted or discarded, in particular depending on a comparison of the output probability or confidence with a predetermined threshold. Method according to one of the preceding claims, characterized in that depending on the probability or confidence output, further reference data is requested or new reference data are generated automatically, in particular depending on a comparison of the probability or confidence with a predetermined threshold. Method according to one of the preceding claims, characterized in that the reference data come from a test on the real product or from a reference simulation. Method according to one of the preceding claims, characterized in that an accuracy or robustness of the simulation is determined as a function of the first classification and the second classification. Method according to one of the preceding claims, characterized in that the first classification and the second classification each include whether the product passed or failed the test or whether the test was not carried out, in particular because conditions specified for carrying out the test were not fulfilled. Method according to one of the preceding claims, characterized in that the determined property comprises a diagnostic or detection function and the first classification and the second classification each comprise whether the diagnostic or detection function results in a false positive, a false negative, a true positive or a true negative result. Method according to one of the preceding claims, characterized in that the specific property includes a safety-critical property of the product and the first classification and the second classification each include whether a predefined, safety-relevant event has occurred or not. Method according to one of the preceding claims, characterized in that the product comprises a safety-critical software function or a safety-critical system for an at least partially automated vehicle or for an at least partially automated robot. Computer program which is set up to carry out a method according to one of the preceding claims. Test environment set up to implement a method according to one of Claims 1 until 11 with a computer program claim 12 to perform.

Images