DE102021202335A1 - Method and device for testing a technical system - Google Patents

Abstract

Method (10) for testing a technical system, in particular an at least partially autonomous robot or vehicle, characterized by the following features: - Combinations of training data and validation data are formed (1), - Samples of the training data (17) and validation data (17) are taken for each of the combinations 23) taken (2), a two-valued classifier trained with the samples (26) and probability distributions of confusion matrices calculated (3), - a number of training data (17) for a simulation of the system is determined (4), - by means of the simulation (11), tests (12) are carried out, - the tests (12) are evaluated with regard to a measure of compliance (13), a quantitative requirement for the system and an error measure (14) of the simulation (11) and - depending on the measure of compliance (13) and The error measure (14) is used to classify the tests (12) as either reliable or unreliable.

Description

The present invention relates to a method for testing a technical system. The present invention also relates to a corresponding device, a corresponding computer program and a corresponding storage medium.

State of the art

In software engineering, the use of models to automate test activities and to generate test artifacts in the test process is summarized under the generic term "model-based testing" (model-based testing, MBT). For example, the generation of test cases from models that describe the target behavior of the system to be tested is well known.

Embedded systems in particular are dependent on coherent input signals from sensors and in turn stimulate their environment through output signals to a wide variety of actuators. In the course of the verification and upstream development phases of such a system, its model (model in the loop, MiL),
Software (software in the loop, SiL), processor (processor in the loop, PiL) or entire hardware (hardware in the loop, HiL) are simulated together with a model of the environment. In automotive engineering, simulators for testing electronic control units based on this principle are sometimes referred to as component, module or integration test benches, depending on the test phase and object.

DE10303489A1 discloses such a method for testing software of a control unit of a vehicle, a power tool or a robotic system, in which a test system is used to at least partially simulate a controlled system that can be controlled by the control unit by generating output signals from the control unit and sending these output signals from the control unit to first hardware components are transmitted via a first connection and signals from second hardware components are transmitted as input signals to the control unit via a second connection, with the output signals being provided as first control values in the software and additionally via a communication interface in real time in relation to the controlled system to the test system be transmitted.

Such simulations are widespread in various fields of technology and are used, for example, to test embedded systems in power tools, engine control units for drive, steering and braking systems, camera systems, systems with artificial intelligence and machine learning components, robotic systems or autonomous vehicles in early phases to check their development for suitability. However, the results of state-of-the-art simulation models are only included in release decisions to a limited extent due to a lack of confidence in their reliability.

The at the German Patent and Trademark Office under the file number 10 2020 205 539.4 Against this background, the pending patent application proposes a “virtual test classifier”.

Disclosure of Invention

The invention provides a method for testing a technical system, a corresponding device, a corresponding computer program and a corresponding storage medium according to the independent claims.

The approach according to the invention is based on the knowledge that the quality of simulation models is crucial for the correct predictability of the test results that can be achieved with them. In the field of MBT, the sub-discipline of validation deals with the task of comparing real measurements with simulation results. Various metrics, measures or other comparators are used for this purpose, which link signals together and which are collectively referred to below as signal metrics (SM). Examples of such signal metrics are metrics that compare magnitude, phase shift, and correlations. Some signal metrics are defined by relevant standards, e.g. B. according to ISO 18571.

More generally, uncertainty quantification techniques aid in the estimation of simulation and model fidelity. The result of an assessment of model performance using a signal metric or more generally using an uncertainty quantification method for a given input X, which can be a parameter or a scenario, is hereafter referred to as the simulation model error metric - error metric for short - SMerrorX. For the generalization (interpolation and extrapolation) of SMerrorX for previously unconsidered inputs, parameters or scenarios X, machine learning models based on so-called Gaussian processes can be used.

During verification, the system under test (SUT) is typically examined against a requirement, specification or key performance indicator. It should be noted that Boolean requirements or specifications can often be converted into quantitative measurements, using formalisms such as signal temporal logic (STL). Such formalisms can serve as a basis for quantitative semantics, which is presented as a generalization of verification insofar as a positive value indicates the fulfillment and a negative value the violation of a requirement. In the following, such requirements, specifications or performance measures are collectively referred to as "quantitative requirements" (QSpec).

Such quantitative requirements can either be checked using the real SUT or a model of the same - like a "virtual SUT". For the purpose of this verification, catalogs are compiled with test cases that an SUT must pass in order to decide whether it has the desired performance and security properties. Such a test case can be parameterized and thus cover any number of individual tests.

Against this background, the proposed approach takes into account the need for reliable test results in order to guarantee the performance and security properties of an SUT. Especially when carrying out tests using a simulation of the system or a subcomponent - instead of the real system - it is important to ensure that the simulation results are trustworthy.

To this end, validation techniques are used to assess the agreement between the simulation model and real measurements. The comparison between the simulation results and the real measurements is done using the validation error metric SMerrorX, which is generalized through interpolation and extrapolation, so that the error measure for new inputs X can be predicted without making corresponding measurements. However, the prediction of SMerrorX involves an uncertainty that can be modeled as an interval or as a probability distribution.

One goal of this approach is therefore to optimally and completely split the entirety of the existing data into validation and training data for the training and validation of the classifier. It is examined how the probabilities for the entries in the so-called truth or confusion matrix change when the ratio M: K of the number M of training data sets and the number K of validation data sets is varied. In this case, the total number of records is N = K + M.

A second use case is to determine if the classifier has been trained on enough data points (convergence). In order to determine whether enough data points have been collected in total, a part of the points is omitted and it is examined how the probabilities (and quantities derived from them) change when the sum of validation data and training data is varied. In this case N > M + K.

A merit of this solution to this task is that it cleverly combines both approaches, as opposed to concepts based solely on validation or solely based on verification. For this purpose, a "virtual test classifier" is introduced, which combines the requirements of model validation and product testing. This is achieved by linking information from the validation of simulation and model quality (SMerrorX) on the one hand and test requirements (QSpec) on the other. In order to be able to evaluate the quality of the statements derived from the classifier, a robustness measure is issued. This robustness measure can be determined using established methods such as cross validation or extreme value analysis. Two possible methods for determining this robustness measure are presented below.

The application of corresponding tests can be considered in a wide variety of fields. For example, the functional safety of automated systems, such as those used to automate driving functions (automated driving), should be considered.

Advantageous further developments and improvements of the basic idea specified in the independent claim are possible as a result of the measures listed in the dependent claims. An automated, computer-implemented test environment can be provided in order to improve the quality of the tested hardware or software products largely automatically.

character list

• 1 einen virtuellen Test-Klassifikator.
• 2 einen ersten Ansatz zur Erzeugung der Entscheidungsgrenze des Klassifikators auf der Grundlage von Daten.
• 3 eine Übersicht des Ansatzes „Statistik von Statistiken“.
• 4 eine Übersicht zur Aufteilung der Grundgesamtheit in Trainingsdaten und Validierungsdaten und eine Übersicht zur Kombination der Aufteilung.
• 5 einen Verlauf zur Evaluierung eines Klassifikators innerhalb von einer Kombination des Datensatzes.
• 6 eine Übersicht vom Vorgang zur Kombination der Grundgesamtheit bis zur Zusammenfassung der Einträge von Konfusionsmatrizen in Form von Verteilungen.
• 7 eine Übersicht zur Wahrscheinlichkeit, dass die Falsch-positiv-Rate exakt I beträgt.
• 8 eine beispielhafte Aufteilung in Trainings- und Validierungssamples.
• 9 eine Kategorisierung der Validierungssamples nach dem Training.
• 10 eine diskrete Wahrscheinlichkeitsverteilung eines Validierungssamples, welches eine diskrete Wahrscheinlichkeitsverteilung der Falsch-positiv-Rate in einem zufälligen Validierungssample enthält.
• 11 eine diskrete Wahrscheinlichkeitsverteilung der jeweiligen Kategorie TP, TN, FP oder FN in Abhängigkeit von der Anzahl des Trainingssamples für s = 1 (eine Teilmenge pro Anzahl an Trainingssamples wurde zufällig aus der Grundgesamtheit ausgewählt).
• 12 eine diskrete Wahrscheinlichkeitsverteilung der jeweiligen Kategorie TP, TN, FP oder FN in Abhängigkeit von der Anzahl des Trainingssamples für r = 1 und s = 5, wobei die Mittelwerte der Mittelwerte und die Mittelwerte der 98-%-Quantile dargestellt sind.
• 13 diskrete Wahrscheinlichkeitsverteilungen der jeweiligen Kategorie TP, TN, FP oder FN in Abhängigkeit von der Anzahl des Trainingssamples in unterschiedlichen Verhältnissen von Training zu Validierungssamples und in unterschiedlichen Berechnungsmethoden für die Quantile.
• 14 bis 17 drei Konvergenzschätzer.
• 18 schematisch eine Arbeitsstation gemäß einer zweiten Ausführungsform.

Embodiments of the invention are shown in the drawings and explained in more detail in the following description. It shows:

• 1 a virtual test classifier.
• 2 a first approach to generate the classifier's decision boundary based on data.
• 3 an overview of the “statistics of statistics” approach.
• 4 an overview of how the population is split into training data and validation data, and an overview of how the split is combined.
• 5 a history for evaluating a classifier within a combination of the data set.
• 6 an overview from the process of combining the population to the summary of the entries of confusion matrices in the form of distributions.
• 7 an overview of the probability that the false positive rate is exactly I.
• 8th an exemplary division into training and validation samples.
• 9 a categorization of the validation samples after training.
• 10 a discrete probability distribution of a validation sample containing a discrete probability distribution of the false positive rate in a random validation sample.
• 11 a discrete probability distribution of the respective category TP, TN, FP or FN depending on the number of training samples for s = 1 (a subset per number of training samples was randomly selected from the population).
• 12 a discrete probability distribution of the respective category TP, TN, FP or FN depending on the number of training samples for r = 1 and s = 5, with the mean values of the mean values and the mean values of the 98% quantiles being shown.
• 13 discrete probability distributions of the respective category TP, TN, FP or FN depending on the number of training samples in different ratios of training to validation samples and in different calculation methods for the quantiles.
• 14 until 17 three convergence estimators.
• 18 schematically a workstation according to a second embodiment.

Embodiments of the invention

According to the invention, as part of a test X, which can be taken from a test catalog as a test case or obtained as an instance of a parametric test, the simulation model error SMerrorX is evaluated and the quantitative specification QSpec is evaluated on the basis of a simulation of the SUT. The virtual test classifier takes SMerrorX and QSpec as input and makes a binary decision as to whether or not to trust the test result based on the simulation.

According to the language commonly used in computer science and in particular pattern recognition, a classifier is any algorithm or any mathematical function that maps a feature space to a set of classes that were formed and differentiated from one another in the course of a classification. In order to be able to decide in which class an object is to be classified or classed (colloquially also: "classify"), the classifier uses so-called class or decision limits. If a distinction between procedure and instance is not important, the term "classifier" in the technical language and also in the following is sometimes used synonymously with "classification" or "classification".

1 illustrates such a classification in the present application example. Each point corresponds to a test that was carried out by way of simulation and for which the measure of fulfillment (13) of the QSpec requirement and the measure of error (14) SMerrorX were calculated. QSpec is defined in this case as taking a positive value if the test suggests that the system meets the requirement in question (reference number 24) and negative if the system fails the requirement (reference number 25).

As can be seen from the figure, the decision limit (19) of the classifier (18) divides the space into four classes A, B, C and D. Class A tests would be passed by the system with high reliability. For tests of classes B and C, the simulation only provides unreliable results; such tests must therefore be carried out on the real system. Class D tests would fail on the system with high confidence.

This virtual test classifier (18) is based on the consideration that a requirement that is only just met in the simulation can only replace the testing of the real system if at most a marginal model error (14) can be assumed. On the other hand, if the degree of fulfillment (13) of the quantitative requirement QSpec is high in terms of amount, i.e. a specification that is far exceeded or clearly missed, a certain deviation of the simulation results from corresponding experimental measurements can be accepted.

Since this approach requires knowledge of the model error SMerrorX of the simulation model, it is assumed that the latter was subjected to verification and validation prior to using the virtual test classifier (18). As part of this validation should - z. B. based on a Gaussian process or otherwise by machine learning - a generalized model that yields SMerrorX for a given X. It should be noted that the reliability of the simulation depends crucially on the correctness of this generalized model.

2 clarifies a possible approach to draw the decision boundary (19 - 1 ) of the classifier (18) based on data. In the simplest case, the boundary (19) runs along a straight line through the origin. The slope of the straight line should preferably be selected in such a way that all points in which the degree of fulfillment (13) of the quantitative requirement QSpec differs in sign between simulation (11) and real measurement (21) - i.e. all tests (12), so to speak, at which the simulation model fails - are in the areas C and B and these areas are also as small as possible.

A more general one, e.g. B. polynomial decision limit (19), whose function curve is adapted by means of linear programming in such a way that it meets the criterion of a classifier (18) VTC. In this case, too, all points in which the degree of fulfillment (13) of the quantitative requirement QSpec differs in sign between simulation (11) and real measurement (21) - i.e. all tests (12), so to speak, in which the simulation model fails - in areas C and B.

The virtual test classifier (hereinafter: "VTC") mentioned in the introduction, which uses the variables QSpec and SMerrorX for the partitioning of test cases, requires a certain amount of data due to its learning-based property on the one hand for the generation of boundary lines for the classification in the training phase. On the other hand, a dataset that differs from the training data is used to validate the classifier. In general, it is advantageous to provide the largest possible amount of data for training the classifier. Thus, there is a high probability that the boundary lines of a classifier will remain unchanged as more training data is added. A measure of this probability is referred to as "robustness" in the present context.

A VTC with high robustness is thus likely to provide a correct classification even when new test data is added. This applies under the assumption that the test data and training data are subject to the same or a similar distribution.

• • Bestimmung der für eine robuste Aussage erforderlichen Mindestanzahl der als Grundwahrheit (ground truth) für das Training und die Validierung dienenden Datensätze. Hierbei sollen die Wahrheitswerte die Kriterien für die Vertrauenswürdigkeit eines Tests beinhalten.
• • Bestimmung der optimalen Aufteilung in Trainingsdaten und Validierungsdaten.
• • Bestimmung der Kennzahl für die Quantifizierung der Robustheit unter Betrachtung vom Konvergenzverhalten.

In practice, however, real tests - e.g. B. for economic reasons - not be carried out in unlimited numbers. On the one hand, it is important to minimize the number of real tests and to replace them with simulation; on the other hand, the confidence in the simulation and in the correctness of the VTC is based on a sufficiently large real data set for training. The invention aims to reconcile these opposites. The following aspects are considered for this challenge:

• • Determine the minimum number of ground truth data sets required for training and validation to make a robust statement. The truth values should contain the criteria for the trustworthiness of a test.
• • Determination of the optimal split into training data and validation data.
• • Determination of the index for the quantification of the robustness considering the convergence behavior.

The virtual test classifier has a special property that sets it apart from traditional learning-based classification methods, e.g. B. in machine learning. Based on a worst-case analysis, the decision limits in the virtual test classifier are generated with the observed support points that are classified as untrustworthy based on the defined criterion. The question of the required amount of training data to build a robust classifier is linked to this peculiarity, since the existence and possible position of such support points varies depending on the partitioning of the training data from the population.

According to the invention, to clarify this question, the classifier is evaluated, so to speak, using statistics from statistics of the data sets. In other words, the relative frequency with which the elements of the confusion matrix assume certain values is evaluated; these relative frequencies are interpreted as probabilities. Ultimately, training and validation datasets in their entirety should also be viewed as a random sample from an unknown (and possibly infinitely large) population of all theoretically possible test cases.

Said statistics of statistics only come into play when the spread of the relative frequencies is examined. Based on this, robustness indicators can be defined for how many points are required in total. In addition, various variants for calculating the key figure for robustness are shown.

3 illustrates the steps (1-4) of a corresponding method (10) together with their respective parameters. First, the population divided into the training and validation data and combined the two datasets (step 1).

After the datasets for training and validation are determined by the combination, the samples are taken within the training and validation dataset. During the validation of the trained test classifier, a confusion matrix is created, which is used to quantitatively assess the error of the binary classifier (step 2).

After iterating these steps 1 and 2, a distribution of entries from confusion matrices is generated (step 3).

Finally, a suitable scope of the training data set is determined using a key performance indicator (KPI) for the robustness of the test classifier (step 4).

These steps (1-4) are explained in detail in the following sections.

In the first step (1), the existing population is divided into training data M and validation data K based on the defined number of training data M and the ratio between training data M and validation data K. As in 4 shown, the union of the split training and validation data can be smaller than the population. Then the divided data is combined. Also such a combination of the breakdown is shown on the right side of the 4 shown as an example.

From this combination (1) arise in the second step (2 - 3 ) different partitions of the population. A split takes data points from a training dataset based on the defined sample size (15 - 3 ) and thus the decision boundary lines of the test classifier are determined. After the learning process, the classifier is validated using a validation data set. In the validation, the truth values are determined from the data set and the binary classification is evaluated using them. The results of the evaluation are presented in the form of a confusion matrix. In this special contingency table, both the frequency of the correct classification by the classifier - true positive (TP), true negative (true negative (TN) - and the frequency of the incorrect classification by the classifier - false positive (false positive, FP), false negative (FN) - shown.

5 shows the course of the evaluation of a classifier within a combination (1) of the data set. On the left side, a combination case is shown from which the training data set (17) and the validation data set (23) are provided, with a remainder (28) of the population not used as the basic truth. A test classifier is taught (26) with the training data, validated (27) and then evaluated using the validation data set (29). As shown on the right side of the figure, the evaluation result is plotted in the form of a confusion matrix (31).

Then in a third step (3 - 3 ) entered the frequency of the respective entries in the confusion matrix. After the combination has been run through in the population, a distribution of the entries from the confusion matrix is produced.

6 first illustrates the process of combining the population on the left diagram. A confusion matrix is created for each combination (1) after evaluating the test classifier (29). The combination of the population leads to the large number of confusion matrices (31, 32) which are plotted in the middle representation. The entries in these confusion matrices are summarized in the form of distributions.

• In einem ersten Anwendungsfall werden die vorhandenen Daten vollständig in Validierungs- und Trainingsdaten aufgeteilt. Es wird untersucht, wie sich die Wahrscheinlichkeiten für die Einträge in der Konfusionsmatrix verändern, wenn das Verhältnis M/K variiert wird. In diesem Fall ist N = K + M.

A method is described below, according to which the probabilities for a given size of the training data set M and the validation data set K can be calculated. Convergence criteria are evaluated based on these probabilities and derived mean values and moments. N is the number of points for which both real and simulation-based testing was performed. Two types of use cases can be distinguished:

• In a first use case, the existing data is completely divided into validation and training data. It is examined how the probabilities for the entries in the confusion matrix change when the ratio M/K is varied. In this case N = K + M.

In a second use case, to determine whether enough data points have been collected, some of the points are omitted and it is examined how the probabilities (and quantities derived from them) change when the sum of validation data and training data is varied. In this case N > M + K.

• In einem ersten Schritt werden die Datenpunkte bezüglich der quantitativen Anforderung QSpec in vier Kategorien unterteilt:
  1. 1. zuverlässig erfüllt (Simulation: QSpec ≥ 0, Messung: QSpec ≥ 0),
  2. 2. vermutlich erfüllt (Simulation: QSpec ≥ 0, Messung: QSpec < 0),
  3. 3. zuverlässig verfehlt (Simulation: QSpec < 0, Messung: QSpec < 0) und
  4. 4. vermutlich verfehlt (Simulation: QSpec < 0, Messung: QSpec ≥ 0).

The procedure for the first use case is described below. For the second case, the probabilities can either be calculated analytically or the selection of the points can be varied numerically and for each choice calculates the probabilities using the following algorithm:

• In a first step, the data points are divided into four categories with regard to the quantitative requirement QSpec:
  1. 1. reliably fulfilled (simulation: QSpec ≥ 0, measurement: QSpec ≥ 0),
  2. 2. probably fulfilled (simulation: QSpec ≥ 0, measurement: QSpec < 0),
  3. 3. reliably missed (simulation: QSpec < 0, measurement: QSpec < 0) and
  4. 4. presumably missed (simulation: QSpec < 0, measurement: QSpec ≥ 0).

In a second step, the SMerrorX/QSpec ratio is calculated for each data point.

In a third step, the points for all four named sets are sorted in ascending order according to their calculated ratio SMerrorX/QSpec.

In a fourth step, the false positive and false negative rates are calculated separately for the right and left sides of the VTC. The true positive and true negative rates can be calculated in an analogous manner.

Finally, in a fifth step, the false positive and false negative rates for the right side of the VTC are calculated as follows. Here TP _sorted is a list of all points that reliably meet the requirement and UP _sorted is a list of all points that are likely to meet the requirement.

Falls I + M ≥ N gilt, beträgt die Wahrscheinlichkeit, dass der betreffende Punkt als Stützpunkt gewählt wird, null. In diesem Fall kann es keine Stichprobe geben, in der der Punkt mit dem Index I enthalten ist und es zugleich keinen Punkt mit einem niedrigeren Index und damit mit einem niedrigeren Verhältnis SMerrorX/QSpec gibt.First, the probability is calculated that the point with index I is contained in the training data set and is also the support point: $$P_1\left(I\right)=M\frac{\left(N-I\right)!\left(N-M\right)!}{N!\left(N-I-M\right)!}$$

If I + M ≥ N, the probability that the point in question will be chosen as a support point is zero. In this case, there cannot be a sample containing the point with index I and at the same time there is no point with a lower index and therefore with a lower SMerrorX/QSpec ratio.

It is possible that no point classified as unreliable in the ground truth is included in the training dataset. In this case, the point classified as reliable with the highest gradient is used as the reference point.

Falls I + M + Zahl_up > N gilt, so ist der Trainingsdatensatz M so groß, dass es nicht möglich ist, dass keine Punkte, welche die Anforderung vermutlich erfüllen, und zugleich keine Punkte, welche die Anforderung sicher erfüllen, mit höherem SMerrorX/QSpec gezogen wurden. In diesem Fall beträgt die Wahrscheinlichkeit null.For purposes of further discussion, let number _up be the total number of points that are likely to meet the requirement and number _Tp be the total number of points that reliably meet the requirement. The probability that the latter group contains the point with the index I and that this also serves as a support point is for J = number _TP + 1 - I $$P_2\left(J\right)=M\frac{\left(N-{\text{Number}}_{uP}-I\right)!\left(N-M\right)!}{N!\left(N-I-M-{\text{Number}}_{uP}\right)!}.$$

If I + M + number _up > N applies, then the training data set M is so large that it is not possible that no points that presumably meet the requirement and at the same time no points that definitely meet the requirement have a higher SMerrorX/ QSpec were drawn. In this case, the probability is zero.

Falls Zahl_TP + Zahl_UP + M > N gilt, so ist dieses Ereignis unmöglich. Die Wahrscheinlichkeit beträgt null.The probability that none of the points in the training set are likely or reliable to meet the requirement is $${\mathrm{<figure-callout\; id="3"\; label="P"\; filenames="DE102021202335A1\_0022.png"\; state="\{\{state\}\}">P</figure-callout>}}_3=M\frac{\left(N-{\text{Number}}_{uP}-{\text{Number}}_{TP}\right)!\left(N-M\right)!}{N!\left(N-M-{\text{Number}}_{TP}-{\text{Number}}_{uP}\right)!}.$$

If number _TP + number _UP + M > N, then this event is impossible. The probability is zero.

• Zunächst wird verglichen, für welche Punkte, welche die Anforderung zuverlässig erfüllen, SMerrorX/QSpec größer ist als der entsprechende Wert desjenigen Punktes mit dem Index fp, welcher die Anforderung vermutlich erfüllt, und zugleich kleiner als der entsprechende Wert desjenigen Punktes mit dem Index fp + 1, welcher die Anforderung vermutlich erfüllt. Die zugehörigen Wahrscheinlichkeiten P₂ werden addiert. Zusätzlich wird der Wert P₁(fp + 1) addiert. Dieser Summand gibt an, mit welcher Wahrscheinlichkeit der Punkt mit dem Index fp + 1 den Stützpunkt für den VTC bildet. Für den Spezialfall fp = 0 wird zusätzlich der Wert von P₃ addiert. Wenn im Trainingsdatensatz kein Punkt auf der rechten Seite enthalten war, werden alle Punkte als unzuverlässig eingestuft. In diesem Fall kann es keine falsch negativen Punkte geben.

The probability that the number of false positive points contained in the validation data set is exactly fp can be determined as follows:

• First, it is compared for which points, which reliably fulfill the requirement, SMerrorX/QSpec is larger than the corresponding value of the point with the index fp, which presumably fulfills the requirement, and at the same time smaller than the corresponding value of the point with the index fp + 1, which presumably meets the requirement. The associated probabilities P ₂ are added. In addition, the value P ₁ (fp + 1) is added. This addend indicates the probability with which the point with the index fp + 1 forms the support point for the VTC. For the special case fp = 0, the value of P ₃ is also added. If there was no point on the right side in the training dataset, all points are considered unreliable. In this case, there can be no false negative points.

As 7 clarified, the probability that the false positive rate is exactly I is equal to the sum of the probabilities of all the reliable points considered, if a point that reliably fulfills the requirement according to the basic truth (33) was selected as the reference point.

The probability that the number of false negative points contained in the validation data set is exactly fp is made up as follows: First, a comparison is made for which points which presumably meet the requirement (34 - 7 ), SMerrorX/QSpec is greater than the corresponding value of the point with the index fp, which reliably fulfills the requirement, and at the same time smaller than the corresponding value of the point with the index fn + 1, which reliably fulfills the requirement. The associated probabilities P ₁ are added. In addition, the value P ₂ (fn) is added. This addend indicates the probability with which the point with the index fn forms the support point for the VTC. For the special case fn = 0, the value of P ₃ is also added. Again, if there was no point on the right side in the training dataset, all points are considered unreliable. In this case, there can be no false negative points.

The probabilities of false positive and false negative cases for the right side of the VTC, i.e. those points which presumably or reliably fail the requirement, can be calculated analogously. The amount of SMerrorX/QSpec must be considered here.

• • richtig positiv (TP), richtig negativ (TN): Vertrauen bzw. Nicht-Vertrauen ist korrekt
• • falsch positiv (FP): Vertrauen ist inkorrekt
• • falsch negativ (FN): Nicht-Vertrauen ist inkorrekt

As an alternative to this purely analytical approach, a hybrid approach using sampling and exact calculation of probability distributions can be considered. The following interpretation is useful here: The VTC is a binary classifier that assesses whether a test performed in the context of a simulation model leads to the same result when performed in the real system (test passed or test failed). The result of the VTC can be interpreted as confidence in a simulation-based test. An established basis for evaluating binary classifiers is the truth or confusion matrix with the following entries:

• • true positive (TP), true negative (TN): trust or lack of trust is correct
• • False Positive (FP): Confidence is incorrect
• • False Negative (FN): non-confidence is incorrect

• • die Richtig-negativ-Rate $$\frac{TN}{TN+FP},$$
  
• • die Falsch-negativ Rate $$\frac{FN}{FN+TP},$$
  
• • die Akkuratheit $$\frac{TP+TN}{TP+TN+FP+FN},$$
  
• • die Präzision $$P=\frac{TP}{TP+FP},$$
  
• • die Trefferquote (recall) $$R=\frac{TP}{TP+FN},$$
  
• • das F1-Maß (F1 score) als harmonisches Mittel aus Präzision und Trefferquote $$\frac{2}{\frac{1}{P}+\frac{1}{R}},$$
  
• • der Fowlkes-Mallows-Index als geometrisches Mittel aus Präzision und Trefferquote $$\text{FMI}=\sqrt{P\cdot R}\text{ oder}$$
  
• • der Matthews-Korrelationskoeffizient $$\text{MCC}=\frac{TP\cdot TN-FP\cdot FN}{\sqrt{\left(TP+FP\right)\left(TP+FN\right)\left(TN+FP\right)\left(TN+FN\right)}}.$$
  

A number of well-established indicators can be derived from these variables, for example

• • the true negative rate $$\frac{TN}{TN+fP},$$
  
• • the false negative rate $$\frac{fN}{fN+TP},$$
  
• • the accuracy $$\frac{TP+TN}{TP+TN+fP+fN},$$
  
• • the precision $$P=\frac{TP}{TP+fP},$$
  
• • the hit rate (recall) $$R=\frac{TP}{TP+fN},$$
  
• • the F1 measure (F1 score) as the harmonic mean of precision and hit rate $$\frac{2}{\frac{1}{P}+\frac{1}{R}},$$
  
• • the Fowlkes-Mallows index as the geometric mean of precision and hit rate $$\text{FMI}=\sqrt{P\cdot R}\text{or}$$
  
• • the Matthews correlation coefficient $$\text{MCC}=\frac{TP\cdot TN-fP\cdot fN}{\sqrt{\left(TP+fP\right)\left(TP+fN\right)\left(TN+fP\right)\left(TN+fN\right)}}.$$
  

1. 1. den zugrundeliegenden Testparametern,
2. 2. dem im Wege der Messung gewonnenen Wert von QSpec,
3. 3. dem im Wege der Simulation gewonnenen Wert von QSpec und
4. 4. dem Simulationsmodellfehler SMerrorX für die gegebenen Parameter.

For the purposes of further discussion, it is assumed that each data point of the ground truth is formed from the following components:

1. 1. the underlying test parameters,
2. 2. the value of QSpec obtained through the measurement,
3. 3. the value of QSpec obtained through the simulation and
4. 4. The simulation model error SMerrorX for the given parameters.

Also given is a ratio r of training to validation samples, limits min and max for the number of training samples, the number t of different numbers of training samples and the number s of subsets per number Training samples randomly selected from the population.

The limits min and max are to be chosen in such a way that the ratio r of training samples to validation samples can be well approximated for a given size of a subset from the population. On this basis, t natural, preferably approximately equidistant numbers between the limits min and max are selected. These indicate the number of training samples - referred to below as #T - that are used in the evaluation. The number of validation samples is therefore $$\text{round}\left(\frac{\#T}{\mathrm{right}}\right).$$

For the purposes of the hybrid approach to evaluating the VTC, let G be the set of tuples (#T, #V) from the number of training and the number of validation samples, which are obtained from the parameters r, min, max and t according to the above procedure result. For each tuple (#T, #V) ∈ G, #T + #V data points are first randomly selected from the population. For all possible partitions into #T training samples and #V validation samples within this selection, the exact discrete probability distribution of TP, TN, FP and FN is determined. Finally, based on the resulting set of discrete probability distributions, the desired statistics and convergence estimators are computed.

In the 8th and 9 the approach is illustrated with an example. Here, let #T + #V = 10 and r = 1. Thus, #T = #V = 5. For a selection of 10 data points from the population, shows 8th a possible division into training data (17) and validation data (23).

einer der Kategorien TP oder TN, mit einer Wahrscheinlichkeit von $\frac{1}{5}$

der Kategorie FP und ebenfalls mit einer Wahrscheinlichkeit von $\frac{1}{5}$

der Kategorie FN zugeordnet.After training the VTC, each validation sample is assigned to the TP, TN, FP, and FN category. An example result is shown 9 . If you randomly choose a validation sample, it has a probability of $\frac{3}{5}$

one of the categories TP or TN, with a probability of $\frac{1}{5}$

of the category FP and also with a probability of $\frac{1}{5}$

assigned to the category FN.

mögliche Aufteilungen in Trainings- und Validierungssamples. Für jede der Kategorien TP, TN, FP und FN ergibt sich damit eine diskrete Wahrscheinlichkeitsverteilung unter der Annahme, dass die Aufteilungen auf eine bestimmte Art verteilt sind, z. B., dass jede Aufteilung gleichwahrscheinlich ist. 10 stellt eine diskrete Wahrscheinlichkeitsverteilung eines Validierungssamples dar, welches eine diskrete Wahrscheinlichkeitsverteilung der Falsch-positiv-Rate in einem zufälligen Validierungssample enthält.For the considered case of #T = #V = 5 there is $\left(\begin{array}{c}10\\ 5\end{array}\right)=252$

possible divisions into training and validation samples. For each of the categories TP, TN, FP and FN this results in a discrete probability distribution assuming that the distributions are distributed in a certain way, e.g. B. that every division is equally probable. 10 represents a discrete probability distribution of a validation sample, which contains a discrete probability distribution of the false positive rate in a random validation sample.

In addition, distributions of derived variables such as FMI or MCC (see formula 4) can also be determined. The special feature of the proposed method is that the discrete probability distributions can be calculated exactly and do not have to be approximated by sampling.

For this purpose, the numbers #T of the training samples and #V of the validation samples as well as a set of #T + #V data points from the population are fixed. In a first step, all samples are sorted according to the angle 1 sorted, ie the sorting is in descending order $${\text{atan}}_2\left(\text{SMerrorX}\left(\text{sample}\right),{\text{QSpec}}_{\text{sim}}\left(\text{sample}\right)\right).$$

The resulting sorted list of samples is split into two lists trust _fail (QSpeC _sim < 0) and trust _pass (QSpec _sim > 0) such that the concatenation of trust _fail and trust _pass gives the original sorted list.

In the following, all divisions of the samples into trust _fail and trust _pass in #T training and #V validation samples are considered. Furthermore, all possible VTCs are considered that are described by a maximum of one fail _sep in trust _fail and a maximum of one pass _sep in trust _pass . In doing so, one excludes those combinations that are not possible with the given values for #T and #V.

In the following, fail _sep and pass _sep are fixed, ie a possible VTC is fixed. This comes about under a certain subset of all divisions into training and validation samples. The set of samples that can appear as validation samples is divided according to the category TP, TN, FP or FN assigned to them and whether the respective sample serves as a validation sample in all cases or only in some cases. In addition, in the following n ₁ denotes the number of must-samples in a specific category X, n ₂ the number of must-samples in all remaining categories, m ₁ the number of optional samples in a specific category X, m ₂ the number of optional Samples of all remaining categories, k = #V, c _min = max(0, k - n ₁ - n ₂ - m ₂ ) and c _max = min(m ₁ , k - n ₁ - n ₂ ).

The probability P _v of randomly choosing a validation sample from class X (see previous example) and the associated probability of occurrence P _a across all partitions of #V validation samples consistent with the fixed VTC are denoted below as tuple (P _v , P _a ) shown. The following set thus represents the exact discrete probability distribution for category X: $$\left\{\left(\frac{{\mathrm{<figure-callout\; id="1"\; label="n"\; filenames="DE102021202335A1\_0022.png,DE102021202335A1\_0026.png"\; state="\{\{state\}\}">n</figure-callout>}}_1+j}{k},\frac{\left(\begin{array}{c}{m}_1\\ j\end{array}\right)\left(\begin{array}{c}{m}_2\\ k-{\mathrm{<figure-callout\; id="1"\; label="n"\; filenames="DE102021202335A1\_0022.png,DE102021202335A1\_0026.png"\; state="\{\{state\}\}">n</figure-callout>}}_1-{\mathrm{<figure-callout\; id="2"\; label="n"\; filenames="DE102021202335A1\_0022.png"\; state="\{\{state\}\}">n</figure-callout>}}_2-j\end{array}\right)}{\left(\begin{array}{c}{\mathrm{<figure-callout\; id="1"\; label="m"\; filenames="DE102021202335A1\_0022.png,DE102021202335A1\_0026.png"\; state="\{\{state\}\}">m</figure-callout>}}_1+m_2\\ k-{\mathrm{<figure-callout\; id="1"\; label="n"\; filenames="DE102021202335A1\_0022.png,DE102021202335A1\_0026.png"\; state="\{\{state\}\}">n</figure-callout>}}_1-n_2\end{array}\right)}\right)|c_{\text{at least}}\le j\le c_{\text{Max}}\right\}$$

11 represents the discrete probability distribution of the respective categories TP and TN (reference 35), FP (reference 36) and FN (37) depending on the number of training samples for s = 1 (a subset per number of training samples was randomly selected from the population) represent.

In this example, r = 1 and #T is varied. The discrete probability distribution is presented for each value of #T in the form of a 98% quantile (45, 46, and 47, respectively) around the mean (35, 36, 37).

If one chooses s > 1, several discrete probability distributions are available for each value of #T. From the discrete probability distribution of the mean values, for example, a statistic can be formed that records the mean value and variance of the mean values. From the discrete probability distribution of the lower and upper q-quantiles, e.g. For example, a statistic can be formed that illuminates the mean, variance, and upper and lower limits of the q-quantiles.

12 illuminates one of the 11 corresponding plot for r = 1 and s = 5, showing the means of the means and the means of the 98% quantile.

A comparison between r = 0.5 (left column) and r = 4 (right column) and the mean of the 98% quantiles (top row) and the upper and lower bounds of the 98% quantiles (bottom row) is shown in 13 shown.

On the basis of these sets of precisely calculated discrete probability distributions, convergence estimators can be defined which, for a given choice of r and s, assess the deviations of the s discrete probability distributions for each choice of #T. Examples of such estimators are the mean or upper and lower bounds of the FN mean values and the maximum distance of all pairs of discrete FN probability distributions. For this purpose, the distance between two distributions ƒ ₁ and ƒ ₂ can be B. be defined as follows: $$\underset{\sigma \in \mathrm{\Sigma }}{\text{Max}}\left|f_1\left(\sigma \right)-f_2\left(\sigma \right)\right|$$

In this context it should be noted that the Kullback-Leibler divergence (information gain) is undefined if ∃σ ∈ Σ: ƒ(σ) = 0.

In the lower halves of 14 until 17 all three convergence estimators are shown using the upper and lower bounds (38) and mean (39) of the FN mean values as well as the maximum distance (40) of all pairs of discrete FN probability distributions. For the the 14 and 15 then r = 0.5 and s = 20, with the range varying at #T values. For the 16 and 17 r = 4 and s = 20 are chosen.

For a given r, the distance of the probability distributions converges with increasing #T and thus also increasing #V. With the help of the convergence estimator it can be decided whether the amount of simulation and measurement data available is sufficient for a robust classification of simulation-based tests. After completing the collection of measurement data, the confidence in the classification result for simulation data that has not yet been seen can be quantified using the distributions of TP, TN, FP, FN and the indicators derived from them.

This method (10) can be implemented, for example, in software or hardware or in a mixed form of software and hardware, for example in a workstation (30), such as the schematic representation of FIG 18 clarified.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 10303489 A1 [0004]
• DE 102020205539 [0006]

Claims (13)

Method (10) for testing a technical system, in particular an at least partially autonomous robot or vehicle and mechatronic vehicle components such as a steering system, braking system and safety-critical driving support systems such as ABS and ESP, characterized by the following features: - Combinations of training data and validation data are formed (1), - for each of the combinations, samples of the training data (17) and validation data (23) are taken (2), a two-valued classifier is trained with the samples (26) and probability distributions of confusion matrices (31, 32) are calculated (3), - a number of the training data (17) for a simulation of the system (4), - tests (12) are carried out by means of the simulation (11), - a robustness measure which evaluates the trustworthiness of the classifier is calculated using the volume of training data, - the tests (12) with regard to a measure of fulfillment (13) of a quant Itative requirement for the system and an error measure (14) of the simulation (11) evaluated and - depending on the compliance measure (13) and error measure (14), the tests (12) are classified as either reliable or unreliable. Method (10) according to claim 1 , characterized by the following feature: - the robustness measure for the classifier is determined using the distribution of the training and test data. Method (10) according to claim 1 or 2 , characterized by the following features: - on the basis of the robustness measure, a decision is made as to whether the underlying totality of the data needs to be enlarged by further simulations and measurements and - if the totality of the data including the measurements is fixed, a simulation without measurements is dependent on the robustness measure Based on the robustness measure, a confidence measure is issued that evaluates how trustworthy the relevant test result is. Method (10) according to any one of Claims 1 until 3 , characterized by the following features: - the classification (15) is carried out by a multi-value classifier (18) using a feature vector (13, 14) and - the measure of compliance (13) and measure of error (14) form components of the feature vector (13, 14). Method (10) according to any one of Claims 1 until 4 , characterized by the following features: - the multi-value classifier (18) forms the feature vector (13, 14) on a case-by-case basis to a first class (A) with a positive degree of compliance (13) in the simulation (11) and the tests (12) on the system, a second class (B) with a positive degree of fulfillment (13) in the simulation (11) and a negative degree of fulfillment (13) in the tests (12) on the system, a third class (C) with a negative degree of fulfillment (13) in the simulation (11 ) and a positive degree of fulfillment (13) in the tests (12) on the system or a fourth class (D) with a negative degree of fulfillment (13) in the simulation (11) and the tests (12) on the system and - the classification takes place within specified Decision limits (19) between the classes (A, B, C, D). Method (10) according to claim 5 , characterized by the following features: - in a preparatory phase (20), the simulation (11) is confirmed by experimental measurement (21) on the system, - the decision limits (19) are drawn in such a way that on the one hand in the simulation (11) and on the other the degree of compliance (13) taken in the measurement (21) deviates as little as possible and - preferably further tests (12) to be carried out in the preparation phase (20) are automatically selected (22). Method (10) according to claim 5 , characterized by the following features: - the multi-value classifier (18) is defined by solving a system of equations and - the system of equations includes definition equations of the measure of compliance (13) and measure of error (14). Method (10) according to any one of Claims 1 until 7 , characterized by the following feature: - the evaluation is carried out in such a way that the degree of fulfillment (13) is positive if the system meets the requirement (24) and negative if the system fails to meet the requirement (25). Method (10) according to any one of Claims 1 until 8th , characterized by the following feature: - for certain parameters of the tests (12), the measure of fulfillment (13) and measure of error (14) are each represented in a feature space spanned by the parameters and - after the evaluation, the classification (15) is visualized in the feature space . Method (10) according to any one of Claims 1 until 9 , characterized in that errors in the system detected by the checking are automatically corrected. Computer program which is set up, the method (10) according to one of Claims 1 until 10 to execute. Machine-readable storage medium on which the computer program claim 11 is saved. Device (30) which is set up, the method (10) according to one of Claims 1 until 10 to execute.

Images