DE202021105594U1 - System for the automatic selection and/or evaluation of at least one AI module and for the classification and/or regression of input data, a computer-readable storage medium and a system for the classification and/or regression of input data - Google Patents

Abstract

System, in particular a device (300), for executing a method (100) for the automatic selection and/or evaluation of at least one AI module (14), comprising:
a. a receiving unit (310) designed to receive input data (11), the input data (11)
◯ Undisturbed data (12) with at least one undisturbed data element (13),
◯ at least one AI module (14) designed for classification and/or regression,
◯ at least one assessment metric (15), and
◯ at least one method for generating disturbances (16, 16') with at least one disturbance parameter (17); specify;
b. a configuration unit (320) which is designed to set a fault parameter value (18) for the at least one to define disturbance parameters (17) and a stop criterion (21), and
c. a computing unit (330), which is designed to calculate deviations (31) by calculating the at least one assessment metric (15) using the undisturbed data (12), the at least one AI module (14) and the at least one method for generating errors (16, 16');
i.e. a robustness determination unit (340) which is designed to determine at least one robustness value (41) for the at least one AI module (14) using the calculated deviations (31);
e. a comparison unit (350) which is designed to compare the at least one robustness value (41) with at least one threshold value (51); and
f. a selection unit (360), which is designed to select the at least one AI module (14) if the robustness value (41) is greater than or equal to the at least one threshold value (51) and is designed to use the computing unit (330) to activate using the configuration unit (320) if the robustness value (41) is less than or equal to the at least one threshold value (51) or to output an indication if the stop criterion (21) is met.

Description

The invention relates to a system for the automatic selection and/or evaluation of at least one AI module and for the classification and/or regression of input data, a computer-readable storage medium and a system for the classification and/or regression of input data.

Methods of machine learning and artificial intelligence (AI models) are now part of the process chain in a large number of economic sectors. Such methods are taking on more and more core tasks. In particular, methods of artificial intelligence are assigned tasks that are safety-critical. For example, in autonomous driving, it is necessary for algorithms to analyze the vehicle's surroundings and issue appropriate control instructions to the vehicle, taking into account the current driving situation.

Many of the artificial intelligence and machine learning methods used are first trained with training data. The parameters of the models or algorithms are set by the training. Due to the complexity of the models, i. H. Due to the number of parameters to be set, it is not possible for a person to predict the behavior of the trained models in practical applications. The models thus work according to a so-called black box principle.

In order to select appropriate models or algorithms or to release them for practical use, it is therefore necessary to assess the quality of the models or algorithms. This is particularly necessary in connection with certification. For liability reasons alone, it is essential that algorithms that work in security-critical areas are objectively certified.

One aspect that is of particular interest with regard to certification is the assessment of the robustness of an AI model. The robustness describes the ability of an AI model to work with incorrect, unknown or adversarial (manipulated) input data. For example, with a high-quality AI model, the output should not be changed by a slight noise in the input data. For example, the lens of a vehicle's camera might be covered with light dirt or a film of dust. This dirt or film of dust should not cause objects within the image to be classified differently.

The assessment of the robustness of an AI model has so far only been insufficiently solved. Existing methods have the disadvantage that no systematic and comparable assessment of robustness is carried out. In addition, a significant problem is that the number of cases and input parameters to be calculated can be very large. This means that, on the one hand, large processor resources are required to carry out the necessary calculations and, on the other hand, a large amount of memory is required to store the results obtained and evaluate them accordingly.

It is also necessary to be able to compare several AI models with each other. This can be done in particular taking into account the robustness. This makes it possible to select the best model with regard to specified quality criteria from a large number of different AI models.

AI models are typically implemented by AI modules, with an AI module then specifying a collection of software programs that implement the AI model.

It is therefore the object of the invention to select an AI module taking its robustness into account. It is in particular the object of the invention to specify an efficient process for evaluating the robustness. It is also a particular object of the invention to create a way of comparing different AI modules with one another.

The object is solved by a system according to claim 1, a computer-readable storage medium according to claim 9 and a system according to claim 18.

1. a. Eine Empfangseinheit (310), die dazu ausgebildet ist, Eingabedaten zu empfangen, wobei die Eingabedaten
   • ◯ Ungestörte Daten mit mindestens einem ungestörten Datenelement,
   • ◯ mindestens ein zur Klassifikation und/oder Regression ausgebildetes KI-Modul,
   • ◯ mindestens eine Beurteilungsmetrik, und
   • ◯ mindestens eine Methode zum Generieren von Störungen mit mindestens einem Störungsparameter;
   angeben;
2. b. Eine Konfigurationseinheit (320), die dazu ausgebildet ist, einen Störungsparameterwert für den mindesten einen Störungsparameter und eines Stoppkriteriums festzulegen, und
3. c. Eine Recheneinheit (330), die dazu ausgebildet ist, Abweichungen durch Berechnen der mindestens einen Beurteilungsmetrik unter Verwendung der ungestörten Daten, des mindestens einen KI-Moduls und der mindestens einen Methode zum Generieren von Störungen zu bestimmen;
4. d. Eine Robustheitsbestimmungseinheit (340), die dazu ausgebildet ist, mindestens einen Robustheitswert für das mindestens eine KI-Modul unter Verwendung der berechneten Abweichungen zu bestimmen;
5. e. Eine Vergleichseinheit (350), die dazu ausgebildet ist, den mindestens einen Robustheitswertes mit mindestens einem Schwellwert zu vergleichen und
6. f. Eine Auswahleinheit (360), die dazu ausgebildet ist, das mindestens eine KI-Modul auszuwählen, wenn der Robustheitswert größer gleich dem mindestens einen Schwellwert ist und dazu ausgebildet ist, die Recheneinheit (330) unter Verwendung der Konfigurationseinheit (320) zu aktivieren, wenn der Robustheitswert kleiner gleich dem mindestens einem Schwellwert ist oder einen Hinweis auszugeben, wenn das Stoppkriterium (21) erfüllt ist.

The object is achieved in particular by a system, in particular a device, for the automatic selection and/or evaluation of at least one AI module, comprising:

1. a. A receiving unit (310) which is designed to receive input data, the input data
   • ◯ Undisturbed data with at least one undisturbed data element,
   • ◯ at least one AI module designed for classification and/or regression,
   • ◯ at least one assessment metric, and
   • ◯ at least one method for generating disturbances with at least one disturbance parameter;
   specify;
2. b. A configuration unit (320) which is designed to define a fault parameter value for the at least one fault parameter and a stop criterion, and
3. c. A computing unit (330) configured to determine deviations by calculating the at least one assessment metric using the undisturbed data, the at least one AI module and the at least one method for generating disturbances;
4. i.e. A robustness determination unit (340) which is designed to determine at least one robustness value for the at least one AI module using the calculated deviations;
5. e. A comparison unit (350) which is designed to compare the at least one robustness value with at least one threshold value and
6. f. A selection unit (360), which is designed to select the at least one AI module if the robustness value is greater than or equal to the at least one threshold value and is designed to activate the computing unit (330) using the configuration unit (320). , if the robustness value is less than or equal to the at least one threshold value, or to output a message if the stop criterion (21) is met.

A core of the invention is that an AI module is checked for robustness until it either no longer meets specified quality criteria or a stop criterion is met. The stop criterion can be, for example, an indication that the AI module is sufficiently robust and further testing is no longer necessary. This check is implemented by the selection unit, the comparison of the robustness value with the threshold value (by the comparison unit) being used as the basis for checking sufficient robustness. Provision can thus be made for the AI module to be checked again with a different parameter if the robustness of the AI module is below the threshold value but the stop criterion is not met. For this purpose, the assessment metrics used, the methods used to generate faults or the associated fault parameters can be changed.

It is conceivable, for example, that an AI module is designed to be robust with regard to a method for generating disturbances, which describes a rotation of an image, for example, but not with regard to a further method for generating disturbances, which indicates noise, for example. However, a high-quality AI module should be designed to be robust with regard to a large number of different faults or methods for generating faults. Since the receiving unit, the configuration unit, the arithmetic unit, the robustness determination unit, the comparison unit and/or the selection unit are each designed to perform their functions sequentially or with feedback, unnecessary calculations are prevented. If it is already determined during a first run that an AI module does not have the necessary robustness with regard to a single type of fault or method for generating faults, then further calculations can be dispensed with, since the AI module as a whole thus has a insufficient robustness.

The undisturbed data can be image data, audio data and/or text data, for example. Image data in particular can be images in chronological order, i. H. trade video data. "Undisturbed" in this context means that no artificial noise has been added to the data. Undisturbed describes the state of the data at a point in time before disturbed data is generated using the method for generating disturbances.

An AI module can be an artificial neural network, for example. However, AI modules are also conceivable, which are designed, for example, as a Bayesian network and/or as a k-nearest-neighbor algorithm.

An assessment metric can indicate the distance of two data items in a vector space. In this case, for example, the distance between a data element of the undisturbed data and a data element of disturbed data can be specified by an assessment metric. It is also possible that an assessment metric indicates the distance of data items indicated by an AI module, e.g. B. by applying the AI module to disturbed and undisturbed data. Thus, the assessment metric can indicate the spacing of data items in a vector space specified by the AI engine.

In one embodiment, the at least one assessment metric may indicate the accuracy of a classification. The undisturbed data can be designed as image data and the at least one AI module can be designed to assign a class to each pixel of an image. The at least one assessment metric may indicate the percentage of pixels whose classification has not changed due to added noise.

In one embodiment, the at least one method for generating perturbations can be embodied as a rotation and/or normal distribution, wherein the at least one perturbation parameter value can indicate an angular magnitude and/or variance. In one embodiment, it is possible for noisy data to be generated by sampling over a parameter space.

A method for generating disturbances can be implemented as a rotation. Using rotation as a method for generating perturbations is particularly advantageous when the undistorted data is image data. A robust AI engine should be able to provide rotationally invariant correct results. A variety of noisy data can be generated using the angular magnitude. In one embodiment, 3° can be specified as the angular size, so that interference for the range from 0° to 360° is generated at 3° intervals.

In one embodiment, the at least one stop criterion can indicate a maximum number of undisturbed data elements, a maximum number of disturbed data elements, a maximum integral value over the deviations and/or the presence of predefined patterns, in particular in the deviations.

The at least one stop criterion can indicate when an AI module has sufficient robustness and can be selected accordingly for further use. This ensures that the selected AI module is sufficiently robust. This is the case, for example, if a sufficient number of data elements has shown that the AI module is robust. Another possibility is to sum the deviations, i. H. form an integral and determine whether the sum or integral is greater than a maximum integral value. In addition, it is conceivable to check whether predefined patterns can be recognized in the deviations. In one embodiment, this pattern recognition can be carried out with a further AI module.

In one embodiment, the configuration unit can also be designed to define the disruption parameter value and the stop criterion by specifying a generation strategy for generating disrupted data. For example, the generation strategy could specify that noisy data is generated by randomly sampling a noise distribution, which may be specified by the noise generation method. It is also possible that noisy data are generated by a grid search (English: "grid search"), in which noisy data are generated for certain parameter ranges.

In one embodiment, the input data can indicate at least two methods for generating disturbances. The configuration unit, the arithmetic unit, the robustness determination unit, the comparison unit and the selection unit can be designed accordingly to process the two methods for generating faults according to their respective function.

It is therefore also conceivable that an AI module can be tested for robustness for a large number of methods for generating faults. In particular, one embodiment provides that the configuration unit, the computing unit, the robustness determination unit, the comparison unit and/or the selection unit are each designed to carry out their functions in a first processing pass, taking into account or using the first method for generating faults and perform their functions in a second processing pass, taking into account or using a second method for generating disturbances.

In this case, the selection unit can be designed to check after each run whether the threshold value has been undercut or exceeded, or whether the stop criterion has been met. Of course, the units can perform their function for more than two methods of generating disturbances. In addition, it is also conceivable that a single method for generating faults is parameterized differently in each run. It is also conceivable that different assessment metrics are used to determine the deviations in different runs. It is thus conceivable to use a first assessment metric in a first processing pass and to use a second assessment metric in a second processing pass.

Overall, a large number of different situations can be checked by setting a few parameters, so that the robustness of the AI module can be comprehensively assessed.

In one embodiment, the configuration unit can be designed to specify a sequence of at least two methods for generating faults, wherein the sequence can be specified taking into account the calculation complexity for each method for generating faults.

As a result of the procedure, the sequence of checking different methods for generating faults in different processing runs can also be defined automatically by the configuration unit. The order can be determined taking into account the computational complexity of the methods for generating disturbances. The calculation complexity can be specified, for example, by the number of parameters of the respective method for generating disturbances (“random parameters”).

In one embodiment, the at least two methods for generating interference can be uncorrelated. The effectiveness of a processing run can be maximized if the methods used to generate interference are designed to be uncorrelated or generate interference uncorrelated with one another. This means that the robustness is checked with regard to different criteria in each run. In this context, uncorrelated can mean that there is no statistical connection between the faults generated by the respective methods for generating faults.

In one embodiment, it is possible, for example, for the at least two methods for generating disturbances to have a minimum distance with regard to a measure of the difference in probability distributions. For example, the at least two methods for generating disturbances using the Kullback-Leibler divergence and/or the Wasserstein metric can be compared with one another or their distance from one another can be determined. If the first and second methods of generating perturbations are sufficiently different from each other, then they can be said to be uncorrelated.

In one embodiment, the distance of the at least two methods for generating disturbances can be compared with a minimum distance, with the run only being continued if the at least two methods for generating disturbances are further than the minimum distance with regard to the measure used for the difference are spaced apart.

In one embodiment, the threshold value can be embodied as a robustness value of a second AI module.

In the embodiment described above, a robustness value of an AI module can be applied as a threshold value within the described process. This makes it possible to compare different AI modules with each other and select the highest-quality AI module. It is conceivable, for example, in one embodiment that the receiving unit, the configuration unit, the computing unit, the robustness determination unit, the comparison unit and/or the selection unit are each designed to carry out their respective function initially for a first AI module and then for a second AI -Execute module. The determined robustness value of the first AI module can be used as a threshold value for the second AI module. The comparison unit can thus be designed to carry out a selection of an AI module by comparing the robustness values of the first and the second AI module.

In one embodiment, in particular, the configuration unit can be designed to select the undisturbed data at least partially by sampling a distribution function.

In addition to the possibility of receiving the undisturbed data directly, it is also possible to sample the undisturbed data from a distribution function. In this case, the parameters of the distribution function can be filled with preferably randomly selected values in order to generate undisturbed data in this way.

• - eine Auswahleinheit zum automatisierten Auswählen eines KI-Moduls, insbesondere unter Verwendung eines vorgegebenen Ziel-Robustheitswertes, der z. B. durch das System wie vorstehend beschrieben als Schwellwert verwendet wird;
• - eine Trainingseinheit, die dazu ausgebildet ist das ausgewählte KI-Modul mit Trainingsdaten zu trainieren;
• - eine Empfangseinheit, die zum Empfangen von Eingangsdaten ausgebildet ist;
• - eine Ausführungseinheit, die zum Ausführen einer Klassifikation und/oder Regression für die Eingangsdaten unter Verwendung des ausgewählten KI-Moduls ausgebildet ist.

The object is also achieved in particular by a system for the classification and/or regression of input data, having the following:

• - a selection unit for the automated selection of an AI module, in particular using a predetermined target robustness value, the z. B. used as a threshold by the system as described above;
• - a training unit that is designed to train the selected AI module with training data;
• - a receiving unit, which is designed to receive input data;
• - an execution unit which is designed to carry out a classification and/or regression for the input data using the selected AI module.

With the system described above it is possible to use a selected AI module for classification and/or regression. This makes it possible to ensure a high quality of the classification or regression.

The input data and/or training data can be chosen in such a way that they are of an identical type as the noisy or noisy data. For example, the input data and/or training data can be in the form of audio data if the disturbed or undisturbed data are also in the form of audio data. The input data and/or training data and the disturbed or undisturbed data can therefore be identical with regard to a large number of features. In particular, the input data and/or training data and the disturbed or undisturbed data can have at least a similar statistical behavior and/or have the same file format.

1. a. Empfangen von Eingabedaten, wobei die Eingabedaten
   • ◯ Ungestörte Daten mit mindestens einem ungestörten Datenelement,
   • ◯ mindestens ein zur Klassifikation und/oder Regression ausgebildetes KI-Modul,
   • ◯ mindestens eine Beurteilungsmetrik, und
   • ◯ mindestens eine Methode zum Generieren von Störungen mit mindestens einem Störungsparameter;
   angeben;
2. b. Festlegen eines Störungsparameterwertes für den mindestens einen Störungsparameter und eines Stoppkriteriums, und
3. c. Bestimmen von Abweichungen durch Berechnen der mindestens einen Beurteilungsmetrik unter Verwendung der ungestörten Daten, des mindestens einen KI-Moduls und der mindestens einen Methode zum Generieren von Störungen;
4. d. Bestimmen mindestens eines Robustheitswertes für das mindestens eine KI-Modul unter Verwendung der berechneten Abweichungen;
5. e. Vergleichen des mindestens einen Robustheitswertes mit mindestens einem Schwellwert; und
6. f. Auswählen des mindestens einen KI-Moduls, wenn der Robustheitswert größer gleich dem mindestens einen Schwellwert ist oder Wiederholen der Schritte b bis f, wenn der Robustheitswert kleiner gleich dem mindestens einen Schwellwert ist oder Abbrechen des Verfahrens, wenn das Stoppkriterium erfüllt ist.

The object is also achieved in particular by a computer-readable storage medium that contains instructions that cause at least one processor to implement a method, the method having the following steps:

1. a. Receiving input data, the input data
   • ◯ Undisturbed data with at least one undisturbed data element,
   • ◯ at least one AI module designed for classification and/or regression,
   • ◯ at least one assessment metric, and
   • ◯ at least one method for generating disturbances with at least one disturbance parameter;
   specify;
2. b. Defining a disturbance parameter value for the at least one disturbance parameter and a stopping criterion, and
3. c. determining deviations by calculating the at least one assessment metric using the undistorted data, the at least one AI module, and the at least one method for generating disturbances;
4. i.e. determining at least one robustness value for the at least one AI module using the calculated deviations;
5. e. comparing the at least one robustness value to at least one threshold value; and
6. f. Selecting the at least one AI module if the robustness value is greater than or equal to the at least one threshold value or repeating steps b to f if the robustness value is less than or equal to the at least one threshold value or aborting the method if the stop criterion is met.

Similar or identical advantages result as have already been described in connection with the system described above.

Further embodiments emerge from the dependent claims.

• 1: ein Flussdiagramm eines Verfahrens zur Auswahl eines KI-Moduls;
• 2: eine schematische Darstellung von Eingabedaten;
• 3: ein Flussdiagramm eines Verfahrens zur Klassifikation und/oder Regression; und
• 4: eine schematische Darstellung einer Vorrichtung zur Auswahl eines KI-Moduls.

The invention is explained in more detail below using exemplary embodiments. show:

• 1 : a flowchart of a method for selecting an AI module;
• 2 : a schematic representation of input data;
• 3 : a flow chart of a method for classification and/or regression; and
• 4 : a schematic representation of a device for selecting an AI module.

In the following, the same reference numerals are used for parts that are the same or have the same effect.

the 1 shows a flow chart that illustrates the sequence of a method 100 for selecting an AI module 14 . In a first reception step 10, input data 11 are received. The input data 11 can be received either via a user interface or the input data 11 can be received via an Internet connection or another communication network. It is also conceivable that the input data 11 is read out from a memory.

The input data 11 are further processed in a determination step 20 . In the determination step 20, at least one disturbance parameter 17 (see 2 ) is assigned a fault parameter value of 18. The disturbance parameter value 18 can be specified by a user, for example. It is also conceivable that the fault parameter value 18 is read out from a memory. At least one method specified by the input data 11 for generating faults 16, 16' is parameterized by the fault parameter value 18.

In addition, a stop criterion 21 is defined in the definition step 20 . The stop criterion 21 defines when the method 100 is to be ended. The stop criterion 21 can also be received by a user input, a communication network or an Internet connection or can be read from a memory.

In the determination step 30, deviations 31 are determined by calculating at least one assessment metric 15 contained in the input data 11. The calculation is carried out taking into account undisturbed data 12 specified in the input data 11, the at least one method for generating disturbances 16, 16' and an AI module 14.

Following the determination step 30, in a further determination step 40, at least one robustness value 41 for the AI module 14 is determined using the calculated deviations 31.

The determined robustness value 41 is compared with a threshold value 51 in a subsequent comparison step 50 . The threshold value 51 can either be specified by a user or read out from a memory. In particular, the threshold value 51 can be embodied as the robustness value of a second AI module. In the comparison step 50 it can either be checked—depending on the selected threshold value 51—whether the determined robustness value 41 is greater than or equal to the threshold value 51 .

If it is determined during the comparison that the robustness value 41 satisfies the comparison criterion, i. H. is greater than or equal to or less than or equal to the threshold value 51, then the method 100 is continued in the determination step 20. When the determination step 20 is carried out a second time, the method for generating faults 16 is parameterized differently or a second method for generating faults 16′ is used, which can also be specified by the input data. In addition, it is also conceivable for other undisturbed data 12 to be used. It is also conceivable that a different stop criterion 21 is assumed when the determination step 20 is carried out a second time.

The comparison of the comparison step 50 thus compares the robustness of the AI module 14 with a reference value, ie the threshold value. In addition, it is checked whether the AI module 14 meets a stop criterion 21 . In the exemplary embodiment shown, it can be provided, for example, that the robustness of the AI module 14 has been checked with regard to a sufficient number of disturbed data or undisturbed data.

If stop criterion 21 is not met and method 100 is not continued in determination step 20 , AI module 14 is selected in selection step 60 .

the 2 shows a schematic representation of the input data 11. The input data 11 can be provided by a user. For example, the user can enter the input data 11 via an input device such as a keyboard. In addition or as an alternative, it is also possible for the input data 11 to be received via a communication network. A programming interface (API) can be provided for this purpose, for example.

The input data 11 indicate the undisturbed data 12, the at least one AI module 14, the at least one assessment metric 15 and the at least one method for generating disturbances 16, 16'. The undisturbed data 12 comprises at least one data element 13. A data element 13 can indicate, for example, an image, an audio or a text file. In the exemplary embodiment shown, the at least one AI module 14 is designed as an artificial neural network. The assessment metric 15 is designed to determine deviations 31, taking into account the undisturbed data 12, the at least one AI module 14 and the at least one method for generating disturbances 16, 16'. The deviations 31 can each indicate how robust the AI module 14 works with regard to the undisturbed data 12 and the at least one method for generating disturbances 16, 16'. This means that the deviations 31 indicate how the AI module 14 behaves when there are disturbances in the data to be processed.

The at least one method for generating faults 16, 16' is parameterized by at least one fault parameter 17. In some exemplary embodiments, it is also conceivable for the method for generating faults 16, 16' to have no fault parameter 17. Therefore, the disturbance parameter 17 is to be regarded as optional. By changing a disturbance parameter value 18 of the disturbance parameter 17, the behavior of the method for generating disturbances 16, 16' can be changed. In the exemplary embodiment shown, the at least one method for generating disturbances 16, 16' describes a rotation and the disturbance parameter 17 describes an angular size. A large number of disturbed data can thus be generated for each data element 13 of the undisturbed data 12 by applying the at least one method for generating disturbances 16, 16'. The corrupted data includes a plurality of corrupted data items. In the example of a rotation, the corrupted data accordingly includes images that are rotated in relation to the at least one data element 13 of the undisturbed data 12 . The perturbation parameter 17 specifies the angular distances at which the rotated images are to be generated. For example, if the perturbation parameter 17 is assigned a perturbation parameter value 18 of 3°, then for each Data element 13 of the undisturbed data 12 generates a large number of disturbed data elements which are each rotated by at least 3° in relation to one another. Thus, a first corrupted data element will be rotated by exactly 3° relative to the at least one undisturbed data element 13 . A second corrupted data element is rotated by 6° compared to the undisturbed data element 13, etc.

In the 2 two methods for generating disturbances 16, 16' are indicated. The first method for generating faults 16 can be used in a first run of the method 100 or processing run, and the second method for generating faults 16′ can be used in a second run of the method or processing run. Of course, more than two methods for generating disturbances 16, 16' are also conceivable, each of which is used in a method run or processing run.

the 3 shows a flow chart for a method 200 for the classification or regression of input data 231 by an AI module 14. In a first selection step 210, an AI module 14 is selected by executing the method 100 of the 1 selected. A target robustness value 201 is specified, which the selected AI module 14 should at least have. This procedure can be viewed as a type of certification of the AI module 14 if the target robustness value 201 is specified by a standardized norm, for example. The target robustness value 201 can be derived from the method 100 of 1 can be used as a threshold value 51 in order to make the selection of the AI module 14 .

In the subsequent training step 220 the AI module 14 is trained with training data 221 . After the training is complete, the trained AI module 14 can be used. For this purpose, input data 231 is received in reception step 230 for which a classification or regression is carried out in execution step 240 . The selected AI module 14 can thus be used in a large number of different scenarios. For example, in autonomous driving, it is necessary to classify objects near a vehicle and segment images accordingly. The security of the system can be ensured by using AI modules 14 that have tested robustness.

the 4 shows an apparatus or a system 300, which is designed to carry out the method 100 of FIG 1 to implement. For this purpose, the device or the system 300 has a receiving unit 310 which is designed to receive input data 11 . A configuration unit 320 is designed to configure the fault parameter 17 of the at least one method specified by the input data 11 for generating faults 16, 16'. In addition, the configuration unit 320 is designed to define a stop criterion 21 . A computing unit 330 is designed to determine deviations by calculating the at least one assessment metric 15 comprised by the input data 11, taking into account the undisturbed data 12, the AI module 14 and the at least one method for generating disturbances 16, 16'.

The device 300 also includes a robustness determination unit 340 which is designed to determine at least one robustness value 41 for the AI module 14 using the calculated deviations 31 . A comparison unit 350 of device 300 is designed to compare the at least one robustness value 41 with a threshold value 51 .

Device 300 also includes a selection unit 360, which is designed to select the at least one AI module 14 when robustness value 41 is greater than or equal to at least one threshold value 51 and is designed to activate computing unit 330 using configuration unit 320 activate if the robustness value 41 is less than or equal to the at least one threshold value 51 or to output a message if the stop criterion 21 is met.

The selection unit 360, the comparison unit 350, the robustness determination unit 340 and the computing unit 330 thus form a feedback loop, so that an optimal AI module 14 is selected. The selection unit 360 can also be designed to activate the configuration unit 320, the configuration unit 320 then being designed to parameterize the method for generating faults 16 differently and/or a second method for generating faults 16' when it is activated again. to use.

At this point it should be pointed out that all parts described above are individually - even without additional features described in the respective context, even if they have not been explicitly identified as optional features in the respective context, e.g. B. by using: in particular, preferably, for example, e.g. B., if necessary, round brackets, etc. - and in combination or any sub-combination are to be regarded as independent configurations or developments of the invention, as defined in particular in the introduction to the description and the claims. deviations from this are possible. Specifically, it should be pointed out that the word "in particular" or round brackets do not denote any characteristics that are mandatory in the respective context.

Reference List

10

receive step

11

input data

12

Undisturbed data

13

data item

14

AI module

15

assessment metric

16, 16'

Method for generating disturbances

17

disturbance parameters

18

fault parameter value

20

determination step

21

stopping criterion

30

determination step

31

deviations

40

determination step

41

robustness score

50

comparison step

51

threshold

60

selection step

100

procedure

200

procedure

201

target robustness value

210

selection step

220

training step

221

training data

230

receive step

231

input data

240

execution step

300

contraption

310

receiving unit

320

configuration unit

330

unit of account

340

robustness determination unit

350

comparison unit

360

selection unit

Claims (18)

System, in particular a device (300), for executing a method (100) for the automatic selection and/or evaluation of at least one AI module (14), comprising: a. a receiving unit (310) designed to receive input data (11), the input data (11) ◯ Undisturbed data (12) with at least one undisturbed data element (13), ◯ at least one AI module (14) designed for classification and/or regression, ◯ at least one assessment metric (15), and ◯ at least one method for generating disturbances (16, 16') with at least one disturbance parameter (17); specify; b. a configuration unit (320) which is designed to define a fault parameter value (18) for the at least one fault parameter (17) and a stop criterion (21), and c. a computing unit (330), which is designed to calculate deviations (31) by calculating the at least one assessment metric (15) using the undisturbed data (12), the at least one AI module (14) and the at least one method for generating errors (16, 16'); i.e. a robustness determination unit (340) which is designed to determine at least one robustness value (41) for the at least one AI module (14) using the calculated deviations (31); e. a comparison unit (350) which is designed to compare the at least one robustness value (41) with at least one threshold value (51); and f. a selection unit (360), which is designed to select the at least one AI module (14) if the robustness value (41) is greater than or equal to the at least one threshold value (51) and is designed to use the computing unit (330) to activate using the configuration unit (320) if the robustness value (41) is less than or equal to the at least one threshold value (51) or to output an indication if the stop criterion (21) is met. system (300) after claim 1 , characterized in that the at least one method for generating disturbances (16, 16') is in the form of a rotation and/or normal distribution, the at least one disturbance parameter value (17) indicating an angular size and/or variance. System (300) according to any one of the preceding claims, characterized in that the at least one stop criterion (21) a maximum number of undisturbed data elements, a maximum number of disturbed data elements, a maximum integral value over the deviations and/or the presence of predefined patterns, in particular in the deviations (31). System (300) according to any one of the preceding claims, characterized in that the input data specify at least two methods for generating faults (16, 16'), the configuration unit, the computing unit, the robustness determination unit, the comparison unit and/or the selection unit each being associated therewith are designed to process the two methods of generating disturbances according to their respective functions. System (300) according to any one of the preceding claims, in particular claim 4 , characterized in that the configuration unit (320) is further designed to define a sequence for at least two methods for generating faults (16, 16 '), wherein the configuration unit (320) is designed to define taking into account a calculation complexity for to carry out any method for generating disturbances (16, 16'). System (300) according to any one of the preceding claims, in particular claim 5 , characterized in that the at least two methods for generating disturbances (16, 16') are designed to be uncorrelated. System (300) according to one of the preceding claims, characterized in that the threshold value (51) is in the form of a robustness value (41') of a second AI module (14'). A system (300) as claimed in any preceding claim, characterized in that the input data includes the undistorted data selected (80) by sampling a distribution function (81). System according to one of the preceding claims, in particular according to claim 5 , characterized in that the threshold value (51) is designed as a robustness value (41) of a second AI module (14). Computer-readable storage medium containing instructions that cause at least one processor to implement a method when the instructions are executed by the at least one processor, the method being a method for automatically selecting and/or evaluating at least one AI module (14) is, comprising: a. a receiving unit (310) designed to receive input data (11), the input data (11) ◯ Undisturbed data (12) with at least one undisturbed data element (13), ◯ at least one AI module (14) designed for classification and/or regression, ◯ at least one assessment metric (15), and ◯ at least one method for generating disturbances (16, 16') with at least one disturbance parameter (17); specify; b. a configuration unit (320) which is designed to define a fault parameter value (18) for the at least one fault parameter (17) and a stop criterion (21), and c. a computing unit (330), which is designed to calculate deviations (31) by calculating the at least one assessment metric (15) using the undisturbed data (12), the at least one AI module (14) and the at least one method for generating errors (16, 16'); i.e. a robustness determination unit (340) which is designed to determine at least one robustness value (41) for the at least one AI module (14) using the calculated deviations (31); e. a comparison unit (350) which is designed to compare the at least one robustness value (41) with at least one threshold value (51); and f. a selection unit (360), which is designed to select the at least one AI module (14) if the robustness value (41) is greater than or equal to the at least one threshold value (51) and is designed to use the computing unit (330) to activate using the configuration unit (320) if the robustness value (41) is less than or equal to the at least one threshold value (51) or to output an indication if the stop criterion (21) is met. computer-readable storage medium claim 10 , characterized in that the at least one method for generating disturbances (16, 16') is in the form of a rotation and/or normal distribution, the at least one disturbance parameter value (17) indicating an angular size and/or variance. Computer-readable storage medium according to any one of Claims 10 until 11 , characterized in that the at least one stop criterion (21) specifies a maximum number of undisturbed data elements, a maximum number of disturbed data elements, a maximum integral value over the deviations and/or the presence of predefined patterns, in particular in the deviations (31). . Computer-readable storage medium according to any one of Claims 10 until 12 , characterized in that the input data specify at least two methods for generating faults (16, 16'), wherein the configuration unit, the computing unit, the robustness determination unit, the comparison unit and/or the selection unit are each designed to perform their functions in a first processing pass to carry out taking into account or using the first method for generating disturbances and in a second processing pass to carry out their functions taking into account or using a second method for generating disturbances. Computer-readable storage medium according to any one of Claims 10 until 13 , especially after Claim 13 , characterized in that the configuration unit (320) is further designed to define a sequence for at least two methods for generating faults (16, 16 '), wherein the configuration unit (320) is designed to define taking into account a calculation complexity for to carry out any method for generating disturbances (16, 16'). Computer-readable storage medium according to any one of Claims 10 until 14 , especially after Claim 14 , characterized in that the at least two methods for generating disturbances (16, 16') are designed to be uncorrelated. Computer-readable storage medium according to any one of Claims 10 until 15 , characterized in that the threshold value (51) is designed as a robustness value (41') of a second AI module (14'). Computer-readable storage medium according to any one of Claims 10 until 16 , characterized in that the input data includes undisturbed data selected (80) by sampling a distribution function (81). System for classification and/or regression of input data, having the following: - a selection unit for the automated selection of an AI module, in particular using a predetermined target robustness value, which z. B. by the system according to one of Claims 1 until 9 used as a threshold; - a training unit that is designed to train the selected AI module with training data; - a receiving unit, which is designed to receive input data; - an execution unit which is designed to carry out a classification and/or regression for the input data using the selected AI module.

Images