WO2021191115A1

WO2021191115A1 - Quantitative rating of the uncertainty of statements by a classifier

Info

Publication number: WO2021191115A1
Application number: PCT/EP2021/057200
Authority: WO
Inventors: Julia Rebecca LUST; Alexandru Paul Condurache
Original assignee: Robert Bosch Gmbh
Priority date: 2020-03-25
Filing date: 2021-03-22
Publication date: 2021-09-30
Also published as: DE102020203838A1

Abstract

A method (100) for rating the uncertainty afflicting the association (3) of measurement data (2) with one or more classes (3a-3c) of a predefined classification by a classifier (1), this classifier (1) comprising at least one artificial neural network, ANN (11), the behaviour of which is characterized by classification parameters (11a), having the steps of: • the measurement data (2), the association (3) ascertained for these measurement data (2) by the classifier (1) and the classification parameters (11a) and architecture (11b) of the ANN (11) are used to ascertain (110) a dependency (4, 4a) of the association (3), and/or a dependency (4, 4b) of a cost function (5) rating the correctness of this association (3), on the classification parameters (11a); • the ascertained dependency (4) is mapped (120) to at least one rating variable (7), which is a measure of the probability p of the association (3) ascertained by the classifier (1) being correct, using at least one trainable rating network (6), the behaviour of which is characterized by rating parameters (6a).

Description

description

Title:

Quantitative evaluation of the uncertainty of statements of a classifier

The present invention relates to the automated classification of physical measurement data with which, for example, image data or other observation data can be analyzed for objects that are present in an observed area.

State of the art

In order for vehicles to be fully or partially automated in road traffic, control systems are required that reliably detect traffic-relevant objects in the vehicle environment and initiate an appropriate response in each case. Such systems usually feed image data or other observation data to a classifier that contains an artificial neural network, ANN. Such a classifier assigns the observation data, such as the image, to one or more classes from a given classification. If, for example, a vehicle can be seen in an image or image area, the image or the image area is assigned to the “vehicle” class. The power of KN Ns to generalize ensures that, for example, vehicles with a modified design that only came onto the market after training the KN Ns are recognized as vehicles.

For operational safety, it is important to know whether this generalization actually leads to the right class in each case. Noise and other disturbances in the observation data as well as deliberate manipulations with so-called "adverse examples" can lead to incorrect classifications. DE 10 2017 218 889 A1 discloses an ANN whose weights are at each use can be drawn from a random distribution. If the same input is fed to this ANN several times, the uncertainty of this output can be inferred from the dispersion of the output.

Disclosure of the invention

In the context of the invention, a method for evaluating the uncertainty associated with the assignment of physical measurement data to one or more classes of a given classification by a classifier was developed.

Here, the term “measurement data” includes not only physical measurement data that were recorded in a physical measurement process with at least one sensor, but also measurement data that were generated by a partial or complete simulation of such a measurement process and / or by a partial or complete simulation of one with such a sensor Measurement process observable technical system. The measurement data can also be, for example, realistic synthetic measurement data that can be generated using Generative Adversarial Networks, GANs, for example. Such synthetic measurement data are often used to alleviate a shortage of training data. When viewed, they are usually difficult to distinguish from “real” measurement data that were actually physically recorded with a sensor.

The classifier comprises at least one artificial neural network, ANN, whose behavior is characterized by classification parameters. The classification parameters can in particular include weights with which inputs that are fed to a neuron or another processing unit of the ANN are calculated to activate this neuron or this other processing unit.

In the method, a dependency of the assignment provided by the classifier and / or a dependency of a cost function evaluating the correctness of this assignment on the classification parameters is determined. This dependency is determined with at least one trainable evaluation network, whose behavior is characterized by evaluation parameters, mapped onto at least one evaluation variable. This evaluation variable is in turn a measure of the probability p that the assignment determined by the classifier is correct. The evaluation variable can in particular be identical to this probability p.

The dependence of the assignment on the classification parameters includes the measurement data itself, the assignment determined for this measurement data by the classifier, the architecture of the ANN and the classification parameters of the ANN, on the basis of which the classifier made the assignment. The method can therefore be used to evaluate the uncertainty of the assignment not only in the fully trained state of the ANN with the final classification parameters, but also at any earlier training level.

It was recognized that an evaluation network is able to provide high-dimensional and therefore very detailed information about the dependency of the assignment or the cost function on the classification parameters on a very low-dimensional evaluation variable, such as the one-dimensional probability p to map. In particular, the dependency that is processed by the evaluation network can be broken down, for example, into layers of an ANN built up in layers, to which the classification parameters (such as weights) relate. It is precisely the distribution of dependency across the layers that can contain qualitative patterns. The evaluation network can recognize these qualitative patterns and convert them into a quantitative statement about the uncertainty.

In principle, the uncertainty could also be determined on the basis of the dependency of the assignment or the cost function on the measurement data itself. However, by choosing the classification parameters as variables, the distribution of the dependency over the layers can be better understood. The classification parameters are not only transparent and accessible parameters in the layer that received the measurement data, but also in the deeper layers. In contrast, the measurement data itself only has an immediate effect on the first layer, while all lower layers work with processing products of this measurement data. The term “cost function” is not to be understood as restrictive to the effect that the correctness of the assignment must be assessed using “ground truth”. “Ground truth” indicates for specific measurement data which target classification the classifier should ideally assign to this specific measurement data, and can be used if it is available. However, it is also possible without “ground truth” to evaluate or at least estimate the correctness of the assignment. For example, the cost function can measure in any way the extent to which the assignment of the measurement data to one or more classes is intrinsically plausible and / or to what extent it has come about in an intrinsically plausible manner.

The evaluation of a cost function is more flexible than a direct evaluation of the assignment itself. In many applications, the assignment ultimately provided by the classifier includes the specification of one or more discrete classes that match the measurement data, and for this reason alone cannot be a continuous function in the classification parameters. The use of discontinuous activation functions, such as ReLU, in the ANN also means that the output of the ANN is no longer a continuous function in the classification parameters. A cost function, on the other hand, can be specifically formulated as a continuous function so that, for example, its gradient always exists in the classification parameters.

In a particularly advantageous embodiment, the dependency comprises at least one gradient of the assignment, or of the cost function, according to at least one of the classification parameters. Since products of inputs and weights are added up when calculating the activation of a neuron or other processing unit in the ANN, the gradient according to the classification parameters also provides information about how sensitively the assignment, or the cost function, depends on the input measurement data .

In a further particularly advantageous embodiment, the dependency is represented by a vector or a matrix. In this vector, or in this matrix, each row or column only depends on Classification parameters from exactly one layer of the ANN. In this way, the dependency can be broken down particularly well according to layers of the ANN, so that the evaluation network, as described above, can recognize patterns in the distribution of the dependency over the layers. For example, each row or column of the matrix can contain the gradients of the assignment, or the cost function, according to all classification parameters (such as weights) of exactly one layer of the ANN.

In a further particularly advantageous embodiment, contributions to the dependency that depend on the classification parameters of each layer are compressed into a scalar assigned to this layer in the vector or the matrix. In this way, the dependency remains broken down into layers of the ANN, while at the same time the amount of information relating to this dependency is significantly compressed.

The scalar can in particular be formed with the contributions, for example, as the norm of a vector. In the example mentioned above, the row or column of the matrix that contains the gradients according to the classification parameters of a layer can be written as a vector and the norm of this vector (for example 1 norm) can be formed.

In a particularly advantageous embodiment, a cost function is selected that depends both on an assignment of the physical measurement data, or a processing product of this physical measurement data, to several classes by the ANN and on a class of the ANN selected by the classifier. In this way, it can be evaluated in particular how strongly the discretization, which follows the processing of the measurement data with the ANN, affects only one class by the classifier on the end result. This is an indicator of the uncertainty that does not require a “ground truth” with regard to a target assignment.

For example, many of the KN Ns used in classifiers provide a vectorial “Softmax Score” of probabilities with which the entered measurement data from the perspective of the ANN of each of the available classes belong. The classifier can then output the class with the highest probability as an assignment.

If, for example, five classes are available and the softmax score determined by the ANN for specific measurement data x is the vector F (x) = (0.05, 0.05, 0.98, 0.05, 0.05), then the classifier uses the probability of 0.98 for the third class to select this third class for the assignment to this specific measurement data. The assignment can then be, for example, the vector y = (0, 0, 1, 0, 0), which is only different in its third component. In this example it is quite clear that the choice of third grade is plausible. If, on the other hand, F (x) = (0.2, 0.19, 0.21, 0.2, 0.2) is determined as the Softmax score for the same measurement data, the classifier returns the same vector ( 0, 0, 1, 0.0). However, it is then much less plausible that the third class in particular should be excellent.

This can be done, for example, with a cost function (“loess function”) L of the form

Express L (0, F (x), y). Here, Q are the classification parameters of the ANN, which in turn can be subdivided into classification parameters qi, ..., q _h , of layers 1 to n of the ANN. The difference between the Softmax score F (x) and the vector y output as an assignment, which always contains a one and four zeros, then depends directly on the gradient

V _6n L (0, F (x), y) of this cost function with respect to the classification parameters q _{h of} the last layer n of the ANN. Starting from this blank for the previous layers n-1, ..., 1, the gradient with respect to the classification parameter q _hi, ..., qi determine, for example by way of back propagation by the NN. Thus, a large difference between the Softmax Score F (x) on the one hand and the final class assignment y on the other hand affects the gradients in all layers 1, ..., n of the ANN.

Instead of the Softmax score F (x) for the actual measurement data x, a Softmax Score F (x) can also be used in the cost function L, which the ANN determines for a processing product x of the measurement data x. In particular, the Class assignment y can still be determined on the basis of F (x), but then compared with F (x). The processing product x can in particular be formed from the actual measurement data x, for example, by any desired noise reduction method. As a noise reduction method, for example

• smoothing with a Gaussian filter kernel with a given standard deviation,

• high-pass filtering and / or low-pass filtering in a frequency space or spatial frequency space, and / or

• a threshold-based filtering (thresholding) of a representation of the measurement data x in wavelet coefficients can be used.

The idea behind this is that a misclassification of measurement data is often caused by small disturbances in the measurement data. The noise removal eliminates small disturbances. If this changes the Softmax Score F (x) so clearly that a completely different class than the class previously output with the vector y would actually be appropriate, this can be “punished” accordingly by the cost function.

_{As explained above} , the gradients V ₀₁ L, ..., V 0n L of the cost function L can be determined with respect to the classification parameters qi, ..., q _h of each layer 1, ..., n of the ANN for each layer Aggregate 1, ..., n with a norm to a scalar. The dependence V ₀ L (0, F (x), y) of the cost function L on the classification parameters Q can then be summarized in a vector:

This vector can be mapped to the evaluation variable by the evaluation network.

In a particularly advantageous embodiment, the evaluation network comprises at least one logistic regression network which maps the dependence on a probability p that the assignment determined by the classifier is correct. Such a logistic regression network is very parameter efficient. In the previously explained Example, in which the dependency V ₀ L (0, F (x), y) for an ANN with n layers is represented by a vector of length n, only n + 1 parameters are required for the logistic regression network. Accordingly, the space in which the optimal configuration of these parameters is to be sought when training the regression network is comparatively low-dimensional.

Alternatively or in combination with this, the evaluation network can comprise at least one classification network which maps the dependency on at least one evaluation level for the probability p. The probability p can, for example, be classified as binary (for example “okay” / “not okay”) or corresponding to the significance for the at least partially automated driving (for example “okay” / “information” / “warning” / “alarm”) .

In a further particularly advantageous embodiment, at least one evaluation network is selected on the basis of the assignment output by the classifier. In this way, the accuracy of the assessment of the extent to which the assignment by the classifier is correct can be improved even further. For example, objects of many different types (e.g. vehicles, traffic signs, people, lane boundaries) must be recognized in a vehicle environment. The assessment of whether the respective assignment to the classes subordinate to these types (e.g. car or motorcycle, stop sign or speed 50 sign) is then expediently divided into types.

If the evaluation variable fulfills a predefined criterion, that is to say, for example, the probability p for a correct assignment falls below a predefined threshold value, various measures can be taken. In the context of at least partially automated driving, for example, a warning can be issued to a user of the vehicle, the user can be requested to take control, or the vehicle can be brought to a stop on the emergency stop trajectory provided for a system failure. Conspicuous uncertainties identified with the method can have various causes. For example, the image quality delivered by a camera system can be too poor for a reliable classification due to unfavorable weather conditions. There may also be a situation for which the ANN of the classifier has not been adequately trained. In the context of at least partially automated driving, these are, for example, “corner cases”, ie traffic situations that occur only rarely, but are then associated with particular hazards. The new creation of traffic signs by the legislator can also give rise to uncertainties in the classification.

For example, when the “Umweltzone” traffic sign was introduced, the appearance of the “Tempo 30 Zone” traffic sign was essentially reused. Only the number 30 was replaced with the word "environment". If the KNN has not yet received the update in a classifier of a vehicle that this new traffic sign "Umweltzone" brings to it, this traffic sign looks most similar to the traffic sign "Tempo 30 Zone" from the perspective of the KNN and is therefore likely to be such be classified. However, it will be reflected in the Softmax scores that this does not quite fit, because the lettering “Environment” looks different from the number “30”, and the additional label, which indicates the required color of the environmental badge, does not fit either. If, as a result, increased uncertainty is recognized, it can be avoided, for example, that on a freeway with a permitted speed limit of 80, which leads into an environmental zone, suddenly braking to 30 km / h and a rear-end collision is triggered.

But uncertainties can also have malicious intent. The power of KN Ns to generalize basically opens up the possibility of attacks with manipulation of the appearance of traffic signs, for example, which a person hardly notices but which lead to a completely wrong classification. With the "right" stickers, for example, it can be provoked that a stop sign is classified as a 70 km / h sign. It has also already been demonstrated that a semi-permeable film with the “correct” point pattern on the camera lens can cause a classifier to completely ignore pedestrians, for example.

In a particularly advantageous embodiment, the method is therefore repeated in response to the fact that the evaluation variable fulfills a predefined criterion, replacing the ANN of the classifier with another ANN which has undergone different training and / or which has a different architecture. If a classification with a significantly lower uncertainty is obtained here, then this can be used for further processing (e.g. for trajectory planning). If the other ANN can classify the same measurement data more accurately, this also provides information on the possible cause of the error. It is most likely that either the training of the CNN used first was not up-to-date or insufficient, or that an attack with an "adversarial example" was attempted.

Alternatively or also in combination with this, further measurement data can be recorded with at least one sensor, and the method can be repeated with this additional measurement data. If, for example, a radar system classifies an object that was previously classified only with great uncertainty on a camera image differently with great certainty, the most likely cause is that the camera image was too bad.

The measurement data can in particular be, for example, optical image data, thermal image data, video data, radar data, ultrasound data, and / or LIDAR data. These are the most important types of measurement data with which at least partially automated vehicles orientate themselves in the traffic area. The measurement data can be obtained through a physical measurement process and / or through a partial or complete simulation of such a measurement process and / or through a partial or complete simulation of a technical system that can be observed with such a measurement process.

However, the application of the method is not restricted to vehicles. In general, a control signal can be formed from the assignment in combination with the evaluation variable, and a vehicle and / or a Classification system, and / or a system for quality control of mass-produced products, and / or a system for medical imaging, and / or an access control system, can be controlled with this control signal.

For example, an attempt to deceive an access control system with a photo of an authorized person or with a copied fingerprint can be detected. The photo or the fingerprint is assigned to the authorized person, but the uncertainty, which is expressed in a reduced probability p for correct classification, reflects that the copy is not perfect.

The invention also relates to a method for training an evaluation network that can be used when the method described above is carried out. In this method, on the basis of training data, the method described above is used to determine an assignment of this training measurement data to one or more classes and an evaluation variable for this assignment. By comparing the assignment determined by the classifier with a nominal assignment known for the training measurement data, an actual state is determined to what extent the assignment is correct. The evaluation parameters are optimized in such a way that the evaluation variable supplied by the evaluation network correlates with the actual state in accordance with an evaluation cost function.

It is not important here to improve the ANN in the classifier. The only thing that matters is that in cases in which the classifier actually makes a false statement, this is reflected in the evaluation variable supplied by the evaluation network.

The possibility of quantitatively determining the uncertainty of a classification can also be used to “labein” training data for classifiers with class assignments. The invention thus also relates to a method for generating training data for classifiers which assign measurement data to one or more classes of a predetermined classification. In this method, based on measurement data, the method described first is used to determine both an assignment of this measurement data to one or more classes and an evaluation variable for this assignment. In response to the fact that the evaluation variable fulfills a predetermined condition, the measurement data are added to the training data in association with the assignment.

The power of KN Ns to generalize lives from the fact that training data with sufficient variability are used during training. For example, the ANN has to process many images of vehicles of different types that were taken under many different conditions in order to be able to recognize new vehicles. The biggest cost driver for the procurement of such images is the manual “labeling” of these images with the class assignment, in this example the assignment to the class “vehicle”. From a large number of unsorted images, the method can now be used to filter out those images that have been classified into the “vehicle” class with a sufficiently high degree of certainty. These images can be added to the training data in association with the "Vehicle" label.

In particular, the methods can be implemented in whole or in part by a computer. The invention therefore also relates to a computer program with machine-readable instructions which, when they are executed on one or more computers, cause the computer or computers to carry out one of the described methods. In this sense, control devices for vehicles and embedded systems for technical devices, which are also able to execute machine-readable instructions, are to be regarded as computers.

The invention also relates to a machine-readable data carrier and / or to a download product with the computer program. A download product is a digital product that can be transmitted via a data network, ie can be downloaded by a user of the data network and that can be offered for sale for immediate download in an online shop, for example. Furthermore, a computer can be equipped with the computer program, with the machine-readable data carrier or with the download product.

Further measures improving the invention are illustrated in more detail below together with the description of the preferred exemplary embodiments of the invention with reference to figures.

Embodiments

It shows:

FIG. 1 exemplary embodiment of the method 100 for evaluating the uncertainty of an assignment 3 of measurement data 2 to classes 3a-3c;

FIG. 2 an illustration of the calculation process from measurement data 2 up to evaluation variable 7 for the uncertainty;

FIG. 3 exemplary embodiment of the method 200 for training an evaluation network 6 for use in the method 100;

FIG. 4 exemplary embodiment of the method 300 for generating training data 2 *.

FIG. 1 is a schematic flow diagram of an exemplary embodiment of the method 100. With this method, the uncertainty with which the assignment 3 of measurement data 2 to one or more classes 3a-3c is afflicted by a classifier is assessed in the form of an assessment variable 7. As measurement data 2, according to step 105, optical image data, thermal image data, video data, radar data, ultrasound data, and / or LIDAR data that are generated by a physical measurement process, and / or by a partial or complete simulation of such a measurement process, and / or by a partial or full simulation of a technical system that can be observed with such a measurement process can be selected.

First, in step 110 the measurement data, the assignment 3 to classes 3a-3c determined for this measurement data 2 by the classifier 1, the classification parameters 11a of the ANN 11 of the classifier 1 and the architecture 11b of this ANN 1 are combined. A dependency 4, 4a of the assignment 3 and / or a dependency 4, 4b of a cost function 5, which evaluates the correctness of this assignment, on the classification parameters 11a is determined. The classifier 1 can be applied to the measurement data 2 during the execution of the method 100. However, this is not absolutely necessary. The method 100 can also work solely on the basis of results from the classifier 1 that have already been completed.

In step 120, this determined dependency 4, 4a, 4b is mapped to at least one evaluation variable 7 with at least one trainable evaluation network 6. This evaluation variable 7 is a measure of the probability p that the assignment 3 determined by the classifier 1 is correct.

In response to the fact that the evaluation variable 7 fulfills a predefined criterion 130 (that is, for example, too great an uncertainty - truth value 1), the method according to step 140 can replace the ANN 11 of the classifier 1 with a differently trained and / or differently structured ANN 11 ' be repeated. Alternatively or also in combination with this, further measurement data 2 'can be recorded with at least one sensor according to step 150, and the method can be repeated according to step 160 with these further measurement data.

From the assignment 3 in combination with the evaluation variable 7, a control signal 8 can be formed in accordance with step 170. According to step 180, a vehicle 50, and / or a classification system 60, and / or a system 70 for quality control of mass-produced products, and / or a system 80 for medical imaging, and / or an access control system 90, can be used with this Control signal 8 can be controlled. A few exemplary configurations for determining the evaluation variable 7 are shown within the box 110.

According to block 111, the dependency 4 can comprise at least one gradient of the assignment 3, or the cost function 5, according to at least one of the classification parameters 11a.

According to block 112, the dependency 4 can be represented by a vector or a matrix. In this case, in this vector, or in this matrix, each row or each column depends only on classification parameters 11a of exactly one layer of the ANN 11. According to block 113, contributions to the dependency 4, which depend on the classification parameters 11a of each layer, can then be compressed into a scalar assigned to this layer in the vector or the matrix. This in turn can be done according to block 113a by forming a norm of a vector with the contributions.

According to block 114, a cost function 5 can be selected which is derived from an assignment 3 'of the measurement data 2, or a processing product 2a of this measurement data 2, to several classes 3a-3c by the ANN 11 as well as from a class 3a selected by the classifier 1 -3c depends.

Two exemplary configurations for determining the evaluation variable 7 are shown within the box 120.

According to block 121, the evaluation network 6

• At least one logistic regression network that maps the dependency 4 to a probability p that the assignment 3 determined by the classifier 1 is correct, and / or

• Include at least one classification network that maps the dependency on at least one evaluation level for this probability p.

According to block 122, at least one evaluation network 6 can be selected on the basis of assignment 3. For example, a separate evaluation network 6 can be used for each class 3a-3c. FIG. 2 uses an exemplary embodiment to once again illustrate the calculation process from measurement data 2 to evaluation variable 7.

The KNN 11 of the classifier 1 is first applied to the measurement data 2, x in the forward direction (arrow from left to right) in order to obtain an assignment F (x), 3 'of this measurement data 2 to classes 3a-3c. On the basis of this assignment 3 ', the classifier selects a single class 3a-3c as the final assignment y, 3.

A processing product 2a, x is generated from the measurement data 2, x by de-noise. The ANN 11 is in turn applied to this processing product 2a, x in the forward direction. The result is again an assignment 3 ', F (x) of the processing product 2a, x to classes 3a-3c.

The cost function 5 has the form L (0, F (x), y). Its gradient V ₀ L (0, F (x), y) is successively calculated by the ANN 11 by means of back propagation (arrow pointing to the left). The gradients with respect to the parameters qi, ..., q _h of each layer 1, ..., n are compressed by forming the 1-norm and combined in a vector with one component per layer 1, ..., n . This vector represents the dependency 4, 4b. It is fed to the evaluation network 6, which in turn determines the evaluation variable 7 (here: the probability p) from this.

FIG. 3 is a schematic flow diagram of an exemplary embodiment of the method 200 for training the evaluation network 6.

In step 210, the method 100 described in connection with FIG. 1 is used to determine an assignment 3 of training measurement data 2 * to one or more classes and an evaluation variable 7 for this assignment 3.

In step 220, by comparing the assignment 3 determined by the classifier 1 with a nominal assignment 3 * known for the training measurement data 2 *, an actual state 7 * is determined to what extent the assignment 3 is correct. In step 230, the evaluation parameters 6a are optimized in such a way that the evaluation variable 7 supplied by the evaluation network 6 correlates with the actual state 7 * in accordance with an evaluation cost function 8. This means that an assignment 3 corresponding to the target assignment 3 * should be assessed by the assessment variable 7 with a higher probability p of being true than an assignment 3 that does not correspond to the target assignment 3 * and is therefore objectively incorrect Extend over a large amount of training measurement data 2 *.

After each optimization step of the evaluation parameter 6a, a branch is made back to step 210 in order to update the evaluation variable 7. If the optimization has ended in accordance with any predetermined termination criterion, the trained state 6a * of the evaluation parameters 6a is present.

FIG. 4 is a schematic flow diagram of an exemplary embodiment of the method 300 for generating training data 2 * for classifiers 1.

In step 310, the method 100 described in connection with FIG. 1 is used to determine both an assignment 3 of measurement data 2 to one or more classes 3a-3c and an evaluation variable 7 for this assignment 3. In step 320 it is checked whether this evaluation variable 7 fulfills a predefined condition. If this is the case (truth value 1), the measurement data 2 in association with the assignment 3 are added to the training data 2 *.

Claims

Expectations

1. Method (100) for evaluating the uncertainty with which the assignment (3) of measurement data (2) to one or more classes (3a-3c) of a given classification is subject to a classifier (1), this classifier (1 ) comprises at least one artificial neural network, ANN (11), the behavior of which is characterized by classification parameters (11a), with the steps:

• Based on the measurement data (2), the assignment (3) determined for this measurement data (2) by the classifier (1) and the classification parameters (11a) and architecture (11b) of the ANN (11), a dependency (4, 4a) the assignment (3), and / or a dependency (4, 4b) of a cost function (5) evaluating the correctness of this assignment (3), determined (HO) from the classification parameters (11a);

• the determined dependency (4) is mapped (120) with at least one trainable evaluation network (6), the behavior of which is characterized by evaluation parameters (6a), to at least one evaluation variable (7), which is a measure of the probability p that the assignment (3) determined by the classifier (1) is correct.

2. The method (100) according to claim 1, wherein the dependency (4) comprises at least one gradient of the assignment (3) or the cost function (5) according to at least one of the classification parameters (11a) (111).

3. The method (100) according to any one of claims 1 to 2, wherein the dependency (4) is represented by a vector or a matrix, with each row or each column of classification parameters in this vector or in this matrix (11a) depends on exactly one layer of the ANN (11) (112).

4. The method (100) according to claim 3, wherein contributions to the dependency (4), which depend on the classification parameters (11a) of each layer, are compressed into a scalar assigned to this layer in the vector or the matrix (113 ).

5. The method (100) according to claim 4, wherein the scalar is formed as a norm of a vector with the contributions (113a).

6. The method (100) according to any one of claims 1 to 5, wherein a cost function (5) is selected (114) which is derived from an assignment (3 ') of the measurement data (2) or a processing product (2a) of this measurement data ( 2), to several classes (3a-3c) by the ANN (11) as well as on a class (3a-3c) selected by the classifier (1).

7. The method (100) according to any one of claims 1 to 6, wherein the evaluation network (6)

• at least one logistic regression network that maps the dependency (4) to a probability p that the assignment (3) determined by the classifier (1) is correct, and / or

• at least one classification network that maps the dependency on at least one evaluation level for this probability p (121).

8. The method (100) according to any one of claims 1 to 7, wherein at least one evaluation network (6) is selected (122) on the basis of the assignment (3).

9. The method (100) according to any one of claims 1 to 8, wherein in response to the fact that the assessment variable (7) meets a predetermined criterion (130),

• the method is repeated (140) by exchanging the ANN (11) of the classifier (1) for another ANN (11 ') that has undergone different training and / or that has a different architecture, and / or

• further measurement data (2 ') are recorded (150) with at least one sensor and the method is repeated (160) with these further measurement data (2').

10. The method (100) according to any one of claims 1 to 9, wherein optical image data, thermal image data, video data, radar data, ultrasound data, and / or LIDAR data generated by a physical measurement process, and / or by a partial or complete simulation of such Measurement process, and / or through a partial or complete simulation of a technical system observable with such a measurement process, can be selected as measurement data (2) (105).

11. The method (100) according to any one of claims 1 to 10, wherein a control signal (8) is formed (170) from the assignment (3) in combination with the assessment variable (7) and wherein a vehicle (50), and / or a classification system (60), and / or a system (70) for quality control of mass-produced products, and / or a system (80) for medical imaging, and / or an access control system (90), with this control signal (8 ) is controlled (180).

12. The method (200) for training an evaluation network (6) for use in the method (100) according to one of claims 1 to 11 with the steps:

• Based on training measurement data (2 *), the method (100) according to one of claims 1 to 11 is used to assign (3) these training measurement data (2 *) to one or more classes and an evaluation variable (7) for them Assignment (3) determined (210);

• By comparing the assignment (3) determined by the classifier (1) with a target assignment (3 *) known for the training measurement data (2 *), an actual state (7 *) is determined (220) to what extent the Assignment (3) is correct;

The evaluation parameters (6a) are optimized (230) in such a way that the evaluation variable (7) supplied by the evaluation network (6) correlates with the actual state (7 *) in accordance with an evaluation cost function (8).

13. Method (300) for generating training data (2 *) for classifiers (1) which assign measurement data (2) to one or more classes (3a-3c) of a given classification, with the steps: • Based on measurement data (2), the method (100) according to one of claims 1 to 11 enables both an assignment (3) of this measurement data (2) to one or more classes (3a-3c) and an evaluation variable (7) for this assignment (3) is determined (310); and »in response to the fact that the evaluation variable (7) is a predetermined

Condition fulfilled (320), the measurement data (2) in association with the assignment (3) are added to the training data (2 *) (330).

14. Computer program, containing machine-readable instructions, which, when executed on one or more computers, the or the

Have the computer set up a method (100, 200, 300) according to one of the

Claims 1 to 13 to carry out.

15. Machine-readable data carrier with the computer program according to claim 14.

16. Computer equipped with the computer program according to claim 14 and / or with the machine-readable data carrier according to claim 15.