DE102021210566A1 - Quantitative assessment of the uncertainty of statements from a classifier based on measurement data and several processing products of the same - Google Patents

Abstract

Method (100) for evaluating the uncertainty with which the assignment (3) of measurement data (2) to one or more classes (3a-3c) of a given classification by a classifier (1) is afflicted, this classifier (1) at least an artificial neural network, ANN (11), comprising the steps: • using the measurement data (2), the assignment (3) determined for this measurement data (2) by the classifier (1) and the classification parameters (11a) and Architecture (11b) of the ANN (11), at least one dependency (4, 4b) of at least one cost function (5) evaluating the correctness of this assignment (3) is determined (110) from the classification parameters (11a); • the determined dependency (4) is mapped (120) to at least one evaluation variable (7) using at least one trainable evaluation network (6), with several dependencies ( 4b, 4b*, 4b**). and mapped to at least one rating variable (7).

Description

The present invention relates to the automated classification of physical measurement data, with which, for example, image data or other observation data can be analyzed for objects that are present in an observed area.

State of the art

In order for vehicles to be able to take part in road traffic fully or partially automatically, control systems are required that reliably recognize traffic-relevant objects in the vehicle's surroundings and initiate an appropriate reaction in each case. Such systems typically feed image data or other observational data to a classifier that includes an artificial neural network, ANN. Such a classifier assigns the observation data, such as the image, to one or more classes from a given classification. For example, if a vehicle can be seen in an image or image area, the image or image area is assigned to the "Vehicle" class. The power of ANNs to generalize ensures that, for example, vehicles with a modified design that only came onto the market after the ANN had been trained are recognized as vehicles.

For operational safety, it is important to know whether this generalization actually leads to the correct class in each case. Noise and other disturbances in the observation data as well as deliberate manipulations with so-called "adversarial examples" can lead to misclassifications. The DE 10 2017 218 889 A1 discloses an ANN whose weights are drawn from a random distribution each time it is used. If one and the same input is fed to this ANN several times, the uncertainty of this output can be deduced from the scatter of the output.

Disclosure of Invention

Within the scope of the invention, a method for evaluating the uncertainty associated with the assignment of physical measurement data to one or more classes of a specified classification by a classifier was developed.

Here, the term "measurement data" includes not only physical measurement data recorded in a physical measurement process with at least one sensor, but also measurement data that is obtained through a partial or complete simulation of such a measurement process, and/or through a partial or complete simulation of one with such a sensor Measurement process observable technical system were obtained. The measurement data can also be realistic synthetic measurement data, for example, which can be generated using Generative Adversarial Networks, GANs. Such synthetic metrics are often used to alleviate training data shortages. As a rule, it is difficult to distinguish them from "real" measurement data that was actually physically recorded with a sensor.

The classifier includes at least one artificial neural network, ANN, whose behavior is characterized by classification parameters. In particular, the classification parameters can include weights, with which inputs that are supplied to a neuron or another processing unit of the ANN are offset against an activation of this neuron or this other processing unit.

In the method, at least one dependency of at least one cost function evaluating the correctness of this assignment on the classification parameters is determined. At least one cost function is selected here, which depends both on an assignment of a processing product of the measurement data to a plurality of classes by the ANN and on a class selected by the classifier. The dependency determined is mapped onto at least one evaluation variable using at least one trainable evaluation network whose behavior is characterized by evaluation parameters. This evaluation variable is in turn a measure of the probability p that the assignment determined by the classifier is correct. In particular, the evaluation variable can be identical to this probability p.

Here, based on several different processing products of the measurement data, several dependencies are determined and mapped to at least one evaluation variable.

The dependency of the assignment provided by the classifier on the classification parameters can optionally also be taken into account.

The measurement data itself, the assignment determined for this measurement data by the classifier, the architecture of the ANN and the classification parameters of the ANN, on the basis of which the classifier has made the assignment, go into the dependency of the assignment on the classification parameters. With the method, the uncertainty of the assignment can not only be reduced in the fully trained state of the ANN with the final class sification parameters, but also be evaluated at any earlier level of training.

It was recognized that an evaluation network is able to also provide high-dimensional and therefore very detailed information about the dependency of the assignment or the cost function on the classification parameters on a very low-dimensional evaluation variable, such as the one-dimensional probability p, to map. In particular, the dependency that is processed by the evaluation network can be broken down, for example, according to layers of a layered ANN to which the classification parameters (such as weights) relate. It is precisely the distribution of the dependency across the strata that can contain qualitative patterns. The evaluation network can recognize these qualitative patterns and convert them into a quantitative statement about the uncertainty.

In principle, the uncertainty could also be determined based on the dependency of the assignment or the cost function on the measurement data itself. However, by choosing the classification parameters as variables, the distribution of the dependency over the strata can be better grasped. The classification parameters are transparent and accessible parameters not only in the layer that received the measurement data, but also in deeper layers. The measurement data themselves, on the other hand, only have a direct effect on the first layer, while all the deeper layers work with the processing products of this measurement data.

The term "cost function" is not to be understood as restrictive in that the correctness of the assignment must always be evaluated using "ground truth". "Ground truth" indicates for specific measurement data which target classification the classifier should ideally assign to this specific measurement data, and can be used if it is available. However, it is also possible without "ground truth" to evaluate or at least estimate the correctness of the assignment. For example, the cost function can measure in any way to what extent the assignment of the measurement data to one or more classes is inherently plausible and/or to what extent it came about in an inherently plausible way.

The evaluation of a cost function is more flexible than a direct evaluation of the assignment itself. In many applications, the assignment ultimately provided by the classifier includes the specification of one or more discrete classes that match the measurement data, and for this reason alone cannot be a continuous function in the classification parameters. The use of discontinuous activation functions, such as ReLU, in the ANN also means that the output of the ANN is no longer a continuous function in the classification parameters. A cost function, on the other hand, can be specifically formulated as a continuous function so that, for example, its gradient always exists in the classification parameters.

In a particularly advantageous embodiment, the dependency includes at least one gradient of the assignment or the cost function according to at least one of the classification parameters. Since products from inputs and weights are summed up when calculating the activation of a neuron or another processing unit in the ANN, the gradient according to the classification parameters also provides information about how sensitively the assignment or the cost function depends on the measurement data entered .

In a further particularly advantageous embodiment, the dependency is represented by a vector or a matrix. In this vector, or in this matrix, each row or each column depends only on the classification parameters of exactly one layer of the ANN. In this way, the dependency can be broken down particularly well according to the layers of the ANN, so that the evaluation network, as described at the beginning, can recognize patterns in the distribution of the dependency across the layers. For example, each row or column of the matrix can contain the gradients of the assignment, or the cost function, according to all classification parameters (e.g. weights) of exactly one layer of the ANN.

In a further particularly advantageous embodiment, contributions to the dependency that depend on the classification parameters of each layer are compressed into a scalar assigned to this layer in the vector or the matrix. In this way, the dependency remains broken down by layers of the ANN, while at the same time the amount of information relating to this dependency is significantly condensed.

In particular, the scalar can be formed, for example, as the norm of a vector with the contributions. In the example mentioned above, the row or column of the matrix, which contains the gradients according to the classification parameters of a slice, can be written as a vector and the norm of this vector (for example 1-norm) can be formed.

As previously explained, a cost function is selected which depends both on an assignment of a processing product of the physical measurement data to a number of classes by the ANN and on a class of the ANN selected by the classifier. In this way, in particular which are evaluated to what extent the discretization downstream of the processing of the measurement data with the ANN to only one class by the classifier affects the final result. This is an indicator of uncertainty that does not require any "ground truth" with regard to a target assignment.

For example, many ANNs used in classifiers provide a vectorial "softmax score" of probabilities with which the input measurement data belongs to each of the available classes from the ANN's point of view. The classifier can then output the class with the highest probability as an assignment.

For example, if five classes are available and the softmax score determined by the ANN for specific measurement data x is the vector F(x)=(0.05, 0.05, 0.98, 0.05, 0.05), then the classifier uses the probability of 0.98 for the third class to select this third class for assignment to this specific measurement data. The assignment can then be, for example, the vector y=(0, 0, 1, 0, 0), which only differs in its third component. In this example it is quite clear that the choice of the third class is plausible. If, on the other hand, F(x)=(0.2, 0.19, 0.21, 0.2, 0.2) is determined as the softmax score for the same measurement data, the classifier returns the same vector ( 0, 0, 1, 0,0). But then it is much less plausible that the third class in particular should be awarded.

ausdrücken. Hierin sind Θ die Klassifikations-Parameter des KNN, die sich wiederum unterteilen lassen in Klassifikations-Parameter θ₁, ..., θ_n, der Schichten 1 bis n des KNN. Die Differenz zwischen dem Softmax-Score F(x) und dem als Zuordnung ausgegebenen Vektor y, der immer eine Eins und vier Nullen enthält, hängt dann direkt mit dem Gradienten $${\nabla }_{{\mathrm{\theta }}_n}L\left(\mathrm{\Theta }\mathrm{,}F\left(x\right),y\right)$$

dieser Kostenfunktion bezüglich der Klassifikations-Parameter θ_n der letzten Schicht n des KNN zusammen. Ausgehend hiervon lassen sich für die vorherigen Schichten n-1, ..., 1 die Gradienten bezüglich der Klassifikations-Parameter θ_n-1, ..., θ₁ beispielsweise im Wege der Rückpropagation durch das KNN ermitteln. Somit wirkt sich ein großer Unterschied zwischen dem Softmax-Score F(x) einerseits und der letztendlichen Klassenzuordnung y andererseits auf die Gradienten in allen Schichten 1, ..., n des KNN aus.This can be done, for example, with a cost function (“loss function”) L of the form $$L\left(\mathrm{\theta },f\left(x\right),y\right)$$

to express. Here Θ are the classification parameters of the ANN, which in turn can be subdivided into classification parameters θ ₁ , . . . , θ _n of layers 1 to n of the ANN. The difference between the softmax score F(x) and the vector y output as assignment, which always contains a one and four zeros, is then directly related to the gradient $${\nabla }_{{\mathrm{\theta }}_n}L\left(\mathrm{\theta }\mathrm{,}f\left(x\right),y\right)$$

this cost function with respect to the classification parameters θ _n of the last layer n of the ANN. Proceeding from this, the gradients with regard to the classification parameters θ _n-1 , _. . . Thus, a large difference between the softmax score F(x) on the one hand and the final class assignment y on the other hand affects the gradients in all layers 1, ..., n of the ANN.

• • eine Glättung mit einem Gaußschen Filterkern mit einer vorgegebenen Standardabweichung, mit einem Wiener-Filter und/oder mit einem Median-Filter, und/oder
• • Überführung der Messdaten in eine informationsreduzierte Repräsentation mit einem Encoder und anschließende Rekonstruktion des Verarbeitungsprodukts aus dieser Repräsentation mit einem zugehörigen Decoder, und/oder
• • auf die Messdaten konditionierte Generierung mit einem generativen Modell, und/oder
• • eine Hochpassfilterung und/oder Tiefpassfilterung in einem Frequenzraum oder Ortsfrequenzraum, und/oder
• • eine schwellwertbasierte Filterung (Thresholding) einer Repräsentation der Messdaten x in Wavelet-Koeffizienten

verwendet werden.Instead of the softmax score F(x) for the actual measurement data x, a softmax score F(x̃) can now be used in the cost function L, which the ANN determines for a processing product x̃ of the measurement data x. In particular, the class assignment y can still be determined on the basis of F(x), but then compared with F(x̃). In this case, the processing product x - can be formed from the actual measurement data x, for example by any noise reduction method. As a denoising method, for example

• • smoothing with a Gaussian filter kernel with a predetermined standard deviation, with a Wiener filter and/or with a median filter, and/or
• • Conversion of the measurement data into an information-reduced representation with an encoder and subsequent reconstruction of the processing product from this representation with an associated decoder, and/or
• • Generation conditioned on the measurement data with a generative model, and/or
• • high-pass filtering and/or low-pass filtering in a frequency space or spatial frequency space, and/or
• • a threshold-based filtering (thresholding) of a representation of the measurement data x in wavelet coefficients

be used.

An arrangement of an encoder and a decoder is also called an autoencoder. The training of such an autoencoder converges to a state in which the encoder selects the most important features of the measurement data for inclusion in the information-reduced representation (also called "latent representation" or "information bottleneck"), while the decoder optimally exploits this representation in order to to create a reconstruction that is as close as possible to the original measurement data.

In particular, a generator can be used as a generative model, which is trained in tandem with a discriminator in a Generative Adversarial Network, GAN. Here, the discriminator is trained to distinguish measurement data generated by the generator from physically generated measurement data with the best possible accuracy, while the generator is trained to degrade precisely this accuracy. By "conditioning" is meant that not any measurement data from that explained by the physical measurement data Domain are generated, but such measurement data that have a semantic content at least similar to the original measurement data. For example, a generated image can be conditioned to contain the same objects as a given original image.

The ulterior motive here is that a misclassification of measurement data is often caused by small disturbances in the measurement data. Denoising eliminates small disturbances. If the softmax score F(x̃) changes so significantly that a completely different class than the class previously output with the vector y would actually be appropriate, this can be "punished" accordingly by the cost function.

In this context, it is particularly advantageous that a number of different dependencies are determined on the basis of a number of different processing products. In particular, it can then be examined, for example, whether certain processing methods are more likely than other processing methods to lead to the class assignment changing. The ultimately determined evaluation variable is then no longer just a mere measure of the probability p that the assignment determined by the classifier is correct. Rather, for example, specific types of disturbances in the measurement data, the elimination of which in a specific processing product leads to a change in the class assignment, can be identified as the cause of a low probability (or a greater uncertainty). This knowledge can be used as feedback, for example, to harden the classifier against these specific disturbances.

One and the same cost function can always be used to compare the softmax score F(x̃) for the various processing products x̃ with the class y selected by the classifier and to evaluate any deviations. In a particularly advantageous embodiment, however, several different cost functions are used, which depend on different processing products. Then, in particular, for example, different faults whose elimination in the processed product leads to a change in the class assignment can be weighted differently.

Several dependencies of the cost function on the classification parameters can be mapped together by one and the same evaluation network onto the at least one evaluation variable. The evaluation network can then learn, for example, to evaluate an overall picture of the uncertainty composed of investigations into several types of disturbances. It is thus conceivable, for example, that a greater degree of uncertainty with regard to one fault is at least partially offset by a lower degree of uncertainty with regard to another fault. Uncertainties with regard to a number of faults can also interact, for example, so that the resulting overall uncertainty becomes particularly large.

Alternatively or also in combination with this, different proposals for the evaluation variable can be determined from various dependencies of the cost function on the classification parameters and aggregated to form a final result of the evaluation variable. This aggregation can then be used to weight the suggestions among one another.

As previously explained, the gradients V _{Θ 1} L, ..., V _{θ n} L of the cost function L with respect to the classification parameters θ ₁ , ..., θ _n of each layer 1, ..., n of the ANN for each layer 1, ..., n with a norm to form a scalar. The dependence ∇ _O L(Θ, F(x̃), y) of the cost function L on the classification parameters Θ can then be summarized in a vector: $${\nabla }_{\mathrm{\theta }}L\left(\mathrm{\theta }\mathrm{,}f\left(\tilde{x}\right),y\right)=\left(\begin{array}{c}{\Vert {\nabla }_{{\theta }_1}L\left(\mathrm{\theta },f\left(\tilde{x}\right),y\right)\Vert }_1\\ ⋮\\ {\Vert {\nabla }_{{\theta }_n}L\left(\mathrm{\theta },f\left(\tilde{x}\right),y\right)\Vert }_1\end{array}\right)$$

This vector can be mapped to the rating size by the rating network.

In a particularly advantageous embodiment, the evaluation network includes at least one logistic regression network that maps the dependency on a probability p that the assignment determined by the classifier is correct. Such a logistic regression network is very parameter efficient. In the example explained above, in which the dependency ∇ _Θ L(Θ, F(x̃), y) for an ANN with n layers is represented by a vector of length n, only n+1 parameters are required for the logistic regression network. Accordingly, the space in which the optimal configuration of these parameters is to be sought when training the regression network is comparatively low-dimensional.

Alternatively or in combination with this, the evaluation network can comprise at least one classification network which maps the dependency to at least one evaluation level for the probability p. The probability p can, for example, be binary (e.g. “okay”/“not ok”) or correspond to the importance for at least partially automated driving (e.g. "OK" / "Information" / "Warning" / "Alarm").

In a further particularly advantageous embodiment, at least one evaluation network is selected based on the assignment output by the classifier. In this way, the accuracy of the assessment of the extent to which the assignment by the classifier is correct can be improved even further. For example, objects of many different types (such as vehicles, traffic signs, people, lane boundaries) must be recognized in a vehicle environment. The assessment of whether the respective assignment to the classes subordinate to these types (e.g. car or motorcycle, stop sign or 50 km/h sign) is likely to be correct is then sensibly divided according to type.

If the evaluation variable meets a specified criterion, that is to say, for example, the probability p for a correct assignment falls below a specified threshold value, various measures can be taken. In the context of at least partially automated driving, for example, a warning can be issued to a user of the vehicle, the user can be asked to take control, or the vehicle can be brought to a standstill on the emergency stop trajectory provided for a system failure.

Significant uncertainties identified with the method can have various causes. For example, the image quality provided by a camera system can be too bad for a reliable classification due to unfavorable weather conditions. There may also be a situation for which the classifier's ANN has not been adequately trained. In the context of at least partially automated driving, these are, for example, "corner cases", i.e. traffic situations that only occur rarely but are then associated with a particular risk. The new creation of traffic signs by the legislature can also give rise to uncertainties in the classification.

For example, with the introduction of the “Umweltzone” traffic sign, the appearance of the “Tempo-30-Zone” traffic sign was essentially reused. Only the number 30 was exchanged for the word "environment". If the ANN in a classifier of a vehicle has not yet received the update that teaches it this new traffic sign "Limum Zone", from the point of view of the ANN this traffic sign looks most similar to the traffic sign "Tempo-30-Zone" and is therefore likely to be classified as such be classified. However, the Softmax scores will show that this does not quite fit, because the lettering "environment" looks different than the number "30", and the additional sign, which indicates the required color of the environmental sticker, does not fit either. If, as a result, increased uncertainty is detected, it is possible, for example, to avoid sudden heavy braking to 30 km/h on an expressway with a permitted speed limit of 80 km/h, which leads to a low-emission zone, and triggering a rear-end collision.

However, uncertainties can also have malicious intentions. The power of ANNs to generalize opens up the possibility of attacks with manipulations of the appearance of traffic signs, for example, which a person hardly notices, but which lead to a completely wrong classification. With the "right" stickers, for example, it can be provoked that a stop sign is classified as a 70 km/h sign. It has also been demonstrated that a semi-transparent film with the "right" dot pattern on the camera lens can cause a classifier to completely ignore pedestrians, for example.

In a particularly advantageous embodiment, the method is therefore repeated in response to the fact that the evaluation variable meets a predetermined criterion, replacing the ANN of the classifier with another ANN that has undergone different training and/or has a different architecture. If a classification with a significantly lower uncertainty is obtained, then this can be used for further processing (e.g. for trajectory planning). If the other ANN can classify the same measurement data more accurately, this also provides information about the possible cause of the error. It is most likely that either the training of the ANN used first was not up-to-date or insufficient, or that an attack with an "adversarial example" was attempted.

Alternatively or also in combination with this, further measurement data can be recorded with at least one sensor, and the method can be repeated with this further measurement data. For example, if a radar system classifies an object differently with great certainty that was previously classified on a camera image only with great certainty, the most likely cause is that the camera image was too bad.

The measurement data can be, for example, optical image data, thermal image data, video data, radar data, ultrasound data and/or LIDAR data. These are the most important types of measurement data, on the basis of which at least partially automated vehicles in Ver orientate the traffic area. The measurement data can be obtained by a physical measurement process and/or by a partial or complete simulation of such a measurement process and/or by a partial or complete simulation of a technical system that can be observed with such a measurement process.

However, the application of the method is not limited to vehicles. In general, a control signal can be formed from the assignment in combination with the evaluation variable, and it can be a vehicle, and/or a classification system, and/or a system for quality control of mass-produced products, and/or a system for medical Imaging, and / or an access control system, are controlled with this control signal.

For example, an attempt to deceive an access control system with a photo of an authorized person or with a copied fingerprint can be detected. The photo or the fingerprint is assigned to the authorized person, but the uncertainty, which is expressed in a reduced probability p for correct classification, reflects the fact that the copy is not perfect.

The invention also relates to a method for training an evaluation network, which can be used when carrying out the method described above. In this method, based on training data, an assignment of this training measurement data to one or more classes and an evaluation variable for this assignment are determined using the method described above. By comparing the assignment determined by the classifier with a target assignment known for the training measurement data, an actual state is determined as to the extent to which the assignment is correct. The evaluation parameters are optimized in such a way that the evaluation variable supplied by the evaluation network correlates with the actual state according to an evaluation cost function.

It is not important to improve the ANN in the classifier. The only thing that matters is that in the cases in which the classifier actually makes a wrong statement, this is reflected in the evaluation variable supplied by the evaluation network.

The possibility of quantitatively determining the uncertainty of a classification can also be used to "label" training data for classifiers with class assignments. The invention also relates to a method for generating training data for classifiers that assign measurement data to one or more classes of a specified classification.

In this method, based on measurement data, both an assignment of this measurement data to one or more classes and an evaluation variable for this assignment are determined using the method described first. In response to the score meeting a predetermined condition, the measurement data associated with the association is added to the training data.

The power of ANNs to generalize depends on the fact that training data with sufficient variability are used during training. For example, the ANN has to process many images of vehicles of different types, taken under many different conditions, in order to be able to recognize new vehicles as well. The biggest cost driver for the procurement of such images is the manual "labelling" of these images with the class assignment, in this example the assignment to the "Vehicle" class. From a large amount of unsorted images, the method can now be used to filter out those images that have been classified in the “vehicle” class with a sufficiently high degree of certainty. These images can be added to the training data in association with the label "vehicle".

In particular, the methods can be fully or partially computer-implemented. The invention therefore also relates to a computer program with machine-readable instructions which, when executed on one or more computers, cause the computer or computers to carry out one of the methods described. In this sense, control devices for vehicles and embedded systems for technical devices that are also able to execute machine-readable instructions are also to be regarded as computers.

The invention also relates to a machine-readable data carrier and/or a download product with the computer program. A downloadable product is a digital product that can be transmitted over a data network, i.e. can be downloaded by a user of the data network and that can be offered for sale in an online shop for immediate download, for example.

Furthermore, a computer can be equipped with the computer program, with the machine-readable data carrier or with the downloadable product.

Further measures improving the invention are presented in more detail below together with the description of the preferred exemplary embodiments of the invention with the aid of figures.

character list

• 1 Ausführungsbeispiel des Verfahrens 100 zur Bewertung der Unsicherheit einer Zuordnung 3 von Messdaten 2 zu Klassen 3a-3c;
• 2 Veranschaulichung des Rechengangs von Messdaten 2 bis zur Bewertungsgröße 7 für die Unsicherheit;
• 3 Ausführungsbeispiel des Verfahrens 200 zum Trainieren eines Bewertungs-Netzwerks 6 zum Einsatz in dem Verfahren 100;
• 4 Ausführungsbeispiel des Verfahrens 300 zum Generieren von Trainingsdaten 2*.

It shows:

• 1 Exemplary embodiment of the method 100 for evaluating the uncertainty of an assignment 3 of measurement data 2 to classes 3a-3c;
• 2 Illustration of the calculation process from measurement data 2 to evaluation variable 7 for the uncertainty;
• 3 Embodiment of the method 200 for training an evaluation network 6 for use in the method 100;
• 4 Embodiment of the method 300 for generating training data 2*.

1 FIG. 12 is a schematic flowchart of an exemplary embodiment of the method 100. With this method, the uncertainty with which the assignment 3 of measurement data 2 to one or more classes 3a-3c is affected by a classifier is evaluated in the form of an evaluation variable 7. According to step 105, the measurement data 2 can be optical image data, thermal image data, video data, radar data, ultrasound data and/or LIDAR data generated by a physical measurement process and/or by a partial or complete simulation of such a measurement process and/or by a partial or complete simulation of a technical system that can be observed with such a measurement process.

First, in step 110, the measurement data, the assignment 3 to classes 3a-3c determined for this measurement data 2 by the classifier 1, the classification parameters 11a of the ANN 11 of the classifier 1 and the architecture 11b of this ANN 1 are combined. At least one dependency 4, 4b of at least one cost function 5 evaluating the correctness of this assignment 3 is determined from the classification parameters 11a. Here, according to block 114, at least one cost function 5 is selected, which depends both on an assignment 3' of a processing product 2a of the measurement data 2 to a plurality of classes 3a-3c by the ANN 11 and on a class 3a-3c selected by the classifier 1. Starting from a number of different processing products 2a, 2a*, 2a** of the measurement data 2, a number of such dependencies 4b, 4b*, 4b** are determined in this way.

The dependency 4, 4a of the assignment 3 supplied by the classifier 1 on the classification parameters 11a can optionally also be taken into account.

The classifier 1 can be applied to the measurement data 2 while the method 100 is being carried out. However, this is not mandatory. The method 100 can also work solely on the basis of results from the classifier 1 that have already been completed.

In step 120, the multiple determined dependencies 4, 4a, 4b, 4b*, 4b** are mapped onto at least one evaluation variable 7 using at least one trainable evaluation network 6. This evaluation variable 7 is a measure of the probability p that the assignment 3 determined by the classifier 1 is correct.

In response to the fact that the evaluation variable 7 meets a predetermined criterion 130 (i.e. too great an uncertainty - truth value 1), the method can be carried out according to step 140 by replacing the ANN 11 of the classifier 1 with a differently trained and/or differently structured ANN 11 ' be repeated. Alternatively or also in combination with this, further measurement data 2′ can be recorded with at least one sensor according to step 150, and the method can be repeated according to step 160 with this further measurement data.

According to step 170, a control signal 8 can be formed from the assignment 3 in combination with the evaluation variable 7 . According to step 180, a vehicle 50, and/or a classification system 60, and/or a system 70 for the quality control of mass-produced products, and/or a system 80 for medical imaging, and/or an access control system 90, with this Control signal 8 are controlled.

A few exemplary configurations for determining the evaluation variable 7 are shown within box 110 .

According to block 111, the dependency 4 can include at least one gradient of the assignment 3, or the cost function 5, according to at least one of the classification parameters 11a.

According to block 112, the dependency 4 can be represented by a vector or a matrix. In this case, each row or each column in this vector or in this matrix depends only on classification parameters 11a of exactly one layer of the ANN 11 . According to block 113, contributions to the dependency 4 which depend on the classification parameters 11a of each layer can then be compressed into a scalar assigned to this layer in the vector or the matrix. this in turn can be done according to block 113a by forming a norm of a vector with the contributions.

According to block 114a, several different cost functions 5 can be used, which depend on different processing products 2a, 2a*, 2a**.

Box 120 shows two exemplary configurations for determining evaluation variable 7 .

• • mindestens ein logistisches Regressionsnetzwerk, das die Abhängigkeit 4 auf eine Wahrscheinlichkeit p, dass die durch den Klassifikator 1 ermittelte Zuordnung 3 korrekt ist, abbildet, und/oder
• • mindestens ein Klassifikationsnetzwerk, das die Abhängigkeit auf mindestens eine Bewertungsstufe für diese Wahrscheinlichkeit p abbildet,

umfassen.According to block 121, the evaluation network 6

• • at least one logistic regression network that maps the dependency 4 to a probability p that the assignment 3 determined by the classifier 1 is correct, and/or
• • at least one classification network that maps the dependency to at least one evaluation level for this probability p,

include.

According to block 122, at least one evaluation network 6 can be selected based on assignment 3. For example, a separate evaluation network 6 can be used for each class 3a-3c.

According to block 123, several dependencies 4b, 4b*, 4b** can be mapped together by one and the same evaluation network 6 onto the at least one evaluation variable 7.

Alternatively or also in combination with this, according to block 124, different suggestions 7a-7c for the evaluation variable 7 can be determined from different dependencies 4b, 4b*, 4b**. According to block 125, these suggestions 7a-7c can then be aggregated to form a final result 7 of the evaluation variable.

2 uses an exemplary embodiment to illustrate the calculation process from the measurement data 2 to the evaluation variable 7.

The ANN 11 of the classifier 1 is first applied to the measurement data 2, x in the forward direction (arrow from left to right) in order to obtain an assignment F(x), 3' of this measurement data 2 to classes 3a-3c. Based on this assignment 3', the classifier selects a single class 3a-3c as the final assignment y,3.

A processing product 2a, x - is generated from the measurement data 2, x by removing noise with a transformation T ₁ . The ANN 11 is in turn applied to this processing product 2a, x - in the forward direction. The result is again an assignment 3', F(x̃) of the processing product 2a, x̃ to classes 3a-3c.

The cost function 5 has the form L(Θ, F(x̃), y). Its gradient ∇ _Θ L(Θ, F(x̃), y) is successively calculated by the ANN 11 by means of back propagation (arrow to the left). The gradients are compressed with respect to the parameters θ ₁ , . . . , θ _n of each slice 1 , summarized. This vector represents dependency 4, 4b. It is fed to the evaluation network 6, which in turn uses this to determine the evaluation variable 7 (here: the probability p).

The same process is also carried out in parallel for further processing products 2a*, 2a**, x - which are obtained from the measurement data 2, x by further transformations T ₂ and T ₃ . In this way, further dependencies 4b*, 4b** also arise. in the in 2 In the example shown, all dependencies 4b, 4b*, 4b** are processed jointly by the evaluation network 6 and mapped to the evaluation variable 7.

3 is a schematic flowchart of an embodiment of the method 200 for training the evaluation network 6.

In step 210, associated with 1 The method 100 described determines an assignment 3 of training measurement data 2* to one or more classes and an evaluation variable 7 for this assignment 3.

In step 220, by comparing the assignment 3 determined by the classifier 1 with a target assignment 3* known for the training measurement data 2*, an actual state 7* is determined to the extent to which the assignment 3 is correct.

In step 230, the evaluation parameters 6a are optimized such that the evaluation variable 7 supplied by the evaluation network 6 correlates with the actual state 7* according to an evaluation cost function 8. This means that an assignment 3 that corresponds to the target assignment 3* should be evaluated by the evaluation variable 7 with a higher probability p of being true than an assignment 3 that does not correspond to the target assignment 3* and is therefore objectively wrong extend over a large amount of training metrics 2*.

After each optimization step of the evaluation parameters 6a, a branch is made back to step 210 in order to update the evaluation variable 7. If the optimization is terminated according to any given termination criterion, the trained state 6a* of the evaluation parameters 6a is present.

4 is a schematic flowchart of an embodiment of the method 300 for generating training data 2* for classifiers 1.

In step 310, associated with 1 The method 100 described above determines both an assignment 3 of measurement data 2 to one or more classes 3a-3c and an evaluation variable 7 for this assignment 3. In step 320 it is checked whether this evaluation variable 7 satisfies a predetermined condition. If this is the case (truth value 1), the measurement data 2 in association with the assignment 3 are added to the training data 2*.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102017218889 A1 [0003]

Claims (19)

Method (100) for evaluating the uncertainty with which the assignment (3) of measurement data (2) to one or more classes (3a-3c) of a given classification by a classifier (1) is afflicted, this classifier (1) at least an artificial neural network, ANN (11), whose behavior is characterized by classification parameters (11a), with the steps: • Based on the measurement data (2), the assignment (3) determined for this measurement data (2) by the classifier (1) and the classification parameters (11a) and architecture (11b) of the ANN (11), at least one dependency (4 , 4b) at least one cost function (5) evaluating the correctness of this assignment (3), determined (110) from the classification parameters (11a), with at least one cost function (5) being selected (114) which is determined both by an assignment ( 3') a processing product (2a) of the measurement data (2) in a plurality of classes (3a-3c) by the ANN (11) and on a class (3a-3c) selected by the classifier (1); • The dependency (4) determined is mapped (120) to at least one evaluation variable (7) which is a measure of the probability using at least one trainable evaluation network (6), the behavior of which is characterized by evaluation parameters (6a). p that the assignment (3) determined by the classifier (1) is correct, with several dependencies (4b, 4b*, 4b**) based on several different processing products (2a, 2a*, 2a**) of the measurement data (2) ) are determined and mapped to at least one evaluation variable (7). Method (100) according to claim 1 , wherein the dependency (4) includes at least one gradient of the assignment (3) or the cost function (5) according to at least one of the classification parameters (11a) (111). Method (100) according to any one of Claims 1 until 2 , where the dependency (4) is represented by a vector or a matrix, where in this vector or in this matrix, each row or each column depends only on classification parameters (11a) of exactly one layer of the ANN (11) ( 112). Method (100) according to claim 3 , where contributions to the dependency (4), which depend on the classification parameters (11a) of each layer, are compressed (113) into a scalar assigned to this layer in the vector or the matrix. Method (100) according to claim 4 , where the scalar is formed as the norm of a vector with the contributions (113a). Method (100) according to any one of Claims 1 until 5 , wherein the evaluation network (6) • at least one logistic regression network that maps the dependency (4) to a probability p that the assignment (3) determined by the classifier (1) is correct, and/or • at least one Classification network that maps the dependency to at least one evaluation level for this probability p includes (121). Method (100) according to any one of Claims 1 until 6 , wherein at least one evaluation network (6) is selected (122) on the basis of the assignment (3). Method (100) according to any one of Claims 1 until 7 , wherein in response to the evaluation variable (7) fulfilling a predetermined criterion (130), • the method of exchanging the ANN (11) of the classifier (1) for another ANN (11') that has undergone a different training and/or which has a different architecture is repeated (140), and/or • further measurement data (2') are recorded with at least one sensor (150) and the method is repeated with this further measurement data (2') (160) . Method (100) according to any one of Claims 1 until 8th , wherein optical image data, thermal image data, video data, radar data, ultrasound data, and/or LIDAR data generated by a physical measurement process, and/or by a partial or full simulation of such a measurement process, and/or by a partial or full simulation of a such a measurement process observable technical system, are selected as measurement data (2) (105). Method (100) according to any one of Claims 1 until 9 , wherein a control signal (8) is formed from the assignment (3) in combination with the evaluation variable (7) (170) and wherein a vehicle (50) and/or a classification system (60) and/or a system (70 ) for the quality control of mass-produced products, and/or a system (80) for medical imaging, and/or an access control system (90), is controlled (180) with this control signal (8). Method (100) according to any one of Claims 1 until 10 , wherein at least one processing product (2a, 2a*, 2a**) of the measurement data (2) by • smoothing the measurement data (2) with a Gaussian filter core with a predetermined standard deviation, with a Wiener filter and/or with a median filter, and/or • Conversion of the measurement data (2) into an information-reduced representation with an encoder and subsequent reconstruction of the processing product (2a, 2a*, 2a**) from this Representation with an associated decoder, and/or • generation conditioned on the measurement data (2) with a generative model, and/or • high-pass filtering and/or low-pass filtering in a frequency space or spatial frequency space, and/or • threshold-based filtering (thresholding) of a Representation of the measurement data x is formed in wavelet coefficients. Method (100) according to any one of Claims 1 until 11 , using several different cost functions (5) (114a) depending on different processing products (2a, 2a*, 2a**). Method (100) according to any one of Claims 1 until 12 , wherein several dependencies (4b, 4b*, 4b**) are mapped (123) together by one and the same evaluation network (6) to the at least one evaluation variable (7). Method (100) according to any one of Claims 1 until 13 , wherein different suggestions (7a-7c) for the evaluation parameter (7) are determined (124) from different dependencies (4b, 4b*, 4b**) and are aggregated (125) to form a final result (7) of the evaluation parameter. Method (200) for training an evaluation network (6) for use in the method (100) according to one of Claims 1 until 11 with the steps: • based on training measurement data (2*), the method (100) according to one of Claims 1 until 14 an assignment (3) of this training measurement data (2*) to one or more classes and an evaluation variable (7) for this assignment (3) is determined (210); • By comparing the classifier (1) determined assignment (3) with a training measurement data (2*) known target assignment (3*), an actual state (7*) is determined (220) to what extent the assignment (3) is correct; • The evaluation parameters (6a) are optimized (230) such that the evaluation variable (7) supplied by the evaluation network (6) correlates with the actual state (7*) according to an evaluation cost function (8). Method (300) for generating training data (2*) for classifiers (1) that assign measurement data (2) to one or more classes (3a-3c) of a specified classification, with the steps: • starting from measurement data (2). with the method (100) according to one of Claims 1 until 14 both an assignment (3) of these measurement data (2) to one or more classes (3a-3c) and an evaluation variable (7) for this assignment (3) are determined (310); and • in response to the evaluation variable (7) fulfilling a predetermined condition (320), the measurement data (2) in association with the association (3) are added to the training data (2*) (330). Computer program containing machine-readable instructions which, when executed on one or more computers, cause the computer or computers to perform a method (100, 200, 300) according to any one of Claims 1 until 16 to execute. Machine-readable data carrier with the computer program Claim 17 . Computer equipped with the computer program according to Claim 17 , and/or with the machine-readable data medium Claim 18 .

Images