DE102017219269A1 - Classification with automatic selection of promising learning data - Google Patents

Abstract

Device (1) for classifying at least one input variable value (11), comprising:
• a plurality of individual evaluators (12a-12c) trainable by means of learning input quantity values (11a) and learning output values (13c), which are pre-trained on identical sets of learning input values (11a) and which are respectively adapted to the input quantity value (11 ) assign a probability distribution (P _a , P _b , P _c ), this probability distribution (P _a , P _b , P _c ) to each of the discrete classes (13a), or to each possible value of the classification size (13b), indicates with which probability the input variable value (11) is to be mapped to the respective class (13a) or to the respective value (13b); and
A confidence module (14), which is designed to determine the uncertainty (15) from the deviations between the probability distributions (P _a , P _b , P _c ) supplied by the individual evaluators (12a-12c), with which the classification is afflicted wherein the device (1) additionally comprises a feedback module (16) adapted to respond to this input variable value (11) as an answer in response to the uncertainty (15) for an input value (11) satisfying a predetermined criterion (15a) Feedback input value (11a) to all individual assessors (12a-12c), and / or to control a perceptible by the user (18) of the device (1) physical warning device (17).

Description

The invention relates to a device for classifying input variables by means of trainable evaluators, a method for operating or training, an associated computer program and uses.

State of the art

In many applications, one or more inputs are to be evaluated for a particular type of situation. For this example, neural networks or other AI modules can be used. For example, the DE 103 42 528 A1 a method and a device for driver assistance, wherein it is classified on the basis of various operating variables of the vehicle, whether an intentional or an unintentional lane change is taking place.

Neural networks and other classifiers are often trained using learning input magnitude values and learning output values. For this purpose, for example, free parameters are optimized so that the learning input values are mapped correctly on average to the known associated learning output values. After a training with a sufficient number of learning input values, it is assumed that future input values are also correctly classified.

( P. Melville, RJ Mooney, "Diverse Ensembles for Active Learning," Proceedings of the 21st Conference on Machine Learning, Banff (2004 )) discloses a method whereby by adding artificial learning input values, the true learning input values can be better utilized.

Disclosure of the invention

In the context of the invention, a device has been developed for classifying at least one input value value into one of a plurality of discrete classes, and / or on the basis of a continuous classification variable. The term input value value is not limited to scalar numerical values, but may also include, for example, vectorial data, matrices or tensors, such as image data.

The device comprises a plurality of individual assessors trainable by means of learning input values and learning output values, which are pre-trained on identical sets of learning input values and each adapted to associate a probability distribution with the input variable. For each of the discrete classes, or for each possible value of the classification variable, this probability distribution indicates with which probability the input variable value is to be mapped to the respective class or to the respective value.

Pretending "on identical sets" does not necessarily require that all individual assessors receive the learning input values from the set in the same order. Rather, by changing the order between the individual evaluators, for example by randomizing the order, specific differences in the behavior of the individual evaluators can be introduced. For example, the amount of learning input quantity values used for the preschool, and possibly associated learning output value values, may reflect a basic model that is refined during further training of the device during operation.

The term probability distribution includes any assignment which assigns a probability value to the input value value on the one hand and to the class or to the value of the continuous classification value, on the other hand.

It is also not mandatory that this distribution for more than one class, or for more than one value of the continuous classification size, is different from zero.

An example of a discrete probability distribution is that a condition with a probability of 50% in the class "very good", with a probability of 30% in the class "good", with a probability of 10% in the class "satisfactory" as well as with probabilities of 5% each into the classes "sufficient" and "defective".

For example, the individual evaluators may all be of the same type, for example neural networks of the same structure or statistical classifiers, and should then differ at least so far from one another that they are not congruent with one another as learning progresses. However, any mixtures of different types of individual evaluators are possible. For example, neural networks can be mixed with statistical classifiers.

It is an important difference to approaches known from the literature that the individual evaluators are present in the device from the beginning and not an individual evaluator is automatically generated to create a quasi-committee out of nowhere. Also, only real learning Input values for use and no artificially generated.

The device further comprises a confidence module, which is designed to determine from uncertainties between the probability distributions provided by the individual evaluators the uncertainty with which the classification is subject.

In this case, the uncertainty of the respective class, or the value of the continuous classification size, depend. For example, in the example given above, two individual assessors may very well agree that the state "Poor" exists only with 5% probability. At the same time, the one individual evaluator can assign the input value value with 95% probability of the class "very good", while the other individual evaluator assigns the same input value value with 95% of the adjacent class "good".

The apparatus additionally includes a feedback module configured to, in response to the uncertainty for an input variable value satisfying a predetermined criterion, return that input variable value to all individual evaluators as a learning input variable value and / or to actuate a physical warning device perceptible by the user of the device ,

The default criterion may be, for example, a threshold. However, it is also possible, for example, to arrange a plurality of input variable values in ascending or descending order according to associated uncertainties, in which case, for example, a predefined number of input variable values is fed back according to this sequence.

It has been recognized that the feedback module, for example, allows to use preferably those input values for the training of which the greatest improvement in the accuracy of the classification is to be expected. This in turn makes it possible to significantly reduce the effort required for the training of individual assessors in two respects.

On the one hand, depending on the application, it may be very costly to obtain the corresponding learning output variable value for a learning input value to be fed back, ie to "label" the learning input value value with the associated correct classification result. For example, if the input quantity value is an image from medical imaging and the classification task is to check for the presence of anomalies in the image, then the associated ground truth in the form of the learning output value may be limited to one Interpretation of such images specialist doctor. This makes learning output values a scarce and expensive commodity. The production of a fully trained device as a whole is therefore more cost-effective, if only those images are presented to the physician for interpretation, of which the largest learning effect is expected in the individual assessors.

On the other hand, even if learning output values are available in a straightforward manner, training can already be complicated by a large number of individual assessors. For example, if the individual evaluators are "weak learners" in the sense that the probability distribution output by each individual evaluator differs only slightly from a random distribution on its own, a larger number of such individual evaluators are needed to statistically significant the single probability distributions aggregate an overall probability distribution. The training effort grows at least proportionally with the number of individual assessors.

Finally, with the feedback module in the normal operation of the device, it can be recognized in the first place that there is a need to continue to train the individual evaluators. For example, the classification device can be designed to recognize traffic signs in image data from the surroundings of a vehicle. The device may then have been trained, for example, with all the traffic signs that were valid at the time of delivery of the vehicle. If the legislator now introduces new traffic signs, such as "toll" or "environmental zone", an individual assessor may even recognize this new traffic sign with high confidence as a similar sign. For example, the traffic sign 391 indicating a tollbooth, the traffic sign 392 , which indicates a customs office, quite similar. The more differently constructed individual assessors are used, the more likely it is to be expected, however, that these individual evaluators will recognize the traffic sign as something else. For example, another individual assessor may be the unknown traffic sign 391 with high confidence as the traffic sign 367 recognize that prohibits the entry into a one-way street in the wrong direction. Only from the disagreement of the two individual evaluators can be seen that neither the classification as a "customs office" nor the classification as "entrance forbidden" is correct.

A similar situation can occur if the prescriber renews a traffic sign completely, such as the traffic sign 151 " Railroad Crossing". Since September 2009, instead of a steam locomotive, an electrically powered intercity train has been on view, and from a different perspective as well.

Even so far can be the detection of traffic signs in part by the smudging, spraying or sticking with stickers in the wrong way. From the point of view of a human driver, these changes do not change the character of the traffic sign and are therefore hardly noticed.

In such situations, when a physical warning device is triggered, the driver may be prompted, for example, to enter the correct classification into the system or to prepare to take over manual control. The warning device can be, for example, optical, acoustic or haptic.

The effect of the feedback module is comparable to that achieved in so-called "Bayesian neural networks" (Bayes-CNN). Ultimately, it is a question of reconstructing a function which is unknown in the course of time and at the same time only very complicated to evaluate by the training of the individual evaluators, of which only a few pairs of learning input values and associated learning output values are known as "support points". In the example of the evaluation of medical images, the function to be imitated is the expert knowledge of the physician, and the evaluation of an image by the physician corresponds to an evaluation of the unknown function at a specific input variable value.

The selection of the learning input variable values based on the uncertainty fulfills a function similar to that which the so-called "acquisition functions" have with Bayesian optimization of elaborate functions to be evaluated. In the case of Bayesian optimization, instead of the value of the complex original function, the value of the "acquisition function" is optimized, which indicates where the original function is meaningfully evaluated in the next step.

The feedback may be equally active during initial training of the device and during operation of the device. In both cases, the model of the said unknown function, and thus of the real process to which the input quantity values and the classification belong, is successively refined. In each case, only the further learning input value values are used, of which actually an improvement of the model is to be expected.

Advantageously, between 5 and 50 individual judges are provided. With at least 5 individual assessors, the statement as to whether there is any uncertainty in comparison between the individual assessors becomes statistically significantly more significant. With a maximum of 50 individual assessors, both the storage costs for the individual assessors on hardware and the computational outlay for training remain within the framework.

In a further particularly advantageous embodiment, the feedback module is additionally designed to request a learning output value value belonging to the learning input value value from the user of the device and to link it back to all individual evaluators together with the learning input value value. In this way, the device has the potential to continue to learn with increasing use, so that the accuracy of the classification can always be better, regardless of whether the manufacturer of the device provides updates and whether these are actually recorded.

In a further particularly advantageous embodiment, at least one of the individual evaluators includes an artificial neural network. Such a network includes an internal processing chain defined by parameters. For example, these parameters can control the behavior of filters in a "convolutional neural network" or weight connections in a fully dense layer between neurons on adjacent layers. Especially with neural networks as individual evaluators, the number of available free parameters tends to be very large, so that usually a large number of learning input values with associated learning output values are necessary for the supervised learning to conclusively converge to a set of parameters. By selecting the most meaningful learning input values based on uncertainty, this process can be speeded up. It may well even be beneficial to the accuracy of the classification to take into account only these learning input values: If less relevant learning input values are taken into account, their comparatively weak expressiveness may also dilute the validity of the more relevant values.

In the device, not all neural networks need to be of the same type. For example, "Convolutional Neural Networks", which essentially contain convolutional layers, may be mixed with "deep learning" networks, which essentially include fully cross-linked layers.

In addition, an aggregator is advantageously provided, which is designed to aggregate the probability distributions supplied by the individual evaluators to form an overall probability distribution. This simplifies the Interpretation of classification by downstream equipment, such as systems for at least partially automated driving. The selection of the learning input value value to be labeled can then take place, for example, with measures such as "max entropy", "variation ratio" or "mean standard deviation".

According to what has been described above, the individual evaluators are advantageously designed to receive image data as input variable values and to classify them with regard to the expression of at least one predetermined feature, and / or with regard to the presence of at least one predetermined object. Especially with image data and other very high-dimensional data, there are many sources of error that may influence the classification. For example, image data may be noise, or image parts may have been saturated by improper exposure.

As described above, the invention also relates to a method for operating and / or training a device for classifying at least one of an input variable value into one of a plurality of discrete classes, and / or using a continuous classification variable.

In the method, a probability distribution is assigned to each input variable value from an identical set by a plurality of individual evaluators trainable by means of learning input values and learning output values, this probability distribution indicating each of the discrete classes or to each possible value of the classification variable which probability the input value value is to be mapped to the respective class or to the respective value.

Furthermore, deviations between the probability distributions provided by the individual evaluators determine the uncertainty with which the classification is subject.

In response to the uncertainty for an input value satisfying a predetermined criterion, the input value is fed back to all individual evaluators as the learning input value and / or a physical warning device perceptible by the user of the device is actuated.

The training may be performed within the device itself, but is not limited thereto. It is quite possible and due to the computational effort for the training also makes sense to replicate the device and in particular the individual evaluator software on an external computer or in a cloud, where a larger computing capacity is available than in the device itself. For example, there is the use The device in a partially or fully automated moving vehicle usually specifications regarding the space and / or power consumption, which in turn limit the available computing capacity.

For example, after the training has been completed, the parameters obtained by the individual evaluators can be transferred to hardware that implements the individual evaluators. This creates a ready-trained device for classification. This device can then have a confidence module and a feedback module according to what has been described above in order to be able to further refine the model contained in it during operation. However, this is not mandatory. The externally pre-trained device according to the method can alternatively also remain on the level achieved in this preschool. Regardless of whether the device has its own confidence module and its own feedback module, it is possible to update it later by external re-learning. For this purpose, for example, the current parameters which the individual evaluators set in the device can be transferred back to the external computer or to the cloud in order subsequently to perform the method for training with further learning input variables values again.

Advantageously, a device is selected in which at least two individual evaluators are of different types. If the individual evaluators nonetheless assign essentially the same probability distribution to a specific input variable value, then this input variable value is with high probability of only low significance for the training.

During training, at least two individual assessors with different start configurations are advantageously initialized. Then it is ensured that the individual evaluators are not identical in the same way during training, although they each receive identical input values.

During training, at least two individual evaluators with different sequences of the same learning input variable values are advantageously acted on. As described above, it is possible in particular to introduce differences in the behavior of, for example, otherwise identically constructed individual evaluators by randomizing the order.

As described above, the device, and / or the method, can be completely or partially implemented in software whose distribution as a stand-alone product makes sense. Therefore The invention also relates to a computer program comprising machine-readable instructions which, when executed on a computer and / or on a control unit, upgrade the computer and / or the control unit to a device according to the invention, and / or cause it to perform a method according to the invention. Likewise, the invention relates to a machine-readable data carrier or a download product with the computer program.

As described above, the device, the method, and / or the computer program are advantageously used in an at least partially automated vehicle, wherein the input variable value is obtained from at least one sensor which detects a physical measured variable in the vehicle or in the surroundings of the vehicle , and wherein the classes, or the classification size, an operating state of the vehicle, and / or a traffic situation, characterize. In this way, especially in new situations for which the individual evaluators have not been trained in the device (such as new traffic signs), the reliability improved.

As described above, the device, the method, and / or the computer program are advantageously used in diagnostics, wherein the input variable value is obtained from at least one sensor which records a physical measured variable on an object or patient to be examined, and wherein the classes , or the classification size, characterize the degree of expression of at least one anomaly of the object or patient. In this way, only a minimum of sometimes very complex available "ground truth" in the form of learning output values (labels) for the training is needed.

Likewise, the device, the method, and / or the computer program, can be advantageously used in the monitoring technique, wherein the input variable value is obtained from at least one sensor that detects a physical measurand from the area to be monitored, and wherein the classes Classification size, characterize the degree of severity of at least one hazardous situation. In this way, for example, false alarms can be avoided in unknown situations. For example, scenes in a video that are only detected with high uncertainty can be reported back to the operator.

worin der Index j=1,...,K über alle K Einzelbewerter ist. p_ij ist die vom Einzelbewerter j zu dem Eingangsgrößenwert i ermittelte Wahrscheinlichkeit für den zugehörigen Ausgangsgrößenwert (Vorhersage-Wahrscheinlichkeit). p _i ist die dem Eingangsgrößenwert i zugeordnete Vorhersage-Wahrscheinlichkeit aggregiert über alle Einzelbewerter j. Die Aggregation kann beispielsweise durch den Mittelwert, durch den Median oder durch die Mode einer geschätzten Verteilung erfolgen.A first exemplary formula for the uncertainty U _i with which the classification of an input variable value i is associated is: $$U_i=\sqrt{{\displaystyle \underset{j=1}{\overset{K}{\Sigma }}\frac{{\left(p_{ij}-{\overline{p}}_i\right)}^2}{N}.}}$$

where the index j = 1, ..., K over all K is individual judges. p _ij is the probability for the associated output variable value (prediction probability) determined by the individual evaluator j relative to the input variable value i. p _i is the prediction probability assigned to the input variable i aggregated over all individual evaluators j. The aggregation can be done, for example, by the mean, by the median or by the mode of an estimated distribution.

Where p _ij , p _i and U _i in the case of discrete classes Vectors containing one element per class. In the case of continuous probability distributions, they are functions of the classification size. Both are omitted for readability in the formula. U _i according to this formula is a mean square deviation.

worin ${\overline{p}}_i^{\left(1\right)}$

der höchste dem Eingangsgrößenwert i nach Aggregation über alle Einzelbewerter zugeordnete Vorhersage-Wahrscheinlichkeitswert und ${\overline{p}}_i^{\left(2\right)}$

der zweithöchste diesem Eingangsgrößenwert i nach der Aggregation zugeordnete Vorhersage-Wahrscheinlichkeitswert ist. Analog zum ersten Beispiel ist auch hier die Vektor- bzw. Funktionseigenschaft weggelassen. U_i gemäß dieser Formel bewertet spezifisch, mit welchem Abstand die zweitwahrscheinlichste Klassifikation auf die wahrscheinlichste Klassifikation folgt.A second exemplary formula for U _i is: $$U_i={\overline{p}}_i^{\left(1\right)}-{\overline{p}}_i^{\left(2\right)}.$$

wherein ${\overline{p}}_i^{\left(1\right)}$

the highest prediction probability value associated with the input variable i after aggregation over all individual evaluators ${\overline{p}}_i^{\left(2\right)}$

is the second highest predicted probability value associated with this input value i after aggregation. Analogous to the first example, the vector or functional property is also omitted here. U _i according to this formula specifically assesses by which distance the second most probable classification follows the most probable classification.

ausgedrückt werden. Welche dieser beispielhaften Formeln am besten geeignet ist, hängt von der jeweiligen Anwendung ab.The uncertainty U _i can also be called variance $$U_i=\frac{1}{N}{\displaystyle \underset{k=1}{\overset{K}{\Sigma }}\left(p_{ij}^2-{\overline{p}}_i^2\right)}$$

be expressed. Which of these exemplary formulas is best suited depends on the particular application.

The uncertainty regarding an input quantity value x can also be calculated as Jenson-Shannon divergence JSD: $$JSD\left(x\right)=H\left({\displaystyle {\Sigma }_{j=1}^K{\pi }_j{P}_j\left(x\right)}\right)-{\displaystyle \underset{j=1}{\overset{K}{\Sigma }}{\pi }_{j}H\left(P_j\left(x\right)\right),}$$

H here is the Shannon entropy function, and P _j is the probability distribution that the individual evaluator j assigns to the input value x. The π _j are weights.

Analogous to the previous formulas for U _i , for a discrete classification, P _j can be a vector containing a probability for each discrete class. P _j may be on the input variable x is a discrete function, but may also be a continuous function, the real occurring at each x _i, i = 1, ..., K, having a support point.

The first sum sums up probabilities, the second sum sums up Shannon entropies with the same weights.

If two individual assessors are to be compared with one another, for example, the Kullback-Leibler Divergence D _{KL can also be} chosen as a measure of the uncertainty: $$D_{KL}\left(P_1\Vert P_2\right)={\displaystyle \underset{c=1}{\overset{C}{\Sigma }}{P}_1\left(c\right)\text{log}\frac{P_1\left(c\right)}{P_2\left(c\right)}},$$

In this example, c is an index over the discrete classes 1 , ..., C. For a continuous classification size, the sum becomes an integral.

Further measures improving the invention will be described in more detail below together with the description of the preferred embodiments of the invention with reference to figures.

embodiments

• 1 Beispielhafte Skizze der Vorrichtung 1;
• 2 Beispielhafte Skizze des Verfahrens 100;
• 3 Beispielhafte Skizze der Verwendung in einem zumindest teilweise automatisiert fahrenden Fahrzeug 2;
• 4 Beispielhafte Skizze der Verwendung in der Diagnostik eines Objekts oder Patienten 32;
• 5 Beispielhafte Skizze der Verwendung in der Überwachung eines Bereichs 42.

It shows:

• 1 Exemplary sketch of the device 1 ;
• 2 Exemplary sketch of the procedure 100 ;
• 3 Exemplary sketch of the use in an at least partially automated moving vehicle 2 ;
• 4 Exemplary sketch of use in the diagnosis of an object or patient 32 ;
• 5 Exemplary sketch of use in monitoring an area 42 ,

To 1 includes the device 1 three exemplified individual assessors 12a . 12b and 12c , each one and the same input value 11 classify with what probability P _a ( 13a . 13b ) this input value 11 one of several discrete classes 13a , and / or a particular value of a continuous classification size 13b , is to be attributed. The three individual judges 12a . 12b and 12c were based on identical sets of learning input values 11a pre-trained, with at least some of these learning input values 11a also the associated learning output value 13c present.

In the confidence module 14 becomes from the three probability distributions P _a . P _b . P _c the uncertainty 15 That is a measure of how divided the individual evaluators are 12a . 12b and 12c among themselves. The confidence module continues to extend the three probability distributions P _a . P _b . P _c to the aggregator 19 from which a single probability distribution P ( 13a . 13b ) determines with which probability the input value value 11 the class 13a , or the value of the classification size 13b , is to be attributed.

The feedback module 16 checks if a given condition 15a for the uncertainty 15 is satisfied. This condition 15a may for example be a predetermined threshold. In the context of the condition 15a However, for example, a "best of" selection of those input value values 11 be taken for the uncertainty 15 is greatest.

Is the condition 15a meets, becomes a warning device 17 driven by the user 18 the device 1 is perceptible. This gives the user the opportunity to get to the input value value 11 matching learning output value 13c nachzuliefern, so the input value 11 subsequently to "label". The input variable value 11 is then used as a learning input value 11a , along with the learning output value 13c , the individual evaluators 12a . 12b and 12c fed to the individual assessors 12a . 12b and 12c to train.

2 shows an embodiment of the method 100 , In step 110 becomes the input variable value 11 about the probability distributions P _a . P _b and classifies P _c according to which probability it is which of the classes 13a , or which of the possible values of the classification size 13b , is to be attributed. From this information will be in step 120 the uncertainty 15 the individual evaluator 12a . 12b . 12c determined among each other. In step 130 will check if the criterion 15a is satisfied. If so (truth value 1 ), is in step 140 the input variable value 11 as a learning input 11a on the individual assessors 12a . 12b . 12c fed back. Unless the user 18 the matching learning output value 13c delivered, ie the learning input value 11a "Labeled" becomes the learning output value 13c also to the individual judges 12a . 12b . 12c transmitted. Alternatively or in combination, in step 150 by the user 18 Perceptible physical warning device 17 be controlled.

3 schematically shows an application of the device 1 , or of the method 100 in an at least partially automated vehicle 2 that is in a traffic situation 24 located. A sensor 21 captures the vehicle environment 22 and gives a physical measure 21a , For example, image data from. The measured quantity 21a is used as input value 11 used and classified. The probability P ( 13a . 13b ), that the input value value 11 a particular class of classes 13a , or a specific value of the classification size 13b , is attributed to, as a probability that a certain operating condition 23 of the vehicle 2 , and / or a specific traffic situation 24 , is present. Furthermore, the associated uncertainty 15 determined, with which the classification is afflicted. A high level of uncertainty can be, for example, an unknown and / or potentially dangerous traffic situation 24 , and / or a possible abnormal operating condition 23 of the vehicle 2 , Show.

4 schematically shows an application of the device 1 , or of the method 100 , in the diagnosis of an object 32 , A sensor 31 captures the object 32 and gives a physical measure 31a out, which is analogous to 3 as input value 11 used and classified. The probability P (13a, 13b) that the input value value 11 a particular class of classes 13a , or a specific value of the classification size 13b , is attributed as a probability that an in 4 exemplified anomaly 32a in the object 32 is present. Likewise, the uncertainty 15 of the classification. A high uncertainty 15 for example, as a first sign of an anomaly 32a but their concrete nature does not even have to be known in advance.

5 schematically shows an application of the device 1 , or of the method 100 , in surveillance technology. Here is a sensor marked as a camera 41 an area 42 monitors, in this example, a bus stop. The signal 41a of the sensor 41 again becomes the input value 11 used and analogous to the 3 and 4 classified. The probability P ( 13a . 13b ), that the input value value 11 a particular class of classes 13a , or a specific value of the classification size 13b , is attributed as a probability that an in 5 exemplified as a suitcase bomb hazard situation 43 is present. Furthermore, the uncertainty 15 of the classification. A high value of uncertainty 15 can then show an unusual or completely unknown situation. For example, this could conceal the attempt of a perpetrator, a deliberately created danger situation 43 to conceal.

QUOTES INCLUDE IN THE DESCRIPTION

This list of the documents listed by the applicant has been generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 10342528 A1 [0002]

Cited non-patent literature

• P. Melville, RJ Mooney, "Various ensembles for Active Learning", Proceedings of the 21 ^st Conference on Machine Learning, Banff (2004 [0004]

Claims (14)

Apparatus (1) for classifying at least one input quantity value (11) into one of a plurality of discrete classes (13a), and / or on the basis of a continuous classification quantity (13b), comprising: • a plurality of by means of learning input values (11a) and learning output values (13c) trainable individual evaluators (12a-12c), which are pre-trained on identical sets of learning input values (11a) and each adapted to assign a probability distribution (P _a , P _b , P _c ) to the input variable (11), wherein this probability distribution (P _a , P _b , P _c ) for each of the discrete classes (13a), or for each possible value of the classification variable (13b), indicates with what probability the input value (11) of the respective class (13a ), or to the respective value (13b), is to be mapped; and • a confidence module (14) adapted to determine from deviations between the probability distributions (P _a , P _b , P _c ) provided by the individual evaluators (12a-12c) the uncertainty (15) with which the classification is subject characterized in that the device (1) additionally comprises a feedback module (16) adapted to respond to this input variable value in response to the uncertainty (15) for an input variable value (11) satisfying a predetermined criterion (15a) (11) as a learning input variable value (11a) to all feedback to individual evaluators (12a-12c), and / or a by the user (18) of the device (1) perceptible physical warning device (17). Device (1) according to Claim 1 , characterized in that between 5 and 50 individual evaluators (12a-12c) are provided. Device (1) according to one of Claims 1 to 2 characterized in that the feedback module (16) is additionally adapted to request a learning output value (13c) belonging to the learning input value (11a) from the user (18) of the device (1) and together with the learning input value ( 11a) to all individual evaluators (12a-12c). Device (1) according to one of Claims 1 to 3 , characterized in that at least one of the individual evaluators (12a-12c) includes an artificial neural network. Device (1) according to one of Claims 1 to 4 , characterized in that in addition an aggregator (19) is provided which is adapted to aggregate the probability distributions (P _a , P _b , P _c ) supplied by the individual evaluators (12a-12c) to an overall probability distribution (P) , Device (1) according to one of Claims 1 to 5 , characterized in that the individual evaluators (12a-12c) are adapted to receive image data as input values (11) and to classify them with regard to the expression of at least one predetermined feature, and / or with regard to the presence of at least one predetermined object. Method (100) for operating and / or training a device (1) for classifying at least one input variable value (11) into one of a plurality of discrete classes (13a), and / or on the basis of a continuous classification variable (13b), wherein • by a plurality of by means of learning input values (11a) and learning output values (13c) trainable individual evaluators (12a-12c) each input value (11) from an identical set a probability distribution (P _a , P _b , P _c ) is assigned (110), wherein this probability distribution (P _a , P _b , P _c ) to each of the discrete classes (13a), or to each possible value of the classification variable (13b), indicates the probability with which the input variable value (11) relates to the respective class (13a) , or to the respective value (13b), is to be mapped; and • from deviations between the probability distributions (P _a , P _b , P _c ) provided by the individual evaluators (12a-12c) the uncertainty (15) is determined (120), which is associated with the classification, characterized in that in response in that the uncertainty (15) for an input variable value (11) satisfies (130) a predetermined criterion (15), the input variable value (11) is fed back as feedback learning input variable value (11a) to all individual evaluators (12a-12c) (140) , and / or a physical warning device (17) perceptible by the user (18) of the device (1) is actuated (150). Method (100) according to Claim 7 , characterized in that a device (1) is selected in which at least two individual evaluators (12a-12c) of different types. Method (100) according to one of Claims 7 to 8th , characterized in that when training at least two Einzelbewerter (12a-12c) are initialized with different startup configurations. Method (100) according to one of Claims 7 to 9 , characterized in that during training at least two individual evaluators (12a-12c) with different sequences of the same learning input values (11a) are applied. A computer program comprising machine-readable instructions which, when executed on a computer and / or on a control device, move the computer and / or the control device to a device (1) according to one of the Claims 1 to 6 upgrade, and / or cause, a method (100) according to one of Claims 7 to 10 perform. Use of a device (1) according to one of Claims 1 to 6 , a method (100) according to one of Claims 1 to 10 and / or a computer program Claim 11 in an at least partially automated vehicle (2), wherein the input variable value (11) is obtained from at least one sensor (21) having a physical measured variable (21a) in the vehicle (2) or in the environment (22) of the vehicle (2 ), and wherein the classes (13a) or the classification variable (13b) characterize an operating state (23) of the vehicle (2) and / or a traffic situation (24). Use of a device (1) according to one of Claims 1 to 6 , a method (100) according to one of Claims 1 to 10 and / or a computer program Claim 11 in diagnostics, wherein the input variable value (11) is obtained from at least one sensor (31) which detects a physical measurand (31a) on an object or patient (32) to be examined, and wherein the classes (13a), Classification size (13b), characterize the degree of severity of at least one abnormality (32a) of the object or patient (32). Use of a device (1) according to one of Claims 1 to 6 , a method (100) according to one of Claims 1 to 10 and / or a computer program Claim 11 in the monitoring technique, wherein the input variable value (11) is obtained from at least one sensor (41) which detects a physical measured variable (41a) from the region (42) to be monitored, and wherein the classes (13a) or the classification variable (13) 13b), characterize the degree of severity of at least one dangerous situation (43).

Images