WO2020216622A1

WO2020216622A1 - Detecting and removing noise in labels of learning data for trainable modules

Info

Publication number: WO2020216622A1
Application number: PCT/EP2020/060006
Authority: WO
Inventors: William Harris Beluch; Maximilian AUTENRIETH; Jan Mathias Koehler
Original assignee: Robert Bosch Gmbh
Priority date: 2019-04-26
Filing date: 2020-04-08
Publication date: 2020-10-29
Also published as: DE102019206049A1

Abstract

The invention relates to a method (100) for training a trainable module (1), having the following steps: a plurality of modification processes (1a-1c) of the trainable module (1), said modification processes differing from one another to such a degree that the modification processes do not completely lead over to one another during a progressive learning process, are each pre-trained (110) at least using a sub-quantity of learning data sets (2); • learning input variable values (11a) of at least one learning data set (2) are fed (120) to all of the modification processes (1a-1c) as input variables (11); • the degree of uncertainty (13b) of output variable values (13) is ascertained (130) from the deviation of the output variable values (13), into which each of the learning input variable values (11a) are converted by the modification processes (1a-1c), from one another, and • in response to the uncertainty (13b) satisfying a specified criterion (140), the weighting of the learning data set (2) when training the trainable module (1) is adapted (180) and/or one or more learning output variable values (13a) of the learning data set (2) are adapted (190). The invention also relates to a method (200) in which the trainable module is further operated (220) and actuates a system (50, 60, 70, 80) using an actuation signal (5).

Description

description

Title:

Detection and elimination of noise in labels of learning data for trainable modules

The present invention relates to the training of trainable modules, such as are used, for example, for classification tasks and / or object recognition in at least partially automated driving.

State of the art

The driving of a vehicle in traffic by a human driver is usually trained by repeatedly confronting a learner driver with a certain canon of situations as part of his training. The learner driver has to react to these situations and receives feedback from comments or even intervention by the driving instructor as to whether his reaction was correct or incorrect. This training with a finite number of situations is intended to enable the learner driver to master even unfamiliar situations while driving the vehicle independently.

In order to allow vehicles to participate fully or partially automatically in road traffic, the aim is to control them with modules that can be trained in a very similar way. These modules receive, for example, sensor data from the vehicle environment as input variables and, as output variables, supply control signals with which the operation of the vehicle is intervened, and / or preliminary products from which such control signals are formed. For example, a classification of objects in the vicinity of the vehicle can be such a preliminary product. For this training, a sufficient amount of learning data sets is required, each of which includes learning input variable values and associated learning output variable values. For example, the learning input variable values can include images and can be labeled as learning output variable values with the information about which objects are contained in the images.

Disclosure of the invention

In the context of the invention, a method for training a

trainable module. The trainable module translates one or more input variables into one or more output variables.

A trainable module is viewed in particular as a module that embodies a function parameterized with adaptable parameters with great force for generalization. During the training of a trainable module, the parameters can in particular be adapted in such a way that when learning input variable values are input into the module, the associated learning output variable values are reproduced as well as possible. The trainable module can in particular contain an artificial neural network, ANN, and / or it can be an ANN.

The training takes place on the basis of learning data sets which contain learning input variable values and associated learning output variable values as labels. In this case, at least the learning input variable values include measurement data obtained through a physical measurement process and / or through a partial or complete simulation of such a measurement process and / or through a partial or complete simulation of a technical system that can be observed with such a measurement process. In contrast to purely synthetic data, after the machine recording of learning input variable values, the associated learning output variable values are not immediately available as labels, but these labels must be determined in a process that is more or less complex depending on the technical application. Most of the time, this process requires human work and is accordingly prone to errors. The term “learning data set” does not designate the entirety of all available learning data, but a combination of one or more learning input variable values and learning output variable values assigned to precisely these learning input variable values as labels. With one for them

Classification and / or regression used trainable module, a learning data record, for example, an image as a matrix of learning input variable values in combination with the Softmax scores, which the trainable module should ideally generate therefrom, as a vector of learning output variable values.

As part of the method, a plurality of modifications of the trainable module are each pre-trained with at least a subset of the learning data records. The modifications differ so widely that they are not transferred congruently into one another as the learning progresses. The modifications can be structurally different, for example. For example, several modifications of ANNs can be generated by deactivating different neurons as part of a "dropout". However, the modifications can also be generated, for example, by pre-training with sufficiently different subsets of the total learning data sets that are present, and / or by pre-training based on sufficiently different initializations.

The modifications can, for example, be pre-trained independently of one another. However, it is also possible, for example, to bundle the pre-training in that only one trainable module or a modification is trained and further modifications are only generated from this module or this modification after this training has been completed.

After the preliminary training, learning input variable values of at least one learning data set are fed to all modifications as input variables. These identical learning input variable values are used by the different

Modifications translated into different output values. A measure for the uncertainty of these output variable values is determined from the deviation of these output variable values from one another. The output variable values can be, for example, Softmax scores which indicate the probabilities with which the learning data set is classified into which of the possible classes.

Any statistical function can be used to determine the uncertainty from a large number of output variable values. Examples of such statistical functions are the variance, the standard deviation, the mean value, the median, a suitably chosen quantile, the entropy and the variation ratio.

Provided that the modifications of the trainable module have been generated in different ways, for example on the one hand by "dropouts" and

on the other hand through other structural changes or through a different initialization of the pre-training, in particular, for example, the

Deviations between those output variable values which are supplied by modifications generated in different ways are compared separately from one another. For example, the

Deviations between output variable values that were supplied by modifications resulting from "dropouts" and the deviations between output variable values that were supplied by modifications that were otherwise structurally changed are considered separately from one another.

In this context, the terms “deviations” and “uncertainty” are not restricted to the one-dimensional, univariate case, but encompass sizes of any dimension. For example, there can be several

Uncertainty features are combined to get a multivariate uncertainty. This increases the accuracy of differentiation between learning data sets with an appropriate assignment of the learning output variable values to the learning input variable values (i.e. "appropriately labeled" learning data sets) on the one hand and learning data sets with an incorrect assignment (i.e.

“Incorrectly labeled” learning data sets) on the other hand.

In response to the fact that the uncertainty fulfills a predetermined criterion, the weighting of the learning data set is adjusted in the training of the trainable module, and / or one or more learning output variable values of the learning data set are adjusted. It was recognized that with an appropriate assignment of the learning output variable values to the learning input variable values, the different modifications of the trainable module have a tendency to

to issue unanimous "opinions" regarding the initial size. The information contained in the relevant assignment asserts itself in the pre-training, as it were, and has the effect that the differences between the modifications manifest themselves little or not at all in different output variables. The less accurate the assignment, the more precisely this effect is missing and the greater are the deviations between the output variable values that deliver the modifications to the same learning input variable values.

If all learning data sets are analyzed in this way, then it will typically emerge that the assignment is more accurate for some learning data sets than for other learning data sets. This mainly reflects the fact that the assignment, i.e. the label, is made by people in most applications of trainable modules and is accordingly prone to errors. For example, in the interest of a high throughput per learning data set, humans can only have a very short time available so that in cases of doubt they cannot investigate more precisely, but have to make some kind of decision. Also can

For example, different editors interpret the criteria according to which they should be labeled differently. For example, if an object casts a shadow in an image, an operator can count this shadow as part of the object, since it was caused by the presence of the object. However, another editor cannot count the shadow as part of the object with the

Reason that the shadow is not something that a person or vehicle can collide with.

Likewise, for example, there are medical data such as

for example image data, ambiguities. For the recognition of many

Diseases and the respective degree of severity, a concrete method is recognized as the "gold standard" to make the respective statement with the greatest possible

To meet accuracy. However, this accuracy often goes with such a high level Effort is associated with the fact that it is not practicable to labein all of the learning data sets required for training the trainable module in this way.

Possible negative effects that a possibly inaccurate labeling of learning data sets may have can be counteracted by adapting the weighting of the learning data set in the training of the trainable module.

In particular, contradictions that arise during training through the

Processing of correctly labeled learning data sets on the one hand and incorrectly labeled learning data sets on the other hand result, mitigate or completely dissolve. The adjustment of the weighting can go so far that a learning data set recognized as incorrectly labeled is no longer taken into account in further training.

Alternatively or also in combination with the adaptation of the weighting, one or more learning output variable values of the learning data set can also be adapted. As explained above, a partially or completely incorrect assignment of learning output variable values to learning input variable values is the ultimate cause of a greater uncertainty in the output variable values. An adjustment of the learning output values tackles the evil of the high uncertainty, so to speak, at the root.

In the simplest case, the adaptation of the learning output variable value can be aimed specifically at reducing the uncertainty. So it can

For example, the learning output variable value can be varied in accordance with any optimization algorithm or any other search strategy with the optimization aim of reducing the uncertainty. Such a correction is self-consistent and does not require any prior knowledge as to which new learning output variable is correct.

A combination of both measures can be useful, for example, if the efforts to obtain more accurate learning output variable values (labels) are only successful for some of the learning data sets. Learning data sets whose learning output variable values prove to be incorrect and also cannot be improved can then, for example, be underweighted or completely disregarded. In a particularly advantageous embodiment, adaptable parameters that characterize the behavior of the trainable module are optimized. The aim of this optimization is to improve the value of a cost function. The cost function measures the extent to which the trainable module maps the learning input variable values contained in the learning data sets to the associated learning output variable values.

In conventional training of trainable modules, all learning data sets have equal rights in this regard, i.e. the cost function measures how well the learning output variable values are reproduced. Any desired error measure can be used to assess the extent to which the learning output variable values are reproduced as desired, such as the cross entropy or the method of the smallest error sum of squares.

This process is modified in such a way that, in response to the fact that the specified criterion is met, the weighting of at least one learning data set in the cost function is reduced.

For example, a learning data set can be weighted less, the higher the uncertainty of the output variable values determined on the basis of its learning input variable values. This can go to the point that, in response to the fact that the uncertainty fulfills a given criterion, this learning data set falls out of the cost function entirely, i.e. is no longer used for further training of the trainable module. This is based on the knowledge that the additional benefit brought about by taking into account another learning data set is the result of an imprecise or incorrect learning output value in the training process

Contradictions can be fully or partially compensated, or even overcompensated. So no information can be better than wrong

Information.

The specified criterion can in particular include, for example, that the uncertainty is greater or smaller than a specified quantile of the learning input variable values determined from a large number of other learning data sets Uncertainties or as a predetermined threshold. For example, the criterion can include that the learning data record belongs to those k% learning data records whose learning input variable values have

Output variable values are translated with the highest uncertainties. This means that the uncertainties for which these k% learning data sets are responsible are at least as great as the uncertainties that arise from all other learning data sets that do not belong to the k%. This is based on the knowledge that selective measures, such as adjusting the weighting and / or the label, have the greatest effect on those learning data sets that are labeled the most inappropriately.

The adaptation of one or more learning output variable values, that is to say the label, can in particular include requesting the input of at least one new learning output variable value via an input device.

In particular, the assignment of the new learning output variable value to the learning input variable values can be carried out by an expert with regard to the correct interpretation of the learning input variable values. For example, images obtained by medical imaging can be labeled by specialists with regard to the anomaly, the presence or form of which is to be determined with the trainable module. Measurement data recorded on mass-produced products can be labeled for these products by experts who, for example, saw through a copy of the product and examine it from the inside. Images of traffic situations that the trainable module is supposed to classify for the purpose of at least partially automated driving can be labeled, for example, by an expert in traffic law who also knows how to correctly interpret complicated traffic situations with a combination of several traffic signs.

The examples mentioned show that it can be time-consuming to obtain a new, more appropriate label for an incorrectly labeled learning data set with the help of an expert. In terms of content, this may be the “gold standard” in the respective subject area, but the expert is not always immediately available. Therefore, in a further particularly advantageous embodiment, the adaptation of one or more learning output variable values includes at least one output variable value that the trainable module and / or a modification of this trainable module assigns to the learning input variable values of the learning data set at its current training level, and /or one

Offsetting several such output variable values to set this learning data set as a new learning output variable value. The offsetting can include, for example, a mean value or median.

For this purpose, it can be particularly advantageous for the pre-training of the

Modifications to choose an epoch number e, in which the

Basics of the contextual relationship between learning input values and learning output values have been learned, but that in

In individual cases the wrong label of a learning data set has not yet been finally learned.

The situation is in some ways comparable to the fact that juvenile justice education measures are most likely to bring about the desired behavioral change if administered at the appropriate stage in the developmental process. A child who is too young would only be frightened off by arrest or social hours, for example, without understanding the point, while a young person, whose criminal misconduct is already established, may only comment on the new measure with the words “always bring it on, I'll collect it”.

Therefore, in a particularly advantageous embodiment, in response to the fact that the course of at least one uncertainty as a function of the epoch number e of the preliminary training assumes a minimum for an epoch number ei, the

Modifications for determining the output variable values on a

Be operated, which corresponds to the epoch number e = ei. For example, the uncertainty whose course is examined can be measured with a different statistical measure than the uncertainty with which it is established at all that there is a need for improvement with regard to at least one learning output variable. It was recognized that this minimum marks the transition between the aforementioned learning of the basics, which is not yet affected by the influence of false labels, and the increasing dilution of this

Successful learning by learning from wrong labels. In particular, the trainable module, or the modification, provides a comparatively accurate estimate of which label is applicable instead, at a minimum at e = ei for an incorrectly labeled learning input variable value.

If the assignments between the learning input variable values and the learning output variable values are essentially correct in all learning data sets, then no or only a very weak minimum is to be expected in the course of the uncertainty. Rather, the uncertainty will then decrease monotonically with the number of epochs e, only to level off at some point.

As a rule, in the state with epoch number e = ei, it is not yet possible to recognize that the minimum is precisely here. This only becomes apparent when the determined uncertainty increases again for values e> ei. If this is recognized, however, then the training status in the modifications can be reset to the epoch e = ei for further investigation.

The training level that is optimal in this respect can alternatively or also in combination with this also be determined via the accuracy with which the validation input variable values from validation data sets are transferred to associated

Validation output values are mapped.

The validation data records are data records which, analogous to the learning data records, contain an assignment of input variable values to target output variable values. However, the trainable module is deliberately not trained on the validation data sets. Therefore, the accuracy determined with the help of the validation data sets measures the ability of the trainable module to generalize the knowledge learned from the learning data sets. So good values for the accuracy cannot be determined by mere

"Trickle" to "memorize" this knowledge. The validation data records can also advantageously be characterized in that the assignment of the validation output variable values to the validation input variable values comes from a particularly reliable source. The validation data records can therefore be labeled in particular, for example, with a particularly reliable and therefore complex method, by a specially designated expert for the respective application, and / or according to a “gold standard” recognized for the respective application. The effort per label is therefore usually considerably greater for the validation data sets than for the learning data sets. Accordingly, there are typically significantly fewer

Validation data sets are available as learning data sets.

If the relevant learning output variable values are assigned to the learning input variable values in all learning data records, then it is to be expected that when testing the trainable module with the validation data records, the accuracy increases monotonically with increasing epoch number e of the training until it is at some point in a saturation goes. If, on the other hand, the assignment is incorrect for some of the learning data sets, then the accuracy will assume a maximum after the said learning of the basics, before it decreases again due to learning from the wrong labels.

Therefore, in a further advantageous embodiment, in response to the fact that the accuracy achieved by the trainable module and / or by at least one modification as a function of the epoch number e of the respective training, the validation input variable values from validation data sets are related to the associated validation Output variable values are mapped, assuming a maximum at an epoch number b2, the modifications for determining the output variable values are operated on a training level which corresponds to the epoch number e = e ₂ . Any of the following can be used to determine the accuracy

Error measure are used, such as a mean

absolute deviation, a mean square deviation, the

Cross entropy, or the least square sum method.

Analogous to the determination of the minimum of the uncertainty, one will not immediately recognize at e = e ₂ that the maximum is here. This only becomes apparent when the accuracy drops again for higher epoch numbers e. The modifications are then turned back to the training level e = e ₂ for further investigation. The route via the accuracy provides a determination of the epoch number e, on which the modifications should sensibly be carried out, which is secured by means of the validation data records. However, it was recognized that the value ei determined on the way via the uncertainty is a good approximation for the value

is. This approximation is also available if

Validation data records cannot be obtained or only with too much effort.

As explained above, the qualitative course of both the uncertainty and the accuracy changes significantly as a function of the number of epochs e if a certain proportion of incorrectly labeled learning data sets is added to correctly labeled learning data sets. Therefore, in a further particularly advantageous embodiment, the course of the uncertainty, and / or the accuracy, as a function of the epoch number e of the preliminary training thereupon

concluded to what extent the assignment of the learning output variable values to the learning input variable values in the learning data sets is correct.

In particular, in response to the fact that the uncertainty predominantly decreases as a function of the epoch number e of the preliminary training, it can be determined that the assignment of the learning output variable values to the learning input variable values is essentially correct in all learning data sets.

Likewise, in response that accuracy can be a function of

The epoch number e of the pre-training predominantly increases, it can be established that the assignment of the learning output variable values to the learning input variable values is essentially correct in all learning data sets.

In this context, “predominantly” is to be understood as meaning, for example, an essentially monotonic curve which converges to a constant value. Small statistical fluctuations in the other direction do not affect this.

Under “essentially applicable in all learning data sets” is to be understood in particular that incorrect labels are at most only so low There is scope so that the contradictions arising from this during the training of the trainable module do not prevail as the number of epochs of the training progresses against the learning success achieved on the basis of the applicable labels.

As explained above, learning data sets that are not labeled appropriately are often responsible for the deterioration in the performance of a trainable module trained with them. Most of the errors that occur are individual errors and not systematic errors. Therefore, the measurement signal contained in the uncertainties with regard to incorrectly labeled learning data sets is essentially concentrated in the highest uncertainties.

Therefore, in a further particularly advantageous embodiment, the course of the uncertainty as a function of the epoch number e of the preliminary training is evaluated only for those uncertainties that are greater or smaller than a predetermined quantile of the uncertainties determined from learning input variable values from a large number of learning data sets or as a given

Threshold.

The course of the uncertainty can, for example, be evaluated with a summary statistic. For example, a mean, a median, a variance and / or a

Standard deviation used to determine the uncertainties of the output variable values. The summarizing statistics can be kept separately, for example, on those output variable values which are applicable or not applicable in the light of the respective learning output variable values in accordance with a predetermined criterion. If the trainable module is designed as a classifier, for example, a first mean value or median of the uncertainties of those can be used

Output variable values are formed which assign the respective learning input variable values to the correct class. A second mean value or median can be formed from the uncertainties of those output variable values that assign the respective learning input variable values to the wrong class. The output variables supplied by the trainable module can in particular contain a classification, regression and / or semantic segmentation of the input measurement data. Especially in determining this

Information from the entered measurement data comes down to the

Ability to generalize trainable modules, such as ANN.

The invention also relates to a parameter set with parameters which characterize the behavior of a trainable module and were obtained with the method described above. These parameters can

For example, weights can be used to activate inputs from neurons or other processing units in an ANN.

Arithmetic units are offset. This parameter set embodies the effort that has been invested in the training and is therefore an independent product.

The invention also relates to a further method which the

Continuation of the chain of effects started with the training up to the control of physical systems.

In this method, a trainable module is first trained with the method described above. This trainable module is then operated by feeding it input variable values. This

Input variable values include measurement data obtained by a physical measurement process and / or by a partial or complete simulation of such a measurement process and / or by a partial or complete simulation

Simulation of a technical system that can be observed with such a measurement process. Depending on the output variable values supplied by the trainable module, a vehicle and / or a

Classification system, and / or a system for quality control of mass-produced products, and / or a system for medical imaging, controlled with a control signal.

In particular, the methods can be implemented entirely or partially by computer. The invention therefore also relates to a computer program machine-readable instructions which, when executed on one or more computers, cause the computer or computers to carry out one of the methods described. In this sense are too

Control units for vehicles and embedded systems for technical devices, which are also able to execute machine-readable instructions, are to be regarded as computers.

The invention also relates to a machine-readable one

Data carrier and / or on a download product with the computer program. A download product is a product that can be transmitted over a data network, i.e. digital product downloadable by a user of the data network that

for example, it can be offered for immediate download in an online shop.

Furthermore, a computer with the computer program with which

machine-readable data carrier or equipped with the download product.

Further measures improving the invention are shown in more detail below together with the description of the preferred exemplary embodiments of the invention with reference to figures.

Embodiments

It shows:

Figure 1 embodiment of the method 100 for training;

FIG. 2 exemplary embodiment of the method 200 with continuation of the functional chain up to the control of physical systems 50, 60, 70, 80;

FIG. 3 recognition of whether incorrectly labeled learning data records are still available via the course of the uncertainty 13b as a function of the epoch number e; FIG. 4 recognition of whether incorrectly labeled learning data records are still available via the course of the accuracy 15 as a function of the epoch number e.

FIG. 1 shows an exemplary embodiment of the method 100 for training a trainable module 1. In step 110, a plurality of modifications 1a-1c of the trainable module 1 are pretrained with at least a subset of the existing learning data sets 2. Each learning data record 2 contains learning input variable values 11a and associated learning output variable values 13a.

In step 120, learning input variable values 11a from learning data records 2 are supplied to all modifications 1a-1c as input variables 11. Each modification la-lc generates its own output variable value 13 from this. According to block 121, the modifications la-lc can be operated on a training level which corresponds to an epoch number e = ei, for which the uncertainty 13b of the output variables 13 is minimal. Alternatively or also in combination with this, the modifications la-lc can be operated on a training level according to block 122, which corresponds to an epoch number e = e2, in which the

Accuracy 15, with which the learning input variable values 11a are mapped onto the learning output variable values 13a, is maximum.

In step 125, one or more modifications la-lc are tested on the basis of validation data sets 3. For this purpose, the modification la-lc is supplied with the validation input variable values 11a * of each validation data record 3 as input variables 11. The accuracy 15 is determined with which the modification la-lc reproduces the respective validation output variable values 13a * from this. This accuracy 15 has depending on the

The epoch number e of the preliminary training 110 has a time course 15 (e).

A measure for the uncertainty 13b of the output variable values 13 is determined in step 130 from the deviations of these output variable values 13 from one another. The accuracy 15 can be derived from the direct comparison of the

Output variable values 13 can be determined with the learning output variable values 13a with any degree of error. The uncertainty 13b, its course 13b (e) Depending on the epoch number e of the preliminary training 110 and the accuracy 15 can be evaluated in the following steps individually or in combination, as described below.

In step 140 it is checked whether the uncertainty 13b of the output variable values 13, which were determined using the learning input variable values 11a of this learning dataset 2, meets a predetermined criterion for at least one learning data record 2. If this is the case (truth value 1), then in step 180 the weighting of the learning data record 2 is adapted in the training of the trainable module 1, and / or one or more learning output variable values 13a of the learning data record 2 are adapted in step 190 .

In step 150 it is checked at which epoch number e = ei of the preliminary training 110 the course 13b (e) of the uncertainty 13b assumes a minimum. In step 155, this epoch number e = ei is set as the training level at which the modifications la-lc according to block 121 are to be operated.

In step 160 it is checked at which epoch number e = e2 of the pre-training the course 15 (e) of the accuracy 15 determined on the basis of the validation data sets 3 assumes a maximum. In step 165, this epoch number e = e2 is set as the training level at which the modifications la-lc according to block 122 are to be operated.

In step 170, the extent to which the assignment of the learning output variable values 13a to the learning input variable values 11a in the learning data records 2 is evaluated from the course 13b (e) of the uncertainty 13b, or from the course 15 (e) of the accuracy 15 is applicable overall. That is, it is checked whether the existing learning data records 2 are essentially all correctly labeled or whether the correctly labeled learning data records 2 are also joined by incorrectly labeled learning data records 2 to a significant extent.

According to block 171, in response to the fact that the uncertainty 13b decreases monotonically as a function of the epoch number e, it is established that essentially all learning data sets 2 are correctly labeled. As explained above, the presence of incorrectly labeled learning data sets 2 leads to the contradictions generated in this way, initial training successes are at least partially nullified and the uncertainty 13b increases again.

According to block 172, in response to the fact that the precision 15 increases monotonically as a function of the number of epochs, it is determined that essentially all learning data sets 2 are correctly labeled. If incorrectly labeled learning data sets 2 are present to a significant extent, this accuracy 15 drops again after an initial increase if the influence of the aforementioned

Makes contradictions noticeable.

According to block 173, when checking the course 13b (e), the focus is specifically on those uncertainties 13b that are greater or smaller than a predetermined quantile of the uncertainties 13b determined from learning input variable values 11a of a plurality of learning data records 2 or as a

predetermined threshold. For example, only the largest 25% of the uncertainties 13b can be taken into account.

If in step 180 the weighting of the learning data record 2 is adjusted in the training of the trainable module 1, then this can be integrated into the training with a cost function 14, for example. According to block 181, adjustable parameters 12 which characterize the behavior of the trainable module 1 are optimized with the aim of improving the value of the cost function 14. The cost function 14 measures the extent to which the trainable module 1 maps the learning input variable values 11a contained in learning data sets 2 to the associated learning output variable values 13a. According to block 182, in response to the predefined criterion 140 being met, the weighting of at least one learning data record 2 in the cost function 14 is reduced. According to block 182a, for example, this can go up to the point at which the learning data record 2 is no longer taken into account in the cost function 2.

If one or more learning output variable values 13a of the learning data record 2 are adapted in step 190, at least one new learning output variable value 13a can for example according to block 191 via a

Input device are requested. The new learning output variable value 13a can be used, for example, by an expert on the basis of the learning input variables values 11a can be determined and entered. Alternatively or in combination with this, at least one output variable value 13, which the trainable module 1 and / or one of its modifications 1a-1c determines to the learning input variable values 11a, can be determined as the new learning output variable value 13a. The trainable module 1 can in a certain way use a “self-healing power”. As explained above, this works particularly well if the trainable module 1 has not yet learned too much from incorrectly labeled learning data sets 2, that is to say the training status of a suitable epoch (for example ei or b2) is selected for this.

FIG. 2 shows an exemplary embodiment of the method 200. In step 210 of this method 200, a trainable module 1 is trained using the method 100 described above. The module trained in this way is operated in step 220 in that input variable values 11 with physically recorded and / or simulated measurement data that relate to a technical system are supplied to it. A control signal 5 is formed in step 230 from the output variable values 13 then supplied by the trainable module 1. A vehicle 50 and / or a classification system 60 and / or a system 70 for quality control of mass-produced products and / or a system 80 for medical imaging is controlled with this control signal 5.

FIG. 3 illustrates by way of example how, on the basis of the course 13b (e) of the uncertainty 13b as a function of the epoch number e, it can be determined whether the

existing learning data sets 2 are essentially all correctly labeled.

Curve a represents the case in which both correctly labeled and incorrectly labeled learning data sets 2 are present. As explained above, the uncertainty 13b initially decreases in the course of the training, since the positive initialization of the training is based on a generally random initialization

Learning effect on the basis of the appropriately labeled learning data sets 2. Beyond the minimum of the uncertainty 13b for the epoch number e = ei, the contradictions caused by the incorrectly labeled learning data sets 2 become noticeable in that the uncertainty 13b increases again. Curve b represents the case in which essentially only appropriately labeled learning data sets 2 are available. Here there are no contradictions in training, so that the positive learning effect that results in a steady

decreasing uncertainty 13b until it continues to converge towards saturation.

FIG. 4 illustrates by way of example how, on the basis of the course 15 (e) of the accuracy 15 determined with the validation data records 3 as a function of the epoch number e, it can be determined whether the existing learning data records 2 in

Essentially all are properly labeled.

Analogously to FIG. 3, curve a represents the case in which both correctly labeled and incorrectly labeled learning data records 2 are present. Starting from the random initialization of the trainable module 1, the accuracy 15 initially increases because the positive effect due to the appropriately labeled learning data sets 2 outweigh the negative effect due to the contradictions with the incorrectly labeled learning data sets 2. In the

The number of epochs e = e2 reaches a maximum. Beyond this maximum, the contradictions become increasingly noticeable, and the accuracy decreases again.

Curve b represents the case in which essentially only appropriately labeled learning data sets 2 are available. Here the positive learning effect continues until the accuracy 15 finally converges towards saturation.

Claims

Expectations

1. Method (100) for training a trainable module (1) which translates one or more input variables (11) into one or more output variables (13) by means of learning data sets (2), the learning input variable values (11a) and the associated Contain learning output variable values (13a), at least the learning input variable values (11a) comprising measurement data obtained by a physical measurement process and / or by a partial or complete simulation of such a measurement process and / or by a partial or complete simulation of a such a measuring process observable technical system, with the steps:

• a plurality of modifications (la-lc) of the trainable module (1), which differ from one another so far that they are not transferred congruently into one another as learning progresses, are each pre-trained with at least a subset of the learning data sets (2) ( HO);

• Learning input variable values (11a) of at least one learning data set (2) are fed (120) to all modifications (la-lc) as input variables (11);

• from the deviation of the output variable values (13) into which the

Modifications (la-lc) each translate the learning input variable values (11a); a measure for the uncertainty (13b) of these output variable values (13) is determined (130) from one another;

• in response to the fact that the uncertainty (13b) is a given

Criterion (140) is met, the weighting of the learning data set (2) is adjusted (180) in the training of the trainable module (1), and / or one or more learning output variable values (13a) of the learning data set (2) adapted (190).

2. The method (100) according to claim 1, wherein adaptable parameters (12) which characterize the behavior of the trainable module (1) are optimized

(181), with the aim of improving the value of a cost function (14), this cost function (14) measuring the extent to which the trainable module (1) affects the learning input variable values (11a) contained in learning data sets (2) associated learning output variable values (13a), the weighting of at least one learning data set (2) in the cost function (14) being reduced

(182) if the specified criterion (140) is met.

3. The method (100) according to claim 2, wherein in response to the fact that the predetermined criterion (140) is met, the learning data record (2) is no longer taken into account in the cost function (14) (182a).

4. The method (100) according to any one of claims 1 to 3, wherein the criterion (140) includes that the uncertainty (13b) is greater or smaller than a predetermined quantile of the learning input variable values (11a) of a large number of other learning data sets (2) determined uncertainties (13b) or as a predetermined threshold value.

5. The method (100) according to any one of claims 1 to 4, wherein the

Adapting (190) one or more learning output variable values (13a) includes requesting (191) the input of at least one new learning output variable value (13a) via an input device.

6. The method (100) according to any one of claims 1 to 5, wherein the

Adapting (190) one or more learning output variable values (13a) includes at least one output variable value (13) that the trainable module (1), and / or a modification (la-lc) of this trainable module (1), at its current one Assigning training status to the learning input variable values (11a) of the learning data record (2) and / or setting (192) a calculation of several such output variable values as a new learning output variable value (13a) of this learning data record (2).

7. The method (100) according to any one of claims 1 to 6, wherein in response to the fact that the course of at least one uncertainty (13b) as a function of The epoch number e of the preliminary training (110) when an epoch number ei assumes a minimum (150), the modifications (la-lc) for determining (120) the

Output variable values (13) are operated (121, 155) on a training stand which corresponds to the number of epochs e = ei.

8. The method (100) according to any one of claims 1 to 7, wherein in response to that of the trainable module (1), and / or of at least one modification (la-lc), as a function of the epoch number e of the respective training Achieved accuracy (15) with which the validation input variable values (11a *) from validation data records (3) are mapped onto the associated validation output variable values (13a *), with an epoch number b2 assumes a maximum (160), the modifications ( la-lc) to determine (120) the

Output variable values (13) are operated (122, 165) on a training stand which corresponds to the epoch number e = e2.

9. The method (100) according to any one of claims 1 to 8, wherein from the course of the uncertainty (13b), and / or the accuracy (15), as a function of the epoch number e of the preliminary training (110) it is concluded (170), the extent to which the assignment of the learning output variable values (13a) to the learning input variable values (11a) in the learning data sets (2) is correct.

10. The method (100) according to claim 9, wherein in response to the fact that the uncertainty (13b) predominantly decreases as a function of the epoch number e of the preliminary training (110), it is established (171) that the assignment of the learning output variable values (13a) is essentially applicable to the learning input variable values (11a) in all learning data sets (2).

11. The method (100) according to any one of claims 9 to 10, wherein in response to the fact that the accuracy (15) as a function of the epoch number e des

Pre-training (110) predominantly increases, it is found (172) that the

Assignment of the learning output variable values (13a) to the learning input variable values (11a) is essentially correct in all learning data sets (2).

12. The method (100) according to any one of claims 7 to 11, wherein the course of the uncertainty (13b) as a function of the epoch number e of the preliminary training (110) is evaluated (173) only for those uncertainties (13b) that are larger or smaller as a predetermined quantile of the uncertainties (13b) determined from learning input variable values (11a) of a multiplicity of learning data sets (2) or as a predetermined threshold value.

13. The method (100) according to any one of claims 1 to 12, wherein the output variables (13) supplied by the trainable module (1) contain a classification, regression and / or semantic segmentation of the input measurement data.

14. Method (200) with the steps:

• a trainable module (1) is trained (210) with the method (100) according to one of claims 1 to 13;

• the trainable module (1) is operated (220) by him

Input variable values (11) are supplied, these

Input variable values (11) comprise measurement data obtained through a physical measurement process and / or through a partial or complete simulation of such a measurement process and / or through a partial or complete simulation of a technical system that can be observed with such a measurement process;

• depending on the one supplied by the trainable module (1)

Output variable values (13) is a vehicle (50), and / or a classification system (60), and / or a system (70) for the

Quality control of mass-produced products and / or a system (80) for medical imaging, controlled (230) with a control signal (5).

15. Parameter set with parameters (12) that determine the behavior of a

characterize trainable module (1) obtained with the method according to one of claims 1 to 13.

16. Computer program containing machine-readable instructions that, when executed on one or more computers, the or the Causing computers to carry out a method according to any one of claims 1 to 13.

17. Machine-readable data carrier and / or download product with the computer program according to claim 16.

18. Computer equipped with the computer program according to claim 16, and / or with the machine-readable data carrier and / or download product according to claim 17.