DE102019206050A1 - Selection of new unlabeled learning data sets for active learning - Google Patents

Abstract

Method for training a trainable module (1), with the following steps: the trainable module (1) is trained (110) with labeled learning data sets (2 *), so that their learning input variable values (11a) are related to the respective learning -Output variable values (13a) are mapped; • for at least one previously unlabeled learning data set (2) from a pool (2p) learning output variable values (13a) are requested (180) so that this unlabeled learning data set (2) becomes a labeled learning data set (2 *); • with the set of labeled learning data sets (2 *) expanded in this way, it branches back (190) to training (110), where • learning input variable values (11a) at least one learning data set (2nd , 2 *), and / or validation input variable values (11a *) at least one validation data set (3), several modifications (1a-1c) of the trainable module (1) are supplied as input variables (11) (120); the deviation of the output variable values (13) into which the modifications (1a-1c) each translate the learning input variable values (11a), a measure of the uncertainty (13b) of these output variable values (13) is determined (130) from one another; and • using the uncertainty (13b), at least one new unlabeled learning data record (2) is selected (150, 160, 170) for addition to the pool (2p). Method (200) with extended functional chain up to the control of physical systems ( 50, 60, 70, 80).

Description

The present invention relates to the training of trainable modules, such as those used, for example, for classification tasks, for quality control and / or for object recognition in at least partially automated driving.

State of the art

Trainable modules, such as artificial neural networks, can be trained with a finite number of situations and the knowledge gained therefrom can be generalized to a large group of unknown situations. For example, an object recognition system for an at least partially automated vehicle can be trained on the basis of a finite number of traffic situations in such a way that it then shows the desired behavior even in unknown, untrained situations. Likewise, for example, a system for quality control of mass-produced products can learn from a finite number of product examples when products are to be classified as defective, and can also recognize this if it manifests itself differently than in the product examples.

When training a trainable module, it is an essential driver of the costs to label the learning data sets, which include, for example, traffic situations or measurement data obtained from product examples, with the conclusion that the trainable module should ideally draw from this. This often has to be done manually. This is why so-called active learning starts with learning data sets, only a very small part of which is labeled. The trainable module then automatically selects those learning data sets that should be labeled next in order to achieve maximum training progress with as few labeled learning data sets as possible.

Disclosure of the invention

Within the scope of the invention, a method for training a trainable module was developed. The trainable module translates one or more input variables into one or more output variables.

A trainable module is viewed in particular as a module that embodies a function parameterized with adaptable parameters with great force for generalization. During the training of a trainable module, the parameters can in particular be adapted in such a way that when learning input variable values are entered into the module, the associated learning output variable values are reproduced as well as possible. The trainable module can in particular contain an artificial neural network, ANN, and / or it can be an ANN.

The training works with active learning. It is therefore based on two types of learning data sets. On the one hand, there is a pool of unlabeled learning data sets that only contain learning input variable values. On the other hand, there are labeled learning data sets which include both learning input variable values and associated learning output variable values. During active learning, unlabeled learning data sets from the pool are initially labeled afterwards after it has been recognized that the labeling of these learning data sets promises a gain in terms of the training result. Further unlabeled learning data records can also be added to the pool in order to search for candidates who are worth labeling in this sense.

The term “learning data set” does not refer to the entirety of all available learning data. An unlabeled learning data record comprises one or more learning input variable values from which the trainable module is to generate one or more output variable values as a statement.

A labeled learning data record also includes the learning output variable values, onto which the trainable module is to map the learning input variable value or values, as a label. In the case of a trainable module used for classification and / or regression, an unlabeled learning data record can include, for example, an image as a matrix of learning input variable values. If this learning data set is labeled, values are added, for example, which the trainable module should ideally generate from this in the form of Softmax scores, as a vector of learning output variable values.

The validation data sets are data sets which, analogously to the learning data sets, contain an assignment of validation input variable values to validation output variable values as target output variable values. However, the trainable module is deliberately not trained on the validation data sets. Therefore, the accuracy determined with the aid of the validation data sets measures the ability of the trainable module to generalize the knowledge learned from the learning data sets. Good values for the accuracy cannot be “obtained” by mere “memorization” of this knowledge. The validation data records can also advantageously be characterized in that the assignment of the validation output variable values to the Validation input values come from a particularly reliable source. The validation data records can therefore be labeled in particular, for example, with a particularly reliable and therefore complex method, by a specially designated expert for the respective application, and / or according to a “gold standard” recognized for the respective application. The effort per label is therefore usually considerably greater for the validation data sets than for the learning data sets. Accordingly, there are typically significantly fewer validation data sets available than learning data sets.

The measurement data were obtained through a physical measurement process and / or through a partial or complete simulation of such a measurement process and / or through a partial or complete simulation of a technical system that can be observed with such a measurement process. This means that the measurement data are fraught with uncertainties and imponderables. In contrast to purely synthetic data, the associated learning output variable values are not available from the outset as labels for all learning input variable values, but these labels must be determined subsequently in a more or less complex process, depending on the technical application.

As part of the method, the trainable module is trained with the labeled learning data sets, so that the learning input variable values contained in these learning data sets are mapped to the respective associated learning output variable values. Any desired error measure can be used to assess the extent to which the learning output variable values are reproduced as desired, such as the cross entropy or the method of the smallest error sum of squares.

Active learning provides that, based on a specific training level of the trainable module, learning output variable values are requested for at least one previously unlabeled data set from the pool, so that this previously unlabeled learning data set becomes a labeled learning data set. As a result, the amount of labeled learning data sets is expanded, and at the same time the freshly labeled learning data set is eliminated from the pool of unlabeled learning data sets.

Here, “request” is to be understood as any possibility of obtaining learning output values as a label. For example, an operator can be requested to enter such labels via a local or remote input device. For example, a physical measurement process can also be triggered, which delivers the label as a result. The label can also be called up from a cloud, for example.

This process is now intervened in such a way that, on the basis of the current training status of the trainable module, the procurement of replenishment of unlabeled learning data sets for the pool is specifically controlled. In particular, unlabeled learning data sets are sought, among which there are more likely promising candidates for labeling.

The level of training in active learning is made up of the number of iterations, also known as “acquisition steps”, which differ in the number of labeled learning data sets, and the individual epochs within such an iteration. The amount of labeled learning data sets is thus increased from iteration to iteration, and within the iteration, training is carried out epoch by epoch with the respective existing labeled learning data sets. Thus the iterations (“acquisition steps”) form a rough division of the training level, while the epochs form a fine division of the training level.

Learning input variable values of at least one labeled or unlabeled learning dataset, and / or validation input variable values of at least one validation dataset, are supplied to several modifications of the trainable module as input variables, so that the different modifications each provide different output variable values. The modifications differ so widely that they are not transferred congruently into one another as the learning progresses.

The modifications can, for example, have the same or comparable training levels in order to make the output variable values supplied in each case better comparable. Depending on the application, however, a difference between the training levels of the modifications can also be expressly desired, for example.

The modifications can be structurally different, for example. For example, several modifications of ANNs can be generated by deactivating different neurons as part of a "dropout". However, the modifications can also be generated, for example, by pre-training with equally powerful but sufficiently different subsets of the overall learning data sets available, and / or by pre-training based on sufficiently different initializations. Again, however, it is not imperative that the subsets of the learning data sets used in the preliminary training are equally powerful.

For example, the modifications can be derived from the trainable module after it has been trained to a certain level of training. The modifications can also, for example, be trained independently of one another up to a desired training level.

A measure for the uncertainty of these output variable values is determined from the deviation of the output variable values into which the modifications translate their input variable values.

Determining the uncertainty for labeled learning data sets or validation data sets is advantageous insofar as the respective existing label can be used as a criterion for the later search for unlabeled learning data sets. Otherwise, however, there is complete freedom as to the type of input variable values on the basis of which the uncertainty was determined. This means that further processing of the uncertainty, and decisions based on the uncertainty, depend only on the uncertainty itself and not on whether it is determined for a labeled learning data set, for an unlabeled learning data set or for a validation data set has been.

Using the uncertainty, at least one new unlabeled learning data set is now selected for addition to the said pool. This means that the pool of unlabeled learning data sets that is available for labeling active learning is increased by this newly selected unlabeled learning data set.

The selection can be drawn from one or more sources of any kind, such as from a pool of unlabeled learning data records that have already been recorded but not yet contained in the pool. However, a continuous data stream of such learning data sets or a cloud can also serve as a source, for example. The learning data record to be selected can also be generated artificially on demand, which can be particularly advantageous because the desired type (class or metaclass) of the unlabeled data record can be controlled directly during artificial generation. The only decisive factor is that a new learning data record of a certain type, which was not previously included in the pool and was therefore not yet involved in active learning, comes into the pool and is thus involved in active learning.

It was recognized that it depends on the current training status of the trainable module and on the previous history of the training which unlabeled learning data sets can advance the training of the trainable module. In particular, it depends on the variability of the situations that the trainable module has already got to know in the previous training.

For example, in the context of at least partially automated driving, it is essential that the detection of lane markings, lane boundaries, traffic signs, other road users and other objects work reliably under all circumstances. Regardless of the time of day, the season or the weather conditions, a stop sign must always be classified as a stop sign and a pedestrian must always be classified as a pedestrian. In order to work towards this, for example during test drives, pictures from the vehicle environment are recorded in all conceivable situations and made available as learning data records to a trainable module responsible for the classification of objects. The greater the variability of the overall situation in which a certain object occurs, the greater the probability that the trainable module will appropriately generalize this learning experience and recognize the object in completely unknown situations.

How great the variability presented in the learning data sets must be, and in what respect variability is required, is difficult to see here. In the case of active learning, there is also the fact that the variability of the unlabeled learning data sets offered does not necessarily affect the actual training of the trainable module. In active learning, only a selection of the unlabeled learning data sets is labeled, and only the labeled learning data sets are used for the monitored training of the trainable module. Therefore, a sufficiently diverse pool of unlabeled learning data sets can ultimately be evaporated into a set of labeled learning data sets that is too one-sided with regard to one or more aspects.

In this situation, the method provides a remedy in that, on the basis of a current training status and based on the determined uncertainty, it is at least limited which types of unlabeled learning data sets are likely to advance further training. In this way, it can be recognized, for example, if only very few of an originally generously dimensioned set of learning data sets with images of traffic situations recorded in autumn in sunshine made it into the selection for labeling, so that these traffic situations ultimately Training of the trainable module are underrepresented. Work can be done to add more such images to the pool and, if necessary, to label them.

The active control of the types of learning data records which are to be newly added to the pool leads to the fact that the newly added learning data records contain a higher probability of promising candidates for labeling. For each ultimately labeled learning data set, a total of fewer unlabeled learning data sets from the pool have to be checked. This reduces the computational effort for active learning.

Furthermore, the costs for the physical or simulatory procurement of the unlabeled learning data sets also decrease. Any unlabeled learning data set that remains unlabeled in the long term does not go into the final training of the trainable module. The final state of the trainable module achieved at the end of the training would also have been achieved if the acquisition of the learning data sets in the pool, which were still unlabeled in this state, had been saved.

An exemplary application in which the acquisition of unlabeled learning data sets requires increased effort is the analysis of medical images for the presence or the degree of severity of certain diseases. New unlabeled learning data sets can only be obtained here through new physical examinations of patients, which requires their cooperation. The image acquisition process itself can, for example, require the use of expensive large-scale equipment for radiological imaging and time-consuming sample preparation for microscopic tissue examinations.

Even with at least partially automated driving, in addition to labeling learning data sets, the procurement of unlabeled learning data sets can be complex. For example, test drives of such vehicles may require the use of special vehicles with additional sensors (such as lidar or radar) or the ride of a monitoring human test driver.

In a particularly advantageous embodiment, the selection of the new unlabeled learning data set for addition to the pool is based at least on the fact that the uncertainty is greater or smaller than a predetermined quantile of the uncertainties determined from learning input variable values of a large number of unlabeled learning data sets or as a predetermined threshold. This is a sign that the trainable module finds it particularly difficult to make a statement about certain learning input variable values. As the learning input variable values are fed to the various modifications of the trainable module, the statement made by the trainable module is put on the "shaking level", as it were. The greater the discrepancy between the output variable values that the various modifications provide, the lower the probability that the output variable values supplied by the actual trainable module are exactly correct.

In a further particularly advantageous embodiment, labeled or unlabeled learning data records or validation data records for which uncertainties have been determined are classified into a plurality of classes. At least one new unlabelled learning data set belonging to a class selected on the basis of the uncertainty is selected for addition to the pool. This is based on the knowledge that, particularly in the case of learning data sets with physically or simulatively collected measurement data, an increased uncertainty of the information provided by the trainable module can often be determined by one or more specific phenomena that were effective when the measurement data was collected. For example, if the training status of the trainable module is not yet sufficient, a difference between two times of the day or seasons at which images were recorded can be misinterpreted as a difference in the semantic object content. The division into classes, in particular, facilitates the search for candidates for a new, unlabeled learning data record to be added to the pool, or the examination of whether such a candidate corresponds to the selection criterion.

Labeled learning data sets and validation data sets can be classified particularly easily. The respective label can, for example, be used directly as an indicator for class membership.

In a particularly advantageous manner, a summarizing statistic of the uncertainties is determined separately for the individual classes about the learning data records that have not yet been labeled and / or about the learning data records that have already been labeled and / or specified labeled validation data records. The summary statistics for the various classes are compared with one another. This comparison can provide a “ranking” of the classes with regard to the uncertainty, which is a measure of how worthwhile it is to add further unlabeled learning data sets for these classes to the pool with a view to achieving further training progress.

Therefore, a class is selected based on the comparison. At least one new unlabeled learning data set belonging to this selected class is selected for addition to the pool.

For example, pictures taken from a vehicle can be classified according to the combination of time of year and time of day at which they were be classified. For example, the combinations of spring and early morning and of autumn and early morning can lead to a particularly high level of uncertainty. One possible reason for this is that with these combinations of season and time of day, the low-lying sun shines directly into the camera, so that the available dynamic range of the image sensor is largely exhausted and the brightness levels of the actual image features are only discretized with a comparatively poor resolution.

For example, whenever a new unlabeled learning data record is to be added to the pool, an unlabeled learning data record can be selected from that class for which the greatest uncertainty was previously determined.

In a further particularly advantageous embodiment, several unlabeled learning data sets, the distribution of which to the individual classes is weighted according to the summarizing statistics for the various classes, are selected for addition to the pool. In this way, further training is focused on the classes with the greatest uncertainties, while at the same time the variability of the learning data sets is preserved.

In a further advantageous embodiment, a preselection of unlabeled learning data sets is determined on the basis of the uncertainty. Several unlabeled learning data sets, the distribution of which to the individual classes is weighted according to the distribution of the pre-selection to the classes, are then selected for addition to the pool. With this approach, the information contained in the preselection, with which type of learning data sets is particularly difficult for the trainable module, is converted into a “wish list” expressed in classes, which can be easily implemented when procuring new unlabeled learning data sets.

In a further particularly advantageous embodiment, the assignment of unlabeled learning data records to the classes is determined by the trainable module on the basis of its current training status. This is not to be equated with the later labeling in the context of active learning. However, this subdivision into classes by no means requires the level of accuracy that is striven for in the actual training of the trainable module with the labeled learning data sets. Rather, the subdivision into classes at this point only serves to roughly limit which unlabeled learning data records are likely to contain information that is useful for further training of the trainable module. The "pseudo-labels" that the trainable module delivers with its current training status are sufficient for this. Any incorrect assignments are "caught" by the fact that the unlabeled learning data records added to the pool must later undergo a selection process as part of active learning in order to ultimately be labeled and included in the training of the trainable module.

In a particularly advantageous embodiment, the classes represent different types of situations in which the learning input variable values of the labeled or unlabeled learning data sets or validation input variable values of the validation data sets were recorded. As explained above, this can be, for example, seasons, times of day, weather conditions, lighting conditions or other external conditions that interact in some form with the physical process that supplies the measurement data (in reality or simulated). For example, a changed temperature as such does not have a direct effect on the recording of images, apart from the temperature dependence of the electronic noise of image sensors. A change of season from summer to winter, on the other hand, changes the lighting conditions and thus has a direct effect on the recording of the images.

A division of the classes according to situations can in particular be independent of the classes that are defined by the labels contained in the labeled learning data sets and validation data sets, and in this respect convey a new “perspective” on the data.

For example, the labeled or unlabeled learning data sets, or the validation data sets, can be assigned to the situations in which they were recorded on the basis of metadata. Such metadata are often recorded as learning input variable values or as validation input variable values when the measurement data are recorded. For example, camera modules often record the type and basic settings, such as ISO sensitivity or zoom factor, of the camera module and the date and time. In conjunction with a GPS module, the location is often also recorded. In this way, for example, the weather conditions are also determined at the time the image was taken, because these can be retrieved from databases afterwards using the date, time and location, for example.

In a further particularly advantageous embodiment, the classes represent the presence of different types of impairments in the labeled or unlabeled learning data records or in the validation data records. These impairments can for example statistical errors such as noise, but also systematic errors such as blurring of an image due to defocus blurring or motion blurring. In the case of a simulation, impairments can arise, for example, through discretization and / or rounding within the framework of the model used. In general, an impairment can be understood to mean any transfer function effective when recording the measurement data as learning input variable values or as validation input variable values, according to whose effect on the measurement data the recognizability of at least one feature relevant to the respective application in the measurement data is worse than before.

A division of the classes according to impairments can in particular be independent of the classes that are defined by the labels contained in the labeled learning data sets and validation data sets, and in this respect convey a further new “perspective” on the data.

The aforementioned impairments are a main source of uncertainty in practice. Different impairments act in different ways on the measurement data, so that accordingly different countermeasures are indicated in order to reduce the influence of the impairments. For example, the influence of noise can be at least partially eliminated by blurring, while the influence of defocus blurring or motion blurring is exacerbated by blurring.

Furthermore, learning data records or validation data records can be classified with comparatively little effort in terms of the types of impairments they are afflicted with. For example, it is much easier to classify an image as having defocus blurring than it is to create a complete semantic segmentation of this image.

If the different classes represent different types of impairments, the analysis of the uncertainties broken down by class may, for example, reveal that the uncertainty is particularly great for images with shot noise. During further training, additional images with shot noise can be added to the pool in order to give the trainable module “tutoring” to reliably recognize the features relevant to the respective application in the images despite the shot noise.

In a further particularly advantageous embodiment, the affiliation of at least one candidate for a new unlabeled learning data set to be added to the pool is checked for at least one class on the basis of a similarity comparison of this candidate with learning data sets and / or validation data sets belonging to this class. This similarity comparison can take place, for example, with a correlation or also, for example, with an unsupervised, ie without labeled learning data records, trained trainable module. Depending on the specific design of the similarity search, it can be faster, for example, to search through a large pool of images that are still unlabeled using the similarity search than to first classify the entire pool and use this classification to select an image.

In a further particularly advantageous embodiment, at least one new unlabeled data record is selected for addition to the pool on the basis of at least one specific feature. In particular, a preselection of unlabeled learning data records can be determined on the basis of the uncertainty, for example. At least one common characteristic can then be determined from the learning input variable values of the unlabeled learning data records in the preselection. A new, unlabeled learning data set, which has the at least one common characteristic, can then be selected for addition to the pool.

As explained above, mostly individual features in the learning input variable values of the learning data sets are responsible for the fact that the trainable module finds it particularly difficult to process them. By specifically selecting further unlabeled learning data sets with these characteristics, the trainable module “tutoring” is given precisely in the “subjects” where its “grades” have so far been worst.

In a further particularly advantageous embodiment, the new unlabeled learning data record is artificially generated on the basis of the common feature and / or it is obtained by impressing the common feature on measurement data. In this way, the correct processing of learning data sets containing this feature can be trained with high variability even if no “real”, ie physically recorded or simulated, learning data sets with this feature are available. This is particularly advantageous when physical observation of this feature is only rarely possible. For example, extreme heavy rain and extreme snowfall, both of which can impair the detection of objects in images recorded by vehicles, are comparatively rare. The artificial generation of learning data sets with extreme heavy rain or extreme snowfall, or the imprinting of these conditions on "real" images, creates a sufficiently large variability without having to wait for a sufficient number of such extreme events.

The output variables supplied by the trainable module can in particular contain a classification, regression and / or semantic segmentation of the input measurement data. Especially when determining this information from the inputted measurement data, the ability to generalize trainable modules such as ANNs is important.

The invention also relates to a parameter set with parameters which characterize the behavior of a trainable module and were obtained with the method described above. These parameters can, for example, be weights with which inputs from neurons or other computing units are calculated in an ANN to activate these neurons or computing units. This parameter set embodies the effort that has been invested in active learning and is, in this respect, an independent product. This means that if you have this data set available in a comparable situation, you can save yourself the time-consuming labeling of learning data sets.

The invention also relates to a further method, which includes the continuation of the chain of effects started with the training up to the control of physical systems.

In this method, a trainable module is first trained with the method described above. This trainable module is then operated by feeding it input variable values. These input variable values include measurement data obtained through a physical measurement process and / or through a partial or complete simulation of such a measurement process and / or through a partial or complete simulation of a technical system that can be observed using such a measurement process. Depending on the output variable values supplied by the trainable module, a vehicle and / or a classification system and / or a system for quality control of mass-produced products and / or a system for medical imaging is controlled with a control signal.

In particular, the methods can be implemented entirely or partially by computer. The invention therefore also relates to a computer program with machine-readable instructions which, when they are executed on one or more computers, cause the computer or computers to carry out one of the methods described. In this sense, control units for vehicles and embedded systems for technical devices, which are also able to execute machine-readable instructions, are to be regarded as computers.

The invention also relates to a machine-readable data carrier and / or to a download product with the computer program. A download product is a product that can be transmitted over a data network, i.e. A digital product which can be downloaded by a user of the data network and which can be offered for sale for immediate download, for example, in an online shop.

Furthermore, a computer can be equipped with the computer program, with the machine-readable data carrier or with the download product.

Further measures improving the invention are shown in more detail below together with the description of the preferred exemplary embodiments of the invention with reference to figures.

Figure list

• 1 Ausführungsbeispiel des Verfahrens 100 zum Trainieren;
• 2 Ausführungsbeispiel des Verfahrens 200 mit Fortsetzung der Wirkkette;
• 3 Beispielhafte Verläufe der Unsicherheit 13b in Abhängigkeit der Epochenzahl e des Trainings des trainierbaren Moduls 1, ermittelt auf verschiedenen Klassen 20-29.

It shows:

• 1 Embodiment of the method 100 for exercising;
• 2 Embodiment of the method 200 with continuation of the chain of action;
• 3 Exemplary courses of uncertainty 13b depending on the epoch number e of the training of the trainable module 1 , determined on different classes 20-29 .

1 shows an embodiment of the method 100 . In step 110 becomes the trainable module 1 with labeled learning data sets 2 * trained so that the learning records labeled in these 2 * contained learning input variable values 11a to the respective associated learning output variable values 13a be mapped.

The workout 110 can be parameters in particular 12 define the behavior of the trainable module 1 characterize.

In step 120 become various modifications 1a-1c of the trainable module 1 Learning input variable values 11a from labeled learning data sets 2 * and / or from unlabeled learning data sets 2 , and / or validation input values 11a * from validation data sets 3 , fed. The validation records 3 additionally contain validation output values 13a * as labels that are compared to the learning output values 13a of the labeled learning Records 2 * have been determined in a more complex way.

Any variation 1a-1c supplies to the learning input variable values 11a from each unlabeled learning data set used 2 a different output value 13 than the other variations 1a-1c . From the deviation of these output variable values 13 each other is in step 130 the uncertainty 13b the output variable values 13 determined.

The determined uncertainty 13b can now be used in various ways to create a new unlabeled learning data set 2 for adding to the pool 2p to select.

In step 140 can use the already existing unlabeled learning data sets 2 into a plurality of classes 20-29 be classified. It can then step into 150 at least one new unlabeled learning data set 2 the one based on uncertainty 13b selected class 20-29 listened for addition to the pool 2p to be chosen.

Inside the box 140 there are several exemplary ways in which the classification can be carried out in detail.

According to block 141 can assign the unlabeled learning data sets 2 , the labeled learning records 2 * , or the validation data sets 3 , to the classes 20-29 through the trainable module 1 can be determined on the basis of his current training status. As explained above, a rough assignment is sufficient here, and also any incorrect classifications due to the not yet perfect training of the trainable module 1 do not have a noticeable disadvantageous effect on further training.

According to block 142 can the classes 20-29 represent different types of situations in which the learning input values 11a of the unlabeled learning records 2 or the labeled learning data sets 2 * , or the validation input variable values 11a * of the validation records 3 , were recorded. For example, a differentiation can be made here according to times of day, seasons, weather conditions, lighting conditions or combinations of such factors. The assignment can be made according to block 142a can be determined, for example, on the basis of metadata, which when the respective learning input variable values 11a , or the respective validation input variable values 11a * , were recorded. Examples of such metadata are location, date and time.

According to block 143 the classes can detect the presence of various types of impairments, such as, for example, noise, blurring or other imaging errors, in the unlabeled learning data sets 2 , in the labeled learning data sets 2 * , or in the validation data sets 3 , represent. As explained earlier, it is significantly easier to create a learning record 2 , 2 * , or a validation data set 3 to classify with regard to the presence of such impairments than to analyze its semantic meaning, for example.

Inside the box 150 There are several exemplary ways of doing this broken down as one to a particular class 20-29 corresponding new unlabeled learning data set 2 can be selected.

According to block 151 can for each class 20-29 each a separate summary statistic 20a-29a the uncertainties 13b about the as yet unlabeled records 2 , and / or via the already labeled learning data sets 2 * , and / or via predefined labeled validation data sets 3 , be determined. These summary statistics 20a-29a for the different classes 20-29 can according to block 152 be compared with each other. Based on this, according to block 153 a class 20-29 to be chosen. It can then according to block 154 at least one new unlabeled learning data set 2 , of this selected class 20-29 listened for addition to the pool 2p to be chosen.

According to block 155 can based on the uncertainty 13b , so initially detached from the classes 20-29 , a preselection 2a unlabeled learning records 2 be determined. It can then according to block 156 several unlabeled learning records 2 , their distribution to the individual classes 20-29 according to the distribution of the preselection 2a on the classes 20-29 is weighted for addition to the pool 2p to be chosen.

According to block 157 can generally indicate membership of at least one candidate for a new pool 2p learning data set to be added 2 to at least one class 20-29 based on a similarity comparison of this candidate with this class 20-29 associated learning data sets 2 , 2 * , and / or validation records 3 , being checked. This makes it easier in particular to select from a large pool of candidates who have not yet entered the classes 20-29 are divided into one of the desired class 20-29 find affiliated candidates.

Alternatively or in combination to choose from 150 of a new unlabeled learning data set 2 from a certain class 20-29 can in step 160 at least one new unlabeled record 2 based on at least one characteristic specific to the addition to the pool 2p to be chosen. One exemplary way this can be done is inside the box 160 broken down in more detail.

According to block 161 can initially based on the uncertainty 13b a preselection 2a unlabeled learning records 2 be determined. From the learning input variable values 11a of the unlabeled learning records 2 in the preselection 2a can according to block 162 at least one common characteristic 2 B be determined. According to block 163 can then create a new unlabeled learning data set 2 that has at least one common characteristic 2 B for addition to the pool 2p to be chosen. This type of selection is in particular not based on certain types of common features known a priori 2 B limited.

Alternatively or in combination to choose from 150 from one class 20-29 and to choose from 160 using a specific feature, however, can also be done in step 170 in any other way a new unlabeled learning data set 2 for adding to the pool 2p to be chosen.

In step 180 gets out of the pool 2p of the total currently available unlabeled learning data sets 2 at least one learning record 2 selected, and for this learning data set 2 become learning output values 13a requested. This creates exactly this learning data set 2 to a labeled learning data set 2 * . With the amount of labeled learning data sets expanded in this way 2 * becomes training 110 branched back.

This cycle, and thus the training as a whole, can end, for example, when the in training 110 achieved accuracy meets a specified criterion. Active learning then has the desired accuracy, ideally with a minimum of effort for labeling initially unlabeled learning data sets 2 achieved. Corresponding termination conditions are in 1 not shown for the sake of clarity.

2 shows an embodiment of the method 200 which continues the chain of effects from training to the control of technical systems.

In step 210 becomes the trainable module 1 using the procedure described above 100 trained. In step 220 this trainable module is operated by giving it input variable values 11 are fed. These input variable values include measurement data obtained through a physical measurement process and / or through a partial or complete simulation of such a measurement process and / or through a partial or complete simulation of a technical system that can be observed using such a measurement process.

The trainable module 1 delivers at his operation 220 Output values 13 . In step 230 becomes dependent on these output variable values 13 a control signal 5 generated. With this control signal 5 becomes a vehicle 50 , and / or a classification system 60 , and / or a system 70 for quality control of mass-produced products, and / or a system 80 for medical imaging.

3 shows examples of a number of courses of summary statistics 20a-29a the uncertainties 13b that are on different classes 20-29 were created, depending on the epoch number e of the training of the trainable module 1 . The training levels of the modifications 1a-1c that is responsible for determining the uncertainties 13b are used, for example, can be based on this epoch number e. The summary statistics 20a-29a diverge significantly with increasing epoch number e.

Claims (20)

Method for training a trainable module (1), which translates one or more input variables (11) into one or more output variables (13), by means of a pool (2p) of unlabeled learning data sets (2), the learning input variable values (11a) Include measurement data, as well as labeled learning data sets (2 *), the learning input variable values (11a) and associated learning output variable values (13a), the measurement data being obtained by a physical measurement process and / or by a partial or complete simulation of such a measurement process , and / or by a partial or complete simulation of a technical system that can be observed with such a measurement process, with the following steps: the trainable module (1) is trained (110) with the labeled learning data sets (2 *), see above that their learning input variable values (11a) are mapped onto the respective associated learning output variable values (13a); • For at least one previously unlabeled learning data set (2) from the pool (2p), learning output variable values (13a) are requested (180) so that this unlabeled learning data set (2) becomes a labeled learning data set (2 *) becomes; • With the set of labeled learning data sets (2 *) expanded in this way, it branches back to training (110) (190), where • learning input variable values (11a) of at least one learning data set (2, 2 *), and / or validation Input variable values (11a *) of at least one validation data set (3), several modifications (1a-1c) of the trainable module (1) are supplied (120) as input variables (11), these modifications (1a-1c) being so far apart distinguish that they are not transferred congruently into one another as learning progresses; • from the deviation of the output variable values (13) into which the modifications (1a-1c) each translate their input variable values (11), a measure of the uncertainty (13b) of these output variable values (13) is determined (130); and • using the uncertainty (13b), at least one new unlabeled learning data set (2) is selected (150, 160, 170) for addition to the pool (2p). Method (100) according to Claim 1 wherein the selection (170) of the new unlabeled learning data set (2) for addition to the pool (2p) is based at least on the fact that at least one uncertainty (13b) is greater or smaller than a predetermined quantile of the learning input variable values (11a ) a plurality of unlabeled learning data sets (2) ascertained uncertainties (13b) or as a predetermined threshold value. Method (100) according to one of the Claims 1 to 2 , learning data sets (2, 2 *) and / or validation data sets (3), for which uncertainties (13b) were determined, being classified (140) into a plurality of classes (20-29) and at least one new unlabeled learning data set (2), which belongs to a class (20-29) selected on the basis of the uncertainty (13b), is selected (150) for addition to the pool (2p). Method (100) according to Claim 3 , whereby • for the individual classes (20-29) separately summarizing statistics (20a-29a) of the uncertainties (13b) about the learning data sets (2) that have not yet been labeled and / or about the learning data sets that have already been labeled (2 *), and / or using predefined labeled validation data sets (3), is determined (151), • the summarizing statistics (20a-29a) for the different classes (20-29) are compared with one another (152), • on the Based on this comparison (152), a class (20-29) is selected (153) and • at least one new unlabeled learning data record (2) belonging to this selected class (20-29) for addition to the pool (2p) is selected (154). Method (100) according to Claim 4 , with several unlabeled learning data sets (2), the distribution of which to the individual classes (20-29) is weighted in accordance with the summarizing statistics (20a-29a) for the various classes (20-29), for addition to the pool (2p ) can be selected (154a). Method (100) according to one of the Claims 3 to 5 , with a preselection (2a) of unlabeled learning data sets (2) being determined (155) based on the uncertainty (13b) and with several unlabeled learning data sets (2), their distribution to the individual classes (20-29) according to the distribution of the preselection (2a) is weighted on the classes (20-29) are selected (156) for addition to the pool (2p). Method (100) according to one of the Claims 3 to 6 , the assignment of unlabeled learning data sets (2) to the classes (20-29) by the trainable module (1) being determined on the basis of its current training status (141). Method (100) according to one of the Claims 3 to 7th , the classes (20-29) representing different types of situations (142) in which the learning input variable values (11a) of the learning data sets (2, 2 *), or validation input variable values (11a *) of the validation Records (3), were recorded. Method (100) according to Claim 8 , whereby the assignment of the learning data sets (2, 2 *) or the validation data sets (3) to the types of situations is determined on the basis of metadata (142a) which are generated when the respective learning input variable values (11a ) or the respective validation input variable values (11a *). Method (100) according to one of the Claims 3 to 9 , the classes (20-29) representing (143) the presence of different types of impairments in the learning data sets (2, 2 *) or in the validation data sets (3). Method (100) according to one of the Claims 3 to 10 , the affiliation of at least one candidate for a new unlabeled learning data set (2) to be added to the pool (2p) to at least one class (20-29) on the basis of a similarity comparison of this candidate with learning data sets belonging to this class (20-29) ( 2, 2 *) and / or validation data sets (3) is checked (157). Method (100) according to one of the Claims 1 to 11 , wherein at least one new unlabeled data record (2) is selected (160) on the basis of at least one specific feature for addition to the pool (2p). Method (100) according to Claim 12 , where • based on the uncertainty (13b) a preselection (2a) of unlabeled learning data sets (2) is determined (161), • from the learning input variable values (11a) of the unlabeled learning data sets (2) in the preselection (2a) at least one common feature (2b) is determined (162) and • a new, unlabeled learning data set (2) which contains the at least one common feature (2b) is selected (163) to be added to pool (2p). Method (100) according to Claim 13 , the new unlabeled learning data set (2) • being artificially generated (163a) using the common feature (2b) and / or • being obtained by impressing the common feature (2b) on measurement data (163b). Method (100) according to one of the Claims 1 to 13 , wherein the output variables (13) supplied by the trainable module (1) contain a classification, regression and / or semantic segmentation of the input measurement data. Method (200) with the steps: • a trainable module (1) is created with the method (100) according to one of the Claims 1 to 15th trained (210); • the trainable module (1) is operated (220) in that input variable values (11) are fed to it, these input variable values (11) comprising measurement data obtained by a physical measurement process and / or by a partial or complete simulation of such a measurement process, and / or by a partial or complete simulation of a technical system observable with such a measurement process; • A vehicle (50) and / or a classification system (60) and / or a system (70) for quality control of products manufactured in series and / or is dependent on the output variable values (13) supplied by the trainable module (1) a system (80) for medical imaging, controlled (230) with a control signal (5). Parameter set with parameters (12) which characterize the behavior of a trainable module (1), obtained with the method (100) according to one of the Claims 1 to 15th . Computer program containing machine-readable instructions which, when executed on one or more computers, cause the computer or computers to implement a method according to one of the Claims 1 to 16 execute. Machine-readable data carrier and / or download product with the computer program Claim 18 . Computer equipped with the computer program according to Claim 18 , and / or with the machine-readable data carrier and / or download product Claim 19 .

Images