DE102021208727A1 - Training a synthetic measurement data generator with improved variability control - Google Patents

Abstract

Method (100) for training a generator (4) that maps input vectors (22) from a latent distribution (21) to synthetic measurement data (2) of a specified domain, with the steps: • samples (11) of measurement data are generated of the given domain from a first source (3a) as real samples (1) (110); (3a) acquired samples (11) are conditioned as further real samples (1) acquired (120); • input vectors (22) from the latent distribution (21) are sampled (130) and generated by the generator (4 ) are mapped onto fake samples (2) with synthetic measurement data (140);• the real samples (1) are combined with the fake samples (2) to form a pool P (150);• the pool P becomes samples P * drawn (160) and classified (170) by a discriminator (5) as to whether the samples are fake (2) or not;• parameters (5a ) that characterize the behavior of the discriminator (5) are optimized (180) with the aim that the classification accuracy of the discriminator (5) improves; • Parameters (4a) that characterize the behavior of the generator (4) are optimized (190) with the aim that the classification accuracy of the discriminator (5) deteriorates.

Description

The present invention relates to synthetic measurement data generators that can be used, for example, to generate training data for image classifiers.

State of the art

Many driver assistance systems and systems for at least partially automated driving process the measurement data recorded by sensors of a vehicle with classifiers to form classification scores in relation to one or more classes of a specified classification. On the basis of these classification scores, for example, decisions are then made about interventions in the driving dynamics of the vehicle.

The training of such classifiers requires training data with a high degree of variability, so that the classifier can generalize well to measurement data previously unseen in the training. The recording of training data on test drives with the vehicle and especially the largely manual labeling of this training data with target classification scores are time-consuming and expensive.

Therefore, the training data is often enriched with synthetically generated training data. So reveals about the DE 10 2018 204 494 B3 a method with which radar signals can be generated synthetically in order to enrich physically recorded radar signals for training a classifier.

Disclosure of Invention

As part of the invention, a method was developed for training a generator that maps input vectors from a latent distribution to synthetic measurement data of a specified domain. The domain is specified by the respective specific application and can include images of traffic situations, for example.

In this case, the latent distribution can in particular be a random distribution, for example. "Synthetic measurement data" means in particular measurement data that appear so realistic in the context of a given application that they can be used for further processing on an equal footing and can also be used in mixtures with measurement data actually recorded by one or more sensors. This further processing can consist in particular, for example, in the synthetic measurement data being used together with physically recorded measurement data as training data for a classifier.

As part of the method, samples of measurement data of the specified domain are procured from a first source as real samples. In particular, a set of measurement data recorded with at least one sensor can be selected as the first source.

Furthermore, samples of measurement data of the specified domain are procured from a second source as further real samples. These samples are conditioned to the samples already obtained from the first source. Broadly speaking, this means that the selection of samples from the second source depends on which samples have already been obtained from the first source. Thus, the samples obtained from the second source become a logical extension of the set of samples obtained from the first source.

In particular, a synthetic generation and/or modification of measurement data can be selected as the second source. This synthetic generation and/or modification can be carried out, for example, with the aim of providing measurement data that differ from the measurement data from the first source in at least one aspect. However, a set of measurement data recorded with at least one additional sensor can also be selected as the second source, for example.

In the total set of real samples, the combination of using two sources for the samples in connection with the conditioning on the samples from the first source ensures that the set of real samples shows greater variability overall, but at the same time more than a unit can be grasped.

Input vectors are sampled from the latent distribution and mapped to fake samples with synthetic measurement data by the generator. The term "fake" is not to be understood in a (derogatory) way, but as a term commonly used in the field of Generative Adversarial Networks, GAN, for measurement data that was generated by a generator to be trained. Conversely, the term "real samples" should not be understood to mean that all these samples originally have to come from one sensor. Rather, this means those samples from which the samples generated by the generator should be as difficult as possible to distinguish.

The real samples are combined with the fake samples to form a pool P. Samples P* are drawn from this pool P and from one Discriminator classifies whether the samples are fake or not.

Parameters characterizing the behavior of the discriminator are optimized with the aim of improving the classification accuracy of the discriminator. However, parameters that characterize the behavior of the generator are also optimized with the aim of deteriorating the classification accuracy of the discriminator. In this way, the generator is trained to produce “falsifications” of measurement data of ever better quality, while at the same time the discriminator is trained to better and better distinguish between reconstructed real samples on the one hand and fake samples on the other.

In particular, this training can be carried out, for example, on the total set of real samples. For a training step, the pool P can then contain, for example, a batch of real samples in addition to a batch of fake samples, which contains both samples from the first source and samples from the second source. However, pools P can also be used alternately, for example, in which a batch of fake samples is combined only with a batch of samples from the first source or only with a batch of samples from the second source. The samples from both sources can thus each cause their own updates of gradients of the parameters to be optimized. The real samples from both sources can also be combined using any other strategy.

The variability gained by adding the samples from the second source means that the generator is no longer only trained on the modes of training data defined by the samples from the first source, but also on others defined by the samples from the second source fashions. This effectively increases the variability in the output of the generator. The measurement data generated by the generator is therefore no longer concentrated in one or a few modes, but rather covers the specified domain of the measurement data better overall.

The conditioning on the samples from the first source avoids in particular that the advantageous variability in the training data of the generator turns into an inherent contradiction of this training data. When trying to resolve such an inconsistency and "please all training data", the generator can introduce artifacts into the synthetic measurement data it generates as a "lazy compromise". In images as synthetic measurement data, such artifacts can manifest themselves, for example, in textures or shapes that do not occur in realistic images, and/or objects of specific classes that would actually be expected there can be missing in the images. Such traces in the synthetic measurement data offer the discriminator simple starting points to identify the samples with synthetic measurement data as such without in-depth analysis of the semantic content. Furthermore, the conditioning to the samples from the first source has the effect that the generator advantageously "mixes" samples from the first and second source in the same mode and, for example, does not recognize that synthetically generated measurement data from the second source, for example, contain "corner cases" (such as such as near-accident traffic situations) or represent anomalies. On the other hand, if the generator sorts measurement data from the second source (e.g. synthetically generated measurement data) into a separate group without learning their relationship to the sensor data from the first source, the ability to reliably generate "corner cases" can suffer as a result.

For example, if all the samples from the first source are images of conventional automobiles, adding samples showing station wagons, SUVs and other modifications of automobiles from a second source can improve the variability of the generator generated images of vehicles. On the other hand, if the images from the second source show trucks, buses, ships or planes, there are hardly any realistic images that are overall consistent with a union of all training images. The generator can therefore try, for example, to "bend" an image of a truck by texturing it with a specific pixel pattern so that it is also consistent with the cars in the training images. An example of this would be a truck with a car depicted on its body. In general, when a request is made to the generator to generate images with a class or combination of classes that is rarely or never seen in training, there is little control over the result. For example, the generator could simply ignore one of the classes that absolutely doesn't want to fit, or it could produce an overall nonsensical image in an attempt to fit all requested classes into a single image.

In a particularly advantageous embodiment, the conditioning of the samples from the second source to the samples from the first source thus includes that the samples from the second source belong to a distribution defined by the samples from the first source with regard to one or more predetermined properties, and/or which are otherwise similar to the samples from the first source with respect to those one or more properties.

• • die in den Bildern enthaltenen Farben;
• • die in den Bildern enthaltenen Objektklassen; und/oder
• • die in den Bildern enthaltenen Kombinationen von Objektklassen, umfassen.

As explained above, images can be selected as samples of measurement data, for example. Then the predetermined property with respect to which the samples from the second source are conditioned on the samples from the first source, e.g

• • the colors contained in the images;
• • the object classes contained in the images; and or
• • include the combinations of object classes contained in the images.

In the aforementioned example, images of other types of cars (e.g. station wagons or SUVs) in many variants could be obtained from the second source. The real samples would then only relate to passenger cars.

In a further advantageous embodiment, the samples from the second source are modified in such a way that they belong to a lesser extent to a distribution defined by the samples from the first source with regard to at least one predefined property, and/or in relation to this at least one predefined property property otherwise become more dissimilar to the samples from the first source. In this way, the tendency can be increased that the samples from the second source still teach the generator something new, but without resulting in fundamental contradictions to the samples from the first source. Overall, modifying is a particularly easy way to obtain more real samples, which in turn makes it less likely that the generator will recognize a difference between the measurement data recorded by at least one sensor from the first source on the one hand and synthetically generated measurement data from the second source on the other .

In the above example, the shapes and colors of the cars shown in the images from the second source can be changed within wide limits without there being any doubt that these images show cars in each case.

• • den in den Bildern enthaltenen Farben eine oder mehrere Farben hinzuzufügen;
• • den in den Bildern enthaltenen Objektklassen eine oder mehrere Objektklassen hinzuzufügen; und/oder
• • den in den Bildern enthaltenen Kombinationen von Objektklassen eine oder mehrere Kombinationen hinzuzufügen.

The modification can include, for example,

• • add one or more colors to the colors contained in the images;
• • add one or more object classes to the object classes contained in the images; and or
• • add one or more combinations to the combinations of object classes contained in the images.

• • als Funktion der Epoche E, in der der Generator und der Diskriminator trainiert werden, und/oder
• • als Funktion des Indexes T eines aktuellen Batches von Real-Samples, der in dieser Epoche E verarbeitet wird,

variiert. So können beispielsweise zu Beginn des Trainings nur leichte Abwandlungen zugelassen werden und erst im weiteren Trainingsverlauf stärkere Abwandlungen. Auf diese Weise kann die Wahrscheinlichkeit erhöht werden, dass das Training des Generators gerade zu Beginn in die richtige Richtung läuft.In a further particularly advantageous embodiment, the strength of the modification

• • as a function of epoch E, in which the generator and the discriminator are trained, and/or
• • as a function of the index T of a current batch of real samples processed in this epoch E,

varies. For example, only slight modifications can be allowed at the beginning of the training session and stronger modifications only as the training progresses. In this way, the probability can be increased that the training of the generator runs in the right direction, especially at the beginning.

In a further particularly advantageous embodiment, the samples from the second source are determined independently of one another for a plurality of batches of samples from the first source. Each individual training step uses a specific batch of samples. So that each of these steps leads in the right direction, it is important that the real samples within this batch are consistent with one another, so that a statement can be obtained from this batch for the next update of the generator parameters.

If, for example, both the samples from the first source and the samples from the second source each show different variants of cars, trucks and buses, these samples can be optimally used for training the generator by only including images of one of the vehicle types in a batch be summarized. If, on the other hand, cars and buses are mixed in a batch, it may not be possible to derive a meaningful training step from this.

In a further advantageous embodiment, the real samples required for training the generator and the discriminator in a predetermined number of epochs are determined completely in advance, before training of the generator and the discriminator is started. In this way, there is no need to wait for the provision of new samples from the second source during training. If the samples from the second source are generated synthetically with a neural network, for example, this network must be traversed once in the forward direction in order to provide each sample.

For example, a pseudo-random order can be specified in which batches of samples are obtained from the first source. A batch of samples can then be obtained from the second source for each of these batches. This creates an overall sequence of batches of real samples, each containing samples from the first source and samples from the second source. The batches can be stored in a memory and retrieved from this memory when training the generator and the discriminator.

As explained above, the generator can be used in particular to generate synthetic training data for a classifier. Therefore, in a further advantageous embodiment, further input vectors from the latent distribution are mapped onto synthetic measurement data of the specified domain with the completely trained generator. A classifier for measurement data is trained using this synthetic measurement data as training data.

The synthetic measurement data can cover a significantly greater variability than measurement data recorded with a sensor alone. In particular, imbalances between the frequency with which certain situations occur in the training data on the one hand and the importance of these situations on the other hand can be leveled out.

For example, it can be expected that in training data with images of traffic situations, the most common traffic signs for giving priority and speed limits will appear much more frequently than, for example, a traffic sign that announces an unguarded level crossing or a road leading to an unsecured bank. Nevertheless, precisely these signs, if they occur, must be recognized with absolute certainty, since incorrect recognition is more likely to have serious consequences than, for example, incorrect recognition of a speed limit.

Thus, in a further particularly advantageous embodiment, the trained classifier is supplied with further measurement data recorded with at least one sensor. The classifier then maps this measurement data to classification scores related to one or more classes. A control signal is formed from the classification scores. A vehicle and/or a system for product quality control and/or a system for monitoring areas is controlled with this control signal. The training of the classifier with training data of a higher variability ultimately has the effect that the reaction of the controlled technical system to the situation on which the measurement data is based is appropriate with a higher probability of this situation.

In particular, the method can be fully or partially computer-implemented. The invention therefore also relates to a computer program with machine-readable instructions which, when executed on one or more computers, cause the computer or computers to carry out the method described. In this sense, control devices for vehicles and embedded systems for technical devices that are also able to execute machine-readable instructions are also to be regarded as computers.

The invention also relates to a machine-readable data carrier and/or a download product with the computer program. A downloadable product is a digital product that can be transmitted over a data network, i.e. can be downloaded by a user of the data network and that can be offered for sale in an online shop for immediate download, for example.

Furthermore, a computer can be equipped with the computer program, with the machine-readable data carrier or with the downloadable product.

Further measures improving the invention are presented in more detail below together with the description of the preferred exemplary embodiments of the invention with the aid of figures.

exemplary embodiments

• 1 Ausführungsbeispiel des Verfahrens 100 zum Trainieren eines Generators 4;
• 2 Beispiel für die Zusammenstellung von Real-Samples 1 mit Samples 11, 12 aus zwei Quellen 3a, 3b.

It shows:

• 1 Embodiment of the method 100 for training a generator 4;
• 2 Example of the compilation of real samples 1 with samples 11, 12 from two sources 3a, 3b.

1 FIG. 1 is a schematic flowchart of an embodiment of the method 100 for training a generator 4. The generator 4 maps input vectors 22 from a latent distribution 21 onto synthetic measurement data 2. FIG.

In step 110, samples 11 of measurement data of the specified domain are procured as real samples 1 from a first source 3a. According to block 111, for example, images can be selected as samples 11 in particular.

In step 120, samples 12 of measurement data of the specified domain are obtained as further real samples 1 from a second source 3b, which are conditioned to the samples 11 already obtained from the first source 3a.

According to block 121, the conditioning of the samples 12 from the second source 3b to the samples 11 from the first source 3a can include that the samples 12 from the second source 3b with respect to one or more predetermined properties by the samples 11 from the first Belong to source 3a defined distribution, and/or are similar to the samples 11 from the first source 3a in relation to these one or more properties in any other way.

• • die in Bildern enthaltenen Farben;
• • die in Bildern enthaltenen Objektklassen; und/oder
• • die in Bildern enthaltenen Kombinationen von Objektklassen,

umfassen.This predetermined property can, for example, according to block 121a

• • the colors contained in images;
• • the object classes contained in images; and or
• • the combinations of object classes contained in images,

include.

According to block 122, the samples 12 from the second source 3b can be modified so that they belong to a lesser degree to a distribution defined by the samples 11 from the first source 3a and/or in relation to at least one predetermined property at least one predetermined property become more dissimilar in some other way to the samples 11 from the first source 3a.

• • den in Bildern enthaltenen Farben eine oder mehrere Farben hinzuzufügen;
• • den in Bildern enthaltenen Objektklassen eine oder mehrere Objektklassen hinzuzufügen; und/oder
• • den in Bildern enthaltenen Kombinationen von Objektklassen eine oder mehrere Kombinationen hinzuzufügen.

This predetermined property can include, for example, according to block 122a,

• • add one or more colors to the colors contained in images;
• • add one or more object classes to the object classes contained in images; and or
• • add one or more combinations to the combinations of object classes contained in images.

The magnitude of this modification can be varied as block 122b progresses through training.

• • eine Menge von mit mindestens einem weiteren Sensor erfassten Messdaten, und/oder
• • eine synthetische Generierung und/oder Abwandlung von Messdaten, als zweite Quelle 3b für Samples 12 gewählt werden.

According to block 123,

• • a quantity of measurement data recorded with at least one further sensor, and/or
• • a synthetic generation and/or modification of measurement data, can be selected as the second source 3b for samples 12.

According to block 124, the samples 12 from the second source 3b can be determined independently of one another for a plurality of batches of samples 11 from the first source 3a.

In step 130, input vectors 22 from the latent distribution 21 are sampled.

In step 140 the input vectors 22 are mapped by the generator 4 onto fake samples 2 with synthetic measurement data.

In step 150, the reconstructed real samples 1' are combined with the fake samples 2 to form a pool P.

In step 160, samples P* are drawn from the pool P.

In step 170, the samples P* are classified by a discriminator 5 as to whether they are fake samples 2 or not. in the in 1 example shown, each sample P* can be either real sample 1 or fake sample 2.

In step 180, parameters 5a, which characterize the behavior of discriminator 5, are optimized with the aim of improving the classification accuracy of discriminator 5. The fully trained state of the parameters 5a is denoted by the reference symbol 5a*.

In step 190, parameters 4a, which characterize the behavior of generator 4, are optimized with the aim that the classification accuracy of discriminator 5 deteriorates. The fully trained state of the parameters 4a is denoted by the reference symbol 4a*. These parameters 4a* define the completely trained generator 4.

In step 210, further input vectors 22 from the latent distribution 21 are mapped onto synthetic measurement data 2 of the specified domain using the completely trained generator 4*.

In step 220, a classifier 6 for measurement data is trained using this synthetic measurement data 2 as training data. The fully trained state of this classifier 6 is denoted by the reference symbol 6*.

In step 230, the trained classifier 6* is supplied with further measurement data 11', which were recorded with at least one sensor. The classifier then forms this measurement data 11' on classification scores 6a-6c in relation to one or more classes.

In step 240, a control signal is formed from the classification scores 6a-6c.

In step 250, a vehicle 50 and/or a system 60 for product quality control and/or a system 70 for monitoring areas is controlled with this control signal 240a.

In 2 is an example of how samples 11, 12 from two sources can be combined into a set of real samples 1.

The first source 3a is a set of images a, b, c of cars recorded with at least one sensor.

The second source 3b is a set of synthetically generated images d, e, f, g, h and i of vehicles of different types. Pictures d, e and h show different types of cars, which differ significantly from the cars shown in pictures a, b and c. Picture f shows a truck, picture g shows a ship, and picture i shows a school bus.

in the in 2 In the example shown, it makes a lot of sense to combine the images a, b and c as samples 11 from the first source 3a with the images d, e and h to form a total set of real samples 1. If the generator 4 is trained in this way, its ability to generate images of cars with greater variability is significantly improved. This is exactly what happens when the selection of the samples 12 from the second source 3b is conditioned to the samples 11 already present from the first source 3a.

On the other hand, the quality of the images generated by generator 4 would possibly deteriorate if it were trained with a combination of images a, b and c and one or more of images f (truck), g (ship) or i (school bus). As previously explained, the set of training images for generator 4 would then be self-contradictory. This could result in Generator 4 introducing artifacts into the generated images in order to somehow resolve the contradictions and "make all training images more or less right".

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102018204494 B3 [0004]

Claims (16)

Method (100) for training a generator (4), which maps input vectors (22) from a latent distribution (21) onto synthetic measurement data (2) of a specified domain, with the steps: • Samples (11) of measurement data of the specified domain are procured (110) from a first source (3a) as real samples (1); • Samples (12) of measurement data of the specified domain from a second source (3b), which are conditioned to the samples (11) already procured from the first source (3a), are procured as further real samples (1) (120 ); • input vectors (22) are sampled (130) from the latent distribution (21) and mapped (140) by the generator (4) to fake samples (2) with synthetic measurement data; • the real samples (1) are combined with the fake samples (2) to form a pool P (150); • Samples P* are drawn from the pool P (160) and classified (170) by a discriminator (5) as to whether they are fake samples (2) or not; • Parameters (5a) characterizing the behavior of the discriminator (5) are optimized (180) with the aim of improving the classification accuracy of the discriminator (5); • Parameters (4a), which characterize the behavior of the generator (4), are optimized (190) with the aim that the classification accuracy of the discriminator (5) deteriorates. Method (100) according to claim 1 , wherein the conditioning of the samples (12) from the second source (3b) to the samples (11) from the first source (3a) includes (121) that the samples (12) from the second source (3b) with respect to belong to one or more predetermined properties of a distribution defined by the samples (11) from the first source (3a), and/or in relation to these one or more properties in some other way to the samples (11) from the first source (3a) are similar. Method (100) according to any one of Claims 1 until 2 , images being selected (111) as samples (11, 12) of measurement data. Method (100) according to claim 2 and 3 , where the given property • the colors contained in the images; • the object classes contained in the images; and/or • the combinations of object classes contained in the images (121a). Method (100) according to any one of Claims 1 until 4 , wherein the samples (12) from the second source (3b) are modified (122) in such a way that, with regard to at least one predetermined property, they are to a lesser degree of a distribution defined by the samples (11) from the first source (3a). belong and/or become more dissimilar to the samples (11) from the first source (3a) in relation to this at least one predetermined property. Method (100) according to claim 3 and 5 wherein the modifying includes (122a) • adding one or more colors to the colors contained in the images; • add one or more object classes to the object classes contained in the images; and/or • to add one or more combinations to the combinations of object classes contained in the images. Method (100) according to any one of Claims 5 until 6 , where the strength of the modification • as a function of the epoch E, in which the generator (4) and the discriminator (5) are trained, and/or • as a function of the index T of a current batch of real samples (1), the processed in this epoch E is varied (122b). Method (100) according to any one of Claims 1 until 7 , A set of measurement data recorded with at least one sensor being selected as the first source (3a) for samples (11) (113). Method (100) according to any one of Claims 1 until 8th , wherein • a quantity of measurement data recorded with at least one further sensor, and/or • a synthetic generation and/or modification of measurement data is selected as the second source (3b) for samples (12) (123). Method (100) according to any one of Claims 1 until 9 , wherein the samples (12) from the second source (3b) to several batches of samples (11) from the first source (3a) are determined independently (124). Method (100) according to any one of Claims 1 until 10 , wherein the real samples (1) required for training the generator (4) and the discriminator (5) in a predetermined number of epochs are determined completely in advance (155) before training the generator (4) and the discriminator (5) is started. Method (100) according to any one of Claims 1 until 11 , where • further input vectors (22) from the latent distribution with the completely trained generator (4*). (21) are mapped onto synthetic measurement data (2) of the specified domain (210) and • using this synthetic measurement data (2) as training data, a classifier (6) for measurement data is trained (220). Method (100) according to claim 12 , where • the trained classifier (6*) is supplied with further measurement data (11') (230), which were recorded with at least one sensor, so that the classifier can convert this measurement data (11') to classification scores (6a-6c) in depicts reference to one or more classes; • a control signal (240a) is formed (240) from the classification scores (6a-6c); and • a vehicle (50), and/or a system (60) for product quality control, and/or a system (70) for monitoring areas, is controlled (250) with this control signal (240a). Computer program containing machine-readable instructions which, when executed by one or more computers, cause the computer or computers to perform the method (100) according to any one of Claims 1 until 13 to execute. Machine-readable data carrier with the computer program Claim 14 . One or more computers with the computer program after Claim 14 , and/or with the machine-readable data medium and/or download product claim 15 .

Images