DE102020103274A1 - Method and device for generating training data for teaching a neural network - Google Patents

Abstract

A device for providing training data for teaching a neural network for a specific task is described. The device is set up to determine a weighting function on the basis of a synthetic set of synthetic data samples that have an original probability distribution and on the basis of a real set of real data samples that have a target probability distribution. The device is also set up, taking into account the determined weighting function, to learn a Generative Adversarial Network, GAN for short, the GAN comprising a generator network. The training data can then be determined using the trained generator network.

Description

The invention relates to a device and a corresponding method for generating training data for teaching a neural network.

An at least partially automated (motor) vehicle (e.g. a passenger car, a truck or a bus) can be set up to evaluate sensor data from one or more environment sensors of the vehicle (e.g. an image camera, a lidar sensor, a radar sensor, etc.), e.g. to detect one or more objects around the vehicle. The vehicle can then be automatically guided longitudinally and / or transversely as a function of the one or more detected objects.

The detection of objects on the basis of the sensor data from one or more environment sensors can take place on the basis of specifically learned neural networks, in particular so-called deep neural networks. The learning of neural networks for different tasks, e.g. for the recognition of different types of objects, is typically associated with a relatively high effort, especially with regard to the provision of labeled training data, such as labeled images, for the different tasks.

One approach to providing labeled training data is to generate synthetic training data, e.g. labeled synthetic images, by means of computer animation. However, it has been shown that a neural network learned on the basis of synthetic training data has a limited reliability, e.g. a relatively low object recognition rate, when used with real data (in particular with real images).

The present document deals with the technical task of providing synthetically generated training data for learning a neural network (e.g. for the task of object recognition), which is also highly reliable when applied to real data.

The object is achieved in each case by the independent claims. Advantageous embodiments are described, inter alia, in the dependent claims. It is pointed out that additional features of a patent claim dependent on an independent patent claim without the features of the independent patent claim or only in combination with a subset of the features of the independent patent claim can form a separate invention independent of the combination of all features of the independent patent claim, which can be made the subject of an independent claim, a divisional application or a subsequent application. This applies equally to technical teachings described in the description, which can form an invention that is independent of the features of the independent patent claims.

According to one aspect, a device (e.g. a computer or a server) for providing training data for training a neural network for a specific task is described. The neural network learned for the specific task can be designed, for example, for object classification, object recognition and / or object segmentation (within images). In other words, the particular task can be object classification, object recognition and / or object segmentation (within one or more images).

The device is set up to determine a weighting function on the basis of a synthetic set of synthetic data samples that have an original probability distribution and on the basis of a real set of real data samples that have a target probability distribution. The synthetic amount and / or the real amount can each comprise 100 or more, or 1000 or more, or 10,000 or more samples. A synthetic data sample and / or a real data sample can each include or be an image, in particular a labeled image. A synthetic data sample can, for example, have been generated using a computer simulation or a computer animation. A real data sample can, for example, have been recorded by an environment sensor, in particular by a camera (of a vehicle).

The weighting function can be determined in such a way that the weighting function for a synthetic data sample from the synthetic set indicates how close the synthetic data sample is to the target in terms of a (symmetrical and / or asymmetrical) distance measure. The probability distribution is or is, and / or how well the synthetic data sample fits the target probability distribution in the sense of the distance measure. The distance measure can, for example, comprise or be a measure for a divergence (in particular for a Kullback-Leibler divergence).

The weighting function can in particular be determined in such a way that the weighting function displays a weighting value for each of the synthetic data samples from the synthetic set. The weighting value for a synthetic data sample can indicate how close the synthetic data sample is or is to the target probability distribution in terms of the distance measure, and / or how good the synthetic data sample is in terms of the distance measure to the target - Probability distribution fits. A relatively high weighting value can indicate that the synthetic data sample is relatively close to the target probability distribution and / or fits relatively well with the target probability distribution. On the other hand, a relatively low weighting value can indicate that the synthetic data sample is relatively far removed from the target probability distribution and / or fits the target probability distribution relatively poorly. The weighting function can, for example, be normalized to weighting values between zero and one.

The weighting function can in particular be determined in such a way that the weighting function is or represents an approximation of the ratio of the density function of the target probability distribution to the density function of the original probability distribution. In this way, a Generative Adversarial Network, or GAN for short, can be provided which maintains the semantics of the synthetic data samples in a particularly reliable manner.

The device can be set up to determine the weighting function by means of a Kullback-Leibler importance estimation procedure, or KLIEP for short, method. In this case, the distance measure can comprise a Kullback-Leibler distance or a Kullback-Leibler divergence or a Kullback-Leibler distance or a Kullback-Leibler divergence. A particularly precise approximation of the ratio of the density functions and thus a particularly precise weighting function can thus be provided.

As an alternative or in addition, the device can be set up to determine the weighting function on the basis of a weighted sum of basic functions. The basic functions can include radial basis function, RBF for short, cores. Furthermore, the basic functions can depend on the real data samples from the real set. A particularly precise approximation of the ratio of the density functions and thus a particularly precise weighting function can thus be provided.

The device is also set up to learn a GAN, taking into account the weighting function determined. The GAN includes a generator network which is designed to determine a corresponding adapted data sample on the basis of a synthetic data sample. Furthermore, the GAN further comprises a discriminator network which is designed to analyze an adapted data sample generated by the generator network from a synthetic data sample and a real data sample from the real set (in pairs), and a To make a decision for or against the adapted data sample. In particular, a decision can be made as to whether the adapted data sample or the real data sample is a sample from the target probability distribution. The GAN can be learned in such a way that the discriminator network decides (exactly) in 50% of the cases for the respectively adjusted data sample and / or (exactly) in 50% of the cases against the respectively adjusted data sample.

The device can be set up to determine an error function for teaching the GAN on the basis of the weighting function. In particular, the device can be set up to weight an error term of the error function for a synthetic data sample with the weighting value of the weighting function for the synthetic data sample. The error term of the error function for the synthetic data sample can depend on whether the discriminator network of the GAN decides in favor of the adjusted data sample when analyzing the corresponding adjusted data sample and a real data sample from the real quantity Not. Alternatively or additionally, the error function can be derived from the expected value of the decisions of the discriminator network for the synthetic data samples from the synthetic set, in particular from an expected value of the decisions of the discriminator network weighted with the weighting function for the synthetic data samples from the synthetic set , depend.

By weighting the error function, it can be achieved in an efficient manner that the GAN is increasingly trained on the basis of the synthetic data samples, which already fit relatively well with the target probability distribution (and therefore only need to be adapted relatively little). As a result, the extent of the semantic changes caused by the generator network of the GAN can be reduced.

The device can also be set up to determine the training data on the basis of the generator network of the GAN. In particular, the device can be set up to use the generator network to determine an adapted data sample from the synthetic set as part of the training data for the neural network for each synthetic data sample. The neural network can then be learned in a precise manner on the basis of the training data determined. Object recognition in a vehicle can thus be improved and, based on this, the automated longitudinal and / or lateral guidance of the vehicle can be improved.

The device can be set up to determine an inverse weighting function on the basis of the synthetic amount of synthetic data samples and on the basis of the real amount of real data samples. For a real data sample from the real set, the inverse weighting function can show how close the real data sample is or is to the original probability distribution in the sense of a distance measure and / or how well the real data sample is to the original Probability distribution fits.

Furthermore, the device can be set up to learn an inverse GAN, taking into account the ascertained inverse weighting function and together with the GAN. The inverse GAN can include an inverse generator network which is designed to determine a corresponding inversely adapted data sample on the basis of a real data sample.

In a preferred example, the GAN and the inverse GAN can be learned by means of a common error function, the common error function depending on the weighting function and on the inverse weighting function. The error function can include a first error term that depends on a decision by the discriminator network of the GAN, and a second error term that depends on a decision by the inverse discriminator network of the inverse GAN. The first error term can then be weighted with the weighting function and the second error term can be weighted with the inverse weighting function.

By learning a GAN and an inverse GAN at the same time, the robustness of the learned GAN and thus the quality of the training data provided and thus the quality of the learned neural network can be increased.

According to a further aspect, a (computer-implemented) method for providing and / or determining training data for teaching a neural network for a specific task is described. The method comprises determining a weighting function based on a synthetic set of synthetic data samples that have an original probability distribution and based on a real set of real data samples that have a target probability distribution. The method further comprises the training of a GAN taking into account the weighting function determined. The method also includes determining the training data using the generator network of the trained GAN.

According to a further aspect, a software (SW) program is described. The software program can be set up to be executed on a processor and thereby to execute the method described in this document.

According to a further aspect, a storage medium is described. The storage medium can comprise a software program which is set up to be executed on a processor and thereby to execute the method described in this document.

It should be noted that the methods, devices and systems described in this document can be used both alone and in combination with other methods, devices and systems described in this document. Furthermore, any aspects of the methods, devices and systems described in this document can be combined with one another in diverse ways. In particular, the features of the claims can be combined with one another in diverse ways.

• 1 beispielhafte Komponenten eines Fahrzeugs;
• 2a ein beispielhaftes neuronales Netz;
• 2b ein beispielhaftes Neuron;
• 3 ein beispielhaftes Generative Adversarial Network (GAN); und
• 4 ein Ablaufdiagramm eines beispielhaften Verfahrens zur Bereitstellung von Trainingsdaten zum Anlernen eines neuronalen Netzes.

The invention is described in more detail below on the basis of exemplary embodiments. Show it

• 1 exemplary components of a vehicle;
• 2a an exemplary neural network;
• 2 B an exemplary neuron;
• 3 an exemplary Generative Adversarial Network (GAN); and
• 4th a flowchart of an exemplary method for providing training data for teaching a neural network.

As stated at the beginning, the present document deals with the efficient provision of training data for teaching a neural network for a specific target task. Exemplary tasks of a neural network are: classification, object recognition and / or segmentation.

1 shows exemplary components of a vehicle 100 . The vehicle 100 includes one or more environment sensors 102 that are set up, environment data (ie sensor data) relating to the environment of the vehicle 100 capture. Exemplary environmental sensors 102 are an image camera, a radar sensor, a lidar sensor and / or an ultrasonic sensor. The vehicle also includes 100 typically one or more vehicle sensors 103 , which are set up, state data (ie sensor data) in relation to a state variable of the Vehicle 100 capture. Exemplary state variables are the driving speed and / or the (longitudinal) acceleration of the vehicle 100 . The vehicle also includes 100 one or more actuators 104 for automatic longitudinal and / or lateral guidance of the vehicle 100 . Exemplary actuators 104 are a drive motor, a braking device and / or a steering device.

A control unit 101 of the vehicle 100 can be set up one or more actuators 104 to operate automatically on the basis of the environment data and / or on the basis of the status data. In particular, the control unit 101 be set up on the basis of the sensor data of the one or more environment sensors 102 an object in the vicinity of the vehicle 100 to detect. The one or more actuators 104 can then be operated as a function of the detected object, in particular around the vehicle 100 at least partially automated.

To detect an object based on the sensor data from one or more environmental sensors 102 At least one neural network can be used that has been trained on the basis of training data for the task of object recognition. 2a and 2 B show exemplary components of a neural network 200 , especially a feedforward network. The network 200 comprises two input neurons or input nodes in the example shown 202 , which at a certain point in time t each have a current value of an input variable as an input value 201 record, tape. The one or more input nodes 202 are part of an entry layer 211 . In general, the network 200 be designed, input data with one or more input values 201 (eg the pixels of an image) from a data sample.

The neural network 200 also includes neurons 220 in one or more hidden layers 212 of the neural network 200 . Each of the neurons 220 can use the individual output values of the neurons of the previous layer as input values 212 , 211 have (or at least part of them). In each of the neurons 220 processing takes place in order to generate an output value of the neuron as a function of the input values 220 to determine. The output values of the neurons 220 the last hidden layer 212 can be in an output neuron or output node 220 an output layer 213 processed to the one or more output values 203 of the neural network 200 to determine. In general, the network 200 be designed, output data with one or more output values 203 provide.

2 B illustrates the exemplary signal processing within a neuron 220 , especially within the neurons 202 the one or more hidden layers 212 and / or the output layer 213 . The input values 221 of the neuron 220 are with individual weights 222 weighted to in a total unit 223 a weighted sum 224 of the input values 221 to be determined (if necessary, taking into account a bias or offset 227 ). Through an activation function 225 can be the weighted sum 224 to an initial value 226 of the neuron 220 can be mapped. You can use the activation function 225 For example, the range of values can be limited. For a neuron 220 can eg a sigmoid function or a hyperbolic tangent (tanh) function or a rectified linear unit (ReLU), eg f (x) = max (0, x) as activation function 225 be used. If necessary, the value of the weighted sum 224 with an offset 227 be moved.

A neuron 220 thus has weights 222 and / or an offset, if applicable 227 as a neuron parameter. The neuron parameters of the neurons 220 of a neural network 200 can be learned in a training phase to cause the neural network 200 approximates a certain function and / or models a certain behavior.

Learning a neural network 200 can be done using the backpropagation algorithm, for example. For this purpose, a learning algorithm for the input values can be used in a first phase of a q ^{th epoch} 201 at the one or more input nodes 202 of the neural network 200 output values corresponding to a certain training set of data samples 203 at the output of the one or more output neurons 220 be determined. On the basis of the initial values 203 the error value of an optimization or error function can be determined. In particular, the deviations between the network 200 calculated output values 203 and the nominal output values are calculated from the data samples as error values. As an alternative or in addition, the error function can be derived from a statistical distribution of the output values 203 of the neural network 200 (e.g. to teach-in a GAN, as described in this document).

In a second phase of the q ^th epoch of the learning algorithm, the error or the error value is propagated back from the output to the input of the neural network 200 to layer the neuron parameters of the neurons 220 to change. The error function determined at the output can be partially based on each individual neuron parameter of the neural network 200 can be derived in order to determine an extent and / or a direction for adapting the individual neuron parameters. This learning algorithm can be iteratively repeated for a variety of epochs, up to a predefined convergence and / or termination criterion is reached.

The number of data samples in a training set, for example the number of labeled images from a camera that are required for learning a neural network 200 are typically relatively high. The individual data samples must each be labeled in order to indicate the respective target output values for the input values. Labeling, in particular manual labeling, is associated with a relatively high level of effort.

mit i = 1, ...,N_S, typischerweise nicht der (Ziel-) Wahrscheinlichkeitsverteilung P_r(x) von realen Daten-Stichproben $x_j^r,$

mit j = 1, ..., N_r, übereinstimmt.The data samples of a training data set can be generated synthetically. For example, synthetic images can be generated as data samples by means of computer animation. The synthetic images can then be labeled automatically as part of the computer animation. However, it has been shown that a neural network 200 , which was learned on the basis of synthetic data samples, is mostly not applicable to real data samples (eg to real images from a camera). The reason for this is in particular that the (original) probability distribution P _s (x) of the synthetic data samples $x_i^s,$

with i = 1, ..., N _S , typically not the (target) probability distribution P _r (x) of real data samples $x_j^r,$

coincides with j = 1, ..., N _r .

von synthetischen Daten-Stichproben kann derart angepasst werden, dass die angepasste Menge von angepassten, synthetischen Daten-Stichproben (zumindest annähern) eine Wahrscheinlichkeitsverteilung aufweist, die der realen (Ziel-) Wahrscheinlichkeitsverteilung P_r(x) einer realen Menge $D_r={\left\{x_j^r\right\}}_{j=1}^{N_r}$

von realen Daten-Stichproben entspricht. Dies kann z.B. mittels eines Generative Adversarial Networks (GAN) bewirkt werden.A synthetic amount ${\mathrm{D.}}_s={\left\{x_i^s\right\}}_{i=1}^{N_s}$

of synthetic data samples can be adjusted in such a way that the adjusted set of adjusted, synthetic data samples has (at least approximately) a probability distribution that corresponds to the real (target) probability distribution P _r (x) of a real set ${\mathrm{D.}}_r={\left\{x_j^r\right\}}_{j=1}^{N_r}$

of real data samples. This can be achieved, for example, by means of a Generative Adversarial Network (GAN).

311 aus der synthetischen Menge D_s 310 in eine angepasste, synthetische Daten-Stichprobe 312 zu transformieren, die zumindest statistisch von dem Diskriminator-Netzwerk 302 nicht von einer realen Daten-Stichprobe $x_j^r$

321 aus der realen Menge D_r 320 unterschieden werden kann. Das Diskriminator-Netzwerk 302 ist ausgebildet, jeweils zwei Daten-Stichproben 312, 321 am Eingang aufzunehmen, und jeweils zu entscheiden, welche der beiden Daten-Stichprobe 312, 321 die reale Daten-Stichprobe ist. Das Generator-Netzwerk 301 kann derart angelernt werden, dass die Entscheidung 303 des Diskriminator-Netzwerkes 302 (genau) in 50% der Fälle auf die jeweilige angepasste, synthetische Daten-Stichprobe 312 fällt. 3 shows an exemplary GAN 300 with a generator network 301 and a discriminator network 302 . The generator network 301 can be learned in such a way that the generator network 301 is trained to sample a synthetic data $x_i^s$

311 from the synthetic set D _s 310 into an adapted, synthetic data sample 312 to transform, at least statistically, from the discriminator network 302 not from a real data sample $x_j^r$

321 can be distinguished from the real quantity D _r 320. The discriminator network 302 is designed to take two data samples at a time 312 , 321 at the entrance, and decide which of the two data samples 312 , 321 is the real data sample. The generator network 301 can be learned in such a way that the decision 303 of the discriminator network 302 (exactly) in 50% of the cases on the respective adapted, synthetic data sample 312 falls.

For teaching the GAN 300 can be related to 2 The learning algorithm set out above can be used. You can learn the discriminator network 302 an error function can be used, which aims to increase the probability that the discriminator network will move 302 for the real data samples 321 decides.

On the other hand, it is used to train the generator network 301 an error function can be used, which aims to increase the probability that the discriminator network will move 302 for the adapted, synthetic data samples 312 decides.

wobei E[ ] der Erwartungswert für die betrachtete Mengen 310, 320 ist. Diese Fehlerfunktion kann zum Anlernen des Diskriminator-Netzwerks 302 maximiert und zum Anlernen des Generator-Netzwerks 301 minimiert werden.One from the generator network 301 Adapted, synthetic data sample generated from the synthetic data sample x ^s can be denoted by _{g r} ( _x ^s). The decision 303 of the discriminator network 302 can be denoted by _{d r ().} An exemplary error function for teaching can be $$\mathrm{E.}\left[\text{log}\left(d_r\left(x^s\right)\right)+\text{log}\left(1-d_r\left(G_r\left(x^s\right)\right)\right)\right]$$

where E [] is the expected value for the set under consideration 310 , 320 is. This error function can be used to train the discriminator network 302 maximized and for training the generator network 301 be minimized.

As stated above, the synthetic data have samples 311 typically each label has a semantic description of the respective synthetic data sample 311 and / or a sub-area of the synthetic data sample 311 provide. For example, a label can indicate the position of an object in the synthetic data sample 311 are displayed. Through the transformation in the generator network 301 can be a semantic change in a synthetic sample of data 311 caused so that the label of the synthetic data sample 311 no longer on the corresponding adapted (synthetic) data sample 312 is applicable. Consequently, the adjusted data sample 312 not for learning a neural network 200 be used.

1. a) eine gewünschte Ziel-Wahrscheinlichkeitsverteilung P_r(x) aufweisen (die durch die reale Menge 320 beschrieben wird); und
2. b) keine (wesentlichen) semantischen Veränderungen gegenüber den entsprechenden synthetischen Daten-Stichproben 311 aufweisen.

This document describes a method that enables a GAN 300 learn in such a way that the GAN 300 from synthetic data samples 311 generated adjusted data samples 312 ,

1. a) have a desired target probability distribution P _r (x) (that by the real quantity 320 is described); and
2. b) no (significant) semantic changes compared to the corresponding synthetic data samples 311 exhibit.

For this purpose, a weighting function ω (x) can be provided, in particular approximated. The weighting function can be used for a synthetic data sample 311 show how strong the synthetic data sample is 311 from a typical real data sample 321 deviates. In the case of a relatively large deviation, the weighting function preferably has a relatively low weighting value, and in the case of a relatively low deviation, the weighting function preferably has a relatively high weighting value. The weighting function can then be used when teaching the GAN 300 the synthetic data samples 311 from the synthetic crowd 310 that have a relatively high weighting value than the synthetic data samples 311 that have a relatively low weight value.

The weighting function can be derived from the probability density function p _s (x) of the synthetic data samples 311 and from the probability density function P _r (x) of the real data samples 321 depend, in particular on the ratio of the two probability density functions, for example $\omega \left(x\right)=\frac{p_r\left(x\right)}{p_s\left(x\right)}.$

wobei α_i die Gewichte für die Basisfunktionen sind. Als Basisfunktionen können radial basis function (RBF) Kerne K_σ verwendet werden, z.B. jeweils zentriert um eine reale Daten-Stichprobe 321 und mit der Varianz σ_r der realen Menge 320. Die Gewichtungsfunktion kann dann auf Basis der Mengen 310, 320 der Daten-Stichproben 311, 321 ermittelt werden, durch Lösen des folgenden Optimierungsproblems $$\begin{array}{ll}\underset{{\alpha }_l}{{\displaystyle \text{maximize}}}\hfill & {\displaystyle \sum _j\text{log}}{\displaystyle \sum _l{\alpha }_l\phi \left(x_j^r\right)}\hfill \\ \text{subject to}\hfill & {\displaystyle \sum _l{\alpha }_l}{\displaystyle \sum _i{\phi }_l\left(x_i^s\right)=N_s,}\hfill \\ \hfill & \alpha \ge 0.\hfill \end{array}$$

The Kullback-Leibler Importance Estimation Procedure (KLIEP) method can be used to approximate the weighting function. The weighting function can be approximated by a weighted sum of basic functions φ _i (x), for example as $$\widehat{\omega }\left(x\right)={\displaystyle \sum _l{\alpha }_l{\phi }_l\left(x\right)}$$

where α _{i are} the weights for the basis functions. Radial basis function (RBF) kernels K _σ can be used as basic functions, for example each centered around a real data sample 321 and with the variance σ _{r of} the real set 320 . The weighting function can then be based on the quantities 310 , 320 of the data samples 311 , 321 can be determined by solving the following optimization problem $$\begin{array}{ll}\underset{{\alpha }_l}{{\displaystyle \text{maximize}}}\hfill & {\displaystyle \sum _j\text{log}}{\displaystyle \sum _l{\alpha }_l\phi \left(x_j^r\right)}\hfill \\ \text{subject to}\hfill & {\displaystyle \sum _l{\alpha }_l}{\displaystyle \sum _i{\phi }_l\left(x_i^s\right)=N_s,}\hfill \\ \hfill & \alpha \ge 0.\hfill \end{array}$$

When teaching the GAN 300 the (approximated) weighting function ω̂ (x) can be used to make the decisions 303 of the discriminator network 302 to weight. This can make decisions 303 for synthetic data samples 311 that of a typical real-world data sample 312 are relatively similar, are weighted more heavily than decisions 303 for synthetic data samples 311 that is relatively far from a typical real-world data sample 312 are away.

As an error function when teaching the GAN 300 can be used, for example, $$\mathrm{E.}\left[\widehat{\omega }\left(x^s\right)\text{log}\left(d_r\left(x^s\right)\right)+\widehat{\omega }\left(x^s\right)\text{log}\left(1-d_r\left(G_r\left(x^s\right)\right)\right)\right]$$

To increase the robustness of the GAN 300 in the generation of adapted (synthetic) data samples 312 from synthetic data samples 311 , it can be advantageous to learn an inverse GAN at the same time, which is formed from real data samples 321 Generate adapted (real) data samples that show the (original) probability distribution of the synthetic set 310 has (or is approximated to this). The inverse generator network of the inverse GAN can be denoted by g _s (), and the inverse discriminator network of the inverse GAN can be _{denoted by d s} ().

The following error function can be used as a common error function for teaching the networks of both GANs $$\begin{array}{l}={\mathrm{I.}\mathrm{W.}\mathrm{A.}dv}_{}+{\mathrm{I.}\mathrm{W.}\mathrm{C.}yc}_{}\\ =\underset{x^r}{{\displaystyle \mathbb{E.}}}\left[\psi \left(x^r\right)\text{log}{d}_r\left(x^r\right)\right]\\ +\underset{x^s}{{\displaystyle \mathbb{E.}}}\left[\omega \left(x^s\right)\text{log}(1-d_r\left(G_r\left(x^s\right)\right)\right]\\ +\underset{x^s}{{\displaystyle \mathbb{E.}}}\left[\omega \left(x^s\right)\text{log}{d}_s\left(x^s\right)\right]\\ +\underset{x^r}{{\displaystyle \mathbb{E.}}}\left[\psi \left(x^r\right)\text{log}(1-d_s\left(G_s\left(x^r\right)\right)\right]\\ +\underset{x^r}{{\displaystyle \mathbb{E.}}}\left[\psi \left(x^r\right)\Vert G_r\left(G_s\left(x^r\right)\right)-x^r\Vert \right]\\ +\underset{x^s}{{\displaystyle \mathbb{E.}}}\left[\omega \left(x^s\right)\Vert G_s\left(G_r\left(x^s\right)\right)-x^s\Vert \right]\end{array}$$

approximiert wird. Die inverse Gewichtungsfunktion ψ(x) kann durch eine komplementäre bzw. inverse Formulierung des o.g. Optimierungsproblems approximiert werden. Insgesamt können dann die Gewichtungsfunktionen approximiert werden als, $$\widehat{\omega }\left(x^s\right)={\displaystyle \sum _l{\alpha }_l{K}_{{\sigma }_r}\left(x^s,x_l^r\right)}$$

$$\widehat{\psi }\left(x^r\right)={\displaystyle \sum _k{\beta }_k{K}_{{\sigma }_a}\left(x^r,x_k^s\right)}$$

Here, ψ (x) is an inverse weighting function for the inverse GAN, with which the inverse ratio of the density function, ie $\frac{p_s\left(x\right)}{p_r\left(x\right)},$

is approximated. The inverse weighting function ψ (x) can be approximated by a complementary or inverse formulation of the above-mentioned optimization problem. Overall, the weighting functions can then be approximated as, $$\widehat{\omega }\left(x^s\right)={\displaystyle \sum _l{\alpha }_l{K}_{{\sigma }_r}\left(x^s,x_l^r\right)}$$

$$\widehat{\psi }\left(x^r\right)={\displaystyle \sum _k{\beta }_k{K}_{{\sigma }_a}\left(x^r,x_k^s\right)}$$

4th shows a flow diagram of an exemplary (computer-implemented) method 400 for the provision of training data for learning a neural network 200 for a specific task. An exemplary specific task is the object recognition of one or more objects in an image, for example in an image from a vehicle camera 102 .

311, die eine Ursprungs-Wahrscheinlichkeitsverteilung P_s(x) aufweisen, und auf Basis einer realen Menge D_r 320 von realen Daten-Stichproben $x_j^r$

321, die eine Ziel-Wahrscheinlichkeitsverteilung P_r(x) aufweisen.The procedure 400 includes determining 401 a weighting function ω̂ (x), based on a synthetic set D _s 310 of synthetic data samples $x_i^s$

311, which have an original probability distribution P _s (x), and based on a real set D _r 320 of real data samples $x_j^r$

321, which have a target probability distribution P _r (x).

311 aus der synthetischen Menge D_s 310 anzeigen, wie nah die synthetische Daten-Stichprobe $x_i^s$

311 im Sinne eines Abstandsmaßes an der Ziel-Wahrscheinlichkeitsverteilung P_r(x) ist, und/oder wie gut die synthetische Daten-Stichprobe $x_i^s$

311 im Sinne des Abstandsmaßes zu der Ziel-Wahrscheinlichkeitsverteilung P_r(x) passt. Ein relativ hoher Gewichtungswert kann anzeigen, dass die synthetische Daten-Stichprobe $x_i^s$

311 relativ gut zu der Ziel-Wahrscheinlichkeitsverteilung P_r(x) passt (und deshalb mit einer relativ hohen Gewichtung beim Anlernen eines GANs 300 berücksichtigt werden sollte). Andererseits zeigt ein relativ niedriger Gewichtungswert ggf. an, dass die synthetische Daten-Stichprobe $x_i^s$

311 relativ schlecht zu der Ziel-Wahrscheinlichkeitsverteilung P_r(x) passt (und deshalb mit einer relativ niedrigen Gewichtung beim Anlernen eines GANs 300 berücksichtigt werden sollte).The weighting function ω̂ (x) for a synthetic data sample $x_i^s$

311 from the synthetic set D _s 310 indicate how close the synthetic data sample is $x_i^s$

311 is in the sense of a distance measure on the target probability distribution P _r (x), and / or how good the synthetic data sample is $x_i^s$

_{311 fits the target probability distribution P r} (x) in the sense of the distance measure. A relatively high weight value can indicate that the synthetic data sample $x_i^s$

311 fits the target probability distribution P _r (x) relatively well (and therefore with a relatively high weighting when teaching a GAN 300 should be considered). On the other hand, a relatively low weighting value may indicate that the synthetic data sample $x_i^s$

311 fits the target probability distribution P _r (x) relatively poorly (and therefore with a relatively low weighting when teaching a GAN 300 should be considered).

311 eine entsprechende angepasste Daten-Stichprobe $g_r\left(x_i^s\right)$

312 zu ermitteln. Durch das Anlernen 402 kann bewirkt werden, dass die angepassten Daten-Stichproben $g_r\left(x_i^s\right)$

312 statistisch nicht von den realen Daten-Stichproben $x_j^r$

321 unterschieden werden können, und/oder dass die angepassten Daten-Stichproben $g_r\left(x_i^s\right)$

312 die Ziel-Wahrscheinlichkeitsverteilung P_r(x) aufweisen. Des Weiteren kann durch das Anlernen 402 bewirkt werden (insbesondere aufgrund der Verwendung der Gewichtungsfunktion ω̂(x)), dass die Semantik der synthetischen Daten-Stichprobe $x_i^s$

311 in den entsprechenden angepassten Daten-Stichproben $g_r\left(x_i^s\right)$

312 (zumindest statistisch) erhalten bleibt.The procedure 400 also includes teaching 402 of a GAN 300 taking into account the determined weighting function ω̂ (x). The GAN 300 includes a generator network 301 that is formed based on a synthetic data sample $x_i^s$

311 a corresponding adapted data sample $G_r\left(x_i^s\right)$

312 to be determined. By teaching 402 can be effected that the adjusted data samples $G_r\left(x_i^s\right)$

312 not statistically different from the real data samples $x_j^r$

321 can be differentiated, and / or that the adapted data samples $G_r\left(x_i^s\right)$

312 have the target probability distribution P _r (x). Furthermore, by teaching 402 caused (in particular due to the use of the weighting function ω̂ (x)) that the semantics of the synthetic data sample $x_i^s$

311 in the corresponding adapted data samples $G_r\left(x_i^s\right)$

312 (at least statistically) is preserved.

bereitgestellt werden.The procedure also includes 400 determining 403 the training data using the generator network 301 . In particular, the training data can be based on the adapted data samples $G_r\left(x_i^s\right)$

to be provided.

The measures described in this document can be used to efficiently provide training data with which a neural network 200 can be learned in a precise and robust way for a specific task (e.g. object recognition).

The present invention is not restricted to the exemplary embodiments shown. In particular, it should be noted that the description and the figures are only intended to illustrate the principle of the proposed methods, devices and systems by way of example.

Claims (14)

Device for providing training data for training a neural network (200) for a specific task; wherein the device is set up, - on the basis of a synthetic amount (310) of synthetic data samples (311) which have an original probability distribution, and on the basis of a real amount (320) of real data samples (321) which have a target probability distribution, to determine a weighting function which indicates for a synthetic data sample (311) from the synthetic set (310) how close the synthetic data sample (311) is to the target probability distribution in terms of a distance measure, - learning a Generative Adversarial Network, GAN for short, (300) taking into account the weighting function determined; wherein the GAN (300) comprises a generator network (301) which is designed to determine a corresponding adapted data sample (312) on the basis of a synthetic data sample (311); and - to determine the training data using the generator network (301). Device according to Claim 1 , wherein the weighting function is an approximation of a ratio of a density function of the target probability distribution and a density function of the original probability distribution. Device according to one of the preceding claims, wherein the device is set up to determine the weighting function using a Kullback-Leibler importance estimation procedure, or KLIEP for short, method; and - the distance dimension comprises a Kullback-Leibler distance or is a Kullback-Leibler distance. Device according to one of the preceding claims, wherein the device is set up - to determine the weighting function on the basis of a weighted sum of basic functions; and - the basic functions radial basis function, RBF for short, include kernels; and or - The basic functions depend on the real data samples (321) from the real set (320). Device according to one of the preceding claims, wherein the device is set up - to determine an error function for teaching the GAN (300) on the basis of the weighting function; and or - to weight an error term of an error function for training the GAN (300) for a synthetic data sample (311) with a weighting value of the weighting function for the synthetic data sample (311). Device according to Claim 5 , wherein an error term of the error function for a synthetic data sample (311) depends on whether a discriminator network (302) of the GAN (300) is in an analysis of the corresponding adapted data sample (312) and a real data sample (321) decides from the real quantity (320) for the adapted data sample (312) or not. Device according to one of the Claims 5 until 6th wherein - the GAN (300) comprises a discriminator network (302) which is designed, an adapted data sample (312) generated by the generator network (301) from a synthetic data sample (311) and a real one Analyze data sample (321) from the real set (320) and make a decision (303) for or against the adapted data sample (312); and - the error function of an expected value of the decisions (303) for the synthetic data samples (311) from the synthetic set (310), in particular of an expected value of the decisions (303) for the synthetic data samples (311) weighted with the weighting function ) from the synthetic amount (310). Device according to one of the preceding claims, wherein the device is set up - on the basis of the synthetic amount (310) of synthetic data samples (311) and on the basis of the real amount (320) of real data samples (321) to determine an inverse weighting function that is necessary for a real data sample (321) indicates from the real quantity (320) how close the real data sample (321) is to the original probability distribution in terms of a distance measure; and - learning an inverse GAN, taking into account the ascertained inverse weighting function and together with the GAN (300); wherein the inverse GAN comprises an inverse generator network which is designed to determine a corresponding inversely adapted data sample on the basis of a real data sample (321). Device according to Claim 8 , in which - the device is set up to learn the GAN (300) and the inverse GAN by means of a common error function which depends on the weighting function and on the inverse weighting function; - the error function in particular comprises a first error term that depends on a decision (303) of a discriminator network (302) of the GAN (300), and comprises a second error term that depends on a decision of an inverse discriminator network of the inverse GAN; and the first error term is weighted in particular with the weighting function and the second error term is weighted in particular with the inverse weighting function. Device according to one of the preceding claims, wherein the device is set up - to determine the weighting function in such a way that the weighting function displays a weighting value for each of the synthetic data samples (311) from the synthetic set (310); and - The weighting value for a synthetic data sample (311) indicates how close the synthetic data sample (311) is to the target probability distribution in terms of the distance measure. Device according to one of the preceding claims, wherein - a synthetic data sample (311) and / or a real data sample (321) comprise an image, in particular a labeled image; and or - The neural network (200) learned for the specific task is designed for object classification, object recognition and / or object segmentation. Device according to one of the preceding claims, wherein the device is set up - using the generator network (301) to determine an adapted data sample (312) as part of the training data for the neural network (200) for each synthetic data sample (311) from the synthetic set (310); and or - Learn the neural network (200) on the basis of the training data determined. Device according to one of the preceding claims, wherein - A synthetic data sample (311) was generated on the basis of a computer simulation; and or - A real data sample (321) was recorded by an environment sensor (102), in particular by a camera, of a vehicle (100). Method (400) for providing training data for training a neural network (200) for a specific task; wherein the method (400) comprises, - determining (401) a weighting function based on a synthetic set (310) of synthetic data samples (311) which have an original probability distribution, and based on a real set (320) of real data samples (321), which have a target probability distribution; wherein the weighting function for a synthetic data sample (311) from the synthetic set (310) indicates how close the synthetic data sample (311) is to the target probability distribution in terms of a distance measure; - Learning (402) a Generative Adversarial Network, GAN for short, (300) taking into account the weighting function determined; wherein the GAN (300) comprises a generator network (301) which is designed to determine a corresponding adapted data sample (312) on the basis of a synthetic data sample (311); and - Determination (403) of the training data on the basis of the generator network (301).

Images