DE102019134156A1 - Train a multitask neural network - Google Patents

Abstract

The present invention relates to a method for training a neural network (10), in particular a convolutional neural network for use in a driving support system of a vehicle, with an encoder (12) for coding provided input data (14) and several decoders (16), of which each carries out a decoding task (18) with annotated training data (20) as input data (14), each of the multiple decoding tasks (18) having a target application context, with the steps of: providing multiple sets (22) of differently annotated training data (20) for training the multiple decoding tasks (18), arranging the multiple sets (22) of differently annotated training data (20) in data streams (24, 26), wherein for training each of the multiple decoding tasks (18) a first set (38) with training data (20), which correspond to a target application context of the neural network (10), and additionally a second set (40) with traini ngs data (20) are provided which do not correspond to the target application context of the neural network (10), generation of mini-batches (32) of training data (20), the several sub-mini-batches (34, 36) of training data (20) for training each of the decoding tasks (18), generating the sub-mini-batches (34, 36) of training data (20) for training each of the decoding tasks (18) based on the respective first set (38) in combination with the respective one second set (40), the training data (20) for the jointly trained decoding tasks (18) being combined according to a combination scheme in order to provide the respective sub-mini-stack (34, 36), training the neural network (10) using the generated mini-stacks (32), and updating the combination scheme for combining the sub-mini-stacks (34, 36) of training data (20) as a function of a learning development of the neural network (10).

Description

The present invention relates to a method for training a neural network, in particular a convolutional neural network for use in a driving support system of a vehicle, with an encoder for coding input data provided and several decoders, each of which carries out a decoding task, with annotated training data as input data, each the decoding task has a target application context.

A multitask neural network consists of a shared encoder and several task decoders. This enables efficient data processing with reduced resources compared to several parallel single-task neural networks, each of which only handles one task. The decoders are able to decode the information made available in parallel under various aspects, e.g. carrying out segmentation or detection of objects, to name just a few. The input data that are used as input for training the neural network are input images (or can be inputs from other sensors such as LiDAR, radar, etc.), and the output of the neural network is the respective decoding information for each decoder, e.g. acquisition or segmentation annotations. Due to their scalability, the neural multitask networks are particularly well suited for the construction of autonomous driving systems in order to add further decoding tasks if necessary.

Multitask learning (MTL), i.e. the training of such neural multitask networks, is an emerging field in the field of deep learning that focuses on the design and training of a common neural network to solve several tasks. Therefore, MTL is particularly important for the design and training of autonomous driving systems that use such neural multitasking networks. However, training such a multitask neural network requires a lot of training data (hundreds of thousands of data instances) that are annotated for the target decoding tasks. This annotated training data is also known as the ground truth. Ideally, if the input data, i.e. image data sets for training the multitask neural network, are completely annotated for the various tasks represented by the multitask decoders, the network uses the same batch of input data sets to perform all tasks simultaneously work out. This enables the neural network coder to be trained and each decoder to be trained with each of the data sets.

In practice it is difficult to find such a fully annotated data set for all target tasks. Typically, several data sets are available from different sources or projects, some of which are annotated for one or more of the target decoding tasks within a particular application context. Such application contexts relate, for example, to the detection of pedestrians when parking or the detection of lanes in a driving scenario on the motorway. The application context is crucial for any system that needs to be trained appropriately. However, input data from other application contexts can also be used for generalization. For example, when looking at pedestrian detection in a parking scenario, parking spaces are the target application context. However, input data with annotated pedestrians in other urban environments can also be used.

Because of the need for large amounts of training data, a sufficient amount of such training data is particularly important for training neural networks, in particular for deep learning systems. Obtaining a new data set with training data that has all the necessary annotations for training all decoding tasks is very costly, e.g. in the range of millions of euros, which makes the reuse of available training data important. Therefore, mixed strategies are required for training such neural multitask networks so that these neural networks can be trained efficiently using available training data that are not completely annotated for all decoding tasks and / or cover different application contexts.

However, several technical problems can arise when attempting to combine different types of differently annotated records. One question is how annotated and non-annotated data should be managed for each decoding task during training. Another question is how to deal with the different sizes of available data sets in order to avoid a tendency in training towards the largest data set. It is just as important to know how data sets with different application contexts are to be managed, e.g. motorway lane detection vs. lane detection in a parking scenario. Random mixing strategies, which are currently being investigated, do not show sufficient results. These random shuffling strategies can lead to an unbalanced training of the decoding tasks, with one decoding task being updated more frequently than another. This also applies to the encoder, whose update is based on the contribution of the training data from those in a batch of training data Tasks depends. The use of training data sets that differ in size leads to an unbalanced training of the decoding tasks and a tendency towards the larger data set. The use of training data sets with different application contexts is not dealt with.

It is an object of the present invention to provide a method of the above-mentioned type which overcomes at least some of the above-mentioned problems and disadvantages. In particular, the present invention is based on the object of specifying a method for training a neural network, in particular a convolutional neural network for use in a driving support system of a vehicle, which has an encoder for encoding input data provided and several decoders, each of which carries out a decoding task, which enables efficient and balanced training of the encoder and the multiple decoders.

This problem is solved by the independent claim. Advantageous refinements are given in the subclaims.

In particular, the present invention provides a method for training a neural network, in particular a convolutional neural network for use in a driving support system of a vehicle, which has an encoder for coding input data provided and several decoders, each of which performs a decoding task, with annotated training data as input data specified, wherein each of the multiple decoding tasks has a target application context, with the steps of providing multiple sets of differently annotated training data for training the multiple decoding tasks, arranging the multiple sets of differently annotated training data in data streams, wherein for training each of the multiple decoding tasks a data stream with training data, the correspond to the target application context, and those that do not correspond to the target application context are provided for each task (ie a data generator for each task) generating mini-batches of training data having multiple sub-mini-batches of training data for training each decoding task, in generating the sub-mini-batches of training data for training each of the decoding tasks that corresponds to the target application context and those which do not correspond to the target application context, the respective training data are combined according to a combination scheme to provide the respective sub-mini-batches, training the neural network using the generated mini-batches and updating the combination scheme to combine the sub-mini-batches of training data depending on a learning development of the neural network.

The basic idea of the present invention is to use an adaptive scheme for mixing the provided training data for each of the decoding tasks. This is achieved by determining the learning development of the neural network as feedback. The learning evolution enables the generation of the sub-mini-batches of training data to be adapted for training each of the decoding tasks, i.e. combining the training data corresponding to the target application context and the training data which does not correspond to the target application context, for each of the jointly trained decoding tasks. The creation of the sub-mini-stacks and the adaptation of the combination scheme can be done in different ways. Some options for creating the sub-mini-stacks and for adapting the combination scheme are explained in more detail below. The proposed method provides a simple solution for efficiently supplying the neural multitask network with partially and / or fully annotated data by using separate data streams for each decoding task. In addition, a balanced training of the decoding tasks is carried out through the use of the mini-batches which contain training data for each of the decoding tasks. In addition, different training data sets can be processed that cover different application contexts. For example, a training data set can be helpful to train the respective decoding task, even if it relates to a different application context, in order to achieve a better generalization with more training data, e.g. an object like a car retains the same visual properties, regardless of whether it is located in an urban or in a rural application context. The proposed method can easily be scaled to as many training data sets as possible. A dynamic mix of training data that corresponds to the target application context with those that do not correspond to the target application context is achieved.

Each of the decoders decodes the input data provided under different aspects, for example it carries out a parallel segmentation or a detection of objects, to name just a few. The input data that are used as input for training the neural network are input images that are fed to the encoder. The output of the neural network is corresponding decoding information for each decoder, e.g. acquisition or segmentation annotations.

The training data have at least one annotated training data set for each of the decoding tasks in order to carry out efficient and balanced training for the entire neural network.

Each set of training data annotated for training the same decoding task is added to the respective data streams. For each of the decoding tasks, the training data have at least annotated data from the target application context or annotated data that do not correspond to the target application context. For all decoding tasks, however, the training data can contain both training data that correspond to the target application context and training data that do not correspond to the target application context. For each decoding task, all target classes of the training data of the data stream annotated for this decoding task must be annotated, i.e. the respective data stream of the training data is completely annotated for the respective decoding task. In addition, at least training data for this decoding task are annotated for each decoding task. In particular, there are training data for each decoding task that relate to the target application context of the neural network.

In general, the target application context relates to the neural network. In other cases, however, the target application context can be defined independently for each decoding task.

The mini-stacks are small subsets of training data that are used in each training step of an iterative training process of the neural network. The training data is split into N mini-batches during training. Each mini-batch has a batch size m, i.e. the mini-batch contains m samples of training data. Typical values for the stack size m are, for example, 8, 16, 32. The total amount of training data corresponds to an epoch with a data size of N * m samples. The stack size is selected, for example, depending on a GPU capacity for carrying out the training of the neural network. The mini-batches enable efficient processing of large amounts of training data. Due to the memory limitations of GPUs, it is usually not possible to propagate the entire training data back in one step.

Each of the mini-batches of training data preferably has a sub-mini-batch for each decoding task. Each of the sub-mini-stacks preferably has training data for one of the decoding tasks. Therefore, all decoding tasks are trained in each iteration, i.e. with each mini-batch. The sub-mini-batches are generally formed from unseen training data, i.e. training data that has not yet been used during an epoch. An epoch generally relates to the presentation of the complete training data set for the neural network for a training session. However, if the training data have a different size for the different training tasks, this cannot be applied in full, but some of the training data, for example, cannot be used. When training the neural network, several training epochs are typically carried out.

According to a modified embodiment of the invention, combining the training data corresponding to the target application context and not corresponding to the jointly trained decoding tasks according to a combination scheme for providing the respective sub-mini-stack includes providing an individual combination scheme for each of the jointly trained decoding tasks, and has that Updating the combination scheme for combining the sub-mini-stacks of training data as a function of a learning development of the neural network, the individual updating of the combination scheme depending on the learning development of the neural network for each of the jointly trained decoding tasks. Therefore, the individual learning development is determined individually for each of the decoding tasks, so that the individual learning development can be used for updating the combination scheme for each of the decoding tasks. Thus, each of the sub-mini training data sets for training the various decoding tasks can be put together individually, taking into account the respective learning development. The individual combination scheme enables improved training of the neural network, since the training of each of the decoding tasks is adapted independently of one another.

According to a modified embodiment of the invention, combining the training data corresponding and not corresponding to the target application context for the jointly trained decoding tasks according to a combination scheme for providing the respective sub-mini-stack includes selecting the training data corresponding and not corresponding to the target application context in accordance with a ratio, and the updating of the combination scheme for combining the sub-mini-batches of training data as a function of a learning development of the neural network comprises the updating of the ratio. Therefore, training data is sampled according to the ratio of each of the two categories (whether or not corresponding to the target application context). The presentation of the training data for the neural network is therefore dependent on the composition of the controlled by both data categories. The ratio is preferably a value within a number range between zero and one. However, any other value from any number range can also be used, provided that it can indicate a relationship between the training data that are taken from the two data categories.

According to a modified embodiment of the invention, the selection of the training data from the first and the second data stream according to a ratio includes the generation of a random number, in particular between zero and one, for each training data sample that is to be added to the respective sub-mini-batch, and that Taking the training data sample to be added from the first or second data stream as a function of whether the random number is greater than the ratio or not. The random numbers are generated within the same number range in which the ratio is specified, typically a real number between zero and one. Even if the ratio defines the presentation of the training data from the first and second data streams for the neural network, the random numbers allow a more flexible selection of the training data from the first and second data streams. Therefore, the individual sub-mini-batches may contain training data with different percentages from the two data streams than indicated by the ratio. When considering several sub-mini-batches, however, the sub-mini-batches can contain the training data from the two data streams with the defined ratio. Since the sub-mini-batches typically contain only a small number of training data samples, the ratio cannot be applied with high accuracy. For example, if the sub-mini-batch contains ten samples of training data, only ten ratios can be applied. With the random numbers, the method achieves a correct application of the training data from the two data streams in accordance with the ratio for several sub-mini-stacks of training data. Even if the relationship changes during training of the neural network, the same principles apply.

According to a modified embodiment of the invention, the updating of the combination scheme for combining the sub-mini-stacks of training data as a function of a learning development of the neural network includes applying a learning rate. The learning rate is a measure of a change in the composition of the sub-mini-stacks. It is typically a numerical value. The learning rate can be chosen and adjusted to accommodate the updating of the combination scheme. A higher learning rate generally allows faster convergence, but increases the risk of overfitting a certain type of training data. The learning rate can vary during the training of the neural network.

According to a modified embodiment of the invention, combining the training data of the first and second data streams for the jointly trained decoding tasks according to a combination scheme for providing the respective sub-mini-stack includes reusing training data of the first and / or the second data stream of training data for the respective decoding task. If several independently provided training data sets are used, there is a high probability that one or more of the data streams of training data contain more training data than other data streams of training data. This can lead to part of the training data not being used during an epoch if the training is terminated immediately when no more unused training data from one or more data streams of training data are available. By reusing part of the training data that has already been used during an epoch, the use of the available training data can be improved. This is particularly important when most of the data streams contain more training data than a single data stream of training data or only a few data streams of training data. The reuse of training data is therefore particularly useful in the case of unbalanced data streams of training data for the respective decoding task. If the training data for the different tasks are unbalanced, the training data of the first and the second data stream of training data of one or more complete decoding task (s) can also be reused.

According to a modified embodiment of the invention, arranging the multiple differently annotated training data sets in data streams, a first data stream with training data corresponding to a target application context of the neural network being provided for training each of the multiple decoding tasks, and for additional training at least one jointly trained decoding task The multiple decoding tasks are provided with a second data stream with training data that do not correspond to the target application context of the neural network, combining all training data sets that correspond to a specific decoding task and the respective target application context in the respective first data stream and combining all training data sets that correspond to the specific decoding task and do not correspond to the respective target application in the respective second data stream. Thus, all are available Training data that are annotated for training one of the decoding tasks, combined in the respective first and / or second data stream. In particular, if several training data sets are provided which are suitable for training a specific decoding task that corresponds to the target application context, the individual samples from these training data sets are combined in the respective first data stream. Similarly, if several training data sets are provided for training a specific decoding task that does not correspond to the target application context, the samples from these training data sets are combined in the respective second data stream. Mixing strategies can be used to provide the training data from several training data sets in a data stream. Alternatively, the training data of the data streams can be accessed at random.

According to a modified embodiment of the invention, training the neural network using the generated mini-stacks comprises training the neural network in a synchronous back-propagation manner, and comprises updating the combination scheme for the sub-mini-stacks of training data depending on a learning development of the neural network dynamically updating the combination scheme for subsequent mini-stacks of training data as a function of a learning development of the neural network. According to this, the neural network is updated with a mini-stack after each training sequence, whereby for each decoding task e.g. task losses from each sub-mini-stack are determined and an error is simultaneously backpropagated in the respective decoder of this special decoding task and additionally in the encoder. This ensures a good balance of the trained tasks, particularly with regard to training the coder, e.g. training coder features.

According to a modified embodiment of the invention, generating mini-stacks with multiple sub-mini-stacks of training data for training each of the decoding tasks includes balancing the sub-mini-stacks for each of the mini-stacks according to a balancing scheme. Various equalization schemes for equalizing the training of the multiple decoding tasks in parallel are known in the art.

According to a modified embodiment of the invention, the method has a step for determining a learning development of the neural network as a function of a loss of validation and / or of a validation key performance indicator of the neural network. The loss of validation is a quantity that is often used in deep learning to get a good estimate of how the neural network is learning, e.g. a training speed and the like. In particular, the loss of validation provides information as to whether the neural network is still learning or rather an over-adaptation occurs. In order to get a system that generalizes well, especially to get a system that works well for all training data sets, avoid overfitting. Overfitting can be detected when the loss of validation begins to increase instead of decreasing. In fact, any other parameter that gives a good estimate of the neural network overfitting can be used. In particular, the Validation Key Performance Indicator (Validation KPI) can be used in place of the Loss of Validation if it is available.

According to a modified embodiment of the invention, the determination of a learning development of the neural network as a function of a loss of validation and / or of a validation key performance indicator of the neural network comprises the independent determination of the learning development of the neural network for each of the decoding tasks. This enables a good and detailed insight into the learning process of the neural network. The individually determined learning development enables in particular an individual updating of the combination scheme for the training data from the first and the second data stream.

According to a modified embodiment of the invention, the determination of a learning development of the neural network as a function of a loss of validation and / or of a Validation key performance indicator of the neural network on the independent determination of the learning development of the neural network for each of the application contexts. Accordingly, the learning development for the training carried out is determined individually using the respective training data from the first and the second data stream. On the basis of this determined learning development, it can be reliably determined whether the neural network as a whole and / or the respective decoding tasks benefit more from training with training data with the corresponding target application context or with training data without the corresponding target application context. Thus, the training of the neural network can be adapted in the most efficient way on the basis of the updating of the combination scheme as a whole and / or of the respective decoding tasks.

According to a modified embodiment of the invention, the determination of a learning development of the neural network as a function of a loss of validation and / or of a validation key performance indicator of the neural network includes the application of a learning rate. The learning development can thus be determined generically on the basis of a learning rate. The learning rate is typically a numerical value. The learning development can be used as a basis for updating the combination scheme. A higher learning rate generally allows for faster convergence, but increases the risk of overfitting a particular type of training data. For example, the ratio for the selection of the training data from the first and the second data stream can be updated by multiplying a gradient of the validation loss by the learning rate. This product can be subtracted from a current ratio.

According to a modified embodiment of the invention, the method has a step for terminating the training if the learning development indicates that the neural network is no longer improving, in particular if the learning development indicates that the neural network is changing in relation to at least one the decoding tasks not further improved. The training is therefore terminated so that overadaptation of the neural network can be avoided. For example, the neural network can be regarded as not improving if the gradient of the validation loss for training the neural network with training data from the first data stream and / or for training the neural network with training data from the second data stream is greater than zero.

These and other aspects of the invention will be apparent and illustrated by the embodiments described below. Individual features that are shown in the embodiments can form an aspect of the present invention on their own or in combination. Features of the various embodiments can be transferred from one embodiment to another embodiment.

• 1 eine schematische Darstellung eines neuronalen Netzes, das einen einzelnen Kodierer und zwei Dekodierer aufweist, wobei dem Kodierer Eingabedaten mit zwei Datenströmen zugeführt werden und jeder der Dekodierer entsprechend seiner Dekodieraufgabe eine einzelne Ausgabe bereitstellt, gemäß einer ersten bevorzugten Ausführungsform;
• 2 eine Tabelle, die die Anwendbarkeit verschiedener Trainingsdatensätze angibt, die für verschiedene Dekodieraufgaben bereitgestellt werden, und außerdem angibt, ob der Trainingsdatensatz dem Zielanwendungskontext zugehörig ist, in Übereinstimmung mit der ersten Ausführungsform;
• 3 eine schematische Zeichnung, die einen Sub-Mini-Stapel von Trainingsdaten zum Trainieren eines neuronalen Netzes mit zwei Dekodierern für zwei Dekodieraufgaben darstellt, in Übereinstimmung mit der ersten Ausführungsform;
• 4 eine schematische Zeichnung, die zwei in 3 dargestellte Sub-Mini-Stapel von Trainingsdaten zum Trainieren einer Dekodieraufgabe des neuronalen Netzes darstellt, die dem Zielanwendungskontext entsprechen bzw. nicht dem Zielanwendungskontext der Mini-Trainingsdatensätze für die anderen Dekodieraufgaben entsprechen;
• 5 beispielhaft ein Diagramm, das eine Zusammensetzung der Sub-Mini-Stapel von Trainingsdaten mit Trainingsdaten darstellt, die dem Anwendungskontext entsprechen und Trainingsdaten, die nicht dem Anwendungskontext entsprechen, über verschiedene Epochen in Übereinstimmung mit der ersten Ausführungsform; und
• 6 ein Ablaufdiagramm, das ein Verfahren zum Trainieren des vorstehend erwähnten neuronalen Netzes mit einem Kodierer und mehreren Dekodierern mit annotierten Trainingsdaten darstellt, wobei jede der mehreren Dekodieraufgaben einen Zielanwendungskontext aufweist, in Übereinstimmung mit der ersten Ausführungsform.

Show it:

• 1 a schematic representation of a neural network having a single encoder and two decoders, the encoder being supplied with input data with two data streams and each of the decoders providing a single output corresponding to its decoding task, according to a first preferred embodiment;
• 2 a table indicating the applicability of various training data sets provided for different decoding tasks and also indicating whether the training data set is associated with the target application context, in accordance with the first embodiment;
• 3 Fig. 13 is a schematic drawing showing a sub-mini-batch of training data for training a neural network with two decoders for two decoding tasks, in accordance with the first embodiment;
• 4th a schematic drawing showing two in 3 The illustrated sub-mini-stacks of training data for training a decoding task of the neural network, which correspond to the target application context or do not correspond to the target application context of the mini-training data sets for the other decoding tasks;
• 5 is an exemplary diagram illustrating a composition of the sub-mini-stacks of training data with training data that correspond to the application context and training data that do not correspond to the application context over different epochs in accordance with the first embodiment; and
• 6th Figure 12 is a flowchart illustrating a method of training the aforementioned neural network with an encoder and multiple decoders with annotated training data, each of the multiple decoding tasks having a target application context, in accordance with the first embodiment.

6th shows a flow chart of a method for training a neural network 10 , in particular a convolutional neural network for use in a driving support system of a vehicle, according to a first preferred embodiment.

The neural network 10 the first embodiment has an encoder 12th for coding provided input data 14th and several decoders 16 on, each of which has a decoding task 18th executes, with annotated training data 20th as input data 14th , each of the multiple decoding tasks 18th has an application context. The input data 14th that are used as input for training the neural network 10 are used have multiple samples of input images presented to the encoder 12th are fed. Each of the decoders 16 decodes the provided and encoded input data 14th with regard to different decoding tasks, e.g. it carries out a segmentation or a detection of objects in parallel, to name just a few. One issue 28 of the neural network 10 is for every decoder 16 appropriate Decoding information, e.g. acquisition annotations or segmentation annotations.

The training data 20th the encoder 12th of the neural network 10 as input data 14th fed. The training data 20th assign multiple sentences 22nd of training data 20th on, with each sentence 22nd of training data 20th is annotated differently, as in 2 is shown by way of example. 2 shows training data 20th that in four sentences 22nd of training data 20th provided, which are designated as data set A to D. In 2 assumed that the neural network 10 four decoding tasks 18th which are designated as task 1 to task 4. As in 2 shown are the four sentences 22nd of training data 20th annotated differently, and each sentence 22nd of training data 20th is an example for training one of the decoding tasks 18th annotated. In addition, the table in 2 for each sentence 22nd of training data 20th whether or not it corresponds to the target application context. In the example given, two sentences correspond 22nd of training data 20th the target application context and two not.

Hence the sentences 22nd the training data 20th provided independently. In general, any sentence can 22nd of training data 20th one or more of the decoding tasks 18th correspond, with each sentence 22nd of training data 20th independently of the target application context of the neural network 10 may or may not match.

The method begins with step S100, which focuses on providing multiple sentences 22nd differently annotated training data 20th for training the multiple decoding tasks 18th relates. The training data 20th as they are in the multiple sentences 22nd differently annotated training data 20th are provided as a basis for training the neural network 10 used. The neural network 10 in 1 has only two decoders 16 for two different decoding tasks 18th which are examples of tasks 1 and 2 in the table in 2 correspond.

Step S110 relates to arranging the plurality of sentences 22nd differently annotated training data 20th in data streams 24 , 26th , with each of the two decoding tasks for training 18th a first data stream 24 with training data 20th corresponding to the first decoding task (acquisition decoder), and a second data stream 26th with training data 20th that of the second decoding task (segmentation decoder) of the neural network 10 are provided. Therefore, for the decoding task referred to as task 1 18th the first data stream 24 the data set D added, and for the decoding task designated as task 2 18th becomes the second data stream 26th the record C added as the records 22nd of training data 20th for the respective training task 18th are annotated and the target application context of the neural network 10 correspond. Furthermore, the two decoding tasks referred to as task 1 and task 2 pose 18th jointly trained decoding tasks 18th The training data 20th therefore additionally have for each of the two decoding tasks 18th a set 22nd of training data 20th on that for the respective training task 18th are annotated and not the target application context of the neural network 10 and correspond to the second data stream 24 , 26th will be added. In particular, for the decoding task referred to as task 1 18th the first data stream in each case 24 the data set A is added, and for the decoding task designated as task 2 18th becomes the second data stream 26th data stream A is also added as this record 22nd of training data 20th for the two decoding tasks 18th is annotated and not the target application context of the neural network 10 corresponds to.

In another example, in which several of the provided sentences 22nd of training data 20th meet the above conditions for being the first or the second data stream 24 , 26th are added, all sentences 22nd of training data 20th that a specific decoding task 18th and correspond to the respective target application context, combined and the respective first and second data stream 24 , 26th added, and all sentences 22nd of training data 20th that of the particular decoding task 18th and do not correspond to the respective target application, are combined and the respective first and second data stream 24 , 26th added. To provide the training data 20th from several sentences 22nd of training data 20th in a respective data stream 24 , 26th Mixing strategies can be used.

Step S120 relates to creating mini-stacks 32 of training data 20th who have favourited Multiple Sub-Mini Stacks 34 , 36 the training data 20th for training each of the decoding tasks 18th exhibit. A mini pile 32 the training data 20th is in 3 shown as an example. Any mini-stack 32 contains m samples of training data 20th , where n number of samples of training data 20th a decoding task 18th and a number of (m-n) samples of training data 20th another decoding task 18th corresponds to. The neural network described 10 has two decoding tasks 18th on so that the mini pile 32 two sub-mini stacks 34 , 36 contains, one for each of the two decoding tasks 18th . A constraint for providing the mini-batch 32 is that each of the sub-mini stacks 34 , 36 at least one sample of training data 20th having.

Hence the training data 20th in N mini-stacks 32 split for training each epoch. Any mini-stack 32 has a size m, ie the mini-stack 32 contains m samples of training data 20th . Typical values for the stack size are, for example, 8, 16, 32. The entirety of the training data 20th consequently forms the epoch with a data size of N * m samples.

Step S130 relates to creating the sub-mini-batches 34 , 36 of training data 20th for training each of the two decoding tasks 18th based on the respective data stream 24 , 26th . Either of the two decoding tasks 18th is a jointly trained decoding task 18th . The sub-mini stacks 34 , 36 therefore each have training data 20th from the first and second data stream 24 , 26th on. The training data 20th for each of the decoding tasks 18th are combined according to a combination scheme, as explained in more detail below, resulting in a set 38 of training data 20th that corresponds to the target application context and a set 40 of training data 20th that does not match the target application context, as in 4th is shown.

The combination scheme includes providing a ratio and samples of training data 20th are according to the ratio of the amount 38 , 40 taken. The ratio in the first embodiment corresponds to a real value from a number range between zero and one. In addition, when selecting a sample of training data 20th from the first or second data stream 24 , 26th a random number in the same number range, ie between zero and one, is generated. For each sample of training data 20th corresponding to the respective sub-mini-stack 34 , 36 is to be added, the sample of training data 20th depending on the amount 38 , 40 added whether the random number is greater than the ratio or not.

In this embodiment, the combination scheme is an individual combination scheme for each of the jointly trained decoding tasks 18th . Accordingly, each of the sub-mini-stacks becomes 34 , 36 of training data 20th for training the various decoding tasks 18th individually compiled.

Typically, the samples are training data 20th from the first and the second data stream 24 , 26th about samples that have not yet been used for training during an epoch. A reuse of training data 20th of the first and / or the second data stream 24 , 26th of training data 20th can, however, for the respective decoding task 18th are executed. If one or more of the data streams 24 , 26th of training data 20th more training data 20th contains (contain) than other data streams 24 , 26th of training data 20th , is a reuse of training data 20th particularly useful.

The training data 20th for training every decoding task 18th from the first set 38 correspond to D _goal , and the training data 20th for training every decoding task 18th from the second set 40 correspond to D _others . Accordingly, the ratio defines a probability P _{target of} the samples of training data 20th that came from the first lot 38 can be removed. P _goal corresponds directly to the probability P _{others of} the samples of training data 20th that came from the second lot 40 for every decoding task 18th can be removed. The condition P _Ziel = 1 - P _Other is always fulfilled.

If step S130 is carried out for the first time, the values for P _target and P _{other must be} initialized. The initialization can be, for example
P _Goal = P _Other = 0.5 or
P _Target = 1 and P _Other = 0, or
P _Target = 0 and P _Other = 1.
The initialization can be carried out in any way.

Based on this, a random floating point number f between zero and one is generated for each sample. If the random floating point number f is less than P _target , a sample of training data is used 20th from the first set 38 , taken, which relates to D _goal . This sample is then added to the respective sub-mini-stack 34 , 36 added. Otherwise, ie if f is greater than P _target , the sample of training data is used 20th from the second set 40 which refers to D _Others .

Step S140 relates to the training of the neural network 10 using the generated mini-stacks 32 . The workout will be for each of the mini stacks 32 individually executed in a synchronous back propagation manner. An error will therefore appear in the respective decoder 16 the respective decoding task 18th and additionally in the encoder 14th simultaneously backpropagated. This means that with a backward pass, the decoder losses to the respective decoder 16 and in addition to the encoder 14th be backpropagated at the same time. Because every mini-stack 32 Training data 20th for both decoding tasks 18th the weights of the encoder 14th by the gradient descent using the errors of both decoders 16 updated at the same time. The same applies to each of the mini-stacks after training 32 an update of the entire neural network 10 executed.

Step S150 relates to determining a learning history of the neural network 10 as a function of a loss of validation and / or of a validation key performance indicator of the neural network 10 . The loss of validation is a quantity that is often used in deep learning to get a good estimate of how the neural network 10 learns, for example a training speed, etc. In particular, the loss of validation gives information about whether the neural network 10 is still learning or rather an overadaptation occurs.

The learning development of the neural network 10 is used for each of the decoding tasks 18th independently determined. The learning development of the neural network 10 is also determined independently for each of the application contexts, ie for training data 20th corresponding to the target application context and for training data 20th that do not match the target application context. Therefore, the learning progression for each decoding task will be 18th and for the training data 20th from the first and the second data stream 24 , 26th certainly.

Determining the learning development of the neural network 10 includes applying a learning rate. The learning rate corresponds to a numerical value. Further details are provided below.

Step S160 relates to updating the combination scheme to combine the sub-mini-stacks 34 , 36 of training data 20th depending on the learning development of the neural network 10 . The step of updating the neural network 10 comes with a mini-stack after each training sequence 32 executed, for each decoding task 18th e.g. the task losses from each sub-mini-batch 34 , 36 be determined. Thus the ratio for subsequent mini-stacks will be 32 of training data 20th depending on the learning development of the neural network 10 updated.

The ratio, ie P _target , becomes larger when the neural network 10 learns positively, ie L _target decreases, and smaller when it learns negatively, ie L _target increases. At the same time, the neural network should 10 good to generalize _{to D others.} L _goal refers to a smoothed loss of validation (over a certain number of epochs) with respect to D _goal , and L _other refers to a smoothed loss of validation with respect to D _others .

The updating of the combination scheme refers to updating the ratio and becomes dependent on the learning development of the neural network 10 for every decoding task 18th individually executed. The updating of the combination scheme is carried out using a learning rate. The learning rate is a numerical value. The learning rate is a measure of a change in the composition of the sub-mini-stacks 34 , 36 . A higher learning rate generally allows for faster convergence, but increases the risk of overfitting a particular type of training data 20th . Updating the ratio for the selection of the training data 20th from the crowd 38 , 40 takes place in this embodiment by multiplying a gradient of the validation loss by the learning rate. This product is then subtracted from a current ratio, e.g. $${\text{P.}}_{\text{aim},\text{New}}={\text{P.}}_{\text{aim},\text{current}}-\text{Learning rate}*\text{gradient}\left({\text{L.}}_{\text{aim}}\right)$$

A possible development of the relationship, ie a development of the probabilities P _target and P _other , is exemplified in 5 shown over several training periods.

Step S170 relates to ending the training if the learning history indicates that the neural network is working 10 no further improvement, especially if the learning curve shows that the neural network is changing 10 in relation to at least one of the decoding tasks 18th not further improved. The training is therefore aborted so that overadaptation of the neural network is avoided. The neural network 10 is regarded as not improving further if the gradient of the loss of validation for the training of the neural network 10 with training data 20th from the first set 38 and for training the neural network 10 with training data 20th from the second set 40 is greater than zero. An overfitting can be detected, for example, when the validation loss starts to increase instead of decreasing.

Otherwise, the training continues as discussed above.

List of reference symbols

10

neural network

12th

Encoder

14th

Input data

16

Decoder

18th

Decoding task

20th

Training data

22nd

Training data set

24

first (acquisition) data stream

26th

second (segmentation) data stream

28

output

32

Mini pile

34

Sub-mini stacks

36

Sub-mini stacks

38

Set of training data that corresponds to the target application context

40

Amount of training data that does not correspond to the target application context

Claims (14)

Method for training a neural network (10), in particular a convolutional neural network for use in a driving support system of a vehicle, with an encoder (12) for coding input data (14) provided and several decoders (16), each of which has a decoding task (18) with annotated training data (20) as input data (14), each of the multiple decoding tasks (18) having a target application context, with the steps: Providing several sets (22) of differently annotated training data (20) for training the several decoding tasks (18); Arranging the multiple sets (22) of differently annotated training data (20) in data streams (24, 26), wherein for training each of the multiple decoding tasks (18) for a data stream (24, 26) a first set (38) with training data (20) which correspond to a target application context of the neural network (10), and additionally a second set (40) with training data (20) which do not correspond to the target application context of the neural network (10) are provided; Generating mini-batches (32) of training data (20) comprising a plurality of sub-mini-batches (34, 36) of training data (20) for training each of the decoding tasks (18); Generating the sub-mini-batches (34, 36) of training data (20) for training each of the decoding tasks (18) based on the set (38) in combination with the set (40), the training data (20) for the jointly trained decoding tasks (18) are combined according to a combination scheme in order to provide the respective sub-mini-stacks (34, 36); Training the neural network (10) using the generated mini-stacks (32); and Updating the combination scheme for combining the sub-mini-stacks (34, 36) of training data (20) as a function of a learning development of the neural network (10). Method according to the preceding Claim 1 , characterized in that combining the training data (20) of the data stream (24, 26) for the jointly trained decoding tasks (18) according to a combination scheme for providing the respective sub-mini-stack (34, 36) provides an individual combination scheme for each of the jointly trained decoding tasks (18) has; and updating the combination scheme for combining the sub-mini-stacks (34, 36) of training data (20) as a function of a learning development of the neural network (10) individually updating the combination scheme as a function of the learning development of the neural network (10) for each of the jointly trained decoding tasks (18). Method according to one of the preceding Claims 1 or 2 , characterized in that the combining of the training data (20) of the data stream (24, 26) for the jointly trained decoding tasks (18) according to a combination scheme for providing the respective sub-mini-stack (34, 36) the selection of the training data (20 ) from the set (38, 40) according to a ratio; and updating the combination scheme for combining the sub-mini-stacks (34, 36) of training data (20) as a function of a learning development of the neural network (10) comprises updating the ratio. Method according to the preceding Claim 3 , characterized in that the selection of the training data (20) from the first and the second set (38, 40) according to a ratio generating a random number, in particular between zero and one, for each sample of training data (20) corresponding to the respective Sub-mini-batches (34, 36) are to be added and the sampling of training data (20) to be added from the first or the second set (38, 40) depending on whether the random number is greater than the ratio or not. Method according to one of the preceding claims, characterized in that the updating of the combination scheme for combining the sub-mini-stacks (34, 36) of training data (20) as a function of a learning development of the neural network (10) comprises applying a learning rate. Method according to one of the preceding claims, characterized in that the combining of the training data (20) of the first and second sets (38, 40) for the jointly trained decoding tasks (18) according to a combination scheme for providing the respective sub-mini-stack ( 34, 36) comprises reusing training data (20) of the first and / or the second data stream (24, 26) of training data (20) for the respective decoding task (18). Method according to one of the preceding claims, characterized in that the arrangement of the multiple sets (22) of differently annotated training data (20) in data streams (24, 26), a first set (38) of training data (20) corresponding to the target application context of the neural network (10) and additionally a second set (40) of training data (20) being provided for training each of the multiple decoding tasks (18) which do not correspond to the target application context of the neural network (10), combining all sets (22) of training data (20), which correspond to a specific decoding task (18) and the respective target application context, in the respective first set (38) and the Combining all sets (22) of training data (20) which correspond to the specific decoding task (18) and do not correspond to the respective target application in the respective second set (40). Method according to one of the preceding claims, characterized in that the training of the neural network (10) using the generated mini-stacks (32) comprises training of the neural network (10) in a synchronous back-propagation manner; and the updating of the combination scheme for the sub-mini-batches (34, 36) of training data (20) as a function of a learning development of the neural network (10) a dynamic update of the combination scheme for subsequent mini-batches (32) of training data (20 ) as a function of a learning development of the neural network (10). Method according to one of the preceding claims, characterized in that the generation of mini-stacks (32) of training data (20) which have a plurality of sub-mini-stacks (34, 36) of training data (20) for training each of the decoding tasks, comprises balancing the sub-mini-stacks (34, 36) for each of the mini-stacks (32) according to a balancing scheme. Method according to one of the preceding claims, characterized in that the method has a step of determining a learning development of the neural network (10) as a function of a loss of validation and / or of a validation key performance indicator of the neural network (10). Method according to the preceding Claim 10 , characterized in that the determination of a learning development of the neural network (10) as a function of a loss of validation and / or of a validation key performance indicator of the neural network (10) the independent determination of the learning development of the neural network (10) for each of the decoding tasks ( 18). Method according to one of the preceding Claims 10 or 11 , characterized in that the determination of a learning development of the neural network (10) as a function of a loss of validation and / or of a validation key performance indicator of the neural network (10) comprises the independent determination of the learning development of the neural network (10) for each of the application contexts . Method according to one of the preceding Claims 10 to 12th , characterized in that the determination of a learning development of the neural network (10) as a function of a loss of validation and / or of a validation key performance indicator of the neural network (10) includes the application of a learning rate. Method according to one of the preceding claims, characterized in that the method has a step for terminating the training if the learning development indicates that the neural network (10) is no longer improving, in particular if the learning development indicates that the neural network ( 10) not further improved with regard to at least one of the decoding tasks (18).

Images