DE102020105485A1 - Training local artificial neural networks - Google Patents

Abstract

According to a method for training local artificial neural networks that are stored on respective local devices (7a, 7b, 7c), a first and a second central artificial neural network (4, 5) are provided, which are stored on a server system (2) are. The central networks (4, 5) are initially trained based on initial training data (15) using a common loss function, the second central network (5) being trained to imitate the first central network (4). Each of the local networks is initially generated as a respective copy of the initially trained second central network (5) and each of the local networks is generated as a function of local sensor data that is provided by an environment sensor system (10a, 10b, 10c) of the respective local device (7a, 7b, 7c), trained locally.

Description

The invention relates to a method for training one or more local artificial neural networks, a method for at least partially automatic driving of a vehicle, a system for training one or more local artificial neural networks, an electronic vehicle control system, a computer program product, a computer program and a computer-readable storage medium .

Communicating or connected self-driving vehicles can use artificial neural networks that are stored on the vehicle itself, for example to process or interpret various sensor data such as camera data, radar data and so on. Based on a prediction of the artificial neural network, the vehicle can be driven fully or partially automatically and / or the driver of the vehicle can be supported. In particular, lateral and / or longitudinal control of the vehicle can be performed. Furthermore, algorithms based on the artificial neural networks can also be used for various applications in smartphones or other mobile electronic devices.

One potential challenge that developers face with such embedded systems is their limited computing resources in terms of processor and memory resources. As a result, adequate training of the artificial neural networks locally by the embedded systems can be difficult, inefficient, time consuming or, in the case of complex algorithm architectures, even impossible.

If a driver of a semi-automatic motor vehicle takes over the control from an automatic control, this could also indicate a non-optimal prediction of the neural network and consequently there could be a need to update parameters of the local network. The update can be computationally too demanding to be done locally, and it would have to be done on each local device individually. On the other hand, it can be very time-consuming to update the respective models centrally.

The publication "ProjectionNet: Learning Efficient On-Device Deep Networks Using Neural Projections", arXiv: 1708.00630v2, August 9, 2017, by S. Ravi (hereinafter referred to as "Ravi") describes an approach in which using a common Loss function two neural networks can be trained at the same time. For this purpose, a training network, which is a full neural network and the choice of which is flexible and depends on the task, is created together with a projection network, which is a simpler network that codes a set of operations to be calculated efficiently. The projection network is based on projection transformations using precomputed parameterized functions. During training, the simpler projection network learns to select and apply specific projection operations in order to mimic predictions of the full training network, which has more parameters and thus greater predictive capacity. The result is a trained projection network that is highly efficient in mimicking the high-precision output of the training network to some extent, whereby the complexity of the projection network can be adjusted to achieve the required accuracy.

It is therefore an object of the present invention to provide an improved concept for training one or more local artificial networks, in particular local artificial neural networks for at least partially automatic driving of respective vehicles, with each of the local networks on a respective local device, in particular a respective local device of the respective vehicle, is stored, which reduces the computational effort for training the local artificial neural networks on the local devices.

This object is achieved by the respective subject matter of the independent claims. Further designs and preferred exemplary embodiments are the subject of the dependent claims.

The improved concept is based on the idea of creating a first and a second central artificial neural network on a server system and training both networks together using a common loss function so that the second central network is able to imitate the first central network. The respective copies of the initially trained second central network are then trained locally on the respective local devices.

According to the improved concept, a method for training one or more local artificial neural networks is provided. Each of the local networks is stored on a respective local device, in particular on a respective storage element of the respective local device. A first Central artificial neural network, which is stored on a server system, in particular on a server storage element of the server system, and a second central artificial neural network, which is stored on the server system, in particular on the server storage element, are provided. The first central network and the second central network are initially trained, in particular using a server computing unit of the server system, based on initial training data using a predefined common loss function, the second central network being trained by the common initial training to be the first central Imitate network. Each of the one or more local networks is initially generated as a respective copy of the initially trained second central network, in particular by a respective computing unit of the respective local device and / or by the server computing unit. Each of the one or more initially generated local networks is trained locally as a function of local sensor data obtained from an environment sensor system of the respective local device, in particular by the respective computing unit of the respective local device from the respective environment sensor system.

Here and in the following, the term “network” is used equivalent to the term “artificial neural network”, unless otherwise stated.

According to the improved concept, a method for training one or more local artificial neural networks for at least partially automatic driving of respective vehicles is also provided. Each of the local networks is stored on a respective local device of a respective vehicle. A first central artificial neural network stored on a server system and a second central artificial neural network stored on the server system are provided. The first central network and the second central network are initially trained based on initial training data using a predefined common loss function, the second central network being trained to imitate the first central network. Each of the first and the plurality of local networks is initially generated as a respective copy of the initially trained second central network and each of the one or more local networks is trained locally as a function of local sensor data obtained from an environment sensor system of the respective local device.

In the event that the local devices are contained in respective vehicles, the local devices can be implemented, for example, as respective electronic vehicle guidance systems of the vehicle.

An electronic vehicle guidance system can be understood as an electronic system which is set up to guide a vehicle in a fully automatic or fully autonomous manner and in particular without manual intervention or control by a driver or user of the vehicle being necessary. The vehicle automatically carries out required steering maneuvers, braking maneuvers and / or acceleration maneuvers and so on. In particular, the electronic vehicle guidance system can implement a fully automatic or fully autonomous driving mode according to level 5 of the SAE J3016 classification. An electronic vehicle guidance system can also be implemented as a driver assistance system, ADAS, which supports a driver in semi-automatic or semi-autonomous driving. In particular, the electronic vehicle guidance system can implement a partially automatic or partially autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and in the following, SAE J3016 refers to the respective standard dated June 2018.

At least partially automatic driving of the vehicle can therefore include driving the vehicle in accordance with a fully automatic or fully autonomous driving mode in accordance with level 5 of the SAE J3016 classification. At least partially automatic driving of the vehicle can also include driving the vehicle according to a partially automatic or partially autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification.

In particular, each of the local devices contains the respective computing unit of the local device, the respective environment sensor system and a respective memory element which stores the respective local artificial neural network.

In particular, the local devices can be regarded as local, since their spatial positions are independent of one another and independent of a position of the server system, and their spatial positions can also change over time.

In particular, the local device can correspond to or be contained in a mobile electronic device such as a smartphone or another portable computer. As stated, the local device can also be contained in the vehicle or in the electronic vehicle guidance system.

The initial creation of the local network as a copy of the initially trained second central network can be understood, for example, in such a way that the initially trained second central network and the respective copies have the same network architecture and the same parameters as Weightings and / or bias values or other algorithmic parameters of neural networks.

Training the second central network so that it mimics the first central network can be understood, for example, to mean that the initially trained first and second central networks provide respective outputs in response to a given input that differ less than a predefined maximum error . This can be achieved in particular by jointly training the first and the second central network based on the joint loss function.

In particular, reference is made to the disclosure of Ravi. The first central network can be understood as the training network as described in section 3 of Ravi and the second central network can be understood as the projection network described in section 3 of Ravi.

In particular, the common loss function can be chosen analogously to the loss function defined by Ravi in equations (1) and (2). The first and second central networks can be generated according to equations (3) through (5) and the respective description of Section 3 of Ravi.

An environment sensor system of a local device can be understood, for example, as a sensor system that is capable of generating sensor data or sensor signals that display, represent or map an environment of the sensor system, the local device or, if applicable, the vehicle. In particular, the environment sensor system can contain one or more camera systems and / or one or more radar systems and / or one or more lidar systems and / or one or more ultrasonic sensor systems.

Since the local networks are trained locally as a function of the local sensor data, they are trained on the basis of real information about the surroundings of the respective local device and not, for example, exclusively on the basis of specially designed training data. In particular, it is possible to take into account the real behavior of a user of the respective local device in order to verify or evaluate the predictions or conclusions of the local network. A particularly efficient training of the local network is achieved in this way.

By combining the local training with the joint training of the central networks, the second central network mimicking the first central network, the complexity of the second central network and consequently the local networks can be greatly reduced compared to the complexity of the first central network. The local training of the local networks thus requires less computational effort compared to training the first central network.

Therefore, the improved concept is particularly suitable for local devices in the context of embedded systems, for example for vehicles or mobile electronic devices. By combining the initial central training with the individual local training of the local networks by the local devices, on the other hand, a particularly high reliability and accuracy of the predictions of the local networks can be achieved while the local computing requirements are kept low.

According to several implementations of the method, the one or more local networks includes two or more local networks, each of the two or more local networks having a respective set of local parameters. For each of the two or more local networks, the local training includes a modification of the respective set of local parameters, in particular by the respective processing unit of the local device. For each of the two or more local networks, local training data are transmitted as a function of the modified set of local parameters to the sensor system, in particular to the server computing unit, for example through respective communication interfaces of the local devices. The second central network, in particular the initially trained second central network, is modified in particular by the server computing unit as a function of the transmitted local training data, in particular as a function of the local training data from all local devices or all local networks. The modification of the second central networks includes a modification of respective second central parameters of the second central network, which for example contain respective weighting and / or bias parameters of the second central network.

The modification of the second central network can be followed or accompanied, for example, by additional training or retraining of the first and / or the second central network based on the initial training data and / or based on further training data.

The local parameters can, for example, completely define the operation of the respective local network and can, for example, have respective weighting or bias parameters or both. In other words, the local network is fully defined by providing the respective architecture of the local network and the set of local parameters.

The local training can include, for example, generating the sensor data by the environment sensor system of the respective local device, generating respective input data based on the sensor data, for example by the respective computing unit of the local device, applying the local network to the input data to generate local output data to calculate a respective local loss function by the arithmetic unit of the local device based on the local output data and to modify the respective local parameters based on a value of the local loss function calculated in particular by the respective arithmetic unit of the local device.

The local training data to be transmitted to the sensor system can contain, for example, the modified set of local parameters or data depending on the modified set of local parameters. The local training data can also contain the previous set of local parameters, in particular the set of local parameters before the modification, and / or can contain data that depend on the previous set of local parameters.

In particular, the local training data can include parameter gradients, which can be understood as deviations or differences between the modified set of parameters and the previous set of parameters, or data that depend on such deviations or differences.

In such designs, the accuracy of the predictions of local networks can be improved in that the learning is distributed over all local devices and the local training is carried out based on an individual real environment input, since the local sensor data of the local environment sensors are used for the local training.

In particular, it is not necessary to transmit the sensor data or the local input data, such as camera images, from the local devices to the sensor system. It can be sufficient to transmit the data depending on the parameters or the modification of the parameters of the respective local networks. In this way, improved data security and improved data protection can be achieved.

According to some embodiments, the server computing unit can compute a set of averaged parameters based on all transmitted local training data from the different two or more local networks. In particular, the parameter gradients can be averaged by the server computing unit. The server computing unit can then modify the second central network as a function of the result of the averaging.

In this way, information from all of the two or more local area networks can be taken into account to improve the performance of the second central area network.

According to some embodiments, the two or more locally trained local networks are updated in particular by the server computing unit and / or the respective computing unit of the local device in accordance with the modified second central network.

The updating of the local networks as a function of or in accordance with the modified second central network can include, for example, copying the modified second central parameters and replacing the respective local parameters completely or partially with the modified second central parameters.

In this way, each of the two or more local area networks benefits from the local training of all other local area networks.

According to some embodiments, local input data are generated for each of the one or more local networks, in particular by the respective computing unit of the local device, based on the respective local sensor data, in particular in order to carry out the local training. For each of the one or more local networks, local output data are generated by the respective local network as a function of the local input data, in particular as a function of the local input data and the respective set of local parameters of the local network, in particular to carry out the local training. For each of the one or more local networks, the local input data are transmitted to the server system as a function of the local output data, in particular from the respective communication interface of the local device to the server computing unit. The initially trained first central network and the initially trained second central network are in particular retrained by the server computing unit based on the transmitted local input data, in particular all local devices, using the common loss function.

In particular, the local training of the local network is carried out as a function of the local output data.

The local input data can correspond to the unprocessed or processed sensor data.

The retraining of the first and the second central network can include, for example, applying each of the central networks to the transmitted local input data or to data that depend on the transmitted local input data or contain the transmitted local input data in order to generate respective central outputs . Then the common loss function can be calculated depending on the central outputs and depending on the calculated value of the common loss function.

In such designs, the input data from the local devices can also be used explicitly to train the central networks. In particular, data or sensor data that are defined as “new and interesting” can be transmitted directly to the server system in order to enlarge the available training basis and to improve its relevance and consequently its reliability and accuracy.

According to some embodiments, the first and the second central network are retrained based on the transmitted local input data and the initial training data.

In particular, the renewed training of the central networks based on the transmitted local input data can be combined with the modification of the second central network depending on the transmitted local training data. In particular, the renewed training of the first and the second central network can be followed by the modification of the second central parameters of the second central network as a function of the transmitted local training data or vice versa.

According to some embodiments, the initially trained first central network and the initially trained second central network, in particular by the server computing unit, are retrained based on the initial training data, in particular the initial training data and the transmitted local input data from all local devices, using the common loss function.

According to some embodiments, the two or more local networks, in particular each of the two or more locally trained local networks, are updated in particular by the respective computing unit of the local device in accordance with the retrained second central network.

The updating of the local networks in accordance with the retrained second central network can include, for example, copying the second central parameters of the retrained second central network in order to completely or partially replace the respective local parameters.

According to some embodiments, the computing unit of the respective local device is used for each of the one or more local networks to calculate a value of a predefined local loss function depending on the respective local output data and in particular to assign the calculated value of the local loss function with a predefined threshold value to compare. For each of the one or more local networks, the local input data, in particular through the communication interface of the respective local device, are sent to the server system, in particular to the server computing unit, only when, for example precisely when the calculated value of the local loss function is greater than the predefined threshold value.

In particular, if the value of the local loss function is greater than the threshold value, this can be interpreted in such a way that the respective local network should be further improved through additional training. The corresponding local input data, which lead to the local loss function being greater than the threshold value, can be regarded as “new and interesting” and therefore as an advantageous input for training the central networks.

Since the local input data are only transmitted when the loss function is greater than the threshold value, the data transmission between the local devices and the server system is restricted, which improves the efficiency of the method and data protection.

According to some embodiments, the processing unit of the local device defines one or more respective transmission time periods and transmits the local input data to the server system only during the respective transmission time periods.

According to some embodiments, for each of the one or more local networks, an action by the user of a host device that contains the respective local device, for example a user or driver of the respective vehicle, using a further sensor system of the host device, in particular of the vehicle, detected. For each of the one or more local area networks, it involves locally training the respective local area network, a conclusion of the local area network depending on the to generate local sensor data, in particular by the respective processing unit of the local device, and to verify the conclusion based on the action of the user, in particular using the processing unit of the local device.

The conclusion can correspond, for example, to the local output data that were generated as a function of the local input data.

To verify the conclusion based on the action of the user, the computing unit of the local device can, for example, compare the conclusion or the local output data of the local network with the action of a user or with an expected conclusion or expected local output data based on the action of the user. If the comparison shows that the conclusion is the same as the expected conclusion, except for a predefined tolerance, the conclusion can be considered verified, otherwise it can be considered unverified.

In other words, the local training of the respective local area network is carried out depending on a deviation between the conclusion and the action of the user or between the conclusion and the expected conclusion.

The action of the user can include, for example, an input by the user via a user interface of the local device or a change in the position of the local device or the user and so on.

In the case of a vehicle, the user action can include, for example, operating a control device of the vehicle, such as a steering wheel, changing a steering angle, operating a brake system or a brake pedal or an accelerator pedal of the vehicle. In particular, the user action can include any user action for longitudinal and / or lateral control of the vehicle.

Accordingly, the further sensor system of the host device or of the vehicle can include a steering angle sensor, a pedal sensor, an inertia sensor such as a yaw sensor or an acceleration sensor, a sensor for rotational speed and / or a position sensor.

The verification of the conclusion or the comparison of the conclusion with the action of the user can include, for example, calculating the local loss function and comparing the calculated value with the threshold value.

In such implementations, the user action effectively defines the “correct” behavior or the expected conclusion. As a result, the local training can be carried out without providing a labeled set of training data and can be particularly relevant due to the direct influence of the real environment situation of the local device.

The training can therefore be more effective and the trained local networks can provide more accurate or more reliable predictions.

According to some implementations, the first central network includes a first set of central parameters and the second central network includes a second set of central parameters, the first set of central parameters being greater than the second set of central parameters.

For example, the first and second sets of central parameters may correspond to the parameters defining the operation of the first and second networks, respectively. These parameters can contain, for example, respective weighting and / or bias parameters. In other words, the first and second central parameters of the first and second central network can contain all trainable parameters of the respective central network.

That the first set of central parameters is greater than the second set of central parameters can be understood to mean that the number of parameters in the first set is greater than a number of parameters in the second set.

This choice can represent an efficient trade-off between accuracy and local computational requirements. In particular, the requirements with regard to the local computing resources and memory resources are lower, the smaller the second set of central parameters and consequently the respective sets of local parameters of the local networks are. On the other hand, the greater the first set of central parameters, the greater the accuracy of the prediction of the first central network, which is transferred to the accuracy of the second central network and the local networks. However, the accuracy of the local area networks can also be adjusted by adapting the size of the second set of central parameters and thus the local set of parameters.

In this way, by restricting the allowed thresholds for the joint loss function during central training, the second set of central parameters can be made as small as possible while being as large as necessary to meet the required accuracy.

According to some implementations, a predefined level of complexity of the first central network is greater than the level of complexity of the second central network.

The complexity of the first and the second central network can be given, for example, directly or indirectly by the size of the respective set of central parameters.

The degree of complexity can, for example, correspond to a number of hidden layers or a number of activation layers of the respective central network.

The degree of complexity can also be given by the total number of connections between artificial neurons in the respective network.

According to some implementations, the first central network is designed as a full neural network or a deep neural network. In particular, the first central network has at least two, preferably three or more, hidden layers.

According to some embodiments, the first central network is designed as a feed-forward neural network, as a recurrent neural network, RNN, or as a folding neural network, CNN.

According to some embodiments, the second central network contains exactly one hidden layer or exactly one layer of activations.

The second central network can be defined according to equations (4) and (5) of Ravi, for example.

According to the improved concept, a method for at least partially automatic driving of a vehicle is also provided. The vehicle includes a storage element that stores a local artificial neural network, and the method includes training the local network using a method for training one or more local artificial neural networks to at least partially automatically drive respective vehicles according to the improved concept. The method also includes generating further sensor data that represent the surroundings of the vehicle using an environment sensor system of the vehicle. The method also includes generating further output data depending on the further sensor data using the trained local network and at least partially automatically guiding the vehicle depending on the further output data, in particular using a computer unit of the vehicle.

In particular, the vehicle can contain an electronic vehicle guidance system that can be viewed as a local device in accordance with the explanations above, the electronic vehicle guidance system including the environment sensor system, the memory element and the computing unit.

The generation of the further output data can include, for example, generating further input data as a function of the further sensor data using the computing unit of the vehicle and providing the further input data as an input to the trained local network.

The local output data according to the training may or may not be used for at least partially automatic driving of the vehicle. In other words, the local output data for training according to the method for training one or more artificial neural networks according to the improved concept can also correspond to or contain further local output data for at least partially automatic driving of the vehicle.

According to the improved concept, a training system for training one or more local artificial neural networks is also provided. The training system includes one or more local devices, each local device including one, in particular exactly one, of the one or more artificial neural networks. The training system contains a server system, the server system, in particular a server storage element of the server system, storing a first central artificial network and a second central artificial network. The server system includes a server computing unit which is set up to initially train the first central network and the second central network using a predefined common loss function based on the initial training data, the initial training data being stored on the server storage element of the server system, for example the second central network is trained to mimic the first central network. Each of the local devices contains a respective arithmetic unit which is set up to initially generate the respective local network as a respective copy of the initially trained second central network, to receive local sensor data from an environment sensor system of the respective local device and the initially generated local To train the network locally depending on the local sensor data.

According to some embodiments, the one or more local devices are configured as respective local devices for at least partially automatic driving of respective vehicles, for example as respective electronic vehicle guidance systems. Further versions of the training system according to the improved concept result directly from the different versions of the method for training one or more local artificial neural networks according to the improved concept and from the different versions of the method for driving a vehicle according to the improved concept or vice versa.

According to the improved concept, an electronic vehicle guidance system for at least partially automatic guidance of a vehicle is also provided, the vehicle guidance system including a memory element that stores a local artificial neural network that is generated according to a method for training one or more local artificial neural networks according to the improved concept has been trained. The vehicle guidance system also includes an environment sensor system which is set up to generate additional output data as a function of the additional sensor data using the trained local network and to at least partially automatically guide the vehicle as a function of the additional output data.

Further versions of the electronic vehicle guidance system result directly from the various versions of the method for driving a vehicle in accordance with the improved concept and from the various versions of the training system in accordance with the improved concept or vice versa.

According to the improved concept, a computer program product is also provided. The computer program product contains a central set of commands and a respective local set of commands for each of one or more local devices. If the central set of commands is executed by a server system, in particular by a server computing unit, of the training system according to the improved concept and the local sets of commands are executed by respective local devices, in particular by respective processing units of the local devices, of the training system according to the improved concept , the central set of commands and the local sets of commands cause the training system to perform a method of training one or more local artificial neural networks according to the improved concept.

The computer program product can be designed as a set of computer programs which contain a central computer program with a central set of commands and a respective local computer program for each of the local devices with the respective local set of commands.

The computer program can also be designed as a set of respective computer-readable storage media that store respective central and local sets of commands.

According to the improved concept, a further computer program that contains further instructions is provided. If the further commands are executed by an electronic vehicle guidance system in accordance with the improved concept, the further commands cause the vehicle guidance system to carry out a method for driving a vehicle in accordance with the improved concept. According to the improved concept, a further computer-readable storage medium which stores a further computer program according to the improved concept is also provided.

Further features of the invention emerge from the claims, the figures and the description of the figures. The features and combinations of features mentioned above in the description as well as the features and combinations of features mentioned below in the description of the figures and / or shown alone in the figures can be used not only in the specified combination, but also in other combinations without departing from the scope of the invention . There are thus also embodiments of the invention to be regarded as encompassed and disclosed, which are not explicitly shown and explained in the figures, but emerge and can be generated from the explained embodiments by means of separate combinations of features. Designs and combinations of features are also to be regarded as disclosed, which therefore do not have all the features of an originally formulated independent claim. In addition, designs and combinations of features, in particular through the statements set out above, are to be regarded as disclosed which go beyond the combinations of features set forth in the back-references of the claims or differ from them.

• 1 eine schematische Darstellung einer beispielhaften Ausführung eines Trainingssystems gemäß dem verbesserten Konzept; und
• 2 ein Flussdiagramm einer beispielhaften Ausführungsform eines Verfahrens gemäß dem verbesserten Konzept.

In the figures show:

• 1 a schematic representation of an exemplary embodiment of a training system according to the improved concept; and
• 2 a flow diagram of an exemplary embodiment of a method according to the improved concept.

In 1 is an exemplary implementation of a training system 1 shown schematically according to the improved concept.

The training system includes a server system 2 and a variety of local devices 7a , 7b , 7c . The local devices 7a , 7b , 7c can for example in the respective vehicles 11a , 11b , 11c be included and can, for example, as the respective electronic vehicle guidance systems of the respective vehicles 11a , 11b , 11c be trained.

The server system 2 includes a server computing unit 6th and a server storage element 3 . The server storage element 3 stores a first central artificial neural network 4th and a second central artificial neural network 5 .

Any of the local devices 7a , 7b , 7c contains a respective computing unit 8a , 8b , 8c , a respective local storage element 9a , 9b , 9c as well as a respective sensor system 10a , 10b , 10c such as a camera system.

The function of the training system 1 will be described in more detail below on the basis of exemplary embodiments of a method for training one or more artificial neural networks according to the improved concept and in particular with reference to FIG 2 explained.

2 FIG. 3 shows a flow diagram of an exemplary embodiment of a method for training one or more local artificial neural networks according to the improved concept.

In step S1 of the procedure will be the first central network 4th and the second central network 5 from the server computing unit 6th using an approach described by Ravi.

The first and the second central network 4th , 5 can be used as a projection neural network, such as referring to FIG 1 described by Ravi.

In particular, corresponds to the first central network 4th a training network and the second central network 5 corresponds to a projection network. The first central network 4th can have two or more hidden layers 14th include while the projection network 5 or the second central network 5 a single layer of activations 13th may contain. Providing an input feature vector to each of the central networks 4th , 5 leads to a respective prediction of the first central network 4th and a respective prediction of the second central network 5 . Thereby the connection of the feature vector inputs with the individual neurons of the hidden layers 14th and the interlayer coupling between neurons of the different hidden layers 14th as well as the coupling of the hidden layers 14th to a Softmax layer of the first central network 4th by weighting and bias parameters of the first central network 4th Are defined.

In the case of the second central network 5 the feature vector can be attached to the layer of activations 13th by means of a set of projection functions 12th which can convert the input feature vector into d-bit vectors, one per projection function.

$$\text{L}1(.)=\sum \text{ D}\left[{\text{h}}_4\left({\underset{\_}{\text{x}}}_{\text{i}}\right){\text{,}\widehat{\text{y}}}_{\text{i}}\right]$$

$$\text{L2}(.)=\sum \text{ D}\left[{\text{h}}_5\left({\underset{\_}{\text{x}}}_{\text{i}}\right){\text{,h}}_4\left({\underset{\_}{\text{x}}}_{\text{i}}\right)\right]$$

$$\text{L3}(.)=\sum \text{ D}\left[{\text{h}}_5\left({\underset{\_}{\text{x}}}_{\text{i}}\right){\text{,}\widehat{\text{y}}}_{\text{i}}\right]$$

After that, the first and the second central network 4th , 5 based on initial training data 15th that are on the server storage element 3 can be stored, initially trained using a common loss function. In particular, the central networks 4th , 5 trained together to optimize the common loss function as defined by equations (1) and (2) of Ravi: $$\text{L.}=\mathrm{<figure-callout\; id="1"\; label="\lambda "\; filenames="DE102020105485A1\_0006.png"\; state="\{\{state\}\}">\lambda </figure-callout>}1\cdot \text{L.}1(.)+\mathrm{<figure-callout\; id="2"\; label="\lambda "\; filenames="DE102020105485A1\_0006.png"\; state="\{\{state\}\}">\lambda </figure-callout>}2\cdot \text{L.}2(.)+\mathrm{\lambda }3\cdot \text{L.}3(.)$$

$$\text{L.}1(.)=\sum \text{D.}\left[{\text{H}}_{\mathrm{4th}}\left({\underset{\_}{\text{x}}}_{\text{i}}\right){\text{,}\widehat{\text{y}}}_{\text{i}}\right]$$

$$\text{L2}(.)=\sum \text{D.}\left[{\text{H}}_5\left({\underset{\_}{\text{x}}}_{\text{i}}\right){\text{,H}}_{\mathrm{4th}}\left({\underset{\_}{\text{x}}}_{\text{i}}\right)\right]$$

$$\text{L3}(.)=\sum \text{D.}\left[{\text{H}}_5\left({\underset{\_}{\text{x}}}_{\text{i}}\right){\text{,}\widehat{\text{y}}}_{\text{i}}\right]$$

Here, Σ represents a sum with summation index i, which runs over a number of training instances, x _i represents the input feature vector in the event that the first central network 4th is designed as a feedforward network. ŷ _i , denotes the ground truth output classes that are used for training the network. h ₄ is a parametric representation of the corresponding hidden entities in the first central network 4th , which transforms x _i into the respective output prediction y _4i . Furthermore, h _{5 represents} the respective parameters of the second central network 5 which transforms the input into the corresponding predictions y _5i . A respective Softmax activation layer can be applied to each of the central Networks 4th , 5 can be applied to calculate the respective predictions y _4i and y _5i. D denotes a distance function representing the error. For further details, see the disclosure of Ravi, in particular section 3 , referenced.

By training the first and the second central network together 4th , 5 using the common loss function becomes the second central network 5 trained to do so by the first central network 4th mimic predictions made.

That from the first central network 4th the large model shown and that of the second central network 5 The small model shown are therefore trained at the same time, so that the second central network 5 easy and efficient to compute operations in bit space with low memory requirements, as described by Ravi.

For example, the second central network 5 using an LSH process (English: "Locality-Sensitive Hashing", LSH). In this way, the closest neighbor search in high-dimensional spaces can be solved approximately or exactly. As a result, the second central network can 5 correspond to a binary order of the input. On the other hand is the first central network 4th so complex that it includes all the features to be learned or taught.

The common loss function in this model can be a combination of three losses, namely a loss between the teacher network and the ground truth, a loss between the projection network and the ground truth, and a loss between the teacher network and the projection network.

Tests have shown that it is possible to increase the size of the second central network 4th with an acceptable increase in the error rate by up to 10 to 15 times.

In step S2 become the local neural networks of the respective local devices 7a , 7b , 7c initially generated by the respective initially trained second central network 5 copied and downloaded, for example, via the respective communication interfaces.

In step S3 of the method, the local networks generated in this way are dependent on the respective local sensor data from the corresponding environment sensor systems 10a , 10b , 10c are generated, trained locally.

At least two scenarios can be used for this purpose. In the first scenario, the local networks imitate the behavior of the respective driver of the vehicle 11a , 11b , 11c just after, without controlling the vehicle 11a , 11b , 11 c to take over. The second scenario corresponds to a situation in which the respective local network is so well trained that it can control the vehicle 11a , 11b , 11c takes over until the driver interrupts and takes control again.

In any case, the arithmetic units 8a , 8b , 8c the respective input data 16a , 16b , 16c from the environmental sensor systems 10a , 10b , 10c and the local networks provide corresponding predictions and conclusions. The predictions and conclusions can also be viewed as output data from the local area networks.

In particular, the output of the local loss functions is derived from the respective sensor inputs 16a , 16b , 16c result, compared with a respective predefined threshold value.

If the output of the local loss function is higher than the threshold value, the respective local output data were not accurate enough and the local network can, for example, use back propagation to update its own weights.

On the other hand, the computing units detect 8a , 8b , 8c respective actions of the driver, either by taking control of the automatic driving or by manually driving the vehicle 11a , 11b , 11c , while the network runs unproductively in parallel.

Then calculate the arithmetic units 8a , 8b , 8c respective local loss functions by comparing the driver's actions and the respective logits of the respective local network. Then the arithmetic units can 8a , 8b , 8c adapt the local parameters of the respective local networks accordingly. In this way, the local networks are trained locally.

In step S4 transmit the computing units 8a , 8b , 8c local training data depending on the modified local parameters of the local networks to the server computing unit 6th . The server computing unit 6th can perform an averaging depending on the transmitted local training data. In particular, the server computing unit 6th calculate respective average parameters or average weightings from all uploaded local training data to the second parameters of the second central network 5 to update. For example, the weightings of the second central network 5 modified according to the following rule: $${\text{w}}_{\text{t}+1}\leftarrow {\text{w}}_{\text{t}}-\text{n}\sum {\text{G}}_{\text{k}}\ast {\text{n}}_{\text{k}}\text{/ n}$$

The modified weighting w _{t + i is} the previous weighting wt of the second central network 5 subtracted from the learning rate n together with the summation of all gradients g _{k of} each local device and n is the total number of weightings of all uploaded networks. The summation index k runs over all local devices.

The server computing unit 6th combines the uploaded data from the local devices 7a , 7b , 7c with their original data to obtain an enlarged data set for training.

The new jointly initialized weights according to the above equation are called the modified smaller second central network 5 used. The method steps can then be repeated as described with step S2 can be continued.

In some implementations, the optional process steps S5 and S6 are executed. If the output of the local loss function is higher than the threshold value, the respective input data 16a , 16b , 16c can be considered "interesting" and can be stored in the local storage element 9a , 9b , 9c saved for future upload. On the other hand, if the output of the local loss function is lower than the threshold value, it may be that there is no back propagation of the local network and the respective image data are no longer used for training.

The input data considered “interesting” 16a , 16b , 16c can access the server computing unit 6th uploaded.

After that, in step S6 , the server computing unit can 6th the central networks 4th , 5 based on the uploaded interesting image input data 16a , 16b , 16c for example in combination with the initial training data 15th train again.

This retraining can also benefit from the modification that was made above with regard to step S4 is described, accompanied or combined with it.

Since the input data 16a , 16b , 16c are only uploaded in the event that they are considered “interesting”, in other words in the event that the respective local loss function is greater than the corresponding threshold value, the bandwidth utilization can be maximized with minimal requirements.

While training on the server from step S1 can use the initial training data 15th regardless of any contributing vehicle 11a , 11b , 11c be used.

The information for initially generating the local networks can be downloaded using over-the-air technology.

As stated, the improved concept represents lightweight, adaptable local area networks for use in embedded systems, for example in motorcycles.

The uploading of data can be restricted in order to ensure efficient use of the available bandwidth and an improved level of data protection.

Because local networks are lightweight, latency and power consumption can be reduced.

In other words, the computing costs of the local networks are reduced and the performance of the local networks is continuously improved by using new data and through appropriate local training. In particular, each of the local devices or local networks uses information collected from all connected local devices. In other words, the vehicles will learn from the server system and from each other while maintaining privacy and anonymity.

Claims (15)

Computer-implemented method for training one or more local artificial neural networks for at least partially automatic driving of respective vehicles (11a, 11b, 11c), each of the local networks on a respective local device (7a, 7b, 7c) of a respective vehicle (11a, 11b , 11c), a first central artificial neural network (4) stored on a server system (2) and a second central artificial neural network (5) stored on the server system (2) being provided will; - The first central network (4) and the second central network (5) are initially trained based on initial training data (15) using a predefined common loss function, the second central network (5) being trained to use the first central network ( 4) imitate; and - each of the one or more local networks is initially generated as a respective copy of the initially trained second central network (5); and - each of the one or more local networks is locally trained as a function of local sensor data obtained from an environment sensor system (10a, 10b, 10c) of the respective local device (7a, 7b, 7c). Procedure according to Claim 1 characterized in that - the one or more local area networks includes two or more local area networks, each of the two or more local area networks having a respective set of local parameters; - for each of the two or more local area networks - the local training includes a modification of the respective set of local parameters; and - local training data are transmitted to the server system (2) as a function of the modified set of local parameters; and - the second central network (5) is modified as a function of the transmitted local training data. Procedure according to Claim 2 , characterized in that each of the two or more local area networks is updated according to the modified second central network (5). Method according to one of the preceding claims, characterized in that - for each of the one or more local networks - local input data (16a, 16b, 16c) are generated based on the respective local sensor data; - Local output data are generated as a function of the local input data (16a, 16b, 16c) from the respective local network; and - depending on the local output data, the local input data (16a, 16b, 16c) are transmitted to the server system (2); and - the first central network (4) and the second central network (5) are retrained based on the transmitted local input data (16a, 16b, 16c) using the common loss function. Procedure according to Claim 4 , characterized in that the first central network (4) and the second central network (5) are retrained based on the initial training data (15) using the common loss function. Method according to one of the Claims 4 or 5 , characterized in that each of the two or more local area networks is updated according to the retrained second central network (5). Method according to one of the Claims 4 until 6th , characterized in that for each of the one or more local networks, - a computing unit (8a, 8ab, 8c) of the respective local device (7a, 7b, 7c) is used to calculate a value of a predefined local loss function depending on the respective calculate local output data; - The local input data (16a, 16b, 16c) are only transmitted to the server system (2) if the calculated value of the local loss function is greater than a predefined threshold value. Method according to one of the preceding claims, characterized in that for each of the one or more networks, - an action by a user of the respective vehicle (11a, 11b, 11c) is detected using a further sensor system of the vehicle (11a, 11b, 11c) will; the local training of the respective local network includes generating a conclusion of a local network as a function of the local sensor data; and verify the conclusion based on the user's action. Method according to one of the preceding claims, characterized in that - the first central network (4) has a first set of parameters and the second central network (5) has a second set of parameters, the first set of parameters being greater than that second set of parameters; and / or - a predefined degree of complexity of the first central network (4) is greater than the degree of complexity of the second central network (5). A method for at least partially automatic driving of a vehicle (11a, 11b, 11c), the vehicle (11a, 11b, 11c) having a memory element (9a, 9b, 9c) which stores a local artificial neural network, the method including it , - to train the local network using a method according to one of the preceding claims; - to generate further sensor data representing the surroundings of the vehicle (11a, 11b, 11c) using an surroundings sensor system (10a, 10b, 10c) of the vehicle (11a, 11b, 11c); generate further output data depending on the further sensor data using the trained local network; and - to drive the vehicle (11a, 11b, 11c) at least partially automatically depending on the further output data. Training system for training one or more local artificial neural networks, whereby - the training system (1) has one or more local devices (7a, 7b, 7c) for at least partially automatic driving of respective vehicles (11a, 11b, 11c), each local device (7a, 7b, 7c) having one or more local artificial neural Networks stores; - The training system (1) has a server system (2) which stores a first central artificial neural network (4) and a second central artificial neural network (5); - The server system (2) has a server computing unit (6) which is set up to train the first central network (4) and the second central network (5) based on initial training data (15) using a predefined common loss function, with the second central network (5) is trained to imitate the first central network (4); and - each of the local devices (7a, 7b, 7c) has a respective computing unit (8a, 8b, 8c) which is set up to initially assign the respective local network as a respective copy of the initially trained second central network (5) produce; - to receive local sensor data from an environment sensor system (10a, 10b, 10c) of the respective local device (7a, 7b, 7c); and - to train the local network locally as a function of the local sensor data. Electronic vehicle guidance system for at least partially automatic guidance of a vehicle (11a, 11b, 11c), wherein the vehicle guidance system (7a, 7b, 7c) - contains a memory element (9a, 9b, 9c) that stores a local artificial neural network which, according to a the Claims 1 until 9 is trained; - Contains an environment sensor system (10a, 10b, 10c) which is set up to generate further sensor data which represent the environment of the vehicle (11a, 11b, 11c); Contains a computing unit which is set up to generate further output data as a function of the further sensor data using the trained local network; and - to drive the vehicle (11a, 11b, 11c) at least partially automatically as a function of the further output data. Computer program product, which contains a central set of commands and for each one or more local devices (7a, 7b, 7c) a respective local set of commands, wherein, if the central set of commands and the local sets of commands from a training system (1 ) according to Claim 11 are executed, the central set of commands and the local sets of commands cause the training system (1) to implement a method according to one of the Claims 1 until 9 perform. Computer program which contains commands which, when sent by an electronic vehicle control system (7a, 7b, 7c) according to Claim 12 are executed, cause the electronic vehicle guidance system (7a, 7b, 7c) to carry out a method according to Claim 10 perform. Computer-readable storage medium containing a computer program Claim 14 saves.

Images