DE102021100940A1 - Reconstructing the environment of an environment sensor system - Google Patents

Abstract

A computer-implemented method for training an artificial neural network (10') in order to reconstruct a generic state of an environment of an environment sensor system (4), includes the provision of an artificial neural network (10') with an encoder module (6') and a decoder module ( 7'). Corresponding sensor data records, which represent the environment, are obtained from the environment sensor system (4) for a large number of consecutive training frames. A first feature mapping (11a, 11b) is generated for each of the training frames by applying the encoder module (6') to the respective sensor data set (8a, 8b). A reconstructed data record (13) is generated by the decoder module (7') depending on the first imaged feature images (11a, 11b). The artificial neural network (10') is trained unsupervised as a function of the reconstructed data set (13) in order to remove a dynamic object (14) in an input data set representing the environment.

Description

The present invention is directed to a computer-implemented method for training an artificial neural network to reconstruct a generic state of an environment of an environmental sensor system. The invention also relates to a method for at least partially automatically driving a motor vehicle, to an electronic vehicle guidance system and to a computer program product.

Previously generated and stored trajectories can be used for automatic or semi-automatic driving functions for a motor vehicle. For example, in Visual Simultaneous Location and Mapping, VSLAM, trajectories are created and stored and thereafter, when the same situation or environment is recognized, the stored trajectory can be used for at least partially automatic control of the motor vehicle. Possible use cases are, for example, automatic or semi-automatic parking applications.

VSLAM in itself is a well-known technique to map an environment and identify the vehicle's physical location. During a training phase, the VSLAM algorithm can extract key points or key features from images captured by one or more cameras of the motor vehicle and/or from other environmental sensor data, such as lidar point clouds, while simultaneously creating a map for the sensed environment. If during a playback phase it is determined that the vehicle is back in the same environment, the VSLAM algorithm tries to recognize the previously extracted key points. A trajectory generated during the training phase can then be played back.

A problem with this and similar approaches based on previously trained trajectories is that the key points extracted during the rendering phase must be identical to the key points extracted during the training phase. However, over time it may become necessary to change the initially trained trajectories, for example when new objects appear in the environment or already existing objects are no longer present. With the current approaches, the VSLAM algorithm may fail during the playback phase in such cases.

In addition, similar problems can occur when the appearance of the environment is affected by weather conditions that differ from the weather conditions in the training phase. In this case, too, the key points may not be determined with sufficient reliability. In other words, the VSLAM approach and similar approaches only work reliably if the vehicle's environment remains absolutely the same.

It is therefore an object of the present invention to provide an improved concept for processing data from an environment sensor system that enables more reliable detection of relevant features in the environment of the environment sensor system.

This object is achieved by the respective subject matter of the independent claims. Further implementations and preferred embodiments are subject of the dependent claims.

The improved concept is based on the idea of using an artificial neural network to reconstruct a generic state of the environment of an environmental sensor system.

According to the improved concept, a computer-implemented method is specified for training an artificial neural network to reconstruct a generic state of an environment of an environment sensor system. For this purpose, an artificial neural network, in particular an untrained or partially trained artificial neural network, is provided, for example stored on a memory unit. The artificial neural network is designed as a variation autoencoder that contains an encoder module and a decoder module. For each of several consecutive training frames, a sensor data set representing the surroundings is received by the surroundings sensor system, in particular by a computing unit. A first feature representation is generated for each of the training frames, in particular by the computing unit, in that the encoder module is applied to the sensor data set of the respective training frame. A reconstructed data set is generated, in particular by the processing unit, by the decoder module as a function of the first feature representations, in particular as a function of all first feature representations of the training frames. The artificial neural network is trained unsupervised, in particular by the computing unit, as a function of the reconstructed data set in order to remove a dynamic object in an input data set that represents the environment.

After the training of the artificial neural network has been completed or has reached a target state, the trained artificial neural network can be stored on a storage unit, for example as part of a corresponding inference model.

The generic state of the environment of the environment sensor system can be viewed as a version of the environment without dynamic objects. The generic state may correspond to a current prior state of the environment or a hypothetical prior state of the environment. In particular, the generic state can be viewed as a state of the environment in which all dynamic objects or at least approximately all dynamic objects are removed or not present.

A dynamic object can be viewed as an object that is not static in its environment. In other words, the dynamic object moves or changes its position and/or orientation within a static environment coordinate system. In other words, when the environment sensor system is not moving with respect to the environment, a dynamic object has a time-dependent position and/or orientation in a sensor coordinate system that is rigidly connected to the environment sensor system.

The environment of the environment sensor system corresponds to a given scene in the world defined by a pose, i.e. position and orientation, of the environment sensor system in the world, i.e. in the environment coordinate system, and possibly by the field of view of the environment sensor system. Over the course of the successive training frames, the pose of the environment sensor system does not change significantly, or in other words, changes in pose are negligible.

An environment sensor system can be understood to mean a sensor system that is able to generate sensor data or sensor signals that map or represent the environment of the environment sensor system. For example, cameras, lidar systems, radar systems, and ultrasonic sensor systems can be viewed as environment sensor systems.

According to the improved concept, the environment sensor system preferably includes one or more cameras and/or one or more lidar systems. Consequently, here and below, a sensor data set can include one or more camera images and/or one or more lidar images or lidar point clouds. For example, the training frames can correspond to camera frames or video frames or lidar scan frames.

In particular, a sensor data set can be generated by one or more sensors of the surroundings sensor system. In other words, a sensor data set can correspond to a merged sensor data set.

The encoder module and the decoder module can themselves be regarded as artificial neural networks or as sub-networks of the artificial neural network. In addition to the encoder and the decoder module, the artificial neural network can also include one or more further sub-networks or modules.

In general, the encoder module of an autoencoder is designed to generate a code that represents the data provided as input, where the code has a smaller dimension than the data provided as input. The decoder module is designed to reconstruct the data provided as input in dependence on the low-dimensional code. In this context, however, reconstruction does not necessarily mean that the data provided as input is reproduced identically. In particular, the autoencoder can modify or correct the data provided as input in a defined way. According to the improved concept, this modification can include the removal of the dynamic object. In contrast to a regular autoencoder, the encoder module of a variation autoencoder does not generate a single code, but rather a probability distribution over codes. The decoder module of a variational autoencoder can sample a code from this probability distribution and attempt to reconstruct it based on this sampled code. In the context of the improved concept, the first feature representations can represent corresponding parts of the codes.

For the training of the artificial neural network, the encoder module can also be considered as including several encoder sub-modules, one for each of the training frames. The sensor data records of the respective training frames are made available to the corresponding encoder submodule independently of one another.

In particular, the encoder module can generate multiple feature representations for each of the training frames, including the first feature representation. For each of the training frames, one of the respective feature representations represents a specific feature in the environment or in the respective sensor data set. Ins in particular, each encoder submodule can generate the different feature representations for the corresponding training frames. The first feature representations of the different training frames correspond to the same feature in the environment or represent the same feature. The same applies analogously to all other feature representations. In particular, for a given training frame, each feature representation represents a corresponding feature in the environment, and for each training frame, a corresponding feature representation is generated that represents the same feature.

The fact that the artificial neural network is trained unsupervised to remove the dynamic object in the input data set can be understood as training the artificial neural network with the aim of being able to remove the dynamic object in the input data set. The input data set is not one of the sensor data sets corresponding to the training frames, but rather an, in particular hypothetical, input data set that can be fed to the artificial neural network after training and represents the same environment as the sensor data sets of the training frames .

The unsupervised training can be carried out iteratively, for example, with network parameters of the artificial neural network, in particular of the encoder module and/or the decoder module, being adjusted in each iteration or training epoch such that a predefined loss function is minimized. For example, the network parameters may include appropriate weighting parameters or distortion parameters. The loss function can be a standard loss function for training variational autoencoders, such as a cross-entropy loss function.

If the input data set, which represents the environment and includes the dynamic object, is fed to the artificial neural network, in particular the encoder module, after the training has been completed, the input data set is processed by the encoder module and the decoder module in order to generate a corresponding reconstructed data set that dynamic object does not contain.

Here and in the following, all method steps can be carried out by the computing unit, unless stated otherwise. The processing unit can, for example, be a processing unit of the surroundings sensor system or can be coupled to the surroundings sensor system. The environment sensor system can be attached to a motor vehicle, for example, or can be included in a motor vehicle. The computing unit can then be a computing unit of the motor vehicle and, for example, be included in an electronic control unit, ECU, of the motor vehicle.

In addition to the trained artificial neural network or parts thereof, further information about the current environment for which the training has been carried out can be stored in order to generate the inference model. The additional information can include, for example, a pose of the surroundings sensor system and/or the motor vehicle or calibration data for the surroundings sensor system, in particular a pose of the surroundings sensor system in relation to the motor vehicle, a time stamp or other time information and so on.

The improved concept computer-implemented method for training the artificial neural network uses the time-varying information represented by the successive training frames to force the variational autoencoder to learn what dynamic objects are in the current environment of the environmental sensor system and consequently how to reconstruct the generic state of the environment, that is, the environment without dynamic objects.

After the training of the artificial neural network is completed in the manner described, the trained artificial neural network is able to provide a generic template of the environment when the vehicle or the environment sensor system is again in the same environment, regardless of whether the dynamic situation has changed or additional objects have appeared or previously existing objects have disappeared.

The correspondingly reconstructed data set can therefore be used for subsequent trajectory planning algorithms, for example in the context of VSLAM. The reconstruction of the environment or the generic state of the environment can be viewed as a pre-processing step that pre-processes the current data received from the environment sensor system in a generic way to ensure reproducible results and more reliable detection of key features in the environment. Therefore, previously trained trajectories can also be used in later situations when weather conditions have changed or other dynamic objects affect the sensor data.

In other words, it can be ensured that changing external conditions or Environmental conditions between a time when a trajectory is initially generated and stored and a time when the trajectory is played back or is to be used do not affect the reliability or even hinder the use of a corresponding trajectory-based function. For the same reasons, the frequency of required retraining of trajectories can be reduced.

In addition, unsupervised training can be employed by using the variation autoencoder, which is particularly useful in situations where the typical objects for supervised training are difficult to annotate. The variational autoencoder is used to analyze static objects in the multiple consecutive training frames and remove the dynamic objects, effectively regarding them as part of the noise. Generally known architectures can be used for the design of the encoder module or the encoder submodules and the decoder module. In particular, encoder modules or decoder modules based on convolutional neural networks, CNN, for example VGG networks, can be used.

According to several specific embodiments, the environment sensor system is an environment sensor system of a motor vehicle and the environment of the environment sensor system is an environment of the motor vehicle.

According to several embodiments, a feature prioritization module of the artificial neural network is applied to an output of the encoder module, in particular to a common output of all encoder sub-modules. By using the feature prioritization module, a respective first deviation between the first feature representations of the respective pair is determined for each pair of the first feature representations. The first representations of features, in particular each of the first representations of features, are modified as a function of the determined first deviations. The reconstructed data set is generated by applying the decoder module to an output of the feature prioritization module.

The pairs of the first feature representations include all possible pairs of the first feature representations, ie a number of N*(N-1)/2 pairs for a number of N training frames or first feature representations.

The feature prioritization module can be considered part of the artificial neural network that is placed between the encoder module and the decoder module. While conventional variational autoencoders provide the output of the encoders directly to the decoder, the feature prioritization module is placed between them in respective embodiments, at least for the training phase of the artificial neural network. In particular, the feature prioritization module is designed in such a way that certain features or feature representations are reinforced in relation to others. For this purpose, the respective first deviations and possibly corresponding further deviations between pairs of further feature representations that correspond to the same features are calculated and evaluated in order to understand the changes in certain features during consecutive training frames.

In other words, the smaller the first deviation for a given pair of first feature representations, the less the respective feature has changed between the corresponding frames. Therefore, an average of the first deviation can be taken as a measure of how dynamically the respective feature behaves during the successive training frames. This information can be used to reinforce the first feature representations or, in other words, to prioritize the first feature representations, particularly with regard to further feature representations. As a result, during the training of the artificial neural network, the importance or priority of static objects that are consistent across the successive training frames is emphasized, and objects that are inconsistent or dynamic are given a greatly reduced priority in the encoded features in the respective latent space . Dynamic objects can therefore be completely removed in the course of training.

The output of the encoder module to which the feature prioritization module is applied includes all first feature representations and, if applicable, all further feature representations of each of the consecutive training frames. The output of the prioritization module includes the first feature representations modified as a function of the first deviations.

For example, the feature prioritization module can be designed as a self-attention module or, in other words, as a module for applying a self-attention algorithm to the output of the encoder module. In particular, the self-attention engine is designed to reinforce features with the lowest average deviation in their respective feature representations. For example, the self-attention algorithm can include a global average pooling (GAP) algorithm. The GAP algorithm can determine an average neuron score for each of the feature representations and then multiply that average neuron score by the closest-spaced neurons.

According to various embodiments, a Euclidean distance between the first feature representations of the respective pair is calculated to determine the first deviation of the respective pair. In other words, the first deviation corresponds to a first Euclidean distance.

According to several embodiments, a second feature representation is generated for each of the training frames by applying the encoder module to the sensor data set of the respective training frame.

In particular, for each of the training frames, a plurality of feature representations, including the first and second feature representations, are generated by applying the encoder module to the sensor data set of the respective training frame. For each of the feature representations of a given training frame, there is a corresponding feature representation in each of the remaining training frames, with corresponding feature representations representing the same features or types of features. In other words, all first feature representations correspond to the same first feature, all second feature representations correspond to the same second feature, and so on.

According to several embodiments, by applying the feature prioritization module to the output of the encoder module, a respective second deviation between the second feature representations of the respective pair is determined for each pair of the second feature representations and the first feature representations and/or the second feature representations are determined as a function of the determined first deviations and the determined second deviations modified.

In particular, the first and second feature representations and possibly the further feature representations result from the one-time application of the encoder module to the sensor data set of the respective training frame.

According to several embodiments, by applying the feature prioritization module to the output of the encoder module, features corresponding to the first feature representations are prioritized higher than features corresponding to the second feature representations when a mean value of the first deviations is greater than a mean value of the second deviations.

Similarly, if the mean of the first deviations is less than the mean of the second deviations, the features corresponding to the first feature representations are prioritized less than the features corresponding to the second feature representations.

This principle can be extended analogously to all other feature representations that each represent the same features. In particular, the features that correspond to the feature representations with the lowest mean value of the respective deviation are assigned the highest priority.

According to several embodiments, by applying the feature prioritization module to the output of the encoder module, the self-attention algorithm is applied to the output of the encoder module depending on the first deviations and the second deviations to modify the first feature representations and/or the second feature representations.

According to several embodiments, the self-attention algorithm includes a GAP algorithm.

According to various embodiments, the Euclidean distance between the first feature representations of each pair is calculated to determine the first deviation for each pair. The same applies analogously to the second deviation for the pairs of the second representations of features and possibly further deviations for pairs of further representations of features.

According to several embodiments, a further artificial neural network, in particular a further untrained or partially trained artificial neural network, is provided, the further artificial neural network being designed as a variation autoencoder with a further encoder module and a further decoder module. For each of a plurality of consecutive training frames, a further sensor data set, which represents a further environment of the surroundings sensor system, is received, in particular from the surroundings sensor system. A further first feature representation is generated for each of the training frames by applying the further encoder module to the further sensor data set of the respective training frame. Another The reconstructed data set is generated by the further decoder module as a function of the further first feature representations. The further artificial neural network is trained unsupervised as a function of the further reconstructed data set in order to remove a dynamic object in a further input data set which represents the further environment.

All explanations relating to the artificial neural network and the training frames are transferred analogously to the further artificial neural network or the further training frames.

The further environment of the environment sensor system differs from the environment of the environment sensor system. In particular, the pose of the surroundings sensor system according to the further surroundings differs from the pose of the surroundings sensor system according to the surroundings.

Consequently, a further inference model can be generated on the basis of the further trained artificial neural network. In such implementations, two differently trained artificial neural networks are obtained that are able to remove the dynamic objects for different poses of the environmental sensor system. In the same way, additional further artificial neural networks can be trained for additional further environments.

According to various embodiments, the pose of the surroundings sensor system, which corresponds to the surroundings of the surroundings sensor system, is received by the computing unit and a further pose of the surroundings sensor system, which corresponds to the further surroundings of the surroundings sensor system, is received by the computing unit. After the artificial neural network has been trained, an inference model comprising the pose, the encoder module and the decoder module of the trained artificial neural network is generated and stored. After the further artificial neural network has been trained, a further inference model, which includes the further pose, the further encoder module and the further decoder module of the trained further artificial neural network, is generated and stored.

The pose and the additional pose can be calculated, for example, by the computing system based on the output of one or more position sensors of the surroundings sensor system or of the motor vehicle. The pose and the further pose include in particular the position and orientation of the surroundings sensor system or the further position and the further orientation. The positions may be determined, for example, from a Global Navigation Satellite System (GNSS) signal received by a GNSS receiver coupled to the environment sensor system. The orientation can be calculated as a function of sensor readings from acceleration sensors and/or yaw rate sensors based on an odometric calculation scheme.

For example, the inference model can completely include the trained artificial neural network or the artificial neural network without the prioritization module. The same applies analogously to the further inference model and the further artificial neural network.

Analogously to the inference model and to the additional inference model, one or more additional additional inference models can be generated for one or more additional additional poses and environments. In this way, after the generation of the inference models, the computing unit can select one of the inference models during a playback phase that best corresponds to the current pose of the environment sensor system or the motor vehicle, and with the correspondingly trained encoder and decoder module, the generic state of the current environment in the described way to reconstruct.

According to the improved concept, a method for training an artificial neural network for the reconstruction of a generic state of an environment of an environment sensor system is provided, which is a computer-implemented method for training an artificial neural network for the reconstruction of a generic state of an environment of an environment sensor system according to the improved concept and the Step of generating the sensor data sets for the successive training frames by the environment sensor system.

According to the improved concept, a method for reconstructing a generic state of an environment sensor system is provided. A sensor data set that represents a current environment of the surroundings sensor system is generated by the surroundings sensor system. A trained artificial neural network is applied to the sensor dataset by a computing unit to remove a dynamic object in the sensor dataset, the trained artificial neural network being designed as a variational autoencoder including an encoder module and a decoder module. In order to remove the dynamic object in the sensor data set, a first feature representation is generated by Encoder module is applied to the sensor data set, and a reconstructed data set is generated by the decoder module in dependence on the first feature representation.

Consequently, the reconstructed dataset does not include the dynamic object. The method for reconstructing a generic state according to the improved concept can be regarded as an inference phase or as a playback phase for the trained artificial neural network, while the computer-implemented method for training the artificial neural network for the reconstruction of a generic state corresponds to the training phase for the artificial neural network.

In particular, a computer-implemented method for training an artificial neural network for reconstructing a generic state of an environment of an environment sensor system is performed in order to generate the trained artificial neural network for the method for reconstructing a generic state for an environment of the environment sensor system.

In particular, the trained artificial neural network at least partially contains the artificial neural network of the inference model or the further artificial neural network of the further inference model.

According to various embodiments, a current pose of the environment sensor system is determined. The current pose is compared to the pose of the inference model and to the further pose of the further inference model. Depending on the respective results of the comparisons, the artificial neural network of the inference model or the further artificial neural network of the further inference model is selected as the trained artificial neural network.

In alternative implementations, depending on the respective results of the comparisons, the encoder module and the decoder module of the artificial neural network of the inference model are selected or the further encoder module and the further decoder module of the further artificial neural network of the further inference model are selected.

In other words, if the current pose differs from the pose of the inference model by a predefined tolerance or less, the encoder module and the decoder module of the inference model can be selected for the trained artificial neural network. On the other hand, if the current pose differs from the further pose of the further inference model by the predefined tolerance or less, the further encoder module and the further decoder module of the further inference model can be selected.

According to the improved concept, a method for at least partially automatically driving a motor vehicle is also specified. The method includes performing a method for reconstructing a generic state of an environment of an environment sensor system according to the improved concept, wherein the environment sensor system is mounted on the motor vehicle. A trajectory for the motor vehicle is planned on the basis of the reconstructed data set, in particular by the computing unit or by a further computing unit of the motor vehicle. The motor vehicle is at least partially automatically guided according to the planned trajectory.

For example, the trajectory can be planned based on a VSLAM method. For example, one of several previously trained and stored trajectories can be selected by the computing unit depending on the reconstructed data set in order to plan the trajectory.

The motor vehicle can be at least partially automatically controlled, for example, by an electronic vehicle guidance system of the motor vehicle.

An electronic vehicle guidance system can be understood as an electronic system that is set up to guide a vehicle fully automatically or fully autonomously and in particular without manual intervention or manual control by a driver or user of the vehicle. The vehicle automatically carries out all necessary functions, such as steering manoeuvres, deceleration and/or acceleration manoeuvres, as well as monitoring and recording road traffic and the corresponding reactions. In particular, the electronic vehicle guidance system can implement a fully automated or fully autonomous driving mode according to level 5 of the SAE J3016 classification. An electronic vehicle guidance system can also be implemented as a driver assistance system, ADAS, which supports the driver in semi-automatic or semi-autonomous driving. In particular, the electronic vehicle guidance system can implement a semi-automatic or semi-autonomous driving mode according to levels 1 to 4 of the SAE J3016 classification. Here and in the following, SAE J3016 refers to the corresponding standard from June 2018.

According to the improved concept, a computer system is also provided which is set up to carry out a computer-implemented method for training an artificial neural network to reconstruct the generic state of an environment according to the improved concept. The computing system includes a computing unit and a memory unit. The storage unit stores an artificial neural network designed as a variational autoencoder with an encoder module and a decoder module. The storage medium also stores a corresponding data set representing an environment for each of a plurality of consecutive training frames. The computing unit is set up to generate a first feature representation for each of the training frames by applying the encoder module to the respective training frames of the sensor data set and to generate a reconstructed data set using the decoder module as a function of the first feature representations. The computing unit is set up to train the artificial neural network in an unsupervised manner as a function of the reconstructed data set in order to remove a dynamic object in an input data set that represents the environment.

Further embodiments of the computer system follow directly from the various embodiments of the computer-implemented method for training an artificial neural network according to the improved concept and vice versa. In particular, the computer system can be set up to carry out a computer-implemented method based on the improved concept or it carries out a computer-implemented method based on the improved concept.

According to the improved concept, a sensor arrangement having an environment sensor system and a computing unit is also specified. The sensor arrangement is set up to carry out a method for reconstructing a generic state of an environment of the surroundings sensor system according to the improved concept. In particular, the sensor arrangement can include a computer system according to the improved concept and the surroundings sensor system.

An electronic vehicle guidance system is specified according to the improved concept. The vehicle guidance system includes a computing unit and a memory unit in which a trained artificial neural network is stored, and the vehicle guidance system includes an environment sensor system. The environment sensor system is set up to generate a sensor data set that represents a current environment of the environment sensor system. The trained artificial neural network is designed as a variational autoencoder that includes an encoder module and a decoder module. The processing unit is configured to apply the trained artificial neural network to the sensor data set in order to remove a dynamic object in the sensor data set, wherein a first feature representation is generated by applying the encoder module to the sensor data set and a reconstructed data set by the decoder module as a function of the first Feature representation is generated. The processing unit is set up to plan a trajectory for the motor vehicle on the basis of the reconstructed data set and to generate one or more control signals for at least partially automatically guiding a motor vehicle according to the planned trajectory.

The electronic vehicle guidance system can, for example, include a sensor arrangement based on the improved concept, the sensor arrangement including the computing unit and the environment sensor system of the electronic vehicle guidance system.

According to several embodiments of the vehicle guidance system, the vehicle guidance system comprises one or more actuation units in order to at least partially automatically guide the motor vehicle depending on the one or more control signals according to the trajectory.

Further embodiments of the vehicle guidance system according to the improved concept follow directly from the different embodiments of the method for reconstructing a generic state of an environment according to the improved concept or from the different embodiments of the method for at least partially automatically driving a motor vehicle according to the improved concept and vice versa.

In particular, an electronic vehicle guidance system according to the improved concept can be set up or programmed to carry out a method for reconstructing a generic state of an environment or a method for at least partially automatically guiding a motor vehicle according to the improved concept, or it carries out such a method.

According to the improved concept, a first computer program with first commands is specified. If the first computer program or the first instructions from a computer system, in particular from a Compu tersystem according to the improved concept, the first instructions cause the computer system to execute a computer-implemented method for training an artificial neural network according to the improved concept.

According to the improved concept, a second computer program with second commands is provided. If the second commands or the second computer program are executed by an electronic vehicle guidance system according to the improved concept, the second commands cause the electronic vehicle guidance system to use a computer-implemented method for training an artificial neural network according to the improved concept or a method for reconstructing a generic state of a Environment according to the improved concept or a method for at least partially automatically driving a motor vehicle according to the improved concept.

According to the improved concept, a computer-readable storage medium is also specified, which stores a first computer program and/or a second computer program according to the improved concept.

The first and the second computer program and the computer-readable storage medium according to the improved concept can be referred to as respective computer program products which contain the first and/or the second instructions.

Further features of the invention emerge from the claims, the illustrations and the description of the figures. The features and combinations of features mentioned above in the description and the features and combinations of features mentioned below in the description of the figures and/or shown in the figures alone can be covered by the improved concept not only in the combination specified in each case, but also in other combinations. This covers and discloses embodiments of the improved concept that are not explicitly shown or explained in the figures, but result from separate combinations of features from and can be generated by the implementations explained. Embodiments and combinations of features that do not have all the features of an originally formulated claim can be covered by the improved concept. In addition, embodiments and combinations of features that go beyond or deviate from the combinations of features presented in the relationships of the claims can be covered by the improved concept.

• 1 zeigt eine schematische Darstellung eines Kraftfahrzeugs mit einer beispielhaften Ausführungsform eines elektronischen Fahrzeugführungssystems nach dem verbesserten Konzept;
• 2 zeigt ein schematisches Flussdiagramm einer beispielhaften Ausführungsform eines Verfahrens zur Rekonstruktion eines generischen Zustands einer Umgebung nach dem verbesserten Konzept;
• 3 zeigt schematisch das Ergebnis von Rekonstruktionen anhand exemplarischer Ausführungsformen von Verfahren zur Rekonstruktion eines generischen Zustandes einer Umgebung nach dem verbesserten Konzept;
• 4 zeigt schematisch eine Architektur eines künstlichen neuronalen Netzwerkes, das nach einer beispielhaften Ausführungsform eines computerimplementierten Verfahrens zum Training eines künstlichen neuronalen Netzwerkes nach dem verbesserten Konzept trainiert wird;
• 5 zeigt ein schematisches Flussdiagramm einer beispielhaften Ausführungsform eines computerimplementierten Verfahrens zum Trainieren eines künstlichen neuronalen Netzwerkes nach dem verbesserten Konzept; und
• 6 zeigt ein schematisches Flussdiagramm einer beispielhaften Ausführungsform eines Verfahrens zum zumindest teilweise automatischen Fahren eines Kraftfahrzeugs nach dem verbesserten Konzept.

In the pictures:

• 1 12 is a schematic representation of a motor vehicle having an exemplary embodiment of an electronic vehicle guidance system according to the improved concept;
• 2 12 shows a schematic flow diagram of an exemplary embodiment of a method for reconstructing a generic state of an environment according to the improved concept;
• 3 shows schematically the result of reconstructions based on exemplary embodiments of methods for reconstructing a generic state of an environment according to the improved concept;
• 4 12 schematically shows an architecture of an artificial neural network being trained according to an exemplary embodiment of a computer-implemented method for training an artificial neural network according to the improved concept;
• 5 12 shows a schematic flow diagram of an exemplary embodiment of a computer-implemented method for training an artificial neural network according to the improved concept; and
• 6 shows a schematic flow chart of an exemplary embodiment of a method for at least partially automatic driving of a motor vehicle according to the improved concept.

1 FIG. 1 schematically shows a motor vehicle 1 with an exemplary embodiment of an electronic vehicle guidance system 2 according to the improved concept. The vehicle guidance system 2 includes a computing unit 3 and a memory unit 5 storing a trained artificial neural network 10, as shown in the lower part of FIG 2 is shown schematically. In addition, the vehicle guidance system 2 includes an environment sensor system 4, which can contain one or more cameras and/or lidar systems, which may be mounted at different positions on the vehicle 1.

The vehicle guidance system 2 is set up to carry out a method for at least partially automatically guiding the motor vehicle 1 according to the improved concept. In addition, the vehicle guidance system 2 is set up to use a computer-implemented method for training the artificial neural network 10 perform according to the improved concept. The function of the vehicle guidance system 2 is based on such methods, in particular based on 2 until 6 , explained in more detail.

2 shows a flowchart of an exemplary embodiment of the method for at least partially automatic driving of the motor vehicle 1 and a schematic representation of the trained artificial neural network 10.

The vehicle guidance system 2 can be set up, for example, to guide the motor vehicle 1 at least partially automatically as a function of a trajectory-based function, for example using VSLAM, with a large number of trajectories for different surroundings of the vehicle 1 being planned during a previous training phase and stored in the memory unit 5 have been saved. If the vehicle guidance system 2 determines that the vehicle 1 is again in one of these surroundings, it can select a corresponding one of the stored trajectories and control the motor vehicle 1 accordingly. In order to reliably detect the surroundings of the motor vehicle 1, the 2 procedures shown are carried out.

In step S1 of the method, surroundings sensor system 4 generates a sensor data set 8 that represents the current surroundings of surroundings sensor system 4 or motor vehicle 1 . The sensor data set 8 can, for example, correspond to a camera image, as is shown schematically for two exemplary situations in the left column of FIG 3 is shown. It can be seen that the sensor data set 8 depicts dynamic objects 14, for example cyclists in the top row of 3 or a pedestrian in the bottom row of 3 .

The presence of the dynamic objects 14 may make it difficult or impossible to reliably identify the current environment based on certain predefined key features in the environment. The vehicle guidance system 2 therefore carries out an exemplary embodiment of the method for reconstructing a generic state of an environment of the surroundings sensor system 4 according to the improved concept in step S4 and optionally in the optional steps S2, S3.

In step S4 the processing unit 3 applies the trained artificial neural network 10 to the sensor data set 8 in order to remove the dynamic objects 14 from the sensor data set 8 . For this purpose, an encoder module 6 followed by a decoder module 7 of the trained artificial neural network 10, which is implemented in particular as a variation autoencoder, is applied to the input sensor data set 8 in order to generate a reconstructed data set 9. In this case, the reconstructed data set 9 corresponds to the sensor data set 8 without the dynamic objects 14, as in the right-hand column of FIG 3 shown schematically. To put it more precisely, the encoder module 6 generates an output based on the sensor data set 8 which contains a large number of feature representations which each correspond to different features in the sensor data set 8 or the environment. The decoder module 7 uses the output of the encoder module 6 as input to generate the reconstructed data set 9.

In order to be able to remove the dynamic objects 14 for the sensor data set 8, the artificial neural network 10 was trained in advance, in particular using a computer-implemented method for training an artificial neural network according to the improved concept. Such a procedure is discussed below 4 until 6 explained in more detail.

In step S5, the computing unit 3 plans a trajectory for the motor vehicle 1 on the basis of the reconstructed data set 9 and generates one or more control signals in order to at least partially automatically guide the motor vehicle 1 according to the planned trajectory. One or more actuation units (not shown) of motor vehicle 1 are activated in accordance with the one or more control signals.

In particular, the computing unit 3 can extract one or more key points in the area from the reconstructed data set 9 and compare the extracted key points with previously identified key points in order to be able to reproduce a suitable trajectory. Thanks to the improved concept, the key point extraction in step S5 is based on the reconstructed data set 9 without the dynamic objects 14 and not directly on the sensor data set 8. The dynamic objects 14 therefore do not impede the key point extraction, which leads to more reliable automatic or semi-automatic driving.

For the reconstruction in step S4, the computing unit 3 can select a suitably trained artificial neural network 10 or its encoder module and decoder module 6, 7 from a large number of previously stored inference models. The different inference models each correspond to different poses of the motor vehicle 1 or of the surroundings sensor system 4 and thus to different ones environments. In the optional step S2, the computing unit 3 can therefore use the computing unit 3 to determine a current pose of the motor vehicle 1 or of the surroundings sensor system 4. For this purpose, the computing unit 3 can use sensor outputs from yaw rate sensors, acceleration sensors and so on in order to carry out a corresponding pose estimation, for example based on odometric models.

In the optional step S3, the arithmetic unit 3 can select the most suitable inference model, which is stored on the storage unit 5, depending on the current pose determined in step S2. For example, each inference model stored on the storage unit 5 contains a corresponding pose or the corresponding encoder module 6 and the corresponding decoder module 7.

4 12 shows an exemplary representation of an artificial neural network 10' to be trained using a computer-implemented method according to the improved concept. An exemplary flow chart of such a computer-implemented method is in 5 shown. In steps T1, T1′, T1″, respective poses of the motor vehicle 1 or of the surroundings sensor system 4 are determined at different times or training attempts. In step T2, a first inference model is generated by carrying out a computer-implemented method for training an artificial neural network 10' according to the improved concept for the first training attempt. The same is carried out for a second inference model and a third inference model in steps T2′ and T2″, respectively. In step T3, the inference models determined and the corresponding poses are stored in the storage unit 5 .

For each training attempt, a number of training sensor data sets 8a, 8b are generated by the environment sensor system 4 for consecutive training frames and made available to the processing unit 3. The artificial neural network 10' to be trained is designed as a variation autoencoder which contains an encoder module 6' to be trained and a decoder module 7' to be trained. It can be assumed that the encoder module 6' consists of a large number of encoder sub-modules 6a, 6b, one for each training frame. The respective sensor data set 8a, 8b of each training frame is fed as an input to the corresponding encoder sub-module 6a, 6b. Then the encoder sub-modules 6a, 6b generate a first feature representation 11a, 11b and a large number of further feature representations (not shown) for each of the training frames.

The artificial neural network 10' to be trained may also include a feature prioritization module 12 configured to calculate a respective Euclidean distance between each pair of first feature representations 11a, 11b and a corresponding mean value of the Euclidean distances for the first feature representations 11a, 11b . In the same manner, the feature prioritization module 12 calculates a corresponding mean Euclidean distance for all corresponding other mapped feature representations. In particular, the feature prioritization module 12 calculates pairwise Euclidean distances for each pair of second feature representations mapped by the different encoder modules 6a, 6b, for each pair of third feature representations, and so on. A corresponding average Euclidean distance is determined for each group of mapped feature representations. The feature prioritization module 12 enhances or prioritizes the features corresponding to the respective feature representations according to their respective mean Euclidean distance. In particular, the smaller the average Euclidean distance of a particular set of mapped features, the more the corresponding features are enhanced or prioritized.

In this way, the importance of static objects that are consistent across successive training frames is increased, and objects that are inconsistent or dynamic across successive training frames are given much lower priority in the encoded features in latent space . Dynamic objects are therefore completely removed as soon as the encoder sub-modules 6a, 6b have received enough input over different training epochs to build up confidence in the static objects.

In particular, feature prioritization module 12 may be configured to apply a self-attention algorithm to the output of encoder module 6'. For this purpose, standard techniques such as global average pooling can be applied to the neurons below a certain threshold of average Euclidean distance.

The decoder module 7' is applied to an output of the feature prioritization module and based thereon generates a reconstructed data set 13. The artificial neural network 10' is unsupervised by the computing unit 3 using a standard loss function, for example a cross-entropy loss function, during several consecutive training sessions epochs trained. After the convergence, the resulting trained encoder module 6' and decoder module 7' can be stored in the memory unit 5 together with the corresponding pose of the motor vehicle 1 or the environment sensor system 4. In the same way, the steps described are repeated for the different poses corresponding to the different training attempts in T1, T1', T1''.

6 shows a flow chart of a further embodiment of the method for at least partially automatic driving of the motor vehicle 1 according to the improved concept.

In step R1, the method is triggered manually or based on a position of the motor vehicle 1, which is obtained via a GNSS receiver of the motor vehicle 1, for example. For example, the motor vehicle 1 can automatically enter a parking area when the GNSS receiver identifies the location or a user triggers the application accordingly. In step R2, the training computing unit 3 determines whether or not training should be performed. For example, the computing unit 3 can be prompted manually by a user to perform the training, or automatically based on predefined conditions. If the training is to be carried out, the computing unit 3 and the vehicle guidance system 2 carry out a corresponding method for training the artificial neural network 10' by means of the computing unit 3 according to a computer-implemented method according to the improved concept in step R3, as in relation to FIG 4 respectively 5 described.

If there is no explicit trigger for performing a training, the automatic driving repetition phase is started in step R4. In step R5, the computing unit 3 determines whether a suitable inference model is already stored in the memory unit 5. If this is the case, the computing unit 3 carries out a method for reconstructing a generic environmental state according to the improved concept, or as with regard to FIG 2 respectively 3 described above. A trajectory is then planned and the motor vehicle 1 is driven automatically or partially automatically as described.

A local adaptation can optionally be carried out in step R7 in order to improve the existing inference model. In this case the training is triggered again in step R3. Even if it is determined in step R5 that no suitable inference model is available, training can be proposed in step R8 and carried out in step R3.

As described, in particular with regard to the illustrations, the improved concept enables a more reliable detection of relevant features in the environment of the surroundings sensor system.

The improved concept is particularly useful in the context of VSLAM. With VSLAM, the environment is mapped and the precise physical location of the vehicle is identified when the vehicle is on or near a previously trained trajectory. During VSLAM training, for example, key points or features are extracted from the captured images from multiple cameras and a map of the captured environment is generated at the same time. During playback mode of the VSLAM, when the vehicle is back in the same environment, the current keypoints are compared to those used during training and the actual trajectory is compared to the trained trajectories. With the improved concept it can be achieved that only the identical trajectory as during the training is retained. For this purpose, the environment is reconstructed in such a way that the key points extracted during repetition are essentially identical to those during training.

In this way, current limitations in VSLAM can be overcome. For example, if there is a change in trajectory such as the inclusion or exclusion of new objects or minor changes in constructions during playback mode, traditional VSLAM cannot maintain the trajectory. The ability to capture a camera's location as well as the environment without knowing the data points in advance is very difficult when the environment is compromised due to weather conditions, for example. Also, if the map has too many features in training mode due to the presence of many objects, but has fewer features in playback mode, feature adjustment can be impeded. These problems can be overcome with the help of the improved concept.

Instead of feeding the camera images directly to the appropriate VSLAM module for keypoint extraction, a generic state of the current environment can be reconstructed according to the improved concept. The key points are then extracted from the reconstructed environment. Therefore, the planned Trajectory automatically to a trained trajectory during playback mode. For example, before performing the reconstruction, the pose of the vehicle can be estimated and compared to the respective stored poses at the same locations under the trained inference models. The inference model that corresponds to the closest pose can be selected.

With the help of the improved concept, the current environment of the vehicle can be reconstructed into the trained environment based on its actual pose. Moving or dynamic objects that can cause feature point adjustment problems can be removed. In particular, by using the variational autoencoder architecture, small and moderate changes in the environment do not necessarily require retraining of the inference model.

In some implementations, the training supports multiple trials, where for each trial, for each timestamp or frame, some earlier frames are processed using the variational autoencoder. Therefore, the variation autoencoder can learn temporal consistency. For training, a plurality of encoder sub-modules can receive inputs from different consecutive training frames. Once the corresponding features are separately encoded by individual encoder sub-modules, the Euclidean distance between the individual feature maps of the different frames in latent space can be calculated. In the resulting set of feature representations, self-attention can be applied to the neurons below a certain threshold using a standard technique, e.g. GAP. In this way, the importance of static objects that are consistent across frames is expanded, while all objects that are inconsistent or dynamic are given reduced priority in the encoded features in latent space. When the algorithm runs, the variation autoencoder can align the current scene to a previously created template and eliminate parts that don't match the template. In other words, a template of the scene is built under data control and then the scene is aligned to that template at inference/runtime. In this way, moving objects, different weather conditions or small additional objects placed in the scene can be dealt with.

For example, during the inference or playback phase, only one frame can be viewed at a time and input to the trained encoder. The trained variation autoencoder is able to understand the environment well and makes the approach robust to remove any objects not seen during training. It has also been found that varying weather conditions are generally treated as noise, so weather conditions are not a potentially limiting factor when exercising.

Standard visual odometry can be used to estimate the vehicle's pose in appropriate implementations. It can provide a total of six values, namely translation along the x, y, and z axes and rotation about the x, y, and z axes. During VSLAM training, the pose is saved and during playback, the vehicle's position and pose is recognized. Then an inference model trained at a very similar position and orientation is selected for playback.

Claims (15)

Computer-implemented method for training an artificial neural network (10') to reconstruct a generic state of an environment of an environment sensor system (4), characterized in that - an artificial neural network (10') is provided, the artificial neural network (10 ') is designed as a variation autoencoder, which includes an encoder module (6') and a decoder module (7'); - For each of a plurality of consecutive training frames, a respective sensor data set (8a, 8b), which represents the environment, is obtained from the environment sensor system (4); - A first feature representation (11a, 11b) is generated for each of the training frames by applying the encoder module (6') to the sensor data set (8a, 8b) of the respective training frame; - A reconstructed data set (13) is generated by the decoder module (7') as a function of the first feature representations (11a, 11b); and - the artificial neural network (10') is trained unsupervised depending on the reconstructed data set (13) in order to remove a dynamic object (14) in an input data set representing the environment. Computer-implemented method claim 1 , characterized in that - by applying a feature prioritization module (12) of the artificial neural network (10') to an output of the encoder module (6') for each pair of the first feature representations (11a, 11b) a respective first deviation between the first feature representations ( 11a, 11b) of the respective ligens pair is determined and the first feature representations (11a, 11b) are modified depending on the determined first deviations; - the reconstructed data set (13) is generated by applying the decoder module (7') to an output of the feature prioritization module (12). Computer-implemented method claim 2 , characterized in that - for each of the training frames, a second feature representation is generated by applying the encoder module (6') to the sensor data set (8a, 8b) of the respective training frame; - by applying the feature prioritization module (12) to the output of the encoder module (6'), - for each pair of second feature representations, determining a respective second deviation between the second feature representations of the respective pair; and - the first depicted feature representations (11a, 11b) and/or the second feature representations are modified as a function of the determined first deviations and the determined second deviations. Computer-implemented method claim 3 , characterized in that by applying the feature prioritization module (12) to the output of the encoder module (6'), - features corresponding to the first feature representations (11a, 11b) are prioritized higher than features corresponding to the second feature representations when a mean of the first deviations is greater than a mean of the second deviations; and - the features that correspond to the first feature representations (11a, 11b) are prioritized lower than the features that correspond to the second feature representations if the mean value of the first deviations is smaller than the mean value of the second deviations. Computer-implemented method according to one of claims 3 or 4 , characterized in that by applying the feature prioritization module (12) to the output of the encoder module (6'), a self-attention algorithm depending on the first deviations and the second deviations is applied to the output of the encoder module (6') in order to to modify the first feature representations (11a, 11b). Computer-implemented method according to one of claims 2 until 5 , characterized in that a Euclidean distance between the first feature representations (11a, 11b) of the respective pair is calculated in order to determine the first deviation for the respective pair. Computer-implemented method according to one of the preceding claims, characterized in that - a further artificial neural network is provided, the further artificial neural network being designed as a variation autoencoder which contains a further encoder module and a further decoder module; - A respective further sensor data set is obtained for each of a plurality of subsequent further training frames, which represents a further environment of the surroundings sensor system (4); - For each of the further training frames, a further first feature representation is generated by the further encoder module being applied to the further sensor data set of the respective further training frame; - a further reconstructed data set is generated by the further decoder module as a function of the further first feature representations; and - the further artificial neural network is trained unsupervised as a function of the further reconstructed data set in order to remove a dynamic object (14) in a further input data set which represents the further environment. Computer-implemented method claim 7 , characterized in that - a pose of the environment sensor system (4) which corresponds to the environment is obtained and a further pose of the environment sensor system (4) which corresponds to the further environment is obtained; - after training the artificial neural network (10'), an inference model is generated which includes the pose, the encoder module (6') and the decoder module (7'); and - after the further artificial neural network (10') has been trained, a further inference model is generated which contains the further pose, the further encoder module (6') and the further decoder module (7'). Method for reconstructing a generic state of an environment of an environment sensor system (4), characterized in that - a sensor data set (8), which represents a current environment of the environment sensor system (4), is generated by the environment sensor system (4); - a trained artificial neural network (10) is applied to the sensor data set (8) in order to remove a dynamic object (14) in the sensor data set (8), the trained artificial neural network (10) being designed as a variation autoencoder , which includes an encoder module (6) and a decoder module (7); and - in order to remove the dynamic object (14) in the sensor data set (8), a first feature representation (11a, 11b) by using the encoder module (6) is generated on the sensor data record (8) and a reconstructed data record (9) is generated by the decoder module (7) depending on the first feature representation (11a, 11b). procedure after claim 9 , characterized in that a computer-implemented method according to one of Claims 1 until 8th is performed to generate the trained artificial neural network (10). procedure after claim 9 , characterized in that - for training the artificial neural network (10) according to a computer-implemented method claim 8 is carried out; and - the encoder module (6) and the decoder module (7) of the trained artificial neural network (10) the encoder module (6) and the decoder module (7) of the inference model or the further encoder module (6) and the further decoder module (7) further match inference model. procedure after claim 11 , characterized in that - a current pose of the environment sensor system (4) is determined; - the current pose is compared with the pose of the inference model and with the further pose of the further inference model; and - depending on the respective results of the comparisons, the encoder module (6) and the decoder module (7) of the inference model or the further encoder module (6) and the further decoder module of the further inference model as encoder module (6) and decoder module (7) of the trained artificial neural network (10) to be selected. Method for at least partially automatic driving of a motor vehicle (1), characterized in that - a method according to one of claims 9 until 12 is carried out, in which the environment sensor system (1) is mounted on the motor vehicle (1); - A trajectory for the motor vehicle (1) is planned on the basis of the reconstructed data set (9); and - the motor vehicle (1) is driven at least partially automatically according to the trajectory. Electronic vehicle guidance system containing a computing unit (3), a memory unit (5) that stores a trained artificial neural network (10), and an environment sensor system (4) that is set up to generate a sensor data set (8) that represents a current environment of the surroundings sensor system (4), characterized in that - the trained artificial neural network (10) is designed as a variation autoencoder which contains an encoder module (6) and a decoder module (7); and - the computing unit (3) is set up to - apply the trained artificial neural network (10) to the sensor data set (8) in order to remove a dynamic object (14) in the sensor data set (8), wherein a first feature representation (11a , 11b) is generated by applying the encoder module (6) to the sensor data set (8) and a reconstructed data set (9) is generated by the decoder module (7) depending on the first feature representation (11a, 11b); - to plan a trajectory for the motor vehicle (1) based on the reconstructed data set (9); and - at least partially automatically generate one or more control signals for driving a motor vehicle (1) according to the trajectory. Computer program product comprising - first instructions which, when executed by a computer system, cause the computer system to perform a computer-implemented method according to any one of Claims 1 until 8th to perform; and/or - second commands which, when sent by an electronic vehicle guidance system (2). Claim 14 are executed, cause the electronic vehicle guidance system (2), a computer-implemented method according to one of Claims 1 until 8th or a method according to any of claims 9 until 13 to execute.

Images