DE102021200643B3 - Method for environment recognition for semi-autonomous or autonomous driving functions of a motor vehicle using a neural network - Google Patents

Abstract

In order to provide a method for environment recognition using a neural network for semi-autonomous or autonomous driving functions of a motor vehicle, sensor data (11) being supplied to at least one neural network (13) implemented in a data processing device (10) of a motor vehicle (200), the at least one neural Network (13) generates output data (14) from the sensor data (11), which includes information about the surroundings (15) of the motor vehicle (200), the at least one neural network (13) being a robust, compressed neural network (17). , it is proposed in this method (100) that the robustified, compressed neural network (17) is obtained by using a compression method from a baseline model (19) trained for a method for environment recognition and designed as a deep neural network (18) a compressed student network (23) is created that uses a robustification method from the Bas eline model (19) a robustified teacher network (22) is created, and that the compressed student network (23) is trained in a teacher-student training method (24) using the robustified teacher network (22).

Description

The present invention relates to a method for detecting the surroundings for semi-autonomous or autonomous driving functions of a motor vehicle, with sensor data being supplied to at least one neural network implemented in a data processing device of a motor vehicle, with the at least one neural network generating output data from the sensor data which contains information about the surroundings of the motor vehicle include.

Furthermore, the present invention relates to a motor vehicle comprising a data processing device designed to carry out a method for detecting the surroundings for semi-autonomous or autonomous driving functions.

Modern driver assistance systems that perform semi-autonomous or autonomous driving functions are mostly based on machine learning methods to evaluate the vehicle environment. Using the machine learning process, objects such as pedestrians or vehicles can be recognized and differentiated. Many machine learning methods are based on the training of so-called deep neural networks. Trained deep neural networks can process sensor data, such as from cameras, radar or lidar devices, to generate an environment model of the vehicle.

Known functions performed by neural networks are semantic segmentation or object recognition. In semantic segmentation, the neural network generates a pixel-by-pixel labeling of the input images according to a predefined set of class labels. Object detection neural networks generate bounding boxes around detected objects corresponding to defined target classes. In the two cases described above, the neural networks are usually in the form of so-called convolutional neural networks (CNN).

During the training phase of a neural network, a large number of parameters of the network are adjusted. For this purpose, training data provided with a basic truth are supplied to the neural network. The parameters of the network can be modified with the help of a gradient-based optimization method. The aim of the optimization method is to minimize a loss function that quantifies the deviation of the network output from the basic truth. Deep neural networks often consist of a large number of so-called layers or filters and therefore have a very large number of parameters to be trained.

At the time of inference (output of predictions), a large number of arithmetic operations are necessary for the calculation of the output of the neural network. In an in-vehicle implementation of a network, however, only limited computing resources are available, which are dictated by the embedded hardware. The limitation of the computing resources places high demands on the in-vehicle application of neural networks and limits the available runtime for the calculation of the output, since the hardware used cannot be scaled arbitrarily.

There are several approaches to reduce the computational effort of neural networks. These approaches include what are known as cleansing or pruning methods, which aim to reduce the overall size of a trained network by removing filters and thus parameters associated with the filters. Depending on the filters selected for removal, the performance and accuracy of the mesh may change. Cleansing or pruning strategies are therefore aimed at removing those filters that have the least impact on the output of the network and also allow the greatest possible reduction in network size.

In addition to reducing the network size, the robustness of the network to so-called out-of-domain data is also an important criterion. While deep neural networks typically perform well when the input data is similar to the training data, certain new data can prove challenging for a trained deep neural network. In particular, it has been shown that deep neural networks are vulnerable to certain variations in the input data, including so-called augmentations. In the area of image data-based environment recognition for autonomous driving functions, such variations or augmentations can be caused, for example, by environmental influences such as fog or snowfall. Furthermore, motion blurring can occur in the image data at high driving speeds.

The output of a neural network should also be robust against so-called "adversarial attacks". An adversarial attack is based on the use of specially modified input data, so-called adversarial examples, which are suitable for manipulating the classification and localization results and intentionally mislead a neural network into making a false prediction.

the CN 111461226A discloses a teacher-student training method for a neural network to robustify the neural network against adversarial attack data.

the CN 110880036A discloses a neural network compression method using a teacher-pupil training method to reduce network size.

From the U.S. 2019/0244103 A1 a method is known to robustify a compressed neural network. The neural network is first trained using adversarial examples. The mesh is then reduced using a pruning process. In a further step, a new adversarial attack training takes place.

From the DE 10 2019 119 087 A1 a method for training a neural network is known, with components being provided in a first step which represent typical input variables in relation to an application of the neural network. The neural network is then trained at least with the components provided. In addition, feature distributions for the components can be determined and parameters for conclusions can be learned.

The publication "Relational Knowledge Distillation", Wonpyo Park et al., 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 01/09/2020, 3962-3971, discloses a method for transferring knowledge between neural networks, which is described in was acquired in a teacher model and is to be transferred to a student model. The student model is usually smaller than the teacher model. In the method, mutual relationships of data samples are transmitted, rather than the student model imitating output activations of individual data samples given by the teacher model.

The known methods are either designed to compress a neural network or else to robustify the neural network. Insofar as both compression and robustification are dealt with in the prior art, the corresponding compression and robustification methods are applied in succession. However, neural networks resulting from this sequential application are often less robust than neural networks that have not been previously compressed. The sequential execution of the robustification and compression methods thus leads to a disadvantageous prediction quality of the neural networks.

The present invention is based on the object of providing a method for environment recognition for semi-autonomous or autonomous driving functions of a motor vehicle, with sensor data being fed to at least one neural network implemented in a data processing device of a motor vehicle, with the at least one neural network generating output data from the sensor data, which information about an environment of the motor vehicle, wherein the at least one neural network is a robustified, compressed neural network, which has improved robustness compared to known sequentially robustified and compressed neural networks.

In order to achieve the object on which the invention is based, a method for environment recognition for semi-autonomous or autonomous driving functions of a motor vehicle is proposed, with sensor data being fed to at least one neural network implemented in a data processing device of a motor vehicle, with the at least one neural network generating output data from the sensor data, which information about an environment of the motor vehicle, wherein the at least one neural network is a robustified, compressed neural network, wherein the robustified, compressed neural network is obtained by using a compression method from a trained for a method for environment recognition baseline designed as a deep neural network model, a compressed student network is created and a robustified teacher network is created from the baseline model by means of a robustification method, with the compressed student network is trained in a teacher-student training method using the robustified teacher network, the compressed student network being retrained in the teacher-student training method.

The method can be used for partially autonomous or autonomous driving functions of a motor vehicle. The semi-autonomous driving functions can be functions with low levels of autonomy. These can relate, for example, to adaptive cruise control or automatic parking or lane-keeping systems. However, the method for detecting the surroundings can also be used for highly or fully autonomous driving functions in which the motor vehicle performs driving tasks with only little human intervention or completely without human intervention.

The motor vehicle can be a motor vehicle with an internal combustion engine, a hybrid electric vehicle or a battery electric vehicle.

The sensor data are preferably recorded by means of sensors, in particular environmental sensors, of the motor vehicle. The sensors can be camera devices, radar or lidar devices directions and other sensors suitable for detecting the environment.

Provision is preferably made for the output data to be further processed in a system for semi-autonomous or autonomous driving, for example in a driver assistance system.

At the inference time (output of predictions), i.e. at the time the method is carried out in a motor vehicle, the data processing device, for example a computer, of the motor vehicle receives the sensor data from the sensors and feeds them as input data to the at least one neural network, which calculates the output data . The sensor or input data can first go through several pre-processing steps before they are passed on to the neural network.

The at least one neural network is embodied as a robustified and compressed neural network which is obtained beforehand, ie before the inference time, in a previous method step from a baseline model embodied as a deep neural network.

For this purpose, the baseline model is first trained using training data for environment recognition. A compressed student net is then created from the baseline model using a compression method. At the same time, a robust teacher net is created from the uncompressed baseline model using a robust method.

The compressed student network is then trained in a teacher-student training method using the robustified teacher network. In this way, the robustness properties of the teacher's network are transferred to the student's network. The compressed student network trained using the robustified teacher network is then used as a neural network in the method according to the invention.

The robust, compressed neural network obtained in this way is preferably optimized with regard to the computing time requirement and/or the memory requirement and also has the robustness properties of the non-compressed teacher network.

What is essential in the method according to the invention is that the steps of compression and robustification are not carried out sequentially on the baseline model. Rather, two optimized versions of the same baseline model, namely the compressed student network and the robustified teacher network, are used to obtain the robustified compressed neural network.

According to the invention, the compressed student network is retrained in the teacher-student training method.

In other words, the parameters of the student network are not reinitialized in the teacher-student training method, but are preferably modified in a gradient-based optimization method during the teacher-student training method. There is therefore no complete retraining of the compressed student network.

For the teacher-student training method, training data is fed to both the compressed student network and the robustified teacher network. The output of the robustified teacher network is compared to the output of the student network as ground truth. The deviation of the output of the student network from the ground truth of the teacher network is quantified in a loss function. The parameters of the student network are then adjusted using known learning methods in order to minimize the loss function.

With a further advantage it can be provided that the sensor data include or are image data.

The image data are preferably recorded with an image sensor or a camera device of the motor vehicle. Such image data are particularly advantageous for assessing traffic situations. However, the sensor data can also be in any other suitable form. For example, it is possible that the sensor data includes or is radar or lidar data or V2x data.

Provision is preferably made for the at least one neural network and/or the baseline model to be a convolutional neural network (CNN).

Convolutional neural nets are particularly suitable for processing image data.

Provision is preferably made for the at least one neural network to carry out object detection or image segmentation.

With a further advantage, it can be provided that the robustification method is an “adversarial attack” training method, or a method for augmentation robustification, or a stability training method.

As part of an Adversarial Attack training process, so-called “Adversarial Examples” made available to a neural network. Adversarial examples are specially manipulated input data in an artificial neural network, which intentionally misleads this into misclassification.

For augmentation robustness, the neural network is preferably trained with image corruptions (brightness, noise, etc.), which are preferably real-world corruption and occur in the area of application of the neural networks.

In a stability training procedure, the input data is subjected to data augmentation. Using a stability training procedure, the robustness against out-of-domain data is increased.

In addition, depending on the objective of the semi-autonomous or autonomous driving function, any suitable robustification method can be used.

The robustification method is preferably designed in such a way that the calculation of the output data of the at least one neural network is robust to sensor noise or sensor data influenced by environmental conditions, the environmental conditions preferably being snow and/or fog and/or changes in brightness and/or blurred motion and/or rain.

At the inference or production time, i.e. when carrying out the environment recognition for semi-autonomous and autonomous driving functions, the output of the neural network is therefore insensitive to deviations from the original training data.

With a further advantage, it can be provided that training data provided with different data augmentations are used for the robustification method, with the data augmentations preferably relating to snow and/or fog and/or changes in brightness and/or motion blur.

With a further advantage it can be provided that the compression method is a pruning method and/or a quantization method.

The pruning process can be bottom-up or top-down pruning. Within the framework of the pruning method, the parameters of the baseline model are preferably classified with regard to their importance for the output data, and those parameters which have the least influence on the output data are removed. If the baseline model is a convolutional neural net, those filters that have the least influence on the output data can preferably be removed.

In a quantization process, the data types of the parameters are converted into data types that require less memory. For example, parameters stored as floating point numbers can be converted to integers.

In principle, any suitable compression method can be used, depending on the objective of the compression. Compression methods can be used with which the memory requirement of the neural network is reduced, or compression methods can be used with which the computing time for the inference or production time is reduced.

It is therefore preferably provided that the compression method is designed to reduce the memory requirements of the neural network and/or to reduce the computing time for the inference time.

It is particularly advantageous if sensor data is fed to a number of neural networks, with the number of neural networks being robustified, compressed neural networks, with the neural networks having been created using various combinations of robustification methods and compression methods.

For example, different student networks can be created from the baseline model by using different compression methods. Likewise, different teacher networks can be created by using different robustification methods. The teacher-student training method can then be performed with different combinations of the compressed student networks and the robustified teacher networks, so that several different robustified compressed neural networks are available. Each of these robustified, compressed neural networks can be optimized for a specific task.

For example, one of the robust, compressed neural networks can be designed to recognize traffic signs. Since the computing time is less critical for this application, the robust, compressed neural network can be optimized for the required memory requirements. On the other hand, another robustified neural network for person identification can be optimized in terms of computing time.

When carrying out the method, at least one of the several neuro nal networks depending on the environmental conditions.

Provision is also preferably made for the loss function of the student network to contain an additional regularization term during the teacher-student training method, with the additional regularization term enabling the parameters of the student network to be regularized with respect to the parameters of the teacher network.

The regularization term can be an L1 or L2 regularization. Other regularization methods such as "dropout" methods are also possible.

The student network loss function can also directly account for the activations of the teacher network filters. In addition, further information from the teacher network can be used by adapting the loss function of the student network. For example, feature maps of the teacher network can be used.

A further solution to the problem on which the invention is based is a motor vehicle, comprising a data processing device which is designed to carry out an above-described method for environment recognition for semi-autonomous or autonomous driving functions of the motor vehicle using an above-described, robust, compressed neural network, the robust, compressed neural network generates output data from sensor data, which include information about the surroundings of the motor vehicle.

• 1 ein Ablaufdiagramm für ein Verfahren zur Umfelderkennung für teilautonome oder autonome Fahrfunktionen eines Kraftfahrzeugs,
• 2 ein Kraftfahrzeug mit einer Datenverarbeitungseinrichtung zur Durchführung eines Verfahrens zur Umfelderkennung, und
• 3 ein Ablaufdiagramm für die Erstellung eines robustifizierten, komprimierten neuronalen Netzes.

The invention is explained in more detail with reference to the accompanying figures. Show it:

• 1 a flowchart for a method for detecting the environment for semi-autonomous or autonomous driving functions of a motor vehicle,
• 2 a motor vehicle with a data processing device for carrying out a method for detecting the surroundings, and
• 3 a flow chart for creating a robustified, compressed neural network.

1 shows a flow chart for a method 100 for environment recognition for semi-autonomous or autonomous driving functions of a motor vehicle 200. 2 1 shows a motor vehicle 200 with a data processing device 10 for carrying out the method 100 1 .

In method 100, sensor data 11 from at least one sensor 12 of motor vehicle 200 is supplied to data processing device 10 of motor vehicle 200. The data processing device 10 runs a neural network 13 which receives the sensor data 11 and generates output data 14 from the sensor data 11 . The output data 14 contain information about the surroundings 15 of the motor vehicle 200. The output data 14 can be fed to a driver assistance system 16, which carries out semi-autonomous or autonomous driving functions.

The neural network 13 is a compressed, robustified neural network 17, which with the in 3 illustrated process steps is created. First, a baseline model 19 embodied as a deep neural network 18 is trained using training data for environment recognition. The baseline model 19 is designed as a convolutional neural net 20. In a next method step, quasi two copies 21a, 21b are generated from the baseline model 19 . The first copy 21a of the baseline model 19 is then robustified using a robustification method, so that a robustified teacher network 22 is obtained. At the same time, a compressed student network 23 is created from the second copy 21b using a compression method.

Suitable robustification methods are, for example, so-called adversarial attack training methods or stability training methods. The baseline model 19 is supplied with training data provided with data augmentations and is trained to be insensitive to the augmentations. Such data augmentations can, for example, correspond to environmental influences such as snow, fog, changes in brightness or blurred motion. Known compression methods such as pruning or quantization can be used to compress the baseline model 19 .

The compressed student network 23 is retrained using the robustified teacher network 22 in a teacher-student training method 24 . The student network 23 and the teacher network 22 are supplied with the same training data. The output and, if necessary, the activation of the layers or filters of the teacher network 22 are then compared as base truth with the output data of the student network 23 by means of a student network 23 loss function. By adjusting the parameters of the student network 23, the loss function is minimized. The robustness properties of the teacher network 22 are transferred to the student network 23 by the teacher-student training method 24 . The student network 23 robustified in this way is then a robustified, compressed neural network 17, which as a neural network 13 in 1 Method 100 shown can be used for environment recognition.

Reference List

100

procedure

200

motor vehicle

10

data processing device

11

sensor data

12

sensor

13

neural network

14

output data

15

environment

16

driver assistance system

17

Robustized compressed neural network

18

Deep Neural Network

19

baseline model

20

Convolutional Neural Net

21a

First copy

21b

Second copy

22

teacher network

23

student network

24

Teacher-student training process

Claims (10)

Method (100) for environment recognition for semi-autonomous or autonomous driving functions of a motor vehicle (200), with sensor data (11) being supplied to at least one neural network (13) implemented in a data processing device (10) of a motor vehicle (200), the at least one neural network (13) generates output data (14) from the sensor data (11), which includes information about an environment (15) of the motor vehicle (200), the at least one neural network (13) being a robustified, compressed neural network (17), characterized in that the robust, compressed neural network (17) is obtained by using a compression method to create a compressed student network (23 ) is created and a robustified teacher network (22) is first created from the baseline model (19) by means of a robustification method Is ellt, wherein the compressed student network (23) is trained in a teacher-student training method (24) using the robustified teacher network (22), the compressed student network (23) in the teacher-student training method (24) is retrained. Method (100) according to claim 1 , wherein the sensor data (11) includes or is image data, and/or wherein the at least one neural network (13) and/or the baseline model (19) is a convolutional neural network (20), and/or wherein the at least one neural network (13) performs object detection or image segmentation. Method (100) according to claim 1 or 2 , wherein the robustification method is an adversarial attack training method, or an augmentation robustification method, or a stability training method. Method (100) according to one of the preceding claims, wherein the robustification method is designed such that the calculation of the output data (14) of the at least one neural network (13) is robust to sensor noise or sensor data (11) influenced by environmental conditions, the environmental conditions preferably being snow , and/or fog and/or changes in brightness and/or motion blur and/or rain. Method (100) according to one of the preceding claims, wherein the compression method is a pruning method and/or a quantization method. Method (100) according to one of the preceding claims, wherein the compression method is designed to reduce the memory requirements of the neural network (13) and/or to reduce the computing time for the inference time. Method (100) according to one of the preceding claims, wherein a plurality of neural networks (13) sensor data (11) are supplied, wherein the plurality of neural networks (13) are robustified, compressed neural networks (17), wherein the neural networks (13) by different combinations of robustification methods and compression methods have been created. Method (100) according to claim 7 , wherein at least one of the plurality of neural networks (13) is selected depending on the environmental conditions when carrying out the method. Method (100) according to one of the preceding claims, in which the loss function of the student network (23) contains an additional regularization term, the additional regularization term being a regularization of the parameters of the student network (23) with respect to the parameters of the teacher network ( 22) enabled. Motor vehicle (200) comprising a data processing device (10) designed to carry out a method according to one of the preceding claims for environment recognition for partially autonomous or autonomous driving functions of the motor vehicle by means of a robust, compressed neural network.

Images