DE102020211280A1 - Computer-implemented method for environment recognition for an automated driving system, machine learning method, control unit for an automated driving system and computer program for such a control unit - Google Patents

Abstract

Computer-implemented method for environment detection for an automated driving system (AD) comprising the steps of entering data from at least one sensor (S1, S2, S3) for environment detection of the driving system (AD) into a first artificial neural network (MEB, MFB), which is trained, determining first features from the data and obtaining the first features (V1), inputting and processing the first features in respective second artificial neural networks (Head1 - HeadN), the second artificial neural networks (Head1 - HeadN) each being trained for one of various detection types of automated driving to obtain classification, localization and/or prediction results (V2) and obtaining the environment detection based on the results of the second artificial neural networks (Head1 - HeadN), based on the environment detection rule and/or Control signals for the driving system (AD) are determined (V3).

Description

The invention relates to a computer-implemented method for environment recognition for an automated driving system, a machine learning method, a control unit for an automated driving system and a computer program for such a control unit.

Automated driving systems, such as autonomous vehicles, require a number of sensors to recognize their surroundings and to find their way around in their environment. An important sensor for a self-driving vehicle is the camera. It is used, for example, to recognize lanes, vehicles and obstacles and incorporate them into the control of the autonomous vehicle.

The recognition of the environment and surrounding objects is currently being carried out successfully with artificial neural networks. Each image captured by the camera is processed with a neural network and analyzed using a method developed by the programmer. This method can, for example, recognize other road users, display the course of the road or recognize lane markings. Each recognition process has a different output here, that is, it perceives the elements of the environment visible in the image with different processes and different goals.

In order to perceive countable objects such as cars, people and bicycles, methods are used that display the objects using rectangular 2D or 3D boxes as overlays in the recorded video image. Road courses and lane markings can be represented with lines, for example in the form of image coordinates. Countless objects such as bushes or sky are represented as free areas by means of semantic segmentation, i.e. here the correspondingly recognized image areas are provided with a class identifier or class label for each pixel, such as sky. All environmental objects relevant to autonomous driving require individual recognition and representation in order to make them usable for subsequent driving processing steps.

The different algorithms used for the different detections of the different types/classes of road traffic objects process the data from sensors such as cameras or other sensors such as lidar, radar, acoustic sensors, ultrasonic sensors, olfactory sensors all at the same time and in the Rule with the same frame rate. It is imperative that they run parallel to one another, since, for example, lane markings and other road users must be recognized at the same time, but cannot be represented in the same way.

This applies both to assistance systems that relieve or simplify driving functions for the driver, known as advanced driver assistance systems, abbreviated ADAS, for example SAE Level 2+, as well as to highly automated driving, in which computers carry out vehicle control almost or completely automatically autonomous driving, abbreviated AD, for example SAE level 4 or 5.

Neural networks with so-called convolutional kernels, called convolutional neural network, abbreviated CNN, are currently used in image recognition. In many algorithms used, a CNN takes on the task of recognizing the different object classes. Here, the image is used as a matrix with color values and filters with values - learned by the neural network - process the image into a transformed image. This transformation is repeated many times in current neural networks together with other image processing steps, so that a CNN with up to hundreds of layers can be created. At the end of the network, these take on the form of the output, for example the coordinates of a rectangular box if road users are detected.

The layers of the CNN are created by the repeated transformation using different filters. The layers appear in all CNNs and are often very similar technically. This applies to both camera-based image recognition and those based on other sensors. For this reason, CNNs consist of two parts, the so-called backbone, which is an arrangement of layers on top of the image/sensor input, and the head, called head, which processes the last layer of the backbone into the desired output. The backbone contains most of the layers of the network and accordingly takes the most time to calculate the values mathematically.

Many different CNNs are required for complex systems in the field of ADAS or AD, with each individual network already making very high calculation demands. The safety-related requirement to ensure the shortest possible reaction times from autonomous systems, for example less than 10ms, also contributes to the calculation effort. Running the CNNs in parallel incurs high costs, both for powerful hardware and for power consumption. For this reason, the reduction of Computing capacities are highly relevant for a mature system.

Backbone-Head architectures are known, for example, from https://arxiv.org/pdf/1703.06870.pdf. There, a backbone with two closely related types of detection, namely box and mask for an object, is disclosed. The segmentation is a binary instance segmentation.

Other backbone head architectures are disclosed, for example, in https://arxiv.org/pdf/1904.01169.pdf and https://arxiv.org/pdf/1909.03625.pdf. However, there is not a large number of target detection types here.

The object of the invention was to provide a holistic detection system for the AD/ADAS area, which merges the tasks of multiple networks and can be used to predict a large number of target detection types with a single backbone.

The subjects of claims 1, 7, 9 and 10 each solve this problem with a single backbone. This eliminates the majority of arithmetic operations and thus latency times.

• • Eingeben von Daten zumindest eines Sensors zur Umfelderkennung des Fahrsystems in ein erstes künstliches neuronales Netzwerk, das trainiert ist, aus den Daten erste Merkmale zu bestimmen und Erhalten der ersten Merkmale,
• • Eingeben und Verarbeiten der ersten Merkmale in jeweiligen zweiten künstlichen neuronalen Netzwerke, wobei die zweiten künstlichen neuronalen Netzwerke jeweils trainiert sind, für eine von verschiedenen Erkennungsarten des automatisierten Fahrens Klassifikations-, Lokalisierungs- und/oder Prädiktionsergebnisse zu erhalten und
• • Erhalten der Umfelderkennung basierend auf den Ergebnissen der zweiten künstlichen neuronalen Netzwerke, wobei basierend auf der Umfelderkennung Regel- und/oder Steuersignale für das Fahrsystem bestimmt werden

One aspect of the invention relates to a computer-implemented method for environment recognition for an automated driving system. The procedure includes the steps

• • Entering data from at least one sensor for detecting the surroundings of the driving system in a first artificial neural network that is trained to determine first features from the data and to obtain the first features,
• • Input and processing of the first features in respective second artificial neural networks, wherein the second artificial neural networks are each trained to obtain classification, localization and/or prediction results for one of different detection types of automated driving and
• • Obtaining the environment detection based on the results of the second artificial neural networks, with regulation and/or control signals for the driving system being determined based on the environment detection

• • Eingeben eines Datenbeispiels in jeweilige dritte künstliche neuronale Netzwerke, die jeweils trainiert sind, für die jeweilige Erkennungsart spezifische Kennzeichen zu erhalten,
• • Kennzeichnen des Datenbeispiels mit den Kennzeichen und
• • Vorwärtsspeisen der Netzwerkarchitektur mit dem gekennzeichneten Datenbeispiel und Gradienten basiertes Erlernen der zweiten künstlichen neuronalen Netzwerke.

Another aspect of the invention relates to a machine learning method for a network architecture. The network architecture includes a first artificial neural network for determining first features from data from at least one sensor for detecting the surroundings of an automated driving system. Furthermore, the network architecture includes second artificial neural networks for classification, localization and/or prediction for different detection types of automated driving based on the first features. The procedure includes the steps

• • Entering a data sample into respective third artificial neural networks, each of which is trained to receive specific identifiers for the respective type of recognition,
• • Tagging the data sample with the and tags
• • Feed forward the network architecture with the labeled data example and gradient based learning of the second artificial neural networks.

In one aspect, the steps of entering the data example and tagging are repeated until a fully tagged dataset is obtained. The network architecture is then fed forward with the tagged data.

A further aspect of the invention relates to a control unit for an automated driving system. The control unit includes at least one first interface, via which the control unit receives data from at least one sensor for detecting the surroundings of the driving system. Furthermore, the control unit includes a processing unit. The processing unit is designed to implement a network architecture comprising a first artificial neural network for determining first features from the data and second artificial neural networks for classification, localization and/or prediction for different types of recognition of automated driving based on the first features and an environment recognition and To determine control and / or control signals according to a method according to the invention. In addition, the control unit includes second interfaces via which the control unit provides the control and/or control signals to actuators for longitudinal and/or lateral guidance of the driving system.

A further aspect of the invention relates to a computer program for a control unit according to the invention. The computer program includes instructions that cause the control unit to execute a method according to the invention and/or a machine learning method according to the invention when the computer program runs on the control unit.

Advantageous refinements of the invention result from the definitions, the dependent claims, the drawings and the description of preferred exemplary embodiments.

Computer-implemented means that the steps of the method are executed by a data processing device, for example a computer, a computing system, a computer network, for example a cloud system, hardware of a control device, or parts thereof.

Automated driving systems include automated vehicles, road vehicles, people movers, robots and drones.

Sensors for detecting the surroundings include optical sensors such as cameras, including infrared cameras, and lidar, radar sensors, acoustic sensors such as microphones, ultrasonic sensors and olfactory sensors such as electronic noses. The sensors can be arranged, for example, on an outer skin and/or inner skin of the automated driving system. The data from the sensors includes raw data and pre-processed data, for example filtered raw data with an improved signal-to-noise ratio.

The first artificial neural network has the function of a backbone network. According to the invention, the first artificial neural network is a backbone network that predicts a plurality of target recognition types, in particular target recognition types that are not closely related to one another. In this context, the first artificial neural network is called the multi-detection backbone, abbreviated MEB. The backbone network includes layers of a convolutional network. A DLA-34 network, see https://arxiv.org/pdf/1707.06484.pdf, ResNet, see https://arxiv.org/pdf/1512.03385.pdf, Inception, see https:/ /arxiv.org/pdf/1409.4842.pdf or MobileNets, see https://arxiv.org/pdf/1704.04861.pdf. The first layers of the backbone network extract general characteristics from the data, for example from image data, which are used for a large number of other tasks. The first features include general features that are used as input for the different types of recognition.

The second artificial neural networks have the function of detection heads, i.e. they correspond to the heads in a backbone head architecture. The invention proposes the use of multiple detection heads, for example four detection heads. Each detection head represents an output of the backbone. The detection heads share the backbone network. For example, in a pre-trained backbone network, the last layers of the backbone network are replaced by the detection heads. Each recognition head is post-trained for a specific target task, for example a specific recognition type. According to the invention, a number of detection heads are used with a single backbone, with the detection heads being trained/post-trained together.

The second artificial neural networks include layers of a convolutional network, recurrent layers, or fully connected layers, for example.

This proposed architecture has several advantages over the classical multiple mesh approach. In addition to the runtime reduction - and/or the much more efficient use of the processing hardware - a system with several outputs that are related in terms of content is more robust and precise. The inclusion of all sensors and representations of the different but related task types in the automotive context also leads to a safer overall system. In contrast to existing algorithms with multiple outputs, the proposed system does not only represent an in-depth object detection, but can cover several or all necessary detection types of autonomous driving and can be individually adapted to this problem. These types of detection are used in the higher control levels of such an autonomous system, including planning, trajectory prediction, simultaneous localization and mapping, and combine to form the final actuator control.

In the automated driving domain, the MEB anticipates mutually unfamiliar and generally all detection types, such as weather from raindrops in the roadscape and lane markings on asphalt.

The data, in particular the images from a camera that the first artificial neural network receives to determine the first characteristics, show the traffic on the road. Therefore, knowledge about objects and the different types of recognition in the first artificial neural network can only be learned together in order to obtain generally valid features for all types of recognition. Otherwise, the first artificial neural network would already make a distinction based on the different nature of the image domain. However, a backbone network with several heads would not learn with this. It is problematic that an identified or labeled data set in the form of image-line pairs for training the network architecture according to the invention does not generally exist. For example, a data set for training a weather forecast includes a large one Number of image-target pairs in the form (image 0, target 0 weather), (image_1, target_1_weather), (image_2, target 2 weather), etc. However, the MEB should simultaneously recognize weather and lane markings, for example. For such training, a data set of the form (image 0, target 0 weather, target 0 lane marker), (image_1, target 1_weather, target_1_lane marker), (image_2, target_2_weather, target_2_lane marker), etc. would be required. With more than two types of recognition, such a data set would become even more extensive. The entire data set would also have to be adjusted manually for the addition of a sensor or a type of detection.

However, if a labeled data record exists for training the network architecture according to the invention, the network architecture according to the invention is trained directly with the correct labels. It is then not necessary to generate pseudo labels.

The machine learning method according to the invention enables the MEB to be trained with multiple heads. Machine learning is a technology that teaches computers and other data processing devices to perform tasks by learning from data, rather than being programmed to do the tasks. Gradient-based has the usual meaning that training data pairs comprising tagged data are fed forward through an artificial neural network and in a back-feed a cost function of the network is minimized by gradient formation of the network's weights.

According to the solution according to the invention, third artificial neural networks are used, which are trained for individual special tasks, for example the different types of recognition. In this context, the third artificial neural networks are called specialist networks. Publicly accessible data sets, with which the specialist networks are trained, are available for such special tasks.

The specific labels generated by the third artificial neural networks are so-called pseudo labels. The pseudo labels are predictions of the third artificial neural networks on a dataset. The first artificial neural network is trained with these pseudo labels. This corresponds to a so-called transer learning or distillation of a network. According to one aspect of the invention, the prediction with the highest probability is used as the pseudo label. According to a further aspect of the invention, the entire probability distribution from the prediction of the teaching network, i.e. the third artificial neural networks, is given to the learning network, i.e. the first artificial neural network with the second artificial neural networks, which improves the training.

Because each data example, or also called data sample, is marked with respective pseudo labels for all types of recognition, the third artificial neural networks generate all recognition targets in the same data example, for example in the same image. This enables training of the first artificial neural network with multiple recognition heads. The invention thus makes it possible to generate pseudo labels for any number of sensor data without a label. By bundling the knowledge of individual specialist networks on new sensor data, fully identified complex data sets are made available for training the network architecture according to the invention.

The third artificial neural networks include layers of a convolutional network, recurrent layers, or fully connected layers, for example.

The computer program instructions include software and/or hardware instructions. The computer program is loaded into a memory of the control device according to the invention, for example, or is already loaded into this memory.

According to a further aspect of the invention, the computer program according to the invention is executed on hardware and/or software of a cloud facility.

The computer program is loaded into the memory, for example, by a computer-readable data carrier or a data carrier signal. The invention is thus also implemented as an aftermarket solution.

The control unit, abbreviated as ECU, prepares input signals, processes them using an electronic circuit and provides logic and/or power levels as regulation and/or control signals. The control device according to the invention is scalable for assisted driving through to fully automated/autonomous/driverless driving.

The processing unit includes, for example, a programmable electronic circuit. According to one aspect of the invention, the processing unit or the control device is designed as a system-on-chip.

According to a further aspect, the invention relates to a computer system with input and output, a processing unit and storage devices. The computer system is designed to implement the network architecture according to the invention. The invention thus provides a use of the network architecture according to the invention in areas outside of automated driving.

The proposed network architecture is adaptable to specific hardware by considering, for example, main memory of the ECU, CPU and GPU cores, cache memory, and clock for the architecture choice. The built-in hardware can be used better as a result of the split backbone network according to the invention.

In one embodiment of the computer-implemented method, the first artificial neural network determines the first features for the object detection, semantic segmentation, traffic sign detection and lane marking detection types of detection. A first of the second artificial neural networks recognizes objects. A second of the second artificial neural networks breaks down areas of the data into semantically related units. A third of the second artificial neural networks recognizes traffic signs. A fourth of the second artificial neural networks recognizes lane markings.

• • 2D Detektion ausgehend von sog. zweidimensionalen Bounding Boxen, beispielsweise im Videobild einer Kamera,
• • 3D Detektion ausgehend von dreidimensionalen Bounding Boxen, umfassend Distanz- und Rotationsschätzung,
• • Annotationsschätzung für die erkannten Objekte wie zum Beispiel bei Fußgängern den Fußgängerzustand, beispielsweise laufend, stehen, oder bei Fahrzeugen den Fahrzeugzustand, beispielsweise parkend, angehalten, blinkend.

The first of the second artificial neural networks is a recognition head for object detection. Object detection includes

• • 2D detection based on so-called two-dimensional bounding boxes, for example in the video image of a camera,
• • 3D detection based on three-dimensional bounding boxes, including distance and rotation estimation,
• • Annotation estimation for the detected objects such as, for example, the state of pedestrians in the case of pedestrians, for example running, stationary, or the state of the vehicle in the case of vehicles, for example parking, stopped, blinking.

• • Holistische Segmentierung für alle sichtbaren Klassen,
• • Segmentierung eines jeden Objektes einer Detektion individuell, das heißt Instanzsegmentierung, und in Kombination mit nicht-zählbaren Objekten, sogenannte panoptische Segmentierung,
• • Segmentierung von Fahrbahnmarkierung oder Stäben für eine nachfolgende Linien-Regression eines Fahrspurbereiches.

The second of the second artificial neural networks is a semantic segmentation recognizer. The semantic segmentation includes

• • Holistic segmentation for all visible classes,
• • Segmentation of each object of a detection individually, i.e. instance segmentation, and in combination with non-countable objects, so-called panoptical segmentation,
• • Segmentation of lane markings or bars for a subsequent line regression of a lane area.

• • Klassifikation und/oder Lokalisation des Verkehrszeichens,
• • Lage und Zuordnung zu Fahrspuren und Verkehrsteilnehmern,

The third of the second artificial neural networks is a traffic sign recognition head. This detection includes

• • classification and/or localization of the traffic sign,
• • Location and assignment to lanes and road users,

The fourth of the second artificial neural networks is a lane marking recognition head. This recognition includes fitting, extrapolating from appropriate models such as clothoids, polynomials, splines, and the like.

• • Erkennung von Lichtquellen bei Nacht und deren Klassifikation zu Objekten wie anderen Verkehrsteilnehmern, Straßenlaternen,
• • niederdimensionale Klassifizierung, Regression und Vorhersage, beispielsweise für den Egopfad, das heißt eine Vorhersage, welchen Pfad das eigene Fahrzeug nehmen wird, Freespace, das heißt einen für das eigene Fahrzeug sicher zu befahrenden Bereich, Wetterannotation der direkten Umgebung, Failsafe-Annotation der unterschiedlichen messenden Sensoren, beispielsweise Blendungseffekte der Videosensorik durch eine tiefstehende Sonne.

According to one aspect of the invention, more of the second artificial neural networks are used for the following tasks:

• • Detection of light sources at night and their classification to objects such as other road users, street lights,
• • Low-dimensional classification, regression and prediction, for example for the ego path, i.e. a prediction of which path your own vehicle will take, freespace, i.e. an area that can be safely driven on for your own vehicle, weather annotation of the direct environment, failsafe annotation of the different measuring sensors, for example the glare effects of the video sensors caused by a low-lying sun.

According to one aspect of the invention, there is a holistic segmentation, in particular no binary segmentation, for example into pedestrians and non-pedestrians. The segmentation is done on the whole image for all classes.

In a further refinement, when each of the second artificial neural networks is executed, a respectively predetermined number of layers of the first artificial neural network is accessed. This makes it possible for each of the second artificial neural networks to be transformed with a configured number of layers of the backbone before the individual last processing steps necessary for the corresponding type of recognition are carried out, for example non-maximum suppression or softmax. This achieves an individually adjustable depth of cleavage.

In a further refinement, data from a number of sensors for recognizing the surroundings of the driving system are entered and merged into the first artificial neural network in order to obtain the first features. According to one aspect of the invention, the multiple sensors are sensors of one sensor technology or, according to a further aspect, sensors of different sensor technologies. This feeds various sensor data into the MEB, expanding it into a multi-fusion backbone, or MFB for short.

• • Aufspannen einer 3D Umfeldwahrnehmung mittels Kamera, Radar und Lidar;
  • ◯ Detektion von 3D Bounding Boxen, inkl. Distanz- und Rotationsschätzung,
  • ◯ Tracking und von Objekten,
  • ◯ Trajektorienvorhersage von Verkehrsteilnehmern,
• • Objekte im Soundbereich;
  • ◯ Klassifikation, Ortung und Tracking von Geräuschquellen, zum Beispiel Martinshörner von Einsatzfahrzeugen und deren Weg,
• • Objekte im Geruchsbereich;
  • ◯ Klassifikation, Ortung und Tracking von Geruchsquellen, zum Beispiel Brandherde außerhalb und innerhalb des Fahrzeugs.

Sensors such as cameras, lidar and radar as well as other possible sensors such as acoustic or olfactory sensors can be included for the input. This results in the following fields of application, for example:

• • Creating a 3D environment perception using camera, radar and lidar;
  • ◯ Detection of 3D bounding boxes, including distance and rotation estimation,
  • ◯ tracking and of objects,
  • ◯ trajectory prediction of road users,
• • objects in the sound area;
  • ◯ Classification, localization and tracking of noise sources, for example sirens from emergency vehicles and their route,
• • Objects in the odor area;
  • ◯ Classification, localization and tracking of odor sources, for example sources of fire outside and inside the vehicle.

In a further refinement, the data is pre-processed in each case before input. For example, each sensor's input is transformed with an individual number of layers of a convolutional network before entering the shared MFB backbone.

In a further refinement, a number of layers of the first artificial neural network which the second artificial neural networks each access and/or a number of layers of the first artificial neural network in which the first features are determined based on the data of the at least one sensor for detecting the surroundings and the types of detection.

An individual processing depth for inputs and outputs is made possible by these flowing into the backbone after a different number of layers. According to one aspect of the invention, the layer depth is made possible in addition to an architecture selection based on empirical findings using Neural Architecture Search, abbreviated NAS. In NAS, the structure of an artificial neural network is not chosen by the programmer, but the number and types of layers are also learned along with the parameters of the network.

According to one aspect of the invention, the number of slices is determined based on the data from the camera, radar, lidar and other sensors. This achieves a sensor-individually adjustable fusion depth.

In a configuration of the machine learning method that is possible by way of example, a data example is input into a first of the third artificial neural networks. The first of the third artificial neural networks is trained to recognize objects. As an output from the first of the third artificial neural networks, three-dimensional bounding areas around the recognized objects are obtained as identifiers for the objects. Further, the data example is input to a second of the third artificial neural networks. The second of the third artificial neural networks is trained to break down areas of data into semantically related units. As an output from the second of the third artificial neural networks, segmentations are obtained as identifiers for the semantically related units. In addition, the data example is input to a third of the third artificial neural networks. The third of the third artificial neural networks is trained to recognize traffic signs. As an output from the third of the third artificial neural networks, two-dimensional boundary areas around the recognized traffic signs are obtained as identifiers for the traffic signs. Furthermore, the data example is input to a fourth of the third artificial neural networks. The fourth of the third artificial neural networks is trained to recognize lane markings. As an output of the fourth of the third artificial neural networks, coordinates of the lane markers are obtained as identifiers for the lane markers. Pseudo-labels of different recognition types including object detection, semantic segmentation, traffic sign recognition and lane marking recognition are generated on a data sample and thus a completely complex labeled data set for training the MEB or MFB.

In a further embodiment, the computer program comprises first software code sections, through which a first artificial neural network for determining first characteristics from data of at least one sensor for environment recognition of an automated driving system program is mated. Furthermore, the computer program includes second software code sections, which are used to program second artificial neural networks for classification, localization and/or prediction for different detection types of automated driving based on the first features.

• 1 eine schematische Darstellung einer Umfelderkennung,
• 2 eine reale Darstellung einer Umfelderkennung,
• 3 eine Darstellung einer semantischen Segmentierung,
• 4 eine schematische Darstellung einer individuellen Erkennung und Darstellung von Umfeldobjekten,
• 5 eine schematische Darstellung einer Faltungsmatrix,
• 6 ein Ausführungsbeispiel einer erfindungsgemäßen Netzwerkarchitektur,
• 7 ein weiteres Ausführungsbeispiel einer erfindungsgemäßen Netzwerkarchitektur,
• 8 ein Ausführungsbeispiel zur Generierung von Pseudolabeln,
• 9 ein weiteres Ausführungsbeispiel einer erfindungsgemäßen Netzwerkarchitektur,
• 10 ein Ausführungsbeispiel von Klassifikationsergebnissen der erfindungsgemä-ßen Netzwerkarchitektur und
• 11 ein Ausführungsbeispiel eines erfindungsgemäßen Steuergeräts.

The invention is illustrated in the following exemplary embodiments. Show it:

• 1 a schematic representation of an environment detection,
• 2 a real representation of an environment detection,
• 3 a representation of a semantic segmentation,
• 4 a schematic representation of an individual recognition and representation of surrounding objects,
• 5 a schematic representation of a convolution matrix,
• 6 an embodiment of a network architecture according to the invention,
• 7 a further exemplary embodiment of a network architecture according to the invention,
• 8th an example for generating pseudo labels,
• 9 a further exemplary embodiment of a network architecture according to the invention,
• 10 an embodiment of classification results of the network architecture according to the invention and
• 11 an embodiment of a control device according to the invention.

In the figures, the same reference symbols denote the same or functionally similar reference parts. For the sake of clarity, only the relevant reference parts are highlighted in the individual figures.

1 shows schematically how an automated driving system AD uses a sensor S1 in the form of a camera to recognize lane markings Bahn, objects Obj, traffic signs Ver and associated image areas Seg. This recognition flows through the in 1 shown control unit ECU in the control of the automated driving system AD.

2 shows the detection in action with an example.

3 shows the recognition mode of a holistic semantic segmentation Seg. The correspondingly recognized image areas are provided with a class label for each pixel, such as vehicle, lane, person and traffic lights.

4 shows how the individual identification types Bahn, Obj, Ver, Seg are represented by the respective identifiers K_Bahn for lane markings, K_Obj for object recognition, K_Ver for traffic sign recognition and K_Seg for semantically related units. For example, K_Bahn are based on coordinates of lane markings, K_Obj on 3D boxes, K_Ver on 2D boxes and K_Seg on related pixel areas. This makes it clear that each recognition process has a different output, i.e. it perceives the environmental elements visible in the image with different processes and different goals.

5 shows a section of a convolutional network CNN with an input In, for example a matrix whose entries represent brightness values of sensor pixels. The input is transformed with a convolution matrix kernel as shown to get an output out. The convolution matrix is a Sobel filter, for example, which is used to detect edges in an image. According to a further aspect of the invention, the convolution matrix is a smoothing filter, a relief filter, a Laplacian filter or a sharpening filter. As a rule, the weights of the filter are learned freely using the data. It is within the scope of the invention to choose from pre-designed filters, or a combination of learned and pre-designed filters. According to a further aspect, several of these convolution matrices are applied to the input and correspondingly several outputs are obtained, one output per convolution matrix.

6 shows an exemplary embodiment of the method according to the invention and the network architecture according to the invention. In a first method step V1, data from the camera S1 are entered into the first artificial neural network MEB. The MEB determines the first characteristics. The first features are entered into second artificial neural networks Head1 to Head4 and processed in a second method step V2 depending on a respective type of recognition. For example, four of the second artificial neural networks are used here. The invention also relates to the use of several, N, second artificial neural networks Head1, Head2, ..., HeadN. In the second method step V2, the first recognition head Head1 makes predictions comprising classification, localization and prediction of trajectories and fits and extrapolates road markings. The second Detection head Head2 determines 3D boxes around objects for object detection. The third recognition head, Head3, determines 2D boxes around traffic signs for traffic sign recognition. The fourth recognition head, Head4, semantically segments image areas. The individual recognition heads Head1 - Head4 are transformed with a number of Lay layers configured for them from the MEB. In a third method step V3, the surroundings are recognized based on the results of the second artificial neural networks Head1-Head4. Based on the recognition of the surroundings, the control unit ECU determines regulation and/or control signals for the driving system AD.

7 shows the extension of the MEB to the MFB. For example, data from camera S1, radar S2 and lidar S3 are merged. The use of any number of sensors is also within the scope of the invention. The input of each sensor is transformed with an individual number of layers Lay of a convolutional network.

8th shows the generation of pseudo labels K_Bahn, K_Obj, K_Ver and K_Seg using third artificial neural networks KNN3_1-KNN3_4 in a first method step M1 of the machine learning method according to the invention. According to the invention, a plurality, N, of the third artificial neural networks are provided, depending, for example, on the number of different types of recognition. In a second method step M2, a sample data Samp, for example an image from the camera S1, is marked with all the pseudo labels. The MEB or the MFB is trained with this identified data example Samp in a third method step M3.

9 shows another embodiment of the invention. An MEB with two outputs is implemented, which represents an innovation compared to the prior art. A DLA-34 network is used as the backbone for this. DLA-34 is a convolutional network that splits and joins layers in groups at different points to provide an optimal performance-to-runtime utilization ratio. It takes advantage of the layered structure of various current convolutional network backbones, such as concatenating layers from ResNet, using Batchnorm, etc. While DLA-34 is a very efficient convolutional network, it is used here as an example only and could be used substitute with a less performant convolutional network. As an output there is a 2D detection head based on CenterNet and a head for semantic segmentation based on FCN. The MS-Coco 2017 data set, which consists of images and labels for instance segmentation, is used for training. These are redesigned for the task of parallel recognition of semantic segmentation and object detection.

The result of the in 9 shown embodiment shows 10 . As a result, it can be seen that the MEB is able to carry out the desired recognitions with both heads. 10 shows an inference sample of the network with overlay of the two outputs of semantic segmentation and detection. In 10 it can also be seen that the network carries out semantic segmentation, i.e. the pixel labeling of the people for the person class (general and not box-specific). The performance of the object detection is slightly better with 36.7% mAP in multitask training compared to 36.3% mAP for pure object detection in this setup. mAP means mean average precision. The additional output of the semantic segmentation takes up only 25% of the runtime of the network in the empirical test, compared to 100% with a comparable additional segmentation network. This shows the increased efficiency of the invention presented.

This in 11 Control unit ECU shown receives data from camera S1 via first interfaces INT 1 . A processing unit P, for example a CPU, GPU or FPGA, executes the MEB or MFB and receives the environment recognition. Based on the recognition of the surroundings, the processing unit P determines regulation and/or control signals for automated operation of the driving system AD. The control unit ECU provides the regulation and/or control signals to actuators for longitudinal and/or lateral guidance of the driving system AD via second interfaces INT 2 .

Reference List

AD

automated driving system

MEB

first artificial neural network, multi-detection backbone

MFB

first artificial neural network, multi-fusion backbone

Head1-HeadN

second artificial neural networks, heads

KNN3_1-KNN3_4

third artificial neural networks

object

object detection

seg

semantic segmentation

Ver

Traffic Sign Recognition

train

lane marking detection

K_Obj

Indicators for objects

K_Seg

Indicator for semantically related units

K_Ver

Traffic sign markers

K_Bahn

Signs for lane markings

layout

Layer artificial neural network

samples

marked data example

ECU

control unit

INT1

first interfaces

INT2

second interfaces

P

processing unit

In

input

Out

output

core

convolution matrix

CNN

convolution network

DLA-34

Deep layer aggregation network

CenterNet

artificial neural network

FCN

Fully convolutional network

V1-V3

process steps

M1-M3

process steps

Claims (10)

Computer-implemented method for environment recognition for an automated driving system (AD) comprising the steps • Entering data from at least one sensor (S1, S2, S3) for detecting the environment of the driving system (AD) in a first artificial neural network (MEB, MFB), which is trained to determine the first features from the data and to obtain the first features ( V1), • Entering and processing the first features in respective second artificial neural networks (Head1 - HeadN), the second artificial neural networks (Head1 - HeadN) are each trained for one of different types of recognition of automated driving classification, localization and / or to obtain prediction results (V2) and • Obtaining the environment detection based on the results of the second artificial neural networks (Head1 - HeadN), with regulation and/or control signals for the driving system (AD) being determined (V3) based on the environment detection. procedure after claim 1 , where • the first artificial neural network (MEB, MFB) determines the first features for the recognition types object detection (Obj), semantic segmentation (Seg), traffic sign recognition (Ver) and lane marking recognition (Bahn), • a first of the second artificial neural networks ( Head2) recognizes objects, • a second of the second artificial neural networks (Head4) breaks down areas of the data into semantically related units, • a third of the second artificial neural networks (Head3) recognizes traffic signs and • a fourth of the second artificial neural networks (Head1) recognizes lane markings. procedure after claim 1 or 2 , wherein in the execution of each of the second artificial neural networks (Head1 - HeadN) a predetermined number of layers (Lay) of the first artificial neural network (MEB, MFB) is accessed. Procedure according to one of Claims 1 until 3 , In the first artificial neural network (MFB) data from multiple sensors (S1, S2, S3) for environment detection of the driving system (AD) are entered and merged in order to obtain the first features. procedure after claim 4 , whereby the data is pre-processed before input. Procedure according to one of Claims 1 until 5 , wherein a number of layers (Lay) of the first artificial neural network (MEB, MFB) accessed by the second artificial neural networks (Head1 - HeadN) respectively, and/or a number of layers (Lay) of the first artificial neural network (MEB, MFB) in which the first features are determined based on the data of the at least one sensor (S1, S2, S3) for detecting the surroundings and the types of detection. Machine learning method for a network architecture comprising a first artificial neural network (MEB, MFB) for determining first characteristics from data from at least one sensor (S1, S2, S3) for detecting the environment of an automated driving system (AD) and second artificial neural networks (Head1 - Head4) for classification, localization and/or prediction for different types of detection of the automated Driving based on the first features, the method comprising the steps of • Entering a data example in respective third artificial neural networks (KNN3_1 - KNN3_4), each of which is trained to generate specific identifiers (K_Obj, K_Seg, K_Ver, K_Bahn) for the respective type of recognition (M1), • labeling of the data sample with the labels (K_Obj, K_Seg, K_Ver, K_Bahn) (M2) and • forward feeding of the network architecture with the labeled data sample (Samp) and gradient-based learning of the second artificial neural networks (Head1 - Head4) ( M3). machine learning method claim 7 , where • a data example is input into a first of the third artificial neural networks (KNN3_1) that is trained to recognize objects, and three-dimensional bounding areas around the recognized objects are obtained as identifiers (K_Obj) for the objects, • the data example in a second of the third artificial neural networks (KNN3_2) is entered, which is trained to break down areas of the data into semantically related units, and segmentations are obtained as identifiers (K_Seg) for the semantically related units, • the data example into a third of the third artificial neural networks (KNN3_3) is input, which is trained to recognize traffic signs, and two-dimensional boundary areas around the recognized traffic signs are obtained as identifiers (K_Ver) for the traffic signs, • the data example is input into a fourth of the third artificial neural networks (KNN3_4). that is trained to recognize lane markings, and to obtain coordinates of the lane markings as identifiers (K_Bahn) for the lane markings. Control unit (ECU) for an automated driving system (AD) comprising • at least one first interface (INT1) via which the control unit (ECU) receives data from at least one sensor (S1, S2, S3) for detecting the environment of the driving system (AD), • a Processing unit (P), which is executed, a network architecture comprising a first artificial neural network (MEB, MFB) for determining first features from the data and second artificial neural networks (Head1 - HeadN) for classification, localization and / or prediction for different Carry out detection types of automated driving based on the first characteristics and an environment detection and control and / or control signals according to one of the methods Claims 1 until 6 to be determined, and • second interfaces (INT2), via which the control unit (ECU) provides the control and/or control signals to actuators for longitudinal and/or lateral guidance of the driving system (AD). Computer program for a control unit (ECU). claim 9 comprising commands that cause the control unit (ECU) to carry out a method according to one of Claims 1 until 6 and/or a method according to any one of Claims 7 or 8th executes when the computer program is running on the control unit (ECU).

Images