DE102022119711A1 - Method, system and computer program product for checking data sets for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) - Google Patents

Abstract

The invention relates to a method for checking data sets (320) for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function in at least one scenario (SZi) with regard to the completeness of relevant scenarios ( SZi), comprising: - Acquiring sensor data (220) by sensors (210) of at least one sensor device (200), which is connected to at least one moving object (10), when driving on one or more routes; - Generating a plurality of data sets (320) at least from the sensor data (220) from an input module (300); - identifying and classifying parameterized scenarios (SZpi) from the data sets (320) from a scenario analysis module (400); - identifying and classifying concrete scenario parameters ( Pci) from the data sets (320) and/or sensor data (220) from a parameter analysis module (500); - Creating a data representation for the specific scenario parameters (Pci) and for the parameterized scenarios (SZpi), each in the form of at least one Vector in a latent vector space (570);- Performing a vector space analysis and generating output results (770) from the vector space analysis, in particular in the form of diagrams, tables, graphics.

Description

The invention relates to a method, a system and a computer program product for checking one or more data sets for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function.

Modern vehicles are equipped with a variety of driver assistance systems or automated driver assistance functions to support the driver while driving and increase his safety. Driver assistance systems, for example, support speed and distance control and include lane keeping and lane changing functions. A specific maximum speed can be set, which will not be exceeded as long as the speed limit function is activated. Radar sensors and also camera systems are used for distance control, in which a certain distance is set, particularly to a vehicle in front. This allows the distance to vehicles in front, but also to vehicles in the side area, to be monitored. This leads to improved driving comfort and greater safety, especially when driving on the motorway and during overtaking maneuvers.

This trend towards driver assistance systems (Advanced Driver Assistance System, ADAS) and automated driving systems (ADS) in motor vehicles, but also in aircraft or watercraft and other moving objects, requires extensive security strategies, as the responsibility for the Vehicle control is no longer entirely the responsibility of the driver, but active functions are taken over by computer units in the vehicle. Therefore, for objects that move completely or partially autonomously, it must be ensured that these systems have a very low error rate in driving behavior. In order to ensure the safe functionality of an ADAS/ADS system, the detection and classification of other objects and the interpretation of traffic scenarios in the vicinity of a moving object, especially a vehicle, are important prerequisites. This requires targeted testing and training of driver assistance systems and automated driving systems both in extreme and exceptional situations (corner cases) as well as in everyday situations. Such extreme situations arise from a special combination of different factors. Examples of this are infrastructural features such as the type of road, the development on the edge of a road, the quality of the markings but also environmental conditions such as weather conditions, the time of day and the season. The behavior of other road users, the geographical topography and the weather conditions also play a major role.

However, as the performance of ADAS/ADS systems increases, the number of driving scenarios that the ADAS/ADS systems have to handle in road traffic also increases. In order to ensure safe, comfortable and efficient behavior of an ADAS/ADS system, training the algorithms of an ADAS/ADS system based on artificial intelligence is an important component. However, in order to test and train an ADAS/ADS system or a driving function, it is necessary that all relevant scenarios are completely contained in a training data set, because the ADAS/ADS system can only be comprehensively trained with a complete data set. This makes it possible to rule out the possibility that when using the ADAS/ADS system in real road traffic, a scenario occurs that was not present in the training data in this form or a similar form and therefore the ADAS/ADS system cannot react appropriately. Since the capability and performance of an ADAS/ADS system depends on the quality of the training data, checking existing data sets with regard to the completeness of the different scenarios they contain and thus their suitability for use as training data sets is an important prerequisite for achieving this goal to reach.

The DE 10 2019 217952 A1 discloses a method for providing a training data set for training an AI function on an unknown data domain, using a domain distance measure to determine a domain distance between data sets with marked data from known data domains and a data set from the unknown data domain, and wherein the training data set takes into account the respective determined Domain distances are at least partially generated and provided from the marked data of the data sets of the known data domains.

The DE 11 2019 001605 T5 discloses a method for training, testing and verifying an artificial intelligence using model data sets, with an evaluation of new data sets being carried out, for example, by determining a trust value.

The DE 10 2019 218127 A1 discloses a method for optimally providing AI systems men, whereby the determination of an output confidence value for an AI system is intended. Preferably, the output confidence value is estimated based on input data and/or at least one piece of context information.

The DE10 2019 213559 A1 discloses a driver assistance system with a neural network that is trained with data sets of recorded driving scenarios.

The object on which the invention is based is to provide options for checking one or more data sets for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) or a driving function for completeness with regard to the different scenarios they contain, to increase the safety of ADAS/ADS systems and/or a driving function.

This object is achieved according to the invention with regard to a method by the features of patent claim 1, with regard to a system by the features of patent claim 9, and with regard to a computer program product by the features of patent claim 15. The further claims relate to preferred embodiments of the invention.

The present invention checks existing data sets with a variety of different scenarios to determine whether a data set completely contains those relevant scenarios for testing and training an ADAS/ADS system based on artificial intelligence algorithms and/or a driving function that is required for the Training an ADAS/ADS system and/or a driving function are required. By selecting a complete training data set, the safety and functionality of a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a specific driving task can be significantly improved.

• - Erfassen von Sensordaten durch Sensoren zumindest einer Sensoreinrichtung, die mit zumindest einem sich bewegenden Objekt verbunden ist, beim Befahren einer oder mehrerer Fahrstrecken, wobei die Sensoren die Sensordaten von der Umgebung des sich bewegenden Objekts aufnehmen;
• - Generieren einer Mehrzahl von Datensätzen zumindest aus den Sensordaten von einem Eingabemodul;
• - Identifizieren und Klassifizieren von parametrisierten Szenarien aus den Datensätzen von einem Szenarien-Analysemodul;
• - Identifizieren und Klassifizieren von konkreten Szenarioparametern aus den Datensätzen und/oder Sensordaten von einem Parameter-Analysemodul;
• - Erstellen einer Daten-Repräsentation für die konkreten Szenarioparameter in Form zumindest eines Vektors in einem latenten Vektorraum;
• - Erstellen einer Daten-Repräsentation für die parametrisierten Szenarien in Form zumindest eines Vektors in einem latenten Vektorraum;
• - Durchführen einer Vektorraumanalyse insbesondere mittels einer Clusteranalyse der parametrisierten Szenarien in dem Vektorraum in Bezug auf die konkreten Szenarioparameter;
• - Generieren und Ausgeben von Ausgabeergebnissen aus der Vektorraumanalyse, insbesondere in Form von Diagrammen, Tabellen, Graphiken.

According to a first aspect, the invention provides a method for checking one or more data sets for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function in at least one scenario with regard to the completeness of relevant scenarios ready. A scenario represents a traffic event in a temporal sequence and is defined by a selection of scenario parameters and associated scenario parameter values, whereby in a parameterized scenario the scenario parameters and associated scenario parameter values are freely selectable, and in a specific scenario the specific scenario parameters and associated specific scenario parameter values are fixed . A data set contains a variety of scenarios. The procedure includes the following procedural steps:

• - Acquiring sensor data by sensors of at least one sensor device, which is connected to at least one moving object, when driving on one or more routes, the sensors recording the sensor data from the surroundings of the moving object;
• - Generating a plurality of data sets at least from the sensor data from an input module;
• - Identifying and classifying parameterized scenarios from the datasets from a scenario analysis module;
• - Identifying and classifying specific scenario parameters from the data sets and/or sensor data from a parameter analysis module;
• - Creating a data representation for the specific scenario parameters in the form of at least one vector in a latent vector space;
• - Creating a data representation for the parameterized scenarios in the form of at least one vector in a latent vector space;
• - Carrying out a vector space analysis, in particular by means of a cluster analysis of the parameterized scenarios in the vector space in relation to the specific scenario parameters;
• - Generate and output output results from vector space analysis, especially in the form of diagrams, tables, graphics.

In a further development it is provided that the analyzed data sets are provided with a suitability index based on the output results with regard to their suitability as a training data set for testing and training an ADAS/ADS system and/or a driving function.

In an advantageous embodiment, it is provided that the scenario analysis module comprises a software application that uses calculation methods and/or artificial intelligence algorithms, wherein the parameter analysis module comprises a software application that uses calculation methods and/or artificial intelligence algorithms; and wherein the output module comprises a software application, the calculation methods and/or algorithms of artificial intelligence are used.

In a further embodiment it is provided that the calculation methods and/or algorithms of artificial intelligence are used as mean values, minimum and maximum values, lookup tables, models for expected values, linear regression methods, Gaussian processes, fast Fourier transformations, integral and differential calculations, Markov Methods, probability methods such as Monte Carlo methods, temporal difference learning, extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, feedback neural networks, and / or folded neural networks are formed.

It is advantageously provided that a parameter is a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number , a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a highway, a street sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic volume, a topographic Structure such as a slope, a time, a temperature, a precipitation value, a weather and/or a season.

In particular, the sensors are radar systems, LIDAR systems for optical distance and speed measurement, image-recording 2D/3D cameras in the visible, IR and/or UV range, GPS systems, acceleration sensors, speed sensors, capacitive sensors, inductive sensors, voltage sensors , torque sensors, speed sensors, precipitation sensors and / or temperature sensors. In a further embodiment it is provided that the scenario analysis module, the parameter analysis module and/or the output module are integrated in a cloud computing infrastructure.

Advantageously, the sensor device and/or the input module is/are equipped with mobile radio modules of the 5G or 6G mobile radio standard for sending and receiving data.

According to a second aspect, the invention provides a system for checking one or more data sets for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function in at least one scenario with regard to the completeness of relevant scenarios ready. A scenario represents a traffic event in a temporal sequence and is defined by a selection of scenario parameters and associated scenario parameter values, whereby in a parameterized scenario the scenario parameters and associated scenario parameter values are freely selectable, and in a specific scenario the specific scenario parameters and associated specific scenario parameter values are fixed , A data set contains a variety of scenarios. The system includes a sensor device, an input module, a scenario analysis module, a parameter analysis module and an output module. The input module is designed to record sensor data by sensors of at least one sensor device, which is connected to at least one moving object, when driving on one or more routes, the sensors recording the sensor data from the surroundings of the moving object, and a plurality of data sets at least to generate from the sensor data. The scenario analysis module is designed to identify and classify parameterized scenarios from the data sets from a scenario analysis module. The parameter analysis module is designed to identify and classify specific scenario parameters from the data sets and/or sensor data. The parameter analysis module is designed to create a data representation in the form of at least one vector in a latent vector space for the specific scenario parameters and for the parameterized scenarios, and to perform a vector space analysis in particular by means of a cluster analysis of the parameterized scenarios in the vector space with respect to the specific scenario parameters. The output module is designed to generate and output output results from the vector space analysis, in particular in the form of diagrams, tables, graphics.

In an advantageous embodiment it is provided that the analyzed data sets are provided with a suitability index based on the output results with regard to their suitability as a training data set for testing and training an ADAS/ADS system and/or a driving function.

In a further development, it is provided that the scenario analysis module comprises a software application that uses calculation methods and/or artificial intelligence algorithms, wherein the parameter analysis module comprises a software application that uses calculation methods and/or artificial intelligence algorithms; and wherein the output module includes a software application that calculates Artificial intelligence procedures and/or algorithms are used.

The calculation methods and/or algorithms of artificial intelligence are advantageously as mean values, minimum and maximum values, lookup tables, models for expected values, linear regression methods, Gaussian processes, fast Fourier transformations, integral and differential calculations, Markov methods, probability methods such as Monte Carlo -Methods, temporal difference learning, extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, feedback neural networks, and/or convolutional neural networks.

In particular, it is provided that a parameter is a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number , a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a highway, a street sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic volume, a topographic Structure such as a slope, a time, a temperature, a precipitation value, a weather and/or a season.

Advantageously, the sensors are radar systems, LIDAR systems for optical distance and speed measurement, image-recording 2D/3D cameras in the visible, IR and/or UV range, GPS systems, acceleration sensors, speed sensors, capacitive sensors, inductive sensors, voltage sensors , torque sensors, speed sensors, precipitation sensors and / or temperature sensors.

In a further embodiment it is provided that the scenario analysis module, the parameter analysis module and/or the output module are integrated in a cloud computing infrastructure, and wherein the sensor device and/or the input module is connected to mobile radio modules of the 5G or 6G mobile radio standard is/are equipped to send and receive data.

According to a third aspect, the invention provides a computer program product comprising executable program code configured to, when executed, carry out the method according to the first aspect.

The invention is explained in more detail below using exemplary embodiments shown in the drawing.

• 1 ein Blockdiagramm zur Erläuterung eines Ausführungsbeispiels eines erfindungsgemäßen Systems;
• 2 eine schematische Darstellung eines Vektorraumes;
• 3 ein Flussdiagramm zur Erläuterung der einzelnen Verfahrensschritte eines erfindungsgemäßen Verfahrens;
• 4 ein Blockdiagramm eines Computerprogrammprodukt gemäß einer Ausführungsform des dritten Aspekts der Erfindung.

This shows:

• 1 a block diagram to explain an exemplary embodiment of a system according to the invention;
• 2 a schematic representation of a vector space;
• 3 a flowchart to explain the individual method steps of a method according to the invention;
• 4 a block diagram of a computer program product according to an embodiment of the third aspect of the invention.

Additional features, aspects and advantages of the invention or its embodiments are explained in the following description in conjunction with the claims.

Really measured traffic scenarios, but also increasingly virtual traffic scenarios that are created through programming, are used to test, train and secure driver assistance systems (ADAS) and automated driving systems (ADS). In the context of the invention, a scenario is a traffic event in a temporal sequence. An example of a scenario is driving on a highway bridge, turning in a turning lane, driving through a tunnel, turning into a roundabout or stopping in front of a pedestrian crossing. Another example is an overtaking operation, which can be described as a scenario in which a first vehicle is initially behind another vehicle, then changes lanes to the other lane and increases speed in order to overtake the other vehicle. Such a scenario is also known as a cut-in scenario. In addition, specific visibility conditions, for example due to twilight or high levels of sunlight, as well as environmental conditions such as the weather and season, traffic volume and certain geographical topographical conditions can influence a scenario. It is well known that road conditions look different in heavy rain and slippery conditions than on a sunny summer day. The set of scenarios that can occur in the vehicle's driving environment and that must be correctly recognized and processed by an ADAS/ADS system is represented by an Operational Design Domain (ODD). This includes both everyday driving scenarios and very rarely occurring corner cases.

In order to be able to use an ADAS/ADS system or a driving function in a motor vehicle, they must be tested and trained for reliable use. The aim of testing and training is to demonstrate the reliability and robustness of an ADAS/ADS system or a driving function throughout the entire ODD and then to grant approval. However, the prerequisite is that the selected traffic scenarios also cover the entire ODD. To test and train an ADAS/ADS system or a driving function, a number of data sets are created, with each data set containing a large number of different traffic scenarios.

The question arises as to whether the data sets with the selected traffic scenarios are complete and cover the entire ODD in such a way that verifiable statements can be made in a simulation about the safety and functionality of ADAS/ADS systems or driving functions. According to the invention, a check of the data sets with a large number of different traffic scenarios is provided for their completeness using artificial intelligence.

1 shows a system 100 according to the invention for checking one or more data sets for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function. The system 100 according to the invention includes one or more moving objects 10, at least one sensor device 200, an input module 300, a scenario analysis module 400, a parameter analysis module 500 and an output module 700, each of which is provided with a processor and/or a memory unit can.

In connection with the invention, a “module” can be understood to mean, for example, a processor and/or a processor unit and/or a memory unit for storing program instructions. The processor is specifically set up to execute the program instructions in order to implement or realize the method according to the invention or a step of the method according to the invention. In particular, a module can be integrated into a cloud computing infrastructure 250.

In connection with the invention, a “processor” can be understood to mean, for example, a machine or an electronic circuit. A processor can in particular be a main processor (Central Processing Unit, CPU), a microprocessor or a microcontroller, for example an application-specific integrated circuit or a digital signal processor, possibly in combination with a memory unit for storing program instructions, etc . A processor can also be understood as a virtualized processor, a virtual machine or a soft CPU. For example, it can also be a programmable processor that is equipped with configuration steps for carrying out the method according to the invention or is configured with configuration steps in such a way that the programmable processor has the features of the method, the system, the modules, or other aspects and/or other aspects according to the invention. or partial aspects of the invention are realized. In particular, the processor can contain highly parallel computing units and powerful graphics modules.

In the context of the invention, a “memory unit” or a “memory module” can be understood to mean, for example, a volatile memory in the form of a random access memory (RAM) or a permanent memory such as a hard drive or a data carrier or, for example, a removable memory module become. The storage module can also be a cloud-based storage solution.

“Database” means both a storage algorithm and the hardware in the form of a storage unit. In particular, the databases can be designed as part of a cloud computing infrastructure.

The moving object 10 is in particular a motor vehicle, an autonomous motor vehicle, an agricultural vehicle such as a combine harvester, a robot in production or in service and care facilities, or a watercraft or a flying object such as an air taxi. The moving object 10 can also be designed as a motorcycle or an electric bicycle. The moving object 10 can also be referred to as an ego vehicle within the scope of the invention.

The sensor device 200 is connected to the moving object 10 and includes sensors 210 that capture sensor data 220 from the surroundings of the object 10 such as road markings, vehicles, people, crash barriers, traffic signs, etc. and transmit it to the input module 300.

In connection with the invention, sensor data 220 includes both raw data and already processed data from the measurement results of the sensors 220 and, if necessary, others Understanding data sources. In particular, it can be provided that the input module 300 is connected to a database 270 in which further data is contained. Furthermore, the input module 300 can have a user interface 350 or be connected to a user interface 350. The user interface 350 is designed to input and generate data in the form of text messages and/or voice messages and/or images and graphics. In particular, a keyboard, a microphone, a camera and/or a display designed as a touchscreen are provided for entering the data.

The sensors 210 of the sensor device 200 can in particular be one or more radar systems with one or more radar sensors, one or more LIDAR systems (Light Detection and Ranging) for optical distance and speed measurement, one or more image-recording 2D/3D cameras in the visible Area, but also in the IR and UV area, and/or GPS systems. Furthermore, acceleration sensors, speed sensors, capacitive sensors, inductive sensors, voltage sensors, torque sensors, speed sensors, precipitation sensors and/or temperature sensors, etc. can be provided.

In particular, the 2D/3D image-recording camera is designed as an RGB camera in the visible range with the primary colors blue, green and red. However, a UV camera in the ultraviolet range and/or an IR camera in the infrared range can also be provided. The cameras, which differ in their recording spectrum, can therefore optimally record the recording area even in different lighting conditions. Furthermore, it can be provided that a 3D camera is designed as a stereo camera.

The recording frequency of the sensor device 200 is designed in particular for fast speeds of the moving object 10 and can record sensor data 220 with a high image recording frequency. Furthermore, the sensor device 200 can be equipped with a microphone for detecting acoustic signals. This allows rolling noise from tires or engine noise to be recorded.

In addition, it can be provided that the sensor device 200 automatically starts the image recording process when there is a significant change in area in the recording area of the sensor device 200, for example when a clear change in a traffic situation is recognizable. This enables a selective data collection process and only relevant sensor data 220 is processed by the input module 300. This allows computing capacity to be used more efficiently.

In particular, it is intended to use a weatherproof action camera as the camera type for one or more cameras, which can be arranged in particular in the outside area of the object 10. The attachment position of a camera on the object 10 determines which recording area can be recorded by the camera. In particular, it can be provided that the recording areas of two or more cameras overlap in order to generate a panoramic display, for example by means of subsequent image processing. This allows the spatial surroundings of a moving object 10 to be comprehensively recorded.

Radar sensors can be used for longer distances of up to 250 meters and have the advantage of being independent of weather and lighting conditions. The performance of a radar depends on many factors such as the selected hardware components, software processing and the radar echo. For example, radar accuracy is less precise with a lower signal-to-noise ratio than with a high signal-to-noise ratio. In addition, the installation position is crucial for the high performance of a radar sensor, as effects such as multipath propagation and distortion caused by covers affect the detection accuracy.

In addition to image-recording cameras and radar sensors, LIDAR sensors represent an important type of sensor for perceiving the environment for moving objects 10. As with cameras and radar sensors, the environment can be recorded and distances to other surrounding objects can be measured. In particular, 3D LIDAR sensors can record detailed information about an environmental object through a high sampling rate. Compared to radar sensors, LIDAR sensors are characterized by a higher spatial and depth resolution.

Furthermore, a GPS connection is advantageously provided in order to determine the geographical location of the object 10 and to assign it to the recorded sensor data 220.

In particular, it can be provided that the sensor data 220 from several moving objects 10, in particular a fleet of objects 10, are transmitted to the input module 300. However, it can also be provided that the sensor data 220 is first sent to the cloud Computing infrastructure 250 and from there to the input module 300 are passed on. In particular, it can be provided that the input module 300 is integrated into the cloud computing infrastructure 250. This ensures fast calculations, as cloud-based solutions offer the advantage of high and therefore fast computing performance.

The sensor data 220 of the surroundings of the object 10 recorded by the sensor device 200 are passed on to the input module 300 or initially to the cloud computing infrastructure 250 using data connections. In particular, a wireless data connection is provided, which can be designed, for example, as a mobile radio connection and/or a near-field data connection such as ^Bluetooth® , Ethernet, NFC (near field communication) or Wi-Fi®. For the communication of the sensor device 200 with the input module 300 or the cloud computing infrastructure 250, a 5G mobile phone connection or 6G mobile phone connection is used in particular, since data transmission can take place in real time in this way. For this purpose, the sensor device 200 is equipped with the corresponding mobile radio modules.

5G is the fifth generation mobile communications standard and, compared to the 4G mobile communications standard, is characterized by higher data rates of up to 10 Gbit/sec, the use of higher frequency ranges such as 2100, 2600 or 3600 megahertz, increased frequency capacity and thus increased data throughput and real-time data transmission because up to a million devices per square kilometer can be addressed simultaneously. The latency times range from a few milliseconds to less than 1 ms, so that real-time transmission of data and calculation results is possible. Therefore, the sensor data 220 recorded by the sensor device 200 can be sent in real time to the input module 300 or the cloud computing infrastructure 250.

By integrating the input module 300 in a cloud computing infrastructure 250 in conjunction with a 5G mobile phone connection, processing of the sensor data 220 recorded by the sensor device 200 can be ensured in real time. In order to protect the connection to the cloud computing infrastructure 250 using a mobile radio connection, cryptographic encryption methods are provided in particular.

In the input module 300, sensor data 220 is further processed in order to generate one or more manageable data sets 320 from the raw data of the sensor data 220 for further processing. For example, filtering, time synchronization, changing the data formats (e.g. MPEG, JPEG, FP16) or data fusion of the sensor data 220 can be carried out. In addition, the data retrieved from the database 270 and the data entered using the user interface 350 can be integrated into the respective data sets 320 by the input module 300.

The one or more data sets 320 are now passed on to the scenario analysis module 400. In the scenario analysis module 400, scenarios SZ _i are derived from the data sets 320 using a software application 450. An example of a scenario SZ _i is an overtaking operation in which a first vehicle is initially behind another vehicle, then changes lane to the other lane and increases the speed in order to overtake the other vehicle. Such a scenario is also known as a cut-in scenario. Other examples of scenarios include turning into another street, waiting at a traffic light, driving into a construction site, or performing emergency braking, for example when approaching the end of a traffic jam, etc.

In the context of the present invention, a distinction is made between a parameterized scenario SZp and a concrete scenario SZc. A parameterized scenario SZp can also be referred to as a logical scenario. Any parameterized scenario SZp _i is defined by various scenario parameters P ₁ , P ₂ , ..., P _n from a set of possible scenario parameters P _i and associated scenario parameter values PV ₁ , PV ₂ , ..., PV _n from a set of possible ones Scenario parameter values PV _i are defined, with scenario parameter values PV _i defining the value range of a scenario parameter P _i . A scenario parameter P _i represents, for example, a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a highway, a street sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic volume, a topographical structure such as a gradient, a time of day, a temperature, a precipitation value, a weather condition and/or a season. Scenario parameters P _i thus characterize properties and features of a scenario SZp _i within the scope of the present invention. An example of a scenario parameter P _i of a scenario SZp _i is the speed of an ego vehicle and the value range of the associated Sze narioparameter value PV _i can, for example, cover the range from 100 km/h to 180 km/h. For another scenario SZp _k, the value range of the scenario parameter value PV _i can extend from 40 km/h to 70 km/h.

A parameterized scenario SZp comprises a temporal sequence of time intervals Δt ₁ , Δt ₂ ., ..., Δt _n , in which different scenes and events take place. A parameterized scenario SZp begins with a starting scene and then develops further through occurring events, from which new subsequent scenes emerge over time. The starting scene is thus changed by one or more events. An event can be a reaction actively triggered by a road user, such as an acceleration, or an event that recurs cyclically, such as the switching of a traffic light. The start scene and the individual subsequent scenes therefore only cover a small time interval Δt or a snapshot, while a parameterized scenario SZp covers a longer period of time. A possible illustration of a scenario SZp is a graph representation in which the events are represented as edges and the individual scenes as nodes of a graph.

In a parameterized or logical scenario SZp, both the scenario parameters P _i and the associated scenario parameter values PV _i are not all fixed. A scenario SZ in which the concrete scenario parameters Pc _i and the associated concrete scenario parameter values PVc _i or value ranges of the concrete scenario parameter values PVc _i are defined is referred to as a concrete scenario SZc. Both scenarios, both the parameterized scenario SZp and the concrete scenario SZc, are scenarios that are written in particular in a machine-readable code or script.

The software application 450 creates 320 parameterized scenarios SZp _i from the data sets. Using the software application 450, a suitable parameterized scenario SZp _i can be derived from one or more data sets 320 for a specific geographical location. An exemplary identified scenario SZp _i is the implementation of emergency braking when approaching the end of a traffic jam.

The software application 450 uses in particular calculation methods and/or artificial intelligence algorithms to identify and classify the parameterized scenarios SZp _i . The calculation methods and algorithms of artificial intelligence can include mean values, minimum and maximum values, lookup tables, models for expected values, linear regression methods, Gaussian processes, fast Fourier transformations, integral and differential calculations, Markov methods, probability methods such as Monte Carlo methods, temporal Difference learning, but also extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, and/or feedback neural networks. The artificial intelligence algorithms can in particular be encoders and decoders with neural networks and/or at least one reinforcement learning agent (LV).

A neural network consists of neurons arranged in several layers and connected to each other in different ways. A neuron is able to receive information from outside or from another neuron at its input, evaluate the information in a certain way and forward it in a modified form to another neuron at the neuron output or output it as the end result. Hidden neurons are located between the input neurons and output neurons. Depending on the network type, there may be multiple layers of hidden neurons. They ensure the forwarding and processing of the information. Output neurons ultimately deliver a result and output it to the outside world. Neural networks can be trained through unsupervised or supervised learning.

The different arrangement and connection of the neurons creates different types of neural networks, such as in particular a feedforward neural network (FNN), a feedback network (recurrent neural network, RNN) or a folded neural network (convolutional Neural Network, CNN). A convolutional neural network has multiple convolution layers and is very suitable for machine learning and applications in the field of pattern recognition and image recognition. Since a large part of the data recorded by the sensors is available as images, convolutional neural networks (CNN) are used in particular.

The Convolutional Neural Network (CNN) is particularly well suited for machine learning and artificial intelligence (AI) applications in the field of image and speech recognition because it has multiple convolution layers. The way a convolutional neural network works is to a certain extent modeled on biological processes and the structure is comparable to the visual cortex of the brain. Conventional neural network Works consist of fully or partially connected neurons in several levels and these structures reach their limits when processing images, as there would have to be a number of inputs corresponding to the number of pixels. The convolutional neural network is made up of different layers and is basically a partially locally linked feedforward neural network. The individual layers of the CNN are the convolutional layer, the pooling layer and the fully connected layer. The Convolutional Neural Network (CNN) is therefore suitable for machine learning and artificial intelligence applications with large amounts of input data, such as in image recognition. The network works reliably and is insensitive to distortions or other optical changes. The CNN can process images captured under different lighting conditions and from different perspectives. It still recognizes the typical features of an image. Since the CNN includes several local partially connected layers, it has a significantly lower storage space requirement than fully connected neural networks, as the convolutional layers significantly reduce the storage requirements. This also shortens the training time of a CNN, especially when using modern graphics processors.

The parameterized scenarios SZp _i identified and classified by the software application 450 can be stored in a scenario database 470 or passed on directly to the parameter analysis module 700. In particular, the scenario database 470 can be integrated into the cloud computing infrastructure 250, from which the parameterized scenarios SZp _i are retrieved for further processing by the parameter analysis module 700. In particular, both the scenario analysis module 400 and the parameter analysis module 700 are integrated into the cloud computing infrastructure 250.

In the parameter analysis module 500, the specific scenario parameters Pc _i for the respectively identified parameterized scenarios SZp _i are extracted and classified from the sensor data 220 or the data sets 320, so that concrete scenarios SZc _i can be generated. For this purpose, the parameter analysis module 500 uses a software application 550, which also includes calculation methods and/or artificial intelligence algorithms such as, in particular, deep learning with, for example, at least one convolutional neural network (CNN) and/or at least one learning reinforcement agent (Reinforcement Learning Agent, LV) is used. The scenario parameters P _i in the exemplary identified scenario SZp _i "Performing emergency braking when approaching the end of a traffic jam" are, for example, the "distance between the ego vehicle and a target vehicle at the start of braking", "a deceleration of the Target vehicle”, “a period of delay”, etc.

Using a software application 550, the scenario parameters P _i of a parameterized scenario SZp _i are now spanned by latent vector space 570, in which the concrete scenarios SZc _i can be represented as representations. The vector space is created by automated endless learning through the concrete scenario parameters Pc _i and concrete scenarios SZc _i , so that similarity structures such as clusters of concrete scenarios SZc _i develop in a self-learning manner. These clusters of concrete scenarios SZc _i can now be analyzed using a similarity query.

For the transformation of the concrete scenario parameters Pc _i and concrete scenarios SZc _i into data representations in the form of one or more vectors in the latent vector space 570, the software application includes at least one encoder to encode the scenario parameters Pc _i such that the scenario parameters Pc _i as a data representation in the form of one or more vectors in a latent vector space 570 is present. Furthermore, a second encoder is provided with which the software application 550 encodes the concrete scenarios SZc _i in such a way that they are also present as a data representation in the form of one or more vectors in the latent vector space 570. The latent vector space 570 is a multidimensional vector space with a dimension that corresponds to the representation of the concrete scenario parameters Pc _i and concrete scenarios SZc _i .

The encoders of the software application 550 are in particular deep encoders, which include deep neural networks and in particular for coupled neural networks, which can have different structures. In addition, the encoders may include other structures and algorithms such as genetic algorithms and learning reinforcement agents to perform efficient and precise classification and extraction of features of the concrete scenario parameters Pc _i and , which forms the basis for creating the data representations.

The vector space 570 is thus further developed with each additional learning cycle by entering a new data set 320, whereby its dimension can constantly change. Since coding of the respective concrete scenario parameters Pc _i and the concrete scenarios SZp _i is carried out for a large number of data sets 320, this is successively created in a continuous learning mode through new data sets 320, a latent vector space 570 with a large number of concrete scenario parameters Pc _i and concrete scenarios SZp _i .

In the 2 an exemplary vector space 570 is shown. A first parameter P ₁ can relate to a target vehicle deceleration value and a second parameter P ₂ to the duration of the deceleration. Various concrete scenarios SZp _i are now arranged in the vector space 570 spanned by these two parameters P ₁ and P ₂ . For example, the structure of the vector space 570 can now be analyzed as an image of the data set 320 using a cluster analysis. For this purpose, known methods such as triangulation methods can be used to determine distances. The software application 550 can thus determine those areas in the vector space 570 in which no concrete scenarios SZp _i are present, such as the area A ₁ in the 2 . In the area A ₁ shown, there are therefore no concrete scenarios SZp _i that have a high delay value and a short delay duration.

The generated vector space 570 and/or the results of the vector space analysis are passed on to an output module 700, from which they are output as output data 770. In particular, the output module 700 includes a software application 750 with processing and/or evaluation algorithms to present the output data 770 to a user in the form of diagrams, tables, graphics, etc. For this purpose, for example, algorithms such as decoders can be used to transform the data formats of the vector space 570 back into user-friendly data formats. A user can then decide whether the analyzed data set 320 is suitable as a training data set for an ADAS/ADS system. In a further development, however, it is conceivable that an artificial intelligence is also used here, which self-learning uses the analysis results to recognize which data sets 320 are suitable as training data sets. It can then be provided that the unsuitable data sets 320 are automatically removed from a pool of possible training data sets by the artificial intelligence. The remaining training data sets, on the other hand, are characterized by the fact that they completely contain the relevant scenarios SZ _i that are required to test and train an ADAS/ADS system for a specific driving task. In addition, the analyzed data sets 320 can be provided with a suitability index based on the output results 770 with regard to their suitability as a training data set for testing and training an ADAS/ADS system and/or a driving function. For example, the suitability index may include values on a scale of 0 to 1.

In 3 are the method steps for checking one or more data sets 320 for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function in at least one scenario (SZ _i ) with regard to the completeness of the relevant scenarios included SZ _i shown.

In a step S10, sensor data 220 are recorded by sensors 210 of at least one sensor device 200, which is connected to at least one moving object 10, when driving on one or more routes, the sensors 210 recording the sensor data 220 from the surroundings of the moving object 10 .

In a step S20, a plurality of data sets 320 are generated at least from the sensor data 220 by an input module 300.

In a step S30, parameterized scenarios SZp _i and scenario parameters Pc _i from the data sets 320 are identified and classified by a scenario analysis module 400.

In a step S40, concrete scenario parameters Pc _i from the data sets 320 and/or sensor data 220 are identified and classified by a parameter analysis module 500.

In a step S50, concrete scenarios SZc _i with the identified concrete scenario parameters Pc _i are generated for a parameterized scenario SZp _i .

In a step S60, a data representation is created for a specific scenario SZc _i , in particular in the form of a vector in a latent vector space 570, the vector space 570 being spanned at least by the scenario parameters P _i of the parameterized scenario SZp _i .

In a step S70, a vector space analysis, in particular a cluster analysis, of the concrete scenarios SZc _i in the vector space 570 is carried out in relation to the concrete scenario parameters Pc _i .

In a step S80, output results 770 from the vector space analysis, in particular in the form of diagrams, tables, graphics, are generated and output with regard to the completeness and characteristics of the specific scenarios SZc _i in one or more data sets (320).

4 schematically illustrates a computer program product 1000 that includes an executable program code 1050 configured to execute the method according to the first aspect of the present invention.

The present invention checks existing data sets with a variety of different scenarios to determine whether a data set completely contains those relevant scenarios for the creation of a simulation environment for testing and training an ADAS/ADS system and/or a driving function are suitable for training the artificial intelligence-based algorithms of an ADAS/ADS system and/or a driving function. By selecting a complete training data set, the safety and functionality of a driving assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a specific driving task can be significantly improved.

Reference symbols

10

moving object

100

system

200

Sensor device

210

Sensors

220

Sensor data

250

Cloud computing infrastructure

270

Database

300

Input module

320

record

350

User interface

400

Scenario analysis module

450

Software application

470

Scenario database

500

Parameter analysis module

550

Software application

570

Vector space

700

Output module

750

Software application

770

Output data

1000

Computer program product

1050

Program code

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102019217952 A1 [0005]
• DE 112019001605 T5 [0006]
• DE 102019218127 A1 [0007]
• DE 102019213559 A1 [0008]

Claims (15)

Method for checking one or more data sets (320) for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function in at least one scenario (SZ _i ) with regard to the completeness of relevant scenarios ( SZ _i ), where a scenario (SZ _i ) represents a traffic event in a temporal sequence and is determined by a selection of scenario parameters (P ₁ , P ₂ , ... , P _n ) and associated scenario parameter values (PV ₁ , PV ₂ , .. ., PV _n ) is defined, and where in a parameterized scenario (SZp _i ) the scenario parameters (P ₁ , P ₂ , ... , P _n ) and associated scenario parameter values (PV ₁ , PV ₂ , ..., PV _n ) are freely selectable, and for a specific scenario (SZc _i ), the specific scenario parameters (Pc _i , Pc ₂ , ..., P _cn ) and associated specific scenario parameter values (PVc ₁ , PV _c2 , ..., PVc _n ) are defined are, and wherein a data set (320) contains a plurality of scenarios (SZ _i ), comprising: - Acquiring (S10) of sensor data (220) by sensors (210) of at least one sensor device (200) which is connected to at least one moving object (10) is connected when driving on one or more routes, the sensors (210) recording the sensor data (220) from the surroundings of the moving object (10); - Generating (S20) a plurality of data sets (320) at least from the sensor data (220) from an input module (300); - Identifying and classifying (S30) parameterized scenarios (SZp _i ) from the data sets (320) from a scenario analysis module (400); - Identifying and classifying (S40) concrete scenario parameters (Pc _i ) from the data sets (320) and/or sensor data (220) from a parameter analysis module (500); - Creating (S50) a data representation for the specific scenario parameters (Pc _i ) in the form of at least one vector in a latent vector space (570); - Creating (S60) a data representation for the parameterized scenarios (SZp _i ) in the form of at least one vector in a latent vector space (570); - Carrying out (S70) a vector space analysis, in particular by means of a cluster analysis of the parameterized scenarios (SZp _i ) in the vector space (570) in relation to the specific scenario parameters (Pc _i ); - Generate and output (S80) output results (770) from vector space analysis, especially in the form of diagrams, tables, graphics. Procedure according to Claim 1 , wherein the analyzed data sets (320) are provided with a suitability index based on the output results (770) with regard to suitability as a training data set for testing and training an ADAS/ADS system and/or a driving function. Procedure according to Claim 1 or 2 , wherein the scenario analysis module (400) comprises a software application (450) which uses calculation methods and/or algorithms of artificial intelligence, wherein the parameter analysis module (500) comprises a software application (550) which uses calculation methods and/or algorithms of artificial intelligence intelligence used; and wherein the output module (700) comprises a software application (750) that uses calculation methods and/or artificial intelligence algorithms. Procedure according to Claim 3 , where the calculation methods and/or algorithms of artificial intelligence are used as mean values, minimum and maximum values, lookup tables, models for expected values, linear regression methods, Gaussian processes, fast Fourier transformations, integral and differential calculations, Markov methods, probability methods such as Monte Carlo -Methods, temporal difference learning, extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, feedback neural networks, and / or folded neural networks are formed. Procedure according to one of the Claims 1 until 4 , where a parameter (P _i ) is a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a Number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a highway, a street sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic volume, a topographical structure such as a slope, a time, a temperature, a precipitation value, a weather and/or a season. Procedure according to one of the Claims 1 until 5 , whereby the sensors (210) are radar systems, LIDAR systems for optical distance and speed measurement, image-recording 2D/3D cameras in the visible, IR and/or UV range, GPS systems, acceleration sensors, speed sensors, capacitive sensors, inductive Sensors, voltage sensors, torque sensors, speed sensors, precipitation sensors and / or temperature sensors are formed. Procedure according to one of the Claims 1 until 6 , wherein the scenario analysis module (400), the parameter analysis module (500) and / or the output module (700) are integrated in a cloud computing infrastructure (250). Procedure according to one of the Claims 1 until 7 , wherein the sensor device (200) and/or the input module (300) is/are equipped with mobile radio modules of the 5G or 6G mobile radio standard for sending and receiving data. System (100) for checking one or more data sets (320) for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function in at least one scenario (SZ _i ) with regard to the completeness of relevant scenarios (SZ _i ), where a scenario (SZ _i ) represents a traffic event in a temporal sequence and through a selection of scenario parameters (P ₁ , P ₂ , ... , P _n ) and associated scenario parameter values (PV ₁ , PV ₂ , ..., PV _n ), and where in a parameterized scenario (SZp _i ) the scenario parameters (P ₁ , P ₂ , ... ,P _n ) and associated scenario parameter values (PV ₁ , PV ₂ , ... , PV _n ) are freely selectable, and for a specific scenario (SZc _i ) the specific scenario parameters (Pc _i , Pc ₂ , ..., P _cn ) and associated specific scenario parameter values (PVc ₁ , PV _c2 , ..., PVc _n ) are set, and wherein a data record (320) contains a plurality of scenarios (SZ _i ), comprising a sensor device (200), an input module (300), a scenario analysis module (400), a parameter analysis module (500) and an output module (700); wherein the input module (300) is designed to record sensor data (220) by sensors (210) of at least one sensor device (200), which is connected to at least one moving object (10), when driving on one or more routes, the sensors (210) record the sensor data (220) from the surroundings of the moving object (10), and generate a plurality of data sets (320) at least from the sensor data (220); wherein the scenario analysis module (400) is designed to identify and classify parameterized scenarios (SZp _i ) from the data sets (320) from a scenario analysis module (400); wherein the parameter analysis module (500) is designed to identify and classify specific scenario parameters (Pc _i ) from the data sets (320) and / or sensor data (220), and wherein the parameter analysis module (500) is designed for concrete scenario parameters (Pc _i ) and for the parameterized scenarios (SZp _i ) each to create a data representation in the form of at least one vector in a latent vector space (570), and a vector space analysis in particular by means of a cluster analysis of the parameterized scenarios (SZp _i ) in the vector space (570) in relation to the concrete scenario parameters (Pc _i ); and wherein the output module (700) is designed to generate and output output results (770) from the vector space analysis, in particular in the form of diagrams, tables, graphics. System (100) after Claim 9 , wherein the analyzed data sets (320) are provided with a suitability index based on the output results (770) with regard to suitability as a training data set for testing and training an ADAS/ADS system and/or a driving function. System (100) after Claim 9 or 10 , wherein the scenario analysis module (400) comprises a software application (450) which uses calculation methods and/or algorithms of artificial intelligence, wherein the parameter analysis module (500) comprises a software application (550) which uses calculation methods and/or algorithms of artificial intelligence intelligence used; and wherein the output module (700) comprises a software application (750) that uses calculation methods and/or artificial intelligence algorithms. System (100) after Claim 11 , where the calculation methods and/or algorithms of artificial intelligence are used as mean values, minimum and maximum values, lookup tables, models for expected values, linear regression methods, Gaussian processes, fast Fourier transformations, integral and differential calculations, Markov methods, probability methods such as Monte Carlo -Methods, temporal difference learning, extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, feedback neural networks, and / or folded neural networks are formed. System (100) according to one of the Claims 9 until 12 , where a parameter (P _i ) is a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a Number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a highway, a street sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic volume, a represents topographic structure such as a slope, a time, a temperature, a precipitation value, a weather and/or a season; and wherein the sensors (210) are used as radar systems, LIDAR systems for optical distance and speed measurement sung, image-recording 2D/3D cameras in the visible, IR and/or UV range, GPS systems, acceleration sensors, speed sensors, capacitive sensors, inductive sensors, voltage sensors, torque sensors, speed sensors, precipitation sensors and/or temperature sensors. System (100) according to one of the Claims 9 until 13 , wherein the scenario analysis module (400), the parameter analysis module (500) and / or the output module (700) are integrated in a cloud computing infrastructure (250), and wherein the sensor device (200) and / or the input module (300) is/are equipped with mobile radio modules of the 5G or 6G mobile radio standard for sending and receiving data. Computer program product (1000), comprising an executable program code (1050) which is configured so that, when executed, it implements the method according to one of Claims 1 until 8th executes.

Images