DE102022112060B3 - Scenario database for a method and a system for calibrating and validating a driver assistance system (ADAS) and/or an automated driving system (ADS) - Google Patents

Abstract

The invention relates to a scenario database (330) for carrying out a method for calibrating and validating a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a specified driving task in at least one scenario, one scenario involving a traffic event in a time sequence and is defined by a selection of scenario parameters and associated scenario parameter values, the scenario database (330) using a database structure with a table-based relational database model for storing data on scenario parameters, machine-readable scripts and other information, the database model having at least four Columns, with a first column (332) being provided at least for a designation of a scenario parameter and a designation of the associated parameterized scenario, with a second column (333) being provided at least for one parameter category, with a third column (334) being provided at least for one range of values of a scenario parameter such as a minimum value and a maximum value or a default value, and wherein a fourth column (335) is provided for at least information about an implementation method for the scenario parameters in a simulation.

Description

The invention relates to a scenario database for a method and a system for calibrating and validating a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a specified driving task in at least one scenario.

Modern vehicles are equipped with a large number of driver assistance systems or automated driver assistance functions in order to support the driver when driving and to increase his safety. Driving assistance systems support, for example, speed and distance control as well as lane keeping and lane changing functions. A certain maximum speed can be set here, which will not be exceeded as long as the speed limit function is activated. Radar sensors, but also camera systems, are used for distance control, in which a specific distance is set, in particular from a vehicle driving ahead. This allows the distance to vehicles in front, but also to vehicles in the side area, to be monitored. This leads to improved driving comfort and greater safety, especially when driving on the freeway and when overtaking.

However, this trend towards driver assistance systems (Advanced Driver Assistance System, ADAS), automated driving systems (Automated Driving Systems, ADS) in motor vehicles, but also in aircraft or watercraft, requires extensive security strategies, since the responsibility for driving the vehicle no longer lies unrestrictedly with the driver , but active functions are taken over by computing units. It must therefore be ensured that autonomously moving objects only have a very low error rate in driving behavior. The recognition and classification of objects and the interpretation of traffic scenarios in the area surrounding a vehicle are important prerequisites for the secure functionality of ADAS/ADS. This requires targeted testing and training of the driver assistance systems and automated driving systems both with extreme and exceptional situations (corner cases) as well as with everyday situations. Such extreme situations result from a special combination of different factors. Examples of this are infrastructural features such as the type of road, the development on the side of a road, the quality of the markings, but also environmental conditions such as weather conditions, the time of day and the season. The behavior of other road users, the geographical topography and the weather conditions also play a major role.

However, with the increasing performance of driver assistance systems (ADAS) and automated driving systems (ADS), the number of driving scenarios that have to be managed by the systems in road traffic is also increasing. To ensure that ADAS/ADS behaves safely, comfortably and efficiently, individual functions and the entire system undergo a calibration and validation process during vehicle development.

However, such a calibration and validation also poses a major challenge when integrating modern ADAS/ADS into a vehicle due to a functional specification deficit for driver assistance systems. While a requirements-based test process has been established for conventional systems in the automotive industry, in which test cases are created using test specifications, this has so far been lacking for ADAS/ADS, since in contrast to conventional systems, a significantly larger number of influencing variables have to be taken into account, such as the driving environment in particular , which is detected by sensors. The set of scenarios that can occur in the driving environment of the vehicle and that must be correctly recognized and processed by ADAS/ADS is represented by an Operational Design Domain (ODD). This includes everyday driving scenarios as well as very rare corner cases. In order to test, calibrate and validate an ADAS/ADS using conventional methods, the entire ODD for an ADAS/ADS would have to be recorded and documented in a requirements catalogue. This is not possible due to the complexity of the driving environment and the resulting large number of driving scenarios. This problem is known as a functional specification deficit. It complicates both the ADAS/ADS calibration and validation process and requires alternative approaches to the existing methods.

Due to the challenges in the calibration and validation of ADAS/ADS, new approaches are needed in addition to the known methods in order to carry out the vehicle development process with justifiable effort and acceptable costs. It is therefore desirable to increasingly use virtual methods to simulate the calibration and validation of ADAS/ADS and to design these virtual simulation methods in such a way that an objective comparability of different ADAS/ADS systems with different functionalities in terms of performance and security is possible. The challenges arise in particular from the large number of scenarios that simulation methods used and the selection of suitable metrics for evaluating the simulation results.

In addition, a scenario represents a machine-readable data object that includes a large number of characteristics and properties. So far, this data has been stored in a database in a very unstructured manner, so that it is often difficult to specifically select the appropriate scenario from the large number of stored scenarios and to find the information about a scenario in the database that is necessary for a virtual calibration and validation of ADAS/ADS .

The DE 10 2017 009 971 A1 discloses a method for testing a lane departure warning system for a motor vehicle, using an event-based data mining method to generate logical scenarios, which are included in a systematic test case generation with a generic model for a scenario description.

The DE 10 2020 005 507 A1 discloses a method for testing an automated driving function, in particular an emergency braking function, with measurement data from country-specific field tests being stored in a database and information being extracted from the database using an event-driven time series analysis and clustering, with the extracted clusters and their parameter space having a probability of occurrence for each logical scenario and define a probability distribution of associated parameters.

The DE 10 2021 100 149 A1 discloses a method for scenario-based testing of systems and system components for autonomous driving of a motor vehicle, a specific scenario being formed from a logical scenario without specific parameter data by selecting the parameter data.

The U.S. 2021/179124 A1 discloses a system for monitoring the proper behavior of an autonomous motor vehicle, wherein a plurality of agents are created, each describing a physical object, and wherein a scenario models the behavior of the agents.

The DE 10 2018 205 804 A1 discloses a control unit test device for testing, securing and/or developing a function of a driver assistance system or an autonomous vehicle. The control unit test device has a database in which sensor output data from sensors in the vehicle are each assigned to a specific environmental scenario, and a computing unit.

The object on which the invention is based is now to specify a database structure for scenarios for virtual calibration and validation of a driver assistance system (ADAS) and/or an automated driving system (ADS), so that the calibration and validation process requires less time and with an improved efficiency can be carried out in order to increase the security of ADAS/ADS and to be able to save resources and costs.

This object is achieved according to the invention with regard to a scenario database for a method by the features of patent claim 1 and with regard to a scenario database for a system by the features of patent claim 8 . The further claims relate to preferred embodiments of the invention.

The efficiency and security of calibration and validation tasks for ADAS/ADS is improved by an optimized database structure according to the invention of a database for scenarios.

According to a first aspect, the invention provides a scenario database for performing a method for calibrating and validating a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a specified driving task in at least one scenario. A scenario represents traffic events in a time sequence and is defined by a selection of scenario parameters and associated scenario parameter values, and with a parameterized scenario the scenario parameters and associated scenario parameter values can be freely selected, and in a concrete scenario the concrete scenario parameters and associated concrete scenario parameter values are fixed. The scenario database uses a database structure with a table-based relational database model for storing data on scenario parameters, machine-readable scripts and other information, the database model comprising at least four columns, with a first column being provided at least for a description of a scenario parameter and a description of the associated parameterized scenario is provided, wherein a second column is provided at least for a parameter category, wherein a third column is provided at least for a value range of a scenario parameter such as a minimum value and a maximum value or a default value, and wherein a fourth column is provided at least for information about an implementation method for the scenario parameters in a simulation is provided.

In an advantageous development, a fifth column is/are provided for storing dynamic labels for identifying parameterized scenarios, scenario parameters and/or groups of scenario parameters and/or a sixth column is provided for storing information about the stochastic distribution of scenario parameters in real traffic scenarios .

• - Erstellen von Testfällen von einem Testagenten durch Auswählen von parametrisierten Szenarien und Szenarioparametern aus der Szenariendatenbank sowie Kalibrierungsparametern aus einer Kalibrierungsparameterdatenbank mittels einer Teststrategie für die Fahraufgabe;
• - Weitergeben der ausgewählten Testfälle an ein Simulationsmodul; und
• - Durchführen einer Simulation von dem Simulationsmodul zur Bestimmung von Simulationsergebnissen.

In an advantageous embodiment, the method for calibrating and validating a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a specified driving task in at least one scenario includes the following method steps:

• - Creation of test cases by a test agent by selecting parameterized scenarios and scenario parameters from the scenario database and calibration parameters from a calibration parameter database using a test strategy for the driving task;
• - passing the selected test cases to a simulation module; and
• - Performing a simulation by the simulation module to determine simulation results.

In a further embodiment, it is provided that the parameterized scenarios are generated from measurement data from sensors, with the sensors recording a traffic situation, driving behavior and traffic objects in the vicinity of a motor vehicle, and with a software application being used to identify and classify parameterized scenarios from the measurement data.

In particular, the sensors used are radar systems, LIDAR systems for optical distance and speed measurement, image-recording 2D/3D cameras in the visible, IR and/or UV range, GPS systems, acceleration sensors, speed sensors, capacitive sensors, inductive sensors, Voltage sensors, precipitation sensors and / or temperature sensors formed.

The software application advantageously uses artificial intelligence algorithms to identify and classify the parameterized scenarios.

In particular, a scenario parameter and/or a calibration parameter represents a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as an automobile, person, or cyclist, an immovable object such as a building or tree, a road configuration such as a freeway, a road sign, a traffic light, a tunnel, a roundabout, a turning lane, a volume of traffic, represents a topographical structure such as an incline, a time, a temperature, a precipitation value, a weather condition and/or a season.

According to a second aspect, the invention provides a system for calibrating and validating a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a specified driving task in at least one scenario. A scenario represents traffic events in a time sequence and is defined by a selection of scenario parameters and associated scenario parameter values, with the scenario parameters and associated scenario parameter values being freely selectable in a parameterized scenario, and with a concrete scenario the concrete scenario parameters and associated concrete scenario parameter values being fixed . The scenario database uses a database structure with a table-based relational database model for storing data on scenario parameters, machine-readable scripts and other information, the database model comprising at least four columns, with a first column being provided at least for a description of a scenario parameter and a description of the associated parameterized scenario is provided, wherein a second column is provided at least for a parameter category, wherein a third column is provided at least for a value range of a scenario parameter such as a minimum value and a maximum value or a default value, and wherein a fourth column is provided at least for information about an implementation method for the scenario parameters in a simulation is provided.

In one development, a fifth column is/are provided for storing dynamic labels for identifying parameterized scenarios, scenario parameters and/or groups of scenario parameters and/or a sixth column for storing information about the stochastic distribution of scenario parameters in real traffic scenarios.

An advantageous embodiment provides that the system comprises at least one test module with a test agent and a test strategy and a simulation module, the test agent being designed to test cases by selecting parameterized scenarios and scenario parameters from the scenario database and calibration to create test parameters from a calibration parameter database using the test strategy for the driving task and to forward the selected test cases to the simulation module; wherein the simulation module is designed to carry out a simulation to determine simulation results.

In particular, the parameterized scenarios are generated from measurement data from sensors, the sensors recording a traffic situation, driving behavior and traffic objects in the vicinity of a motor vehicle, and parameterized scenarios being identified and classified from the measurement data using a software application.

Advantageously, the sensors used are radar systems, LIDAR systems for optical distance and speed measurement, image-recording 2D/3D cameras in the visible, IR and/or UV range, GPS systems, acceleration sensors, speed sensors, capacitive sensors, inductive sensors, Voltage sensors, precipitation sensors and / or temperature sensors formed.

In a further embodiment, the software application uses artificial intelligence algorithms to identify and classify the parameterized scenarios.

In particular, a scenario parameter and/or a calibration parameter represents a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as an automobile, person, or cyclist, an immovable object such as a building or tree, a road configuration such as a freeway, a road sign, a traffic light, a tunnel, a roundabout, a turning lane, a volume of traffic, represents a topographical structure such as an incline, a time, a temperature, a precipitation value, a weather condition and/or a season.

The invention is explained in more detail below with reference to exemplary embodiments illustrated in the drawing.

• 1 ein Blockdiagramm zur Erläuterung eines Ausführungsbeispiels eines erfindungsgemäßen Systems;
• 2 eine schematische Darstellung eines Graphen;
• 3 eine schematische Darstellung einer Szenariendatenbank.

It shows:

• 1 a block diagram to explain an embodiment of a system according to the invention;
• 2 a schematic representation of a graph;
• 3 a schematic representation of a scenario database.

Additional characteristics, aspects and advantages of the invention or its embodiments are explained in the following description in connection with the claims.

Simulated traffic scenarios created by programming are increasingly used for testing, training and validating driver assistance systems (ADAS) and automated driving systems (ADS). A traffic event in a time sequence is referred to as a scenario within the scope of the invention. An example of a scenario is entering a freeway overpass, turning into a branch lane, driving through a tunnel, entering a roundabout, or stopping at a pedestrian crossing. In addition, specific visibility conditions, for example due to twilight or high levels of sunlight, as well as environmental conditions such as the weather and time of year, traffic volume and certain geographical topographical conditions can influence a scenario. For example, an overtaking maneuver can be described as a scenario where a first vehicle is initially behind another vehicle, then changes lanes to the other lane and increases speed to overtake the other vehicle. Such a scenario is also referred to as a cut-in scenario. The time of year and the weather conditions also play a role, since the road conditions look different when it rains and is icy than on a sunny summer day.

In order to be able to use driver assistance systems (ADAS) and automated driving systems (ADS) in a motor vehicle, they must be calibrated and validated for reliable use. The calibration is used to adapt the ADAS/ADS or individual driving functions to the respective vehicle type and to the desired behavior of a driving function without changing the software code. For this purpose, calibration parameters are modified and made available to the ADAS/ADS in a data record. It is necessary to find a suitable data set of calibration parameters for optimal ADAS/ADS behavior or a selected driving function in as many operational design domain (ODD) scenarios as possible.

1 shows a system 100 for the simulated calibration and validation of a driver assistance system (ADAS) and/or an automated driving system (ADS) with a scenario database 330 according to the invention Furthermore, a test agent module 200 with a test agent 220 and a test strategy 230, a calibration parameter database 320, an evaluation database 340, a test database 300, a simulation module 400, an evaluation module 500 and an output module 700, each of which can be provided with a processor and/or a memory unit .

In connection with the invention, a “module” can be understood to mean, for example, a processor and/or a processor unit and/or a memory unit for storing program instructions. The processor is specifically set up to execute the program instructions in such a way as to implement or realize the method according to the invention or a step of the method according to the invention. In particular, a module can be integrated in a cloud computing infrastructure.

In the context of the invention, a “processor” can be understood to mean, for example, a machine or an electronic circuit. A processor can in particular be a central processing unit (CPU), a microprocessor or a microcontroller, for example an application-specific integrated circuit or a digital signal processor, possibly in combination with a memory unit for storing program instructions, etc . A processor can also be understood to mean a virtualized processor, a virtual machine or a soft CPU. It can also be a programmable processor, for example, which is equipped with configuration steps for executing the mentioned method according to the invention or is configured with configuration steps in such a way that the programmable processor has the inventive features of the method, the system, the modules, or other aspects and/or or implemented partial aspects of the invention. In particular, the processor can contain highly parallel computing units and powerful graphics modules.

In the context of the invention, a “memory unit” or a “memory module” can mean, for example, a volatile memory in the form of a random-access memory (RAM) or a permanent memory such as a hard disk or a data carrier or e.g. B. be understood as a removable memory module. However, the storage module can also be a cloud-based storage solution.

The test agent 220 of the test agent module 200 uses a software application to create a plurality of test cases T _i for one or more driving task(s). The respective driving task is formulated before the start of the simulation, for example by an expert such as an engineer. However, it can also be provided that a list of driving tasks is generated by a software application. This list can then be successively processed by the system 100. An exemplary driving task is changing lanes on a freeway.

The test strategy 230, which specifies how the test agent 220 creates the test cases T _i , is provided for the selection and design of the test cases T _i . Various calculation methods and algorithms, in particular artificial intelligence algorithms, can be provided for determining the test strategy 230 .

The test agent 220 takes the information required for the creation of the test cases T _i from the calibration parameter database 320, the scenario database 330 and the evaluation database 340. Provision can be made within the scope of the invention that further databases are also used. The created test cases T _i are stored in the test database 300 .

"Database" means both a storage algorithm and the hardware in the form of a storage unit. In particular, the databases 300, 320, 330, 340 can be designed as part of a cloud computing infrastructure.

The test cases T _i are created on the basis of parameterized scenarios, which are also referred to as logical scenarios. Within the scope of the invention, a scenario that is written in particular in a machine-readable code is referred to as a parameterized scenario. A parameterized scenario SZp is defined by various scenario parameters P ₁ , P ₂ , ..., P _n from a set of possible scenario parameters P _i and associated scenario parameter values PV ₁ , PV ₂ , ..., PV _n from a set of possible scenario parameter values PV _i defined, with scenario parameter values PV _i specifying the value range of a scenario parameter P _i . A scenario parameter P _i represents, for example, a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a freeway, a road sign, a traffic light, a tunnel, a roundabout, a turning lane, a volume of traffic, a topographical structure like a gradient, a clock time, a temperature, a precipitation value, a weather condition and/or a season.

Scenario parameters P _i thus characterize properties and features of a scenario within the scope of the present invention.

A parameterized scenario SZp includes a chronological sequence of time intervals Δt ₁ , Δt ₂ . . . . , Δt _n , in which different scenes and events take place. A parameterized scenario SZp begins with an initial scene and then develops further as events occur, from which new subsequent scenes emerge over time. The start scene is thus changed by one or more events. An event can be either a reaction actively triggered by a road user, such as acceleration, or an event that recurs cyclically, such as the switching operations of a traffic light. The start scene and the individual subsequent scenes thus each include only a small time interval Δt or a snapshot, while a parameterized scenario SZp includes a longer period of time. In a graph representation of a scenario SZp, the events can be represented as edges and the individual scenes as nodes of a graph, as is shown in 2 is illustrated

Within the scope of the present invention, a distinction is made between a parameterized scenario SZp and a concrete scenario SZc. Within the scope of this invention, a scenario is defined as a parameterized scenario SZp in which both the scenario parameters P _i and the associated scenario parameter values PV _i are not all specified. By choosing concrete scenario parameters Pc _i and associated scenario parameter values PVc _i or value ranges of the scenario parameter values PVc _i , the parameterized scenario SZp becomes a concrete scenario SZc. In a specific scenario SZc, the specific scenario parameters Pc _i and associated specific scenario parameter values PVc _i or value ranges of the scenario parameter values PVc _i are thus specified. Both scenarios, both the parameterized scenario SZp and the concrete scenario SZc, are scenarios that are written in particular in a machine-readable code.

Various sources such as requirement specifications, expert knowledge and/or measurements on public roads or on test sites using sensors that record a traffic situation, driving behavior and the traffic in the vicinity of an ego vehicle can be used to create parameterized scenarios SZp _i . The sensors used can be designed in particular as radar systems, LIDAR systems for optical distance and speed measurement, image-recording 2D/3D cameras in the visible, IR and/or UV range, and/or GPS systems. Furthermore, acceleration sensors, speed sensors, capacitive sensors, inductive sensors, voltage sensors, precipitation sensors and/or temperature sensors, etc. can be used. Using a software application, a suitable parameterized scenario SZp _i can be derived from the recorded sensor signals and data at a specific geographic location. In particular, the software application uses artificial intelligence algorithms to identify and classify the parameterized scenarios SZp _i . The artificial intelligence algorithms can in particular be encoders and decoders with neural networks.

A neural network consists of neurons that are arranged in several layers and connected to each other in different ways. A neuron is able to receive information from outside or from another neuron at its input, to evaluate the information in a certain way and to pass it on to another neuron in a changed form at the neuron output or to output it as the end result. Hidden neurons are located between the input neurons and output neurons. Depending on the network type, there can be several layers of hidden neurons. They ensure the forwarding and processing of the information. Finally, output neurons deliver a result and output it to the outside world. Neural networks can be trained through unsupervised or supervised learning.

The different arrangement and connection of the neurons results in different types of neural networks, such as a feedforward neural network (FNN), a recurrent neural network (RNN) with feedback or a folded neural network .Convolutional Neural Network, CNN). A folded neural network has multiple layers of convolution and is very well suited for machine learning and applications in the field of pattern recognition and image recognition. Since a large part of the data recorded by the sensors is available as images, folded neural networks (CNN) in particular are used.

The parameterized scenarios SZp _i and scenario parameters P _i are stored in the scenario database 330 . According to the invention, the database structure of the scenario database 330 is based on a table-based relational database model in which data on scenario parameters P _i , machine-readable scripts and other information are stored. For example, the database language SQL (Structured Query Language), which uses methods of relational algebra, can be used to query the data.

Like in the 3 shown, the scenario database 330 according to the invention comprises at least four columns for storing the scenario parameters P _i of a parameterized scenario SZp _i and the collection of machine-readable scripts for the implementation of the test agent 220 selected parameterized scenarios SZp _i in the simulation module 400. A first column 332 contains the designation a scenario parameter P _i and the designation of the associated parameterized scenario SZp _i . An example of a scenario parameter P _i is the speed of an ego vehicle during a cut-in scenario. A second column 333 contains the parameter category, which can relate to the driving mode, a traffic situation, an infrastructure, a topology, other road users, etc., for example. There are different models for parameter categories, such as a layered model with a plurality of layers that relate to different aspects of a scenario. A third column 334 contains the value ranges of a scenario parameter P _i such as a minimum value and a maximum value for a continuous scenario parameter P _i or a default value for a discrete scenario parameter. A continuous scenario parameter P _i is, for example, a speed that can change during a traffic scenario, and an example of a discrete scenario parameter P _i is the width of a lane. The fourth column 335 includes information for the simulation module 400 for performing the simulation and relates, for example, to the implementation method of the scenario parameters P _i in a simulation.

In addition to these four columns, a further fifth column 336 can be provided for storing dynamic labels which are assigned by the test agent 220. Examples of labels are weights to weight parameterized scenarios SZp _i , scenario parameters P _i or groups of scenario parameters P _i with regard to various criteria. Furthermore, a sixth column 337 can be provided, in which information about the stochastic distribution of the scenario parameters P _i in real traffic scenarios is stored.

The scenario database 330 thus contains all the necessary parameter data and machine-readable scripts for selecting the required parameterized scenarios SZp _i for test cases T _i . The test agent 220 can access the parameterized scenarios SZp _i and scenario parameters P _i in order to create the required test cases T _i for the simulation module 400 . The table-based relational database model of the scenario database 330 allows all the necessary information to be called up easily. The test agent 220 can thus efficiently create those test cases T _i from the set of possible parameterized scenarios SZp _i that are of interest for the calibration and validation of ADAS/ADS by selected parameterized scenarios SZp _i and suitable scenario parameters P _i from the Scenario database 240 and a data set of calibration parameters Pcal _i from the calibration parameter database 230 are determined. The calibration parameters Pcal _i can also be a physical variable, a chemical variable, a torque, a speed, a voltage, an amperage, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location , a number, a moving object such as a motor vehicle, person, or cyclist, an immovable object such as a building or tree, a road configuration such as a freeway, a road sign, a traffic light, a tunnel, a roundabout, a turning lane, a volume of traffic , a topographical structure such as an incline, a time, a temperature, a precipitation value, a weather condition and/or a season.

These relevant test cases T _i created by the test agent 220 cover the relevant traffic situations defined in the “Operational Design Domain” (ODD) which driver assistance systems and automated driving functions have to be able to handle. A test case T _i is therefore a relevant specific scenario SZc by specifying specific scenario parameters Pc _i and a data set of calibration parameters Pcal _i for the ADAS/ADS or the driving function to be tested.

In particular, it is provided that the database structure of the calibration parameter database 320 is also based on a table-based relational database model. Columns are provided for the parameter designation of the calibration parameters Pcal _i , the parameter category, the value range of a calibration parameter Pcal _i , the implementation method and dynamic labels in which data on the calibration parameters Pcal _i , machine-readable scripts and other information are stored. The test agent 220 and the simulation module 400 can access the calibration parameter database 320 .

The test strategy 230 of the test agent 220 thus decides on the configuration of the test cases T _i with selected combinations of parameterized scenarios SZp _i and concrete scenario parameters Pc _i as well as further calibration parameters Pcal _i for the ADAS/ADS to be tested or the driving function to be tested. The test strategy 230 used is specified by the purpose of the simulation, ie what knowledge is to be gained about the behavior of an ADAS/ADS or a specific driving function when performing the specified driving task in a simulation. For example, for the virtual validation of an ADAS/ADS with a constant data set of calibration parameters Pcal _i . various parameterized scenarios SZp _i and scenario parameter combinations are created using a combinatorial method. Furthermore, an iterative strategy based on mathematical optimizers can be implemented to adjust the calibration parameters Pcal _i for a smaller set of different parameterized scenarios Szp _i . In this way, the behavior of the ADAS/ADS or the selected driving function can be optimized for a small number of parameterized scenarios Szp _i .

The test agent 220 stores the test cases T _i determined in this way in the test database 300. It can be provided that the database structure of the test database 300 is also based on a table-based relational database model. Thus, the data for a test case T _i can contain a test identification number (test ID), a user name, a creation time and the designation for the selected parameterized scenario Szp _i . Furthermore, those concrete scenario parameters Pc _i can be stored which transform the parameterized scenario Szp _i into a concrete scenario Szc _i . In addition, the relevant data set of calibration parameters Pcal _i for the ADAS/ADS to be tested or the driving function to be tested can be stored in the test database 300 . In addition, further data such as performance indicators (KPIs) with regard to the evaluation of the performance of the respective ADAS/ADS or the respective driving function during the simulation as well as evaluation indicators for the evaluation of the simulation carried out can be stored. Furthermore, information on the status of the simulation such as "carried out or not carried out" can be assigned to a test case T _i . In addition, further text messages, video sequences and/or audio sequences, etc. can be stored with further information. Overall, all relevant information about a test case T _i can thus be stored in the test database 300, with the information being arranged in appropriately provided columns of the database model.

To carry out a simulation of the test cases T _i created, the test agent 220 transfers a test case T _i or several test cases T _i to the simulation module 400. In addition, the simulation module 400 can receive information about the calibration parameters Pcal _i from the calibration parameter database 320 and information about the parameterized scenarios SZp _i and the scenario parameters P _i like machine-readable scripts from the scenario database 330 in order to be able to carry out the simulation. The simulation module 400 includes various sub-modules 410, 420, 430, which are designed for the simulation of specific properties of the test cases T _i and can interact with one another. They are designed as interchangeable sub-modules and have the task of simulating individual aspects of a respective test case T _i or a plurality of test cases T _i . A first sub-module 410 can be embodied as an environment model module that relates to the environment and depicts different environments of a motor vehicle 10 . A second sub-module 420 can be embodied as a driver model module that relates to the driving mode or driving style of a driver, such as dynamic or defensive. A third sub-module 430 can be embodied as a vehicle model module that includes different models of a motor vehicle, for example with regard to the different design of the drive train, the driving dynamics and other sub-functions. The sub-modules 410, 420, 430 implement the description of the respective test case T _i for a specific situation or configuration and provide the necessary input data for simulating a driver assistance function in a driving function module 440.

The simulation module 400 or the various sub-modules 410, 420, 430 are connected to one or more databases 480 in which sensor signals from sensors 470 and other data are stored.

In addition, further historical data in the form of images, graphics, time series, parameters, etc. can be stored in the database 480 . For example, target variables and target values can be stored in the database 480, which define a safety standard for the simulation. The database 480 can also be integrated in a cloud computing infrastructure.

The simulation module 400 thus carries out a simulation of the test cases T _i for a specified driving task and transfers the simulation results 450 to the evaluation module 500.

The evaluation module 500 evaluates the simulation results 450 in terms of the performance and operability of one or more Functions or the overall performance of a driver assistance system (ADAS) or an automated driving system (ADS), in particular in the form of performance indicators (KPIs). In addition, the quality of the simulation process carried out is evaluated, in particular in the form of simulation quality criteria (SQCs). The evaluation module 500 creates evaluation results 550 from these evaluations. The test agent 220 can provide the calibration parameters Pcal _i in the calibration parameter database 320, the parameterized scenarios Szp _i and the scenario parameters P _i in the scenario database 330 and the evaluation results 550 with labels such as weights or to identify special properties in order to create new test cases T _k for the to create a specified driving task.

In particular, it is provided that the database structure of the evaluation database 340 is also based on a table-based relational database model and in particular the KPIs and SQCs of the evaluation results 550 are stored in correspondingly provided columns.

In particular, the test strategy 230 can be adjusted using the data in the calibration parameter database 320 , the scenario database 330 and the evaluation database 340 which have been changed by the evaluation results 550 . The new test cases T _k are in turn stored in the test database 300 and can be used for a new simulation cycle in the simulation module 400 and a subsequent evaluation in the evaluation module 500 .

The combination of the simulation module 400 and the evaluation module 500 thus forms the basis for processing test cases T _i as part of the virtual calibration and validation. The test cases T _i created by the test agent 220 are simulated in the simulation module 400, the simulation module 400 having access to the calibration parameter database 320 and the scenario database 330 in order to obtain the necessary information. After a successful simulation, the simulation results 450 are transferred to the assessment module 500 . The assessment module 500 has access to the assessment database 340 and the scenario database 330 to retrieve needed information. Both the simulation module 400 and the evaluation module 500 contain exchangeable sub-modules, so that an adaptation to different functions and use cases as well as the separate further development of the sub-modules is possible.

After running through a first simulation cycle with first test cases T _i , second test cases T _k are created by the test agent 220 using the test strategy 230 adapted to the simulation results 450 and the calculation results of the first simulation run. A new simulation run is started with these second test cases T _k . Based on the evaluation results 550, a decision is made as to whether the second test cases T _k meet a specific evaluation criterion or whether a further adaptation of the test strategy 230 is required for the implementation of a further simulation cycle. If the evaluation results 550 show that the simulation results 450 converge, the test cases T _k of the last simulation cycle are forwarded to the output module 700 .

On the basis of the evaluation results 550 created by the evaluation module 500 in the form of KPIs and SQCs, an optimal data set of calibration parameters _{Pcal i for} the driver assistance system (ADAS) and/or the automated driving system (ADS) and /or the driving function for carrying out the specified driving task can be determined. This data set of calibration parameters Pcal _i defines the behavior of the driver assistance system (ADAS) and/or the automated driving system (ADS) and/or the driving function for the specified driving task and can then be output by the output module 700 as output results 750 . In addition, output results 750 include information about the simulation performed. The quality of the output results 750 depends on the quality of the scenario description and the type of simulation as well as the definition of the evaluation metrics.

The database structure of the scenario database 330 according to the invention allows the features and properties of a parameterized scenario SZp _i to be stored efficiently and clearly, so that a specific parameterized scenario Szp _i that is required for creating a test case T _i can be easily found with a simple search query . This simplifies the implementation of the virtual calibration and validation of an ADAS/ADS, so that resources can be saved, since both the real driving of test routes with standard traffic situations and with specific corner cases can be reduced.

Reference List

10

motor vehicle

100

system

200

test agent module

220

test agent

230

test strategy

300

test database

320

calibration parameter database

330

scenario database

332

first column

333

second column

334

third column

335

fourth column

336

fifth column

337

sixth column

340

rating database

400

simulation module

410

environment model module

420

driver model module

430

vehicle model module

440

driving function module

450

simulation results

470

sensors

480

Database

500

rating module

510

Feature Evaluation Module

520

simulation evaluation module

550

assessment results

700

output module

750

output data

900

computer program product

950

program code

Claims (14)

Scenario database (330) for carrying out a method for calibrating and validating a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a specified driving task in at least one scenario (SZ _i ), wherein a scenario (SZ _i ) represents a traffic event in a time sequence and is defined by a selection of scenario parameters (P ₁ , P ₂ , ..., P _n ) and associated scenario parameter values (PV ₁ , PV ₂ , ..., PV _n ), and with a parameterized scenario (SZp _i ) the scenario parameters (P ₁ , P ₂ , ..., P _n ) and associated scenario parameter values (PV ₁ , PV ₂ , ..., PV _n ) being freely selectable, and with a concrete Scenario (SZc _i ) the specific scenario parameters (Pc ₁ , Pc ₂ , ..., P _cn ) and associated specific scenario parameter values (PVc ₁ , PVc ₂ , ..., PVc _n ) are specified, the scenario database (330) having a Database structure with a table-based relational database model for storing data on scenario parameters (P _i ), machine-readable scripts and other information used, the database model comprising at least four columns, a first column (332) at least for a designation of a scenario parameter (P _i ) and a designation of the associated parameterized scenario (SZp _i ) is provided, with a second column (333) being provided at least for one parameter category, with a third column (334) being provided at least for a value range of a scenario parameter (P _i ) such as a minimum value and a maximum value or a default value is provided, and wherein a fourth column (335) is provided at least for information about an implementation method for the scenario parameters (P _i ) in a simulation. scenario database (330). claim 1 , wherein a fifth column (336) for storing dynamic labels for identifying parameterized scenarios (SZp _i ), scenario parameters (P _i ) and/or groups of scenario parameters (P _i ) and/or a sixth column (337) for storing information about the stochastic distribution of scenario parameters (P _i ) in real traffic scenarios is provided. scenario database (330). claim 1 or 2 , wherein the method for calibrating and validating a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a specified driving task in at least one scenario (SZ _i ) comprises the following method steps: - Creation of test cases (T _i ) by a test agent (220) by selecting parameterized scenarios (SZp _i ) and scenario parameters (P _i ) from the scenario database (330) and calibration parameters (Pcal _i ) from a calibration parameter database (320) using a test strategy (230) for the driving task ; - Passing on the selected test cases (T _i ) to a simulation module (400); and - performing a simulation by the simulation module (400) to determine simulation results (450). Scenario database (330) according to one of the Claims 1 until 3 , wherein the parameterized scenarios (SZp _i ) are generated from measurement data from sensors, the sensors recording a traffic situation, driving behavior and traffic objects in the vicinity of a motor vehicle (10), and using a software application scenarios (SZp _i ) parameterized from the measurement data can be identified and classified. scenario database (330). claim 4 , the sensors used being in particular radar systems, LIDAR systems for optical distance and speed measurement, image-recording 2D/3D cameras in the visible, IR and/or UV range, GPS systems, acceleration sensors, speed sensors, capacitive sensors, inductive sensors , Voltage sensors, precipitation sensors and/or temperature sensors are formed. scenario database (330). claim 4 or 5 , whereby the software application uses artificial intelligence algorithms to identify and classify the parameterized scenarios (SZp _i ). Scenario database (330) according to one of the Claims 1 until 6 , wherein a scenario parameter (P _i ) and/or a calibration parameter (Pcal _i ) is a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, a Angle, a radius, a location, a number, a moving object like an automobile, a person or a cyclist, an immovable object like a building or a tree, a road configuration like a freeway, a road sign, a traffic light, a tunnel, a roundabout, a turning lane, a volume of traffic, a topographical structure such as an incline, a time, a temperature, a precipitation value, a weather condition and/or a season. Scenario database (330) for a system (100) for calibrating and validating a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a specified driving task in at least one scenario (SZ _i ), wherein a scenario ( SZ _i ) represents a traffic event in a time sequence and is defined by a selection of scenario parameters (P ₁ , P ₂ , ..., P _n ) and associated parameter values (PV ₁ , PV ₂ , ..., PV _n ), and wherein in a parameterized scenario (SZp _i ) the scenario parameters (P ₁ , P ₂ , ..., P _n ) and associated scenario parameter values (PV ₁ , PV ₂ , ..., PV _n ) can be freely selected, and in a specific scenario (SZc _i ), the specific scenario parameters (Pc ₁ , Pc ₂ , ..., Pc _n ) and associated specific scenario parameter values (PVc ₁ , PVc ₂ , ..., PVc _n ) are specified, the scenario database (330) uses a database structure with a table-based relational database model for storing data on scenario parameters (P _i ), machine-readable scripts and other information, the database model comprising at least four columns, a first column (332) being used at least for a description of a scenario parameter (P _i ) and a designation of the associated parameterized scenario (SZp _i ) is provided, with a second column (333) being provided at least for one parameter category, with a third column (334) being provided at least for a value range of a scenario parameter (P _i ) such as a minimum value and a maximum value or a default value is provided, and wherein a fourth column (335) is provided at least for information about an implementation method for the scenario parameters (P _i ) in a simulation. scenario database (330). claim 8 , wherein a fifth column (336) for storing dynamic labels for identifying parameterized scenarios (SZp _i ), scenario parameters (P _i ) and/or groups of scenario parameters (P _i ) and/or a sixth column (337) for storing information about the stochastic distribution of scenario parameters (P _i ) in real traffic scenarios is provided. scenario database (330). claim 8 or 9 , wherein the system (100) comprises at least one test module (200) with a test agent (220) and a test strategy (230) and a simulation module (400), wherein the test agent (220) is designed to test cases (T _i ) by selecting to create parameterized scenarios (SZp _i ) and scenario parameters (P _i ) from the scenario database (330) and calibration parameters (Pcal _i ) from a calibration parameter database (320) using the test strategy (230) for the driving task and the selected test cases (T _i ). pass the simulation module (400); wherein the simulation module (400) is designed to carry out a simulation to determine simulation results (450). Scenario database (330) according to one of the Claims 8 until 10 , the parameterized scenarios (SZp _i ) being generated from measurement data from sensors, the sensors recording a traffic situation, driving behavior and traffic objects in the vicinity of a motor vehicle (10), and scenarios (SZp _i ) parameterized from the measurement data using a software application identified and classified. scenario database (330). claim 11 , the sensors used being in particular radar systems, LIDAR systems for optical distance and speed measurement, image-recording 2D/3D cameras in the visible, IR and/or UV range, GPS systems, acceleration sensors, speed sensors, capacitive sensors, inductive sensors , voltage sens Sensors, precipitation sensors and / or temperature sensors are formed. scenario database (330). claim 11 or 12 , whereby the software application uses artificial intelligence algorithms to identify and classify the parameterized scenarios (SZp _i ). Scenario database (330) according to one of the Claims 8 until 13 , wherein a scenario parameter (P _i ) and/or a calibration parameter (Pcal _i ) is a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, a Angle, a radius, a location, a number, a moving object like an automobile, a person or a cyclist, an immovable object like a building or a tree, a road configuration like a freeway, a road sign, a traffic light, a tunnel, a roundabout, a turning lane, a volume of traffic, a topographical structure such as an incline, a time, a temperature, a precipitation value, a weather condition and/or a season.

Images