DE102022116564A1 - Method, system and computer program product for evaluating test cases for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) - Google Patents

Abstract

The invention relates to a method for evaluating test cases (Ti) for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a defined driving task in at least one scenario (SZi), comprising:- Generating (S10) at least one test case (Ti) from a test agent (350) by selecting at least one parameterized scenario (SZpi) and concrete scenario parameters (Pci) and concrete scenario parameter values (PVci) from a scenario database (200) using a test strategy of one Test agents (350) for the driving task; - Generating (S20) a simulation environment with the selected test case (Ti) and carrying out a simulation of the behavior of the driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function in the simulation environment by a simulation module (400) for determining simulation results (450); - evaluating (S30) the simulation results (450) by an evaluation module (500) for determining a criticality index (590) for the test case (Ti).

Description

The invention relates to a method, a system and a computer program product for evaluating test cases for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function.

Modern vehicles are equipped with a variety of driving assistance systems or automated driving assistance functions to support the driver while driving and increase his safety. Driving assistance systems, for example, support speed and distance control and include lane keeping and lane changing functions. A specific maximum speed can be set, which will not be exceeded as long as the speed limit function is activated. Radar sensors and also camera systems are used for distance control, in which a certain distance is set, particularly to a vehicle in front. This allows the distance to vehicles in front, but also to vehicles in the side area, to be monitored. This leads to improved driving comfort and greater safety, especially when driving on the motorway and during overtaking maneuvers.

This trend towards driver assistance systems (Advanced Driver Assistance System, ADAS) and automated driving systems (ADS) in motor vehicles, but also in aircraft or watercraft and other moving objects, requires extensive security strategies, as the responsibility for the Vehicle control is no longer entirely the responsibility of the driver, but active functions are taken over by computer units in the vehicle. Therefore, for objects that move completely or partially autonomously, it must be ensured that these systems have a very low error rate in driving behavior. In order to ensure the safe functionality of an ADAS/ADS system, the detection and classification of other objects and the interpretation of traffic scenarios in the vicinity of a moving object, especially a vehicle, are important prerequisites. This requires targeted testing and training of driver assistance systems and automated driving systems both in extreme and exceptional situations (corner cases) as well as in everyday situations. Such extreme situations arise from a special combination of different factors. Examples of this are infrastructural features such as the type of road, the development on the edge of a road, the quality of the markings but also environmental conditions such as weather conditions, the time of day and the season. The behavior of other road users, the geographical topography and the weather conditions also play a major role.

However, when integrating modern ADAS/ADS systems into a vehicle, such verification, calibration and validation also represents a major challenge, as there is a functional specification deficit in driving assistance systems. While a requirements-based test process has been established for conventional systems in the automotive industry, in which test cases are implemented based on test specifications, this is currently missing for ADAS/ADS systems because, in contrast to conventional systems, a significantly larger number of influencing variables have to be taken into account, such as in particular the driving environment, which is recorded using sensors. The set of scenarios that can occur in the vehicle's driving environment and that must be correctly recognized and processed by an ADAS/ADS system is represented by an Operational Design Domain (ODD). This includes both everyday driving scenarios and very rarely occurring corner cases. In order to test and train an ADAS/ADS system for all possible driving scenarios, the entire ODD for an ADAS/ADS system would have to be recorded and documented in a requirements catalog. This is not possible due to the complexity of the driving environment and the resulting large number of driving scenarios. This problem is called functional specification deficiency. It complicates both the calibration and validation process of an ADAS/ADS system and requires alternative approaches to the existing methods.

Due to the challenges in testing ADAS/ADS systems, in addition to the known methods, virtual simulation methods are increasingly being used for testing and training for verification, calibration and validation of ADAS/ADS systems. However, these virtual simulation methods must be designed in such a way that an objective comparison of different ADAS/ADS systems with different functionalities in terms of performance and safety is possible. For this purpose, test cases are developed with which a simulation environment for testing and training ADAS/ADS systems can be generated. However, not all test cases reflect critical traffic situations. However, since the assessment of performance and performance Since the performance of an ADAS/ADS system depends in particular on its behavior in critical test cases, an evaluation of test cases with regard to their criticality is necessary.

The DE 10 2017 009971 A1 discloses a method for testing a lane departure warning system, whereby a category of adequate and relevant scenarios for field tests is extracted based on ontology.

The DE 10 2019 124018 A1 discloses a method for optimizing tests of control systems for automated vehicle dynamics systems, whereby quasi-random parameter combinations are generated for relevant parameters and corresponding system responses are evaluated as to whether they are critical or non-critical.

The DE 10 2019 217533 A1 discloses a method for estimating coverage of the space of traffic scenarios using metrics such as criticality.

The DE 10 2020 005507 A1 discloses a method for testing an automated driving function, wherein a confidence level for the driving function to be tested is determined over an entire effective range of an operational design domain.

The object underlying the invention is to provide options for evaluating test cases for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS), so that in particular the calibration and validation process requires less time and with a improved accuracy and efficiency.

This object is achieved according to the invention with regard to a method by the features of patent claim 1, with regard to a system by the features of patent claim 9, and with regard to a computer program product by the features of patent claim 15. The further claims relate to preferred embodiments of the invention.

The present invention makes it possible to select from a pool of available test cases for creating a simulation environment for testing and training an ADAS/ADS system and/or a driving function those critical test cases that are necessary for testing and securing an ADAS/ADS system and/or a driving function are particularly relevant. In addition, the evaluation of the test cases results in a border area between critical and non-critical test cases. These test cases, which lie in the borderline range, are particularly interesting for the real testing of an ADAS/ADS system, as they allow the performance of an ADAS/ADS system to be evaluated precisely and comprehensibly and possible measures for improvement can be derived from this. In particular, it can be examined whether this limit area is compatible with the system limits defined during the development of an ADAS/ADS system, for example in a requirements specification. By selecting critical test data sets, the assessment of the safety and functionality of a driving assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a specific driving task can be significantly improved.

• - Generieren von zumindest einem Testfall von einem Testagenten durch Auswählen von zumindest einem parametrisierten Szenario und konkreten Szenarioparametern und konkreten Szenarioparameterwerten aus einer Szenariendatenbank mittels einer Teststrategie von einem Testagenten für die Fahraufgabe;
• - Generieren einer Simulationsumgebung mit dem ausgewählten Testfall und Durchführen einer Simulation des Verhaltens des Fahrerassistenzsystems (ADAS) und/oder eines automatisierten Fahrsystems (ADS) und/oder einer Fahrfunktion in der Simulationsumgebung von einem Simulationsmodul zur Bestimmung von Simulationsergebnissen;
• - Bewerten der Simulationsergebnisse durch ein Bewertungsmodul zur Bestimmung eines Kritikalitätsindex für den Testfall.

According to a first aspect, the invention provides a method for evaluating test cases for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a defined driving task in at least one scenario. A scenario represents a traffic event in a temporal sequence and is defined by a selection of scenario parameters and associated scenario parameter values, whereby in a parameterized scenario the scenario parameters and associated scenario parameter values are freely selectable, and in a specific scenario the specific scenario parameters and associated specific scenario parameter values are fixed . The procedure includes the following steps:

• - Generating at least one test case from a test agent by selecting at least one parameterized scenario and concrete scenario parameters and concrete scenario parameter values from a scenario database using a test strategy from a test agent for the driving task;
• - Generating a simulation environment with the selected test case and carrying out a simulation of the behavior of the driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function in the simulation environment from a simulation module to determine simulation results;
• - Evaluating the simulation results using an evaluation module to determine a criticality index for the test case.

In a further development, it is provided that the evaluation module calculates further evaluation results, in particular in the form of performance indicators (KPls) for evaluating the performance of the ADAS/ADS system and/or the driving function and simulation quality criteria (SQCs) for determining the quality of the simulation for the test case .

In an advantageous embodiment it is provided that the test case and the specific criticality index as well as the further evaluation results are stored in a test database.

In a further embodiment it is provided that a large number of test cases are generated, which arise from a variation of one or more scenario parameters and specific scenario parameter values.

In particular, it is provided that the test cases, the scenario parameters and the concrete scenario parameter values are represented as a data representation in the form of at least one vector in a latent vector space.

in a further embodiment it is provided that the test agent of the test module comprises a software application that uses calculation methods and/or artificial intelligence algorithms, and that the simulation module comprises a software application that uses calculation methods and/or artificial intelligence algorithms; and that the assessment module includes a software application that uses calculation methods and/or artificial intelligence algorithms.

The calculation methods and/or algorithms of artificial intelligence are advantageous as mean values, minimum and maximum values, lookup tables, models for expected values, linear regression methods, Gaussian processes, fast Fourier transformations, integral and differential calculations, Markov methods, probability methods such as Monte Carlo -Methods, temporal difference learning, extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, feedback neural networks, and/or convolutional neural networks.

In particular, it is provided that a scenario parameter is a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number , a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a highway, a street sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic volume, a topographic Structure such as a slope, a time, a temperature, a precipitation value, a weather and/or a season.

According to a second aspect, the invention provides a system for evaluating test cases for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a defined driving task in at least one scenario. A scenario represents a traffic event in a temporal sequence and is defined by a selection of scenario parameters and associated scenario parameter values, whereby in a parameterized scenario the scenario parameters and associated scenario parameter values are freely selectable, and in a specific scenario the specific scenario parameters and associated specific scenario parameter values are fixed . The system includes a scenario database in which at least parameterized scenarios, scenario parameters and associated scenario parameter values are stored, a test module, a simulation module and an evaluation module. The test module is designed to generate at least one test case by selecting at least one parameterized scenario and specific scenario parameters and specific scenario parameter values from the scenario database using a test strategy from a test agent for the driving task. The simulation module is designed to generate a simulation environment with the selected test case and to carry out a simulation of the behavior of the driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function in the simulation environment by a simulation module to determine simulation results. The evaluation module is designed to evaluate the simulation results to determine a criticality index for the test case.

In an advantageous further development, it is provided that the evaluation module provides further evaluation results, in particular in the form of performance indicators (KPLs) for evaluating the performance of the ADAS/ADS system and/or the driving function and of simulation quality criteria (SQCs) for determining the quality of the simulation for the test case calculated.

Advantageously, the test case and the specific criticality index as well as the further evaluation results are stored in a test database.

In a further development it is provided that a large number of test cases are generated, which arise from a variation of one or more scenario parameters and concrete scenario parameter values.

In an advantageous embodiment it is provided that the test cases, the scenario parameters and the concrete scenario parameter values are represented as a data representation in the form of at least one vector in a latent vector space.

In a further embodiment, it is provided that the test agent of the test module comprises a software application that uses calculation methods and/or artificial intelligence algorithms, and that the simulation module comprises a software application that uses calculation methods and/or artificial intelligence algorithms; and that the assessment module includes a software application.

In particular, the calculation methods and/or algorithms of artificial intelligence are used, the calculation methods and/or algorithms of artificial intelligence being used as mean values, minimum and maximum values, lookup tables, models for expected values, linear regression methods, Gaussian processes, fast Fourier transformations, Integral and differential calculations, Markov methods, probability methods such as Monte Carlo methods, temporal difference learning, extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, feedback neural networks, and / or folded neural networks are formed .

Advantageously, it is provided that a scenario parameter is a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number , a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a highway, a street sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic volume, a topographic Structure such as a slope, a time, a temperature, a precipitation value, a weather and/or a season.

According to a third aspect, the invention provides a computer program product comprising executable program code configured to, when executed, carry out the method according to the first aspect.

The invention is explained in more detail below using exemplary embodiments shown in the drawing.

• 1 ein Blockdiagramm zur Erläuterung eines Ausführungsbeispiels eines erfindungsgemäßen Systems;
• 2 eine schematische Darstellung eines Vektorraums;
• 3 ein Flussdiagramm zur Erläuterung der einzelnen Verfahrensschritte eines erfindungsgemäßen Verfahrens;
• 4 ein Blockdiagramm eines Computerprogrammprodukt gemäß einer Ausführungsform des dritten Aspekts der Erfindung.

This shows:

• 1 a block diagram to explain an exemplary embodiment of a system according to the invention;
• 2 a schematic representation of a vector space;
• 3 a flowchart to explain the individual method steps of a method according to the invention;
• 4 a block diagram of a computer program product according to an embodiment of the third aspect of the invention.

Additional features, aspects and advantages of the invention or its embodiments are explained in the following description in conjunction with the claims.

Simulated traffic scenarios that are created through programming are increasingly being used to test, train and validate driver assistance systems (ADAS) and automated driving systems (ADS). In the context of the invention, a scenario is a traffic event in a temporal sequence. An example of a scenario is driving on a highway bridge, turning in a turning lane, driving through a tunnel, turning into a roundabout or stopping in front of a pedestrian crossing. In addition, specific visibility conditions, for example due to twilight or high levels of sunlight, as well as environmental conditions such as the weather and season, traffic volume and certain geographical topographical conditions can influence a scenario. For example, an overtaking operation can be described as a scenario in which a first vehicle is initially behind another vehicle, then changes lanes to the other lane and increases speed in order to overtake the other vehicle. Such a scenario is also known as a cut-in scenario.

In order to be able to use driver assistance systems (ADAS) and automated driving systems (ADS) in a motor vehicle, they must be calibrated and validated for reliable use. The purpose of calibration is to adapt the functions to the respective vehicle type and to the desired behavior of a driving function without changing the software code. For this purpose, the calibration parameters are modified and made available to the ADAS/ADS in a data set. The aim of the validation is to comprehensively test the data set obtained during calibration, to demonstrate the reliability and robustness of a driver assistance system (ADAS) or an automated driving system (ADS) throughout the entire ODD and then to grant approval. It is particularly important to be able to estimate the safety and functionality of an ADAS/ADS system for critical test cases.

1 shows a system 100 according to the invention for evaluating test cases for testing and training a driver assistance system (ADAS) and/or an automated driving system of a driver assistance system (ADAS) and/or a driving function. The system 100 according to the invention includes a scenario database 200, a test module 300 with a test agent 350, a simulation module 400 and an evaluation module 500, each of which can be provided with a processor and/or a storage unit.

In connection with the invention, a “module” can be understood to mean, for example, a processor and/or a processor unit and/or a memory unit for storing program instructions. The processor is specifically set up to execute the program instructions in order to implement or realize the method according to the invention or a step of the method according to the invention. In particular, a module can be integrated into a cloud computing infrastructure 700.

In connection with the invention, a “processor” can be understood to mean, for example, a machine or an electronic circuit. A processor can in particular be a main processor (Central Processing Unit, CPU), a microprocessor or a microcontroller, for example an application-specific integrated circuit or a digital signal processor, possibly in combination with a memory unit for storing program instructions, etc . A processor can also be understood as a virtualized processor, a virtual machine or a soft CPU. For example, it can also be a programmable processor that is equipped with configuration steps for carrying out the method according to the invention or is configured with configuration steps in such a way that the programmable processor has the features of the method, the system, the modules, or other aspects and/or other aspects according to the invention. or partial aspects of the invention are realized. In particular, the processor can contain highly parallel computing units and powerful graphics modules.

In connection with the invention, a “memory unit” or a “memory module” can, for example, mean a volatile memory in the form of a random access memory (RAM) or a permanent memory such as a hard drive or a data carrier or, for example. B. a replaceable memory module can be understood. The storage module can also be a cloud-based storage solution.

“Database” means both a storage algorithm and the hardware in the form of a storage unit. In particular, a database can be designed as part of a cloud computing infrastructure.

In connection with the invention, “data” means both raw data and already processed data from measurement results from sensors and from other data sources.

The creation of test cases T _i for testing and training a driver assistance system (ADAS) and/or an automated driving system of a driver assistance system (ADAS) and/or a driving function is carried out on the basis of parameterized scenarios, which are also referred to as logical scenarios. In the context of the invention, a parameterized scenario refers to a scenario that is written in particular in a machine-readable code. A parameterized scenario SZp _i is defined by various scenario parameters P ₁ , P ₂ , ..., P _n from a set of possible scenario parameters P _i and associated scenario parameter values PV ₁ , PV ₂ , ..., PV _n from a set of possible scenario parameter values PV _i defined, where scenario parameter values PV _i determine the value range of a scenario parameter P _i . A scenario parameter P _i represents, for example, a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a highway, a street sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic volume, a topographical structure such as a gradient, a time, a temperature, a precipitation value, a weather and/or a season. Scenario parameters P _i thus characterize properties and features of a scenario within the scope of the present invention. An example of a scenario parameter P _i of a scenario SZp _i is the speed of an ego vehicle. For this scenario parameter “speed”, the value range of the associated scenario parameter value PV _i can cover the range from 100 km/h to 180 km/h. For another scenario SZp _k , however, the value range of the parameter value PV _i for the scenario parameter “speed” can extend from 40 km/h to 70 km/h.

A parameterized scenario SZp comprises a temporal sequence of time intervals Δt ₁ , Δt ₂ ., ..., Δt _n , in which different scenes and events take place. A parameterized scenario SZp begins with a starting scene and then develops further through occurring events, from which new subsequent scenes emerge over time. The starting scene is thus changed by one or more events. An event can be a reaction actively triggered by a road user, such as an acceleration, or an event that recurs cyclically, such as the switching of a traffic light. The start scene and the individual subsequent scenes therefore only cover a small time interval Δt or a snapshot, while a parameterized scenario SZp covers a longer period of time.

In the context of the present invention, a distinction is made between a parameterized scenario SZp and a concrete scenario SZc. In the context of this invention, a parameterized scenario SZp is defined as a scenario in which both the scenario parameters P _i and the associated scenario parameter values PV _i are not all fixed. A scenario SZ in which the concrete scenario parameters Pc _i and the associated concrete scenario parameter values PVc _i or value ranges of the concrete scenario parameter values PVc _i are defined is referred to as a concrete scenario SZc. Both scenarios, both the parameterized scenario SZp and the concrete scenario SZc, are scenarios that are written in particular in a machine-readable code.

Various sources such as requirement specifications, expert knowledge, and/or measurements in public traffic or on test sites using sensors can be used to create parameterized scenarios SZp _i . The sensors used can in particular be designed as radar systems with one or more radar sensors, LIDAR systems for optical distance and speed measurement, image-recording 2D/3D cameras in the visible, IR and/or UV range, and/or GPS systems. Furthermore, acceleration sensors, speed sensors, capacitive sensors, inductive sensors, voltage sensors, torque sensors, speed sensors, precipitation sensors and/or temperature sensors, etc. can be used. Using a software application, a suitable parameterized scenario SZp _i can be created from recorded data at a specific geographical location. be derived. The software application uses artificial intelligence algorithms in particular to identify the parameterized scenarios SZp _i . The artificial intelligence algorithms can in particular be encoders and decoders with neural networks.

A neural network consists of neurons arranged in several layers and connected to each other in different ways. A neuron is able to receive information at its input outside or from another neuron, evaluate the information in a certain way and forward it in a modified form at the neuron output to another neuron or output it as the end result. Hidden neurons are located between the input neurons and output neurons. Depending on the network type, there may be multiple layers of hidden neurons. They ensure the forwarding and processing of the information. Output neurons ultimately deliver a result and output it to the outside world. Neural networks can be trained through unsupervised or supervised learning.

The different arrangement and connection of the neurons creates different types of neural networks, such as in particular a feedforward neural network (FNN), a feedback network (recurrent neural network, RNN) or a folded neural network (convolutional Neural Network, CNN). A convolutional neural network has multiple convolution layers and is very suitable for machine learning and applications in the field of pattern recognition and image recognition. Since a large part of the data recorded by the sensors is available as images, convolutional neural networks (CNN) are used in particular.

These parameterized scenarios SZp _i generated from various data sources are stored in the scenario database 200. Scenario parameters P _i , scenario parameter values PV _i , specific scenarios SZc _i , specific scenario parameters Pc _i , specific scenario parameter values PVc _i and other information can also be stored in the scenario database 200 .

The test agent 350 of the test case module 300 creates a plurality of test cases T _i for one or more driving tasks using a software application by selecting suitable parameterized scenarios SZp _i and concrete scenario parameters Pc _i as well as concrete scenario parameter values PVc _i and possibly further information. The respective driving task is formulated before the simulation starts, for example by an expert such as an engineer. However, it can also be provided that the driving task is specified by a software application. An example driving task is an overtaking maneuver on a highway, also known as a cut-in maneuver.

Such a cut-in scenario can be represented by the speed v _ego of the ego vehicle, the speed v _target of the target vehicle, a relative speed v _rel between the two vehicles, the spatial distance d _cutin between the two vehicles and the spatial length l _cutin of the overtaking maneuver can be described. The speed of one or both vehicles can also change during the overtaking maneuver by accelerating or braking. This also applies to the spatial distance d _cutin between the two vehicles.

The following table shows possible scenario parameters P _i and scenario parameter values PV _i of a cut-in scenario: Scenario parameters Scenario parameter value v _ego 30, 50, 60, 80 km/h v _target 30, 50, 60, 80 km/h d _cutin 50, 40, 30, 20, 10m v _rel -20, -10, .5 km/h l _cutin 20.40 m

For the selection and design of the test cases T _i , a test strategy is provided which specifies how the test agent 350 creates the test cases T _i . Various calculation methods and algorithms, in particular artificial intelligence algorithms, can be used to determine the test strategy. Algorithms and calculation methods such as mean values, minimum and maximum values, lookup tables, models for expected values, linear regression methods, Gaussian processes, fast Fourier transformations, integral and differential calculations, Markov methods, probability methods such as Monte Carlo methods, temporal difference learning , but also extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, and/or feedback neural networks are used, with which the strategy can be adapted, in particular through iteration processes.

The test strategy of the test agent 350 thus decides on the configuration of the test cases T _i by selecting suitable parameterized scenarios SZp _i and concrete scenario parameters Pc _i and, if necessary, other parameters. The test strategy used is dictated by the purpose of the simulation, i.e. what knowledge should be gained about the behavior of an ADAS/ADS system or a specific driving function when carrying out the specified driving task through a simulation.

The test agent 350 takes the information required for creating the test cases T _i from the scenario database 200 and, if necessary, at least one further database 750. For the cut-in scenario, a large number of concrete scenarios SZc can be created by varying the possible scenario parameters P _i and scenario parameter values PV _{i i} are generated. In particular, the scenario parameter values PV _i can be successively changed with a fixed value, for example for the “speed” parameter by 10 km/h in a value range of 40 km/h to 120 km/h. These concrete scenarios SZc _i represent possible test cases T _i . A test case T _i is therefore a relevant concrete scenario SZc through the definition of concrete scenario parameters Pc _i and possible further parameters. In particular, the relevant test cases T _i created by the test agent 350 should include the relevant traffic situations for a driving task defined in the “Operational Design Domain” (ODD), which must be controlled by an ADAS/ADS system and/or a driving function.

The test agent 350 stores the test cases T _i determined in this way in a test database 370. The data for a test case T _i can include a test identification number (test ID), a user name, a creation time and the name for the selected parameterized scenario SZp _i contain. Furthermore, those concrete scenario parameters Pc _i that transform the parameterized scenario SZp _i into a concrete scenario SZc _i are saved. In addition, calibration parameters and other data such as performance indicators (KPLs) with regard to the evaluation of the performance of the respective driving assistance function during the simulation as well as evaluation indicators for the evaluation of the simulation carried out can be saved. Furthermore, information about the status of the simulation such as “carried out or not done” can be assigned to a test case T _i . In addition, further text messages, video sequences and/or audio sequences, etc. with further information can be saved. Overall, all relevant information about a test case T _i can be stored in the test database 370.

To carry out a simulation of the behavior of the ADAS/ADS system for a specific driving task, the test agent 350 transfers a test case T _i or several test cases T _i to the simulation module 400. In addition, the simulation module 400 can retrieve further data from at least one further database 750, to be able to carry out the simulation.

In particular, 750 pieces of data in the form of images, graphics, time series, parameters, etc. are stored in the database. For example, target variables and target values can be stored in the database 750, which define a security standard for the simulation. Furthermore, data about the road network with road specifications such as lanes and bridges, the road infrastructure such as the road surface, the edge development, the road layout, country-specific properties, weather conditions, etc. can be provided. The database 750 may also be integrated into the cloud computing infrastructure 700.

The simulation module 400 creates a simulation environment from a test case T _i for simulating the behavior of an ADAS/ADS system or a driving function for a specified driving task. For this purpose, the simulation module 400 includes a software application 450 that uses calculation methods and/or artificial intelligence algorithms such as, in particular, neural networks.

The quality of the simulation results 470 depends on the quality of the scenario description and the choice of a suitable test case T _i . In order to be able to evaluate this quality, the simulation results 470 achieved by the simulation are forwarded to the evaluation module 500 for calculating evaluation results 570. For this purpose, the evaluation module 500 includes a software application 550, which evaluates the simulation results 470 with regard to the performance and functionality of one or more functions or the overall performance of a driver assistance system (ADAS) or an automated driving system (ADS), in particular in the form of performance indicators (KPls). . In addition, the quality of the simulation process carried out is evaluated, particularly in the form of simulation quality criteria (SQCs). The software application 550 uses calculation methods and/or algorithms artificial intelligence, such as neural networks in particular, for producing the assessment results 570.

KPIs are used to describe the performance of an ADAS/ADS system under test, with different KPIs being set for different evaluation categories such as comfort, safety, naturalness of driving and efficiency.

Additionally, additional KPIs can be implemented to verify the correct functionality of the ADAS/ADS system under test. An example of a KPI is the evaluation of a minimum distance to another vehicle or an average acceleration in a deceleration scenario.

In addition to KPIs, SQCs are used to evaluate the quality of the simulation. For example, other road users can be represented by modeled simulation representatives who make different decisions depending on the reactions of the ADAS/ADS system being tested. For this reason, an evaluation must be carried out to check whether the simulated scenario matches the scenario description of the test case T _i . For this purpose, metrics are defined for the SQCs that provide feedback on the correct execution of a scenario simulation. The manner in which an overtaking maneuver was carried out can be evaluated using an SQC, for example with regard to the question of whether a collision could be avoided. The KPIs and SQCs can be represented by numerical values but also by Boolean values.

Direct and indirect evaluation metrics can be used for both the KPIs and SQCs. Direct assessment metrics contain data directly from simulated measurement data. Indirect evaluation metrics in particular use scenario parameters as a data source. An example of the application of indirect evaluation metrics is the implementation of simulation models that quantify and evaluate the subjective feeling of a driver when performing a driving assistance function of an ADAS/ADS system. Based on the KPIs and SQCs, the results of a simulated test case T _i can be represented by numerical values or Boolean values.

According to the invention, a test case T _i is also evaluated with regard to its criticality depending on various parameters using a criticality index 590. Criticality is understood to mean the relevance of a test case T _i with regard to the assessment of the performance and functionality of an ADAS/ADS system or a driving function in relation to a specific driving task. In particular, the criticality of a test case T _i is evaluated in relation to the variation of scenario parameter values PV _i of a scenario parameter P _i .

In 2 For example, a vector space is shown, which is spanned by two scenario parameters P ₁ and P ₂ . For the example of the cut-in scenario already considered, the first parameter P ₁ is the distance d _cutin between the ego vehicle and the target vehicle and the second parameter P ₂ is the speed of the ego vehicle v _ego . Depending on both parameters P ₁ , P ₂ , a criticality index 590 can be assigned to the respective form of the test case T _cutin . In particular, it can be provided that the criticality index 590 includes the two categories “critical” and “uncritical”. The criticality index 590 can also be represented by values on a scale of, for example, 0 to 1. For example, the minimum time gap t _risk to a vehicle in front can be calculated as an evaluation criterion as to whether a variant of the test case T _cutin is assessed as critical or non-critical. A test case in which the minimum time gap cannot be maintained for a duration of more than one second is then classified as a critical test case T _risk : $$t_{risk}>1s$$

Since the ego vehicle stays near the target vehicle for more than a second in this case, the test case is rated as critical.

However, those test cases T _cutin for which the minimum time gap is less than one second are classified as non-critical.

For reasons of clarity, in the 2 only two scenario parameters P ₁ and P ₂ are shown. However, it is possible to select any number of concrete scenario parameters Pc _i with which the software application 550 spans a latent vector space in which the test cases T _i can be represented as representations. The vector space is created by automated endless learning through the con creten scenario parameters Pc _i and test cases T _i are created so that similarity structures such as clusters of test cases T _i develop in a self-learning manner. These clusters of test cases T _i can now be analyzed, for example, using a similarity query and/or a cluster analysis.

For the transformation of the concrete scenario parameters Pc _i and test cases T _i into data representations in the form of one or more vectors in the latent vector space, the software application 550 includes at least one encoder to encode the scenario parameters Pc _i such that the scenario parameters Pc _i as one Data representation in the form of one or more vectors in a latent vector space. Furthermore, a second encoder is provided with which the software application 550 encodes the test cases T _i such that they are also present as a data representation in the form of one or more vectors in the latent vector space. The latent vector space is therefore a multidimensional vector space with a dimension that corresponds to the representation of the concrete scenario parameters Pc _i and the test cases T _i .

The encoders of the software application 550 are designed in particular as deep encoders, which include deep neural networks and in particular for coupled neural networks, which can have different structures. In addition, the encoders can include other structures and algorithms such as genetic algorithms and learning reinforcement agents to perform efficient and precise classification and extraction of features of the concrete scenario parameters Pc _i , which forms the basis for creating the data representations.

The vector space is therefore further developed with each subsequent simulation, and its dimensions can constantly change. Since the respective concrete scenario parameters Pc _i and the test cases T _i are encoded for a large number of simulations, a latent vector space with a large number of concrete scenario parameters Pc _i and test cases T _i is gradually created in a continuous learning mode through new simulations.

In a further embodiment it can be provided that the assessment of the criticality depends on the performance of the SQCs. It can thus be provided that the software application 550 of the evaluation module 500 only carries out an evaluation of the criticality if the respective defined SQCs are also fulfilled by the test case T _i .

The evaluation results 550 in the form of performance indicators (KPIs), simulation quality criteria (SQCs) and the criticality index 590 are returned to the test module 300. The test agent 350 can provide the parameterized scenarios SZp _i and the concrete scenario parameters Pc _i in the scenario database 200 with labels such as weights or to identify the criticality based on the evaluation results. In particular, the test strategy can be adapted in the scenario database 200 using the data labeled by the evaluation results 550. For a new simulation cycle in the simulation module 400, for example, the test cases T _i marked as critical can now be used in order to be able to simulate the behavior of an ADAS/ADS system and/or a driving function in these traffic scenarios that are considered critical.

In 3 the method steps for evaluating test cases T _i for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a defined driving task are shown in at least one scenario SZ _i .

In a step S10, at least one test case T _i is generated by a test agent 350 by selecting at least one parameterized scenario SZp _i and concrete scenario parameters Pc _i and concrete scenario parameter values PVc _i from a scenario database 200 using a test strategy by a test agent 350 for the driving task.

In a step S20, a simulation environment with the selected test case T _i is generated and a simulation of the behavior of the driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function in the simulation environment by a simulation module 400 for determining simulation results 450 carried out.

In a step S30, the simulation results 450 are evaluated by an evaluation module 500 to determine a criticality index 590 for the test case T _i .

4 schematically illustrates a computer program product 1000 that includes an executable program code 1050 configured to execute the method according to the first aspect of the present invention.

The present invention makes it possible to select from a pool of available test cases for creating a simulation environment for testing and training an ADAS/ADS system and/or a driving function those critical test cases that are necessary for testing and securing an ADAS/ADS system and/or a driving function are particularly relevant. In addition, the evaluation of the test cases results in a border area between critical and non-critical test cases. These test cases, which lie in the borderline range, are particularly interesting for the real testing of an ADAS/ADS system, as they allow the performance of an ADAS/ADS system to be evaluated precisely and comprehensibly and possible measures for improvement can be derived from this. In particular, it can be examined whether this limit area is compatible with the system limits defined during the development of an ADAS/ADS system, for example in a requirements specification.

By selecting critical test data sets, the assessment of the safety and functionality of a driving assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a specific driving task can be significantly improved. This allows resources to be saved, as the actual driving of test routes with standard traffic situations as well as with specific corner cases can be reduced.

Reference symbols

100

system

200

Scenario database

300

Test module

350

Test agent

370

Test database

400

Simulation module

450

Software application

470

Simulation results

500

Assessment module

550

Software application

570

Assessment results

590

Criticality index

700

Cloud computing infrastructure

750

Database

1000

Computer program product

1050

Program code

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102017009971 A1 [0006]
• DE 102019124018 A1 [0007]
• DE 102019217533 A1 [0008]
• DE 102020005507 A1 [0009]

Claims (15)

Method for evaluating test cases (T _i ) for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a specified driving task in at least one scenario (SZ _i ), wherein a scenario (SZ _i ) represents a traffic event in a temporal sequence and is defined by a selection of scenario parameters (P ₁ , P ₂ , ... , P _n ) and associated scenario parameter values (PV ₁ , PV ₂ , ..., PV _n ), and whereby in a parameterized scenario (SZp _i ), the scenario parameters (P ₁ , P ₂ , ..., P _n ) and associated scenario parameter values (PV ₁ , PV ₂ , ..., PV _n ) are freely selectable, and in a specific one Scenario (SZc _i ) the concrete scenario parameters (Pc ₁ , Pc ₂ , ..., P _cn ) and associated concrete scenario parameter values (PVc ₁ , PV _c2 , ..., PV _cn ) are defined, comprising: - Generate (S10) of at least one test case (T _i ) from a test agent (350) by selecting at least one parameterized scenario (SZp _i ) and concrete scenario parameters (Pc _i ) and concrete scenario parameter values (PVc _i ) from a scenario database (200) using a test strategy of one Test agents (350) for the driving task; - Generating (S20) a simulation environment with the selected test case (T _i ) and carrying out a simulation of the behavior of the driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function in the simulation environment from a simulation module (400). Determination of simulation results (450); - Evaluating (S30) the simulation results (450) by an evaluation module (500) to determine a criticality index (590) for the test case (T _i ). Procedure according to Claim 1 , wherein the evaluation module (500) provides further evaluation results (570) in particular in the form of performance indicators (KPls) for evaluating the performance of the ADAS/ADS system and/or the driving function and of simulation quality criteria (SQCs) for determining the quality of the simulation for the test case (T _i ) is calculated. Procedure according to Claim 1 or 2 , whereby the test case (T _i ) and the specific criticality index (790) as well as the further evaluation results (570) are stored in a test database (370). Procedure according to one of the Claims 1 until 3 , whereby a large number of test cases (T _i ) are generated, which arise from a variation of one or more scenario parameters (Pc _i ) and concrete scenario parameter values (PVc _i ). Procedure according to Claim 4 , where the test cases (T _i ), the scenario parameters (Pc _i ) and the concrete scenario parameter values (PVc _i ) are represented as a data representation in the form of at least one vector in a latent vector space. Procedure according to one of the Claims 1 until 5 , wherein the test agent (350) of the test module (300) comprises a software application that uses calculation methods and/or artificial intelligence algorithms, wherein the simulation module (400) comprises a software application (450) that uses calculation methods and/or artificial intelligence algorithms ; and wherein the evaluation module (500) comprises a software application (550) that uses calculation methods and/or artificial intelligence algorithms. Procedure according to Claim 6 , where the calculation methods and/or algorithms of artificial intelligence are used as mean values, minimum and maximum values, lookup tables, models for expected values, linear regression methods, Gaussian processes, fast Fourier transformations, integral and differential calculations, Markov methods, probability methods such as Monte Carlo -Methods, temporal difference learning, extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, feedback neural networks, and / or folded neural networks are formed. Procedure according to one of the Claims 1 until 7 , where a scenario parameter (P _i ) is a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a highway, a street sign, a traffic light, a tunnel, a roundabout, a turning lane, a traffic volume, represents a topographic structure such as a slope, a time, a temperature, a precipitation value, a weather and/or a season. System (100) for evaluating test cases (T _i ) for testing and training a driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function for a defined driving task in at least one scenario (SZ _i ), wherein a Scenario (SZ _i ) represents a traffic event in a temporal sequence and is defined by a selection of scenario parameters (P ₁ , P ₂ , ... , P _n ) and associated scenario parameter values (PV ₁ , PV ₂ , ..., PV _n ). is, and where in a parameterized scenario (SZp _i ), the scenario parameters (P ₁ , P ₂ , ..., P _n ) and associated scenario parameter values (PV ₁ , PV ₂ , ..., PV _n ) are freely selectable, and in a specific scenario (SZc _i ), the specific scenario parameters (Pc ₁ , Pc ₂ , ..., P _cn ) and associated specific scenario parameter values (PVc ₁ , PV _c2 , ..., PVc _n ) are defined, comprising a scenario database ( 200), in which at least parameterized scenarios (SZp _i ), scenario parameters (P ₁ , P ₂ , ..., P _n ) and associated scenario parameter values (PV ₁ , PV ₂ , ..., PV _n ) are stored, a test module (300), a simulation module (400) and an evaluation module (500); wherein the test module (300) is designed to provide at least one test case (T _i ) by selecting at least one parameterized scenario (SZp _i ) and specific scenario parameters (Pc _i ) and specific scenario parameter values (PVc _i ) from the scenario database (200) using a test strategy to be generated by a test agent (350) for the driving task; wherein the simulation module (400) is designed to generate a simulation environment with the selected test case (T _i ) and to simulate the behavior of the driver assistance system (ADAS) and/or an automated driving system (ADS) and/or a driving function in the simulation environment of a carry out a simulation module (400) to determine simulation results (450); and wherein the evaluation module (500) is designed to evaluate the simulation results (450) to determine a criticality index (590) for the test case (T _i ). System (100) after Claim 9 , wherein the evaluation module (500) provides further evaluation results (570) in particular in the form of performance indicators (KPls) for evaluating the performance of the ADAS/ADS system and/or the driving function and of simulation quality criteria (SQCs) for determining the quality of the simulation for the test case (T _i ) is calculated, and/or wherein the test case (T _i ) and the specific criticality index (590) as well as the further evaluation results (570) are stored in a test database (370). System (100) after Claim 9 or 10 , whereby a large number of test cases (T _i ) are generated, which arise from a variation of one or more scenario parameters (Pc _i ) and concrete scenario parameter values (PVc _i ). System (100) after Claim 11 , where the test cases (T _i ), the scenario parameters (Pc _i ) and the concrete scenario parameter values (PVc _i ) are represented as a data representation in the form of at least one vector in a latent vector space. System (100) according to one of the Claims 9 until 12 , wherein the test agent (350) of the test module (300) comprises a software application that uses calculation methods and/or artificial intelligence algorithms, wherein the simulation module (400) comprises a software application (450) that uses calculation methods and/or artificial intelligence algorithms ; and wherein the evaluation module (500) comprises a software application (550) that uses calculation methods and/or artificial intelligence algorithms, and wherein the calculation methods and/or artificial intelligence algorithms are used as mean values, minimum and maximum values, lookup tables, models for expected values , linear regression methods, Gaussian processes, fast Fourier transformations, integral and differential calculations, Markov methods, probability methods such as Monte Carlo methods, temporal difference learning, extended Kalman filters, radial basis functions, data fields, convergent neural networks, deep neural networks, feedback neural networks and/or folded neural networks are formed. System (100) according to one of the Claims 9 until 13 , where a scenario parameter (P _i ) is a physical quantity, a chemical quantity, a torque, a speed, a voltage, a current, an acceleration, a speed, a braking value, a direction, an angle, a radius, a location, a Number, a moving object such as a motor vehicle, a person or a cyclist, an immovable object such as a building or a tree, a road configuration such as a highway, a street sign, a traffic light, a tunnel, a roundabout, a turning lane, a volume of traffic, a topographical structure such as a slope, a time, a temperature, a precipitation value, a weather and/or a season. Computer program product (1000), comprising an executable program code (1050) which is configured so that, when executed, it implements the method according to one of Claims 1 until 8th executes.

Images