DE102022213262A1 - System and method for creating a virtual exam environment based on a detected frequency of identical or similar scenarios - Google Patents

Abstract

The invention relates to a system (1) for creating a virtual test environment based on a recognized frequency of identical or similar scenarios (2,2b,2c) for testing the autonomous driving functions (3) for an ego vehicle (4), comprising a simulator (5) for generating different scenarios (2,2b,2c) from the perspective of the ego vehicle (4) with an ego trajectory (7) and/or an interface (6) for receiving real data from different scenarios (2,2b,2c) from the perspective of the ego vehicle (4) with an ego trajectory (7), wherein a module (9) is provided which is designed to carry out an automated assignment of a maneuver step for each maneuver type for each time step of the ego trajectory (7) in the scenario (2,2b,2c), and to carry out an automated assignment of a maneuver step for each maneuver type for each time step of each road user trajectory of each road user in the scenario (2,2b,2c). accomplish, wherein the different maneuver types with selectable maneuver steps are provided as a maneuver catalog, so that the assignments generate a maneuver matrix (M, M1, M2) with different maneuver type-dependent maneuver sequences for each maneuver type, and a comparison module (11) is provided which is designed to automatically summarize the same maneuver matrices (M, M1, M2) into a cluster (C1, C2, C3) as maneuvers, and to determine the size of the clusters (C1, C2, C3) as the frequency of identical or similar scenarios (2, 2b, 2c), and wherein a processor (12) is provided which is designed to create a test environment for testing the autonomous driving functions (3) based on the size of the clusters (C1, C2, C3) and is designed to test the autonomous driving functions (3) of the ego vehicle (4) in the test environment.
Furthermore, the invention relates to a method.

Description

The invention relates to a system for creating a virtual test environment based on a recognized frequency of identical or similar scenarios for testing autonomous driving functions for an at least partially autonomously operable ego vehicle, comprising a simulator for generating different scenarios from the perspective of the ego vehicle with an ego trajectory and/or an interface for receiving real data from different scenarios from the perspective of the ego vehicle with an ego trajectory.

Autonomous driving functions in advanced driver assistance systems (ADAS) and automated driving systems (ADS) on public roads have the potential to fundamentally change the transport system in the future. However, validation and verification of such autonomous driving functions are an important concern in order to ensure a sufficiently high level of safety and acceptance.

However, autonomous driving functions must be validated and verified before they can be used in the ego vehicle. Especially with regard to frequently occurring scenarios, these autonomous driving functions must be able to handle such frequently occurring scenarios without errors using all possible set driving parameters.

However, to validate the autonomous driving functions, a test coverage of several billion test kilometers would have to be driven. Even with the use of simulation tools, the required number of test kilometers cannot be achieved.

Testing a partially autonomous vehicle exclusively on the road with journeys of billions of kilometers is also not possible for reasons of time and cost. In addition, many redundant test kilometers would be created.

Therefore, a more efficient method to increase test coverage is needed. Current research projects and industry focus on scenario-based testing in this regard.

This approach takes advantage of the redundancy of the traffic scenarios that occur in distance-based testing: instead of testing the same scenarios multiple times, as they occur randomly in route-based testing, they are specifically selected from a catalog of scenarios. Testing the scenario with different driving parameters is sufficient to ensure the driver's safety with respect to that scenario.

However, similar scenarios must first be identified in order to determine the scenarios to be tested. In particular, frequently occurring scenarios must be able to be handled safely with the autonomous driving functions.

• Trainieren und Verwenden eines Ähnlichkeitsklassifikationsmoduls zur Klassifizierung der Ähnlichkeit zwischen zwei Szenarien basierend auf einer Teilmenge eines Parametersatzes des jeweiligen Szenarios, welches eine virtuelle Repräsentation einer realen Verkehrssituation darstellt; Erzeugen einer Graphenrepräsentation umfassend eine Vielzahl von Szenarien, umfassend die zwei Szenarien, deren Teilmenge von Parametersätzen in einer Speichereinheit gespeichert sind, wobei Knoten jeweils die Teilmenge eines Parametersatzes eines Szenarios repräsentieren und Kanten zwischen zwei Knoten mit einem Ähnlichkeitsklassifikationswert gewichtet sind, der die Ähnlichkeit zwischen den zwei Szenarien, der jeweiligen Knoten indiziert; Erhalten einer Benutzereingabe durch eine elektronische Benutzerschnittstelle und / oder eine automatischen Systemeingabe, wobei die Eingabe einen geforderten Wert für eine Abdeckung, der in der Speichereinheit gespeicherten Szenarien beinhaltet und / oder eine geforderte Anzahl an zu testenden Szenarien, so dass eine Auswahl und Ausführungsreihenfolge der Szenarien und der repräsentierenden Knoten mit der Zielsetzung einer maximalen Abdeckung des Graphen bestimmt wird; Bereitstellen einer Auswahl und Ausführungsreihenfolge und / oder eines Abdeckungswerts einer Gesamt-Abdeckung der gewählten Knoten und damit der zu repräsentierenden Szenarien.

The WO 2022028935 A1 discloses a test unit and a computer-implemented method for providing a scenario test sequence with scenarios to be tested, in particular traffic scenarios, for testing a device for at least partially autonomously guiding a means of transport, comprising the steps:

• Training and using a similarity classification module to classify the similarity between two scenarios based on a subset of a parameter set of the respective scenario, which represents a virtual representation of a real traffic situation; generating a graph representation comprising a plurality of scenarios, comprising the two scenarios, the subset of parameter sets of which are stored in a storage unit, wherein nodes each represent the subset of a parameter set of a scenario and edges between two nodes are weighted with a similarity classification value that indicates the similarity between the two scenarios of the respective nodes; receiving a user input through an electronic user interface and/or an automatic system input, wherein the input includes a required value for a coverage of the scenarios stored in the storage unit and/or a required number of scenarios to be tested, so that a selection and execution order of the scenarios and the representing nodes is determined with the aim of maximum coverage of the graph; providing a selection and execution order and/or a coverage value of an overall coverage of the selected nodes and thus of the scenarios to be represented.

It is therefore an object of the invention to provide a system and a method for creating a virtual test environment based on a recognized frequency of identical or similar scenarios for testing autonomous driving functions.

The object is achieved by a system having the features of claim 1 and a method having the features of claim 7. The subclaims contain further advantageous measures which can be suitably combined with each other to achieve further advantages.

The object is achieved by a system for creating a virtual test environment based on a recognized frequency of identical or similar scenarios for testing autonomous driving functions for an at least partially autonomously operable ego vehicle, comprising a simulator for generating different scenarios from the perspective of the ego vehicle with an ego trajectory and/or an interface for receiving real data from different scenarios from the perspective of the ego vehicle with an ego trajectory, and
wherein a module is provided which is designed to carry out an automated assignment of a maneuver step for each maneuver type for each time step of the ego trajectory in the scenarios, wherein the various maneuver types are provided with selectable maneuver steps as a maneuver catalog, and to carry out an automated assignment of a maneuver step for each maneuver type for each time step of each road user trajectory of each road user in the scenarios, wherein the various maneuver types are provided with selectable maneuver steps as a maneuver catalog, so that the assignments generate a maneuver matrix with different maneuver type-dependent maneuver sequences for each maneuver type,
and a comparison module is provided which is designed to automatically combine the same maneuver matrices into a cluster as maneuvers, and to determine the size of the clusters as the frequency of identical or similar scenarios,
and wherein a processor is provided which is designed to create a test environment for testing the autonomous driving functions based on the size of the clusters and is designed to test the autonomous driving functions of the ego vehicle in the test environment.

The real data can come from drones or from ego vehicles with appropriate sensors.

A scenario is defined by the static environment such as the road network, buildings and the behavior of other road users; i.e. a scenario is an abstract, semantic description, defined by the ego vehicle and the relevant environment.

A frequently occurring scenario is a scenario that occurs frequently compared to another, i.e. a scenario that occurs repeatedly compared to a scenario that occurs sporadically.

The scenario can be restricted to road users who interact or could interact with the ego vehicle, for example to road users who are within a given radius around the ego vehicle.

Autonomous driving functions are all functions that contribute to autonomous driving, this can also include level 3 driving functions (automated mode) or an assisted mode. Such driving functions can be, for example, lane keeping, speed adjustment or driving functions for fully autonomous operation, etc.

A maneuver catalog provides a collection of maneuver types necessary for completing driving tasks. One such maneuver type is, for example, “longitudinal movement” or “turning,” which describes behavior at intersections.

Maneuver steps of the maneuver type "longitudinal movement" can be, for example, "driving", "braking" or "standing still". Maneuver steps of the maneuver type "turning" can be "turn left" or "turn right" for the direction when crossing an intersection or "no turning" if the road user does not perform a turning maneuver. A maneuver matrix contains the time steps as columns and the extracted maneuver steps of each maneuver type along the trajectory as rows.

According to the invention, it was recognized that a test environment must contain at least all relevant, frequently occurring identical or similar scenarios in order to ensure adequate security of the autonomous driving functions. Furthermore, other scenarios must also be reliably identified in terms of their frequency.

However, the recorded data forms a scenario space with a high degree of redundancy, which must be split up and considered in terms of redundancy. For example, all scenarios in which an ego vehicle drives alone on a straight road can be considered as a cluster (scenario group), so that testing the scenario with different driving parameters of the autonomous driving functions is sufficient to ensure the driver's safety with regard to this scenario. This efficiently increases the safety of the autonomous driving functions tested for this scenario.

The system according to the invention allows a semantic analysis of the data by abstracting the trajectory data into maneuvers and maneuver sequences and dividing them into clusters. Maneuvers can be seen as an abstract description of the behavior of a participant in a certain period of time. Compared to a description with trajectories, the maneuvers are more concise and semantically interpretable and can therefore be processed further more easily. According to the invention, the system detects a maneuver step for each maneuver type at any time and thus generates a maneuver matrix with different maneuver sequences depending on the maneuver type. By subsequently clustering the same maneuver matrices, redundancy detection, outlier detection and frequency detection can be achieved. This means that instead of processing redundant scenarios, each scenario can be represented by a member in a test environment generated in this way. The frequency of the same or similar scenarios detected, i.e. the size of the clusters, can serve as a basis for prioritizing test scenarios and thus for generating the test environment.

Furthermore, rare and therefore possibly unknown scenarios can be easily identified through automated processing.

The system according to the invention generates a test environment that enables faster and safer validation of autonomous driving functions. By grouping similar scenarios, the high level of redundancy in testing autonomous driving functions can be reduced.

In a further embodiment, the processor is designed to carry out the testing by simulating the maneuver in the corresponding cluster using the autonomous driving functions. Furthermore, the processor can be designed to carry out the testing using various driving parameters for the maneuver corresponding in the cluster using the autonomous driving functions and to verify the scenario shown in the cluster with the driving parameters used as valid if the maneuver sequence describing the cluster and the maneuver sequence obtained by testing using various driving parameters are the same.

By testing using different driving parameters, a maneuver sequence is obtained by the autonomous driving functions. The scenario shown with the driving parameters and the driving parameters used is verified as valid if the maneuver sequence describing the cluster and the maneuver sequence obtained by the driving functions are the same.

For example, for a maneuver type "longitudinal movement" with the maneuver steps and the maneuver sequence "maintain speed", "stop", "standstill" and "maintain speed", the same order must be maintained for the autonomous driving functions. This also applies to the other maneuver types. If the order is not maintained, the tested scenario is verified as invalid.

This makes it possible to filter out those driving parameter variations that do not correspond to the scenario in the cluster. The maneuver matrix and the testing of the autonomous driving functions in the individual clusters that make up the test environment can thus identify incorrectly tested scenarios. This can prevent incorrect statements regarding the validation process of the autonomous driving functions.

In a further embodiment, the module is designed to recognize the individual maneuver steps for each maneuver type in a scenario, wherein the scenario is assigned to an environment, and wherein the module is designed to use the map data corresponding to the environment to recognize the individual maneuver steps.

Map data are digital maps/road maps. Using map data increases the robustness of maneuver step recognition. For example, it is recognized when an ego vehicle turns right at an intersection by combining the information, in this case the intersection, from the map and the ego trajectory. The recognition of maneuver steps can be carried out safely and automatically based on rules using the map data.

In a further development, the comparison module is designed to view two maneuver matrices as equal or almost equal or similar based on the number of matches and/or mismatches of the individual maneuver sequences of the individual maneuver types in the maneuver matrix. For example, a scenario can be assigned to a cluster even if there is a slight inequality/difference (not 100% match), for example if one ego vehicle stops at an intersection longer than another. Furthermore, a cluster with identical or sufficiently similar scenarios can be formed in this way in a simple manner.

In further development, the comparison module is designed to create a parameter distribution for comparing parameters of the individual maneuvers in the maneuver matrices belonging to a cluster, whereby each of the maneuver matrices has the same maneuver sequences. Since all scenarios in a cluster have the same maneuver sequence, a meaningful parameter distribution can be created for the parameters.

This means that the parameters of the maneuvers, such as the duration of lane-changing maneuvers, speeds, etc., can be used for further analysis.

• - Generieren von verschiedenen Szenarien aus der Sicht des Egofahrzeugs mit einer Egotrajektorie anhand eines Simulators und/oder Bereitstellen von Realdaten von verschiedenen Szenarien aus der Sicht des Egofahrzeugs mit einer Egotrajektorie,
• - Bewerkstelligen für jeden Zeitschritt der Egotrajektorie in den Szenarien eines automatisierten Zuweisens eines Manöverschrittes für jeden Manövertypen, wobei die verschiedenen Manövertypen mit auswählbaren Manöverschritten als Manöverkatalog bereitgestellt werden, und Bewerkstelligen für jeden Zeitschritt von jeder Verkehrsteilnehmertrajektorie jedes Verkehrsteilnehmers in den Szenarien eines automatisierten Zuweisens eines Manöverschrittes für jeden Manövertypen, wobei die verschiedenen Manövertypen mit auswählbaren Manöverschritten als Manöverkatalog bereitgestellt werden, so dass durch die Zuweisungen eine Manövermatrix mit unterschiedlichen manövertypabhängigen Manöverreihenfolgen zu jedem Manövertyp generiert wird, und
• - automatisiertes Zusammenfassen der gleichen Manövermatrizen zu einem Cluster als Manöver,
• - Bestimmen der Größe der Cluster als Häufigkeit von gleichen oder ähnlichen Szenarien,
• - Erstellen einer Prüfungsumgebung zum Testen der autonomen Fahrfunktionen anhand der Größe der Cluster,
• - Testen der autonomen Fahrfunktionen des Egofahrzeugs in der Prüfungsumgebung.

The object is further achieved by a method for creating a virtual test environment based on a recognized frequency of identical or similar scenarios for testing autonomous driving functions of an ego vehicle, comprising the steps:

• - Generating different scenarios from the perspective of the ego vehicle with an ego trajectory using a simulator and/or providing real data of different scenarios from the perspective of the ego vehicle with an ego trajectory,
• - Carrying out an automated assignment of a maneuver step for each maneuver type for each time step of the ego trajectory in the scenarios, whereby the various maneuver types are provided with selectable maneuver steps as a maneuver catalog, and Carrying out an automated assignment of a maneuver step for each maneuver type for each time step of each road user trajectory of each road user in the scenarios, whereby the various maneuver types are provided with selectable maneuver steps as a maneuver catalog, so that the assignments generate a maneuver matrix with different maneuver type-dependent maneuver sequences for each maneuver type, and
• - automated grouping of the same maneuver matrices into a cluster as maneuvers,
• - Determining the size of the clusters as the frequency of identical or similar scenarios,
• - Creating a test environment to test the autonomous driving functions based on the size of the clusters,
• - Testing the autonomous driving functions of the ego vehicle in the test environment.

The method is particularly designed to be carried out on the system according to the invention. Furthermore, the advantages and advantageous embodiments of the system can be transferred to the method.

In a further embodiment, the testing is carried out by simulating the maneuver in the corresponding cluster based on the autonomous driving functions available in the later ego vehicle.

In further training, testing using different driving parameters for the maneuver corresponding in the cluster is carried out by the autonomous driving functions and the scenario presented in the cluster with the driving parameters used is verified as valid if the maneuver sequence describing the cluster and the maneuver sequence obtained by testing using different driving parameters are the same.

In a further development, the map data corresponding to the environment is used to recognize the individual maneuver steps for each maneuver type in a scenario, whereby the scenario is assigned to an environment. The maneuver steps can be recognized safely and automatically in a rule-based manner using the map data.

In further training, the start and end of a scenario is determined by changing a maneuver step of a single, predetermined maneuver type. This makes it easy to determine a scenario. To create a cluster, the columns and rows of the maneuver matrices are compared. In further training, two maneuver matrices are considered to be the same or similar based on the number of matches and/or mismatches in the individual maneuver sequences of the individual maneuver types in the maneuver matrix.

Furthermore, a score can be awarded at least for matches, whereby two maneuver matrices can be considered equal depending on the score.

In further training, each time step is designed as a predetermined time period. This allows overly difficult maneuver steps to be avoided.

In further training, a parameter distribution is created to compare parameters of the individual maneuvers in the maneuver matrices belonging to a cluster, with each of the maneuver matrices having the same maneuver sequences. Since all scenarios have the same maneuver sequences, a meaningful parameter distribution for the parameters can be created.

• 1: ein erfindungsgemäßes System schematisch,
• 2: einen Manöverkatalog mit Manövertypen dazu passenden Manöverschritten, Parametern und Beschreibung des Manövertyps,
• 3: ein Szenario mit dem Manöverkatalog für das Egofahrzeug,
• 4: ein weiteres Szenario mit dem Manöverkatalog für das Egofahrzeug,
• 5: verschiedene Manövertypen mit den auswählbaren verschiedenen Manöverschritten,
• 6: eine Arbeitsweise eines Vergleichsmoduls,
• 7: mögliche Fahrparametervariationen in den autonomen Fahrfunktionen bei Übereinstimmung mit dem entsprechenden Cluster,
• 8: mögliche Fahrparametervariationen in den autonomen Fahrfunktionen bei nicht Übereinstimmung mit dem entsprechenden Cluster.

Further features and advantages of the present invention will become apparent from the following The following description is given with reference to the accompanying figures, in which:

• 1 : a system according to the invention schematically,
• 2 : a maneuver catalog with maneuver types, corresponding maneuver steps, parameters and description of the maneuver type,
• 3 : a scenario with the maneuver catalog for the ego vehicle,
• 4 : another scenario with the maneuver catalog for the ego vehicle,
• 5 : different maneuver types with selectable different maneuver steps,
• 6 : a way of working of a comparison module,
• 7 : possible driving parameter variations in the autonomous driving functions in accordance with the corresponding cluster,
• 8th : possible driving parameter variations in the autonomous driving functions if there is a discrepancy with the corresponding cluster.

In order to be able to estimate the risk of autonomous driving functions, knowledge of possible traffic scenarios is necessary. A scenario is defined by the static environment such as the road network, buildings and the behavior of other road users; i.e. a scenario is an abstract, semantic description, defined by the ego vehicle and the relevant environment. Autonomous driving functions are all functions that contribute to autonomous driving, this can also include driving functions of level 3 (automated mode) or an assisted mode. Such driving functions can be, for example, lane keeping, speed adjustment, etc.

As many of the scenarios that may occur when the driving function is in operation as possible must be known. In order to estimate the risk, information on the frequency of these scenarios is also required. Traditionally, this information is not collected, but decisions are made on the basis of possibly incomplete expert knowledge. Alternatively, data collections such as accident databases can be used, but these often do not describe things sufficiently and only represent a small part of the traffic situation. The problem is determining the traffic scenarios that occur and their frequency, based on a known route.

1 shows a system 1 according to the invention for creating a virtual test environment based on a recognized frequency of identical or similar scenarios for testing the autonomous driving functions 3 for an ego vehicle 4.

The system 1 comprises a simulator 5 for generating different scenarios 2 from the perspective of the ego vehicle 4, which travels along an ego trajectory 7.

Such simulations can be created, for example, by commercial providers. These recreate realistic traffic scenarios from the perspective of the ego vehicle 4, as well as the interaction with other road users. This allows a large number of scenarios 2 to be generated within a short period of time.

Furthermore, the scenarios 2 can be provided using real data via an interface 6. Such real data can come from traffic monitoring, ego vehicles 4 equipped with sensors, drones, etc.

Furthermore, a module 9 is designed to carry out an automated assignment of a maneuver step for each maneuver type for each time step of the ego trajectory 7 in a scenario 2. Furthermore, the module 9 is also designed to carry out an automated assignment of a maneuver step for each maneuver type for each time step of the road user trajectory of each road user in the scenario 2. The various maneuver types with selectable maneuver steps are provided as a maneuver catalog.

The two assignments result in a maneuver matrix M ( 5 ) with different maneuver type-dependent maneuver sequences for each maneuver type for each road user and the ego vehicle 4.

A maneuver catalog provides a collection of maneuver types necessary to complete the driving task.

One such maneuver type is, for example, “longitudinal movement” or “turning,” which describes behavior at intersections.

Maneuver steps of the "longitudinal movement" maneuver type can be "steady motion", "braking" or "standstill". Maneuver steps of the "turning" maneuver type can be "turn left" or "turn right" for the direction when crossing an intersection or "no turn" if the participant does not perform a turning maneuver.

2 shows such a maneuver catalog with corresponding possible maneuver steps as also the parameters that can be recorded. Furthermore, a description of the maneuver step is given in the last column.

The maneuver catalog is not complete. The maneuver catalog is used for each scenario and filled out accordingly for each road user and the ego vehicle 4.

3 shows a scenario 2, here a turning maneuver, with a maneuver matrix M created for it. A maneuver step is detected at any time for each maneuver type. This semantically abstracts the trajectories of both the ego vehicle 4 and the road users, here the pedestrian 8. A maneuver matrix M contains the time steps as columns and the extracted maneuver steps of each maneuver type as rows and can thus serve as a formalized abstract representation for further processing.

For each maneuver type, a maneuver step is specified for each road user and the ego vehicle 4; this is shown here in the maneuver matrix M only for the ego vehicle 4. Likewise, such a maneuver matrix is created for the pedestrian 8, which is not specified here or is not added to the maneuver matrix M.

4 shows another scenario 2c. This shows an overtaking maneuver of the ego vehicle 4, as well as the maneuver matrix M2 of the ego vehicle 4. The associated maneuver matrix M2 is only given for the ego vehicle 4, not for the preceding vehicle 10 involved in scenario 2c.

The maneuver types “longitudinal movement”, “lateral movement”, “intersection”, “pedestrian right” and “gone ahead” are displayed over time t.

In 5 For better understanding, further possible maneuver steps are shown that can be selected for maneuver types. The maneuver types "longitudinal movement", "lateral movement", "intersection", "intersection context", "road user context" are shown here with the possible maneuver steps: "constant speed", approach, follower, stop and standstill for the maneuver type "longitudinal movement".

For the maneuver type “lateral movement,” however, “keep lane” and “change lane” are displayed, from which a suitable maneuver step is selected for each time step.

For the maneuver type “intersection,” however, “turn right” and “turn left,” as well as “cross intersection” and “no intersection” are displayed, from which a suitable maneuver step is selected for each time step.

For the maneuver type “crossing context,” however, “crossing,” “threading,” and “unthreading” are displayed, from which a suitable maneuver step is selected for each time step.

For the maneuver type “Road user context”, however, “Following” and “Left” as well as “Right” are displayed, from which a maneuver step is selected at each time step.

The road users belonging to the ego vehicle 4 and the scenario 2,2c can be determined by algorithm, for example, only road users who are within a specified radius of the ego vehicle 4 can be considered. Maneuver steps for uninvolved, distant road users, for example, do not have to be displayed in the maneuver matrix M and are not considered to be part of scenario 2.

To recognize the individual maneuver steps for each maneuver type in a scenario, the module 9 can use the corresponding digital map data (digital map/road map) for the environment in which the scenario 2,2c takes place. This increases the robustness of the maneuver step recognition by enriching map data. For example, it is recognized when an ego vehicle 4 turns right at an intersection by combining the information from the map and the ego trajectory 7. Follow-up maneuvers are recognized, for example, via the difference in speed between two vehicles in the same lane.

The recognition of the maneuver steps is rule-based and automated.

A maneuver step is selected for each time step, whereby a time step can also be a time duration.

Furthermore, a comparison module 11 ( 1 ) is available for the automated grouping of the same maneuver matrices M into a cluster C as maneuvers, and for determining the size of the clusters C as the frequency of identical or similar scenarios.

6 shows the operation of such a comparison module 11. This compares the maneuver matrices M generated by module 9 and clusters identical maneuver matrices M into a cluster, here C1. This means that scenarios 2 with identical maneuver sequences are grouped together into cluster C1.

Scenarios 2b, which also have a pedestrian 8 and an ego vehicle 4 but a different trajectory 7a, generate a maneuver matrix M1 and are clustered into a cluster C2.

The start and end of a scenario 2,2b,2c is determined by changing the maneuver steps of a single maneuver type. Depending on the focus of the analysis, sensible definitions of start and end points can vary. To obtain the maneuver matrix of a scenario, the corresponding columns are cut out. A pattern matching algorithm can be used to compare maneuver matrices, which can identify sufficient similarity and group the same maneuver matrices into a cluster. Minor deviations can be permitted.

The technical implementation as a formal maneuver description language enables an automated process that enables the processing of large amounts of data.

In addition, the parameters of the maneuvers, such as the duration of lane change maneuvers and speeds, can be used for further analyses. Since all scenarios 2, 2b and 2c have the same maneuver sequences, a meaningful parameter distribution can be created for the parameters.

Subsequently, a processor 12 ( 1 ) a test environment for testing the autonomous driving functions 3 is generated based on the size of the clusters. Scenarios 2,2b,2c must be handled reliably with high frequency by the autonomous driving functions 3 with a wide variety of driving parameters, so that this can be taken into account in the generation of a test environment.

A frequently occurring scenario is a scenario that occurs frequently compared to another, i.e. a scenario that occurs again and again. Instead of working with the complete set of scenarios, each cluster can be represented by one member. The frequency of the same or similar scenarios recognized thus serves as a basis for prioritization when generating the test environment. For example, scenarios that occur frequently are carried out by the autonomous driving functions 3 with particularly high driving parameter variations in order to guarantee a safe autonomous driving function 3 in such a frequently occurring scenario. This means that, for example, a turning maneuver with a pedestrian is carried out with the driving parameter variation high/low/medium speed when turning the ego vehicle 4, high speed of the pedestrian, black ice, etc.

When testing scenarios 2, 2b, 2c, the autonomous driving functions 3 always store specific values for the road users and the ego vehicle 4 itself. During such testing, the scenarios 2, 2b belonging to cluster C1, C2 are varied and must be mastered by the autonomous driving functions 3. This means that with a maneuver sequence in the maneuver matrix M (maneuver type longitudinal movement), here for example "maintain speed (cruise)", "stop", "standstill" and "maintain speed (cruise)", which for example has the maneuver sequence in the cluster C1 to be tested, the autonomous driving functions 3 for this cluster C1 with the driving parameters set in the autonomous driving functions 3 must have exactly the same maneuver sequence; otherwise the scenario is considered not to have been mastered or the driving parameters set in the driving functions 3 are incorrect. This means that the driving parameter space to be tested is too wide. This means that the same/desired scenario was not tested. For example, if the driving parameters in the autonomous driving functions 3 in the cluster C1 to be tested are set incorrectly, the sequence "maintain speed (Cruise)", "Stop", "Standstill" and "maintain speed (Cruise)" is not tested, because in the simulation with "incorrect" driving parameters the pedestrian 8 crosses the road after passing the ego vehicle 4.

Therefore, by testing using different driving parameters, the maneuver describing the cluster, which is carried out by the autonomous driving functions 3, the scenario 2,2b shown in the cluster C1,C2 is verified as valid with the driving parameters used if the maneuver sequence describing the cluster C1,C2 is the same as the maneuver sequence generated by the driving parameters. For example, for the maneuver type "longitudinal movement", the sequence "maintain speed (cruise)", "stop", "standstill" and "maintain speed (cruise)" must be followed. This also applies to the other maneuver types. If the sequence is not adhered to, the tested scenario is verified as not valid.

Thus, the maneuver matrix M,M1 can be used to filter out those driving parameter variations that do not correspond to the original and expected scenario 2, 2b, i.e. a scenario that is not assigned to the cluster C1,C2, for example.

The maneuver matrix M, M1 and the testing of the autonomous driving functions 3 in the individual clusters C1, C2, which make up the test environment, can thus identify false scenarios. This can prevent false statements regarding the validation process of the autonomous driving functions 3.

In this way, driving parameter variations that belong to an incorrect scenario, i.e. that do not correspond to the original and expected scenario 2,2b, can be filtered out, which accelerates or improves a safeguarding process; for example, if the speed of the ego vehicle 4 in cluster 2 is increased, the pedestrian 8 can cross the street after the ego vehicle 4 has crossed the intersection, which corresponds to a different scenario than that in cluster 1. If driving parameter variations do not belong to a scenario in a cluster, this means that the driving parameter space to be checked has been expanded too widely. By checking the sequence of maneuvers, such incorrect settings can be quickly identified.

7 shows an example of testing the scenario 2c with the autonomous driving functions 3, wherein the scenario 2c is assigned to the cluster C3, under different driving parameter variations F1,F2,F3, wherein each of the selected driving parameter variations shows the desired cluster C3, ie the same maneuver matrix M2 ( 4 ). Scenario 2c corresponds to the overtaking maneuver of ego vehicle 4.

8th shows possible driving parameter variations in the autonomous driving functions 3 of scenario 2c, in which the created maneuver matrix no longer corresponds to the desired maneuver matrix. For example, Sa corresponds to scenario 2c with driving parameter variations F1, F2, F3; however, in scenario Sb no overtaking maneuver takes place, for example the speed of the ego vehicle 4 may be too slow (driving parameter variation F4). Likewise, in scenario Sc no overtaking maneuver takes place (driving parameter variation F5). Thus, the variations F4, F5 of the driving parameters during testing do not correspond to the originally desired scenario 2c present in the cluster C3 to be tested.

A maneuver matrix can easily filter out such “wrong” scenarios, which produce an incorrect result with regard to testing the autonomous driving functions 3.

By means of the system 1 according to the invention and the method according to the invention, traffic scenarios 2, 2b, 2c which are contained in the simulation data/real data can be processed and analyzed via a maneuver matrix M, M1, M2 and the maneuver sequences contained therein. The abstraction of the trajectory data into maneuvers allows a semantic analysis of the data. Clusters C1, C2, C3 are generated based on the maneuver sequences. The scenarios 2, 2b, 2c contained in the clusters C1, C2, C3 can be used to recognize common scenarios 2, 2b, 2c and relevant information on the common scenarios 2, 2b, 2c. Based on the obtained common scenarios 2, 2b, 2c, a test environment for testing autonomous driving functions 3 can be created, which serves to safeguard the autonomous driving functions 3. The maneuver sequence stored behind the cluster C1, C2, C3 can be used to detect when the driving parameter space being tested is too wide and a scenario is being tested that no longer corresponds to the original scenario 2,2b,2c.

Automatically varied and simulated scenarios can be validated automatically and without manual effort at an abstract level using System 1 and the procedure. This saves a lot of manual effort and therefore costs, since the variation can lead to a large number of simulated scenarios. Furthermore, the test result can be easily validated, since the sequence of maneuvers tells us whether the results are based on valid scenarios. The technical implementation as a formal description language enables an automated process that is designed for large amounts of data.

List of reference symbols

1

system

2,2b,2c

scenario

3

autonomous driving functions

4

Ego vehicle

5

simulator

6

interface

7

Ego trajectory

8th

pedestrian

9

module

10

vehicle ahead

11

Comparison module

12

processor

M,M1,M2

Maneuver matrix

C1,C2,C3

Cluster

F1,..,F6

Driving parameter variations

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited patent literature

• WO 2022028935 A1 [0009]

Claims (15)

System (1) for creating a virtual test environment based on a recognized frequency of identical or similar scenarios (2,2b,2c) for testing the autonomous driving functions (3) for an at least partially autonomously operable ego vehicle (4), comprising a simulator (5) for generating different scenarios (2,2b,2c) from the perspective of the ego vehicle (4) with an ego trajectory (7) and/or an interface (6) for receiving real data from different scenarios (2,2b,2c) from the perspective of the ego vehicle (4) with an ego trajectory (7), characterized in that a module (9) is provided which is designed to carry out an automated assignment of a maneuver step for each maneuver type for each time step of the ego trajectory (7) in the scenarios (2,2b,2c), wherein the different maneuver types with selectable maneuver steps are provided as a maneuver catalog and for each time step of each road user trajectory each Road user in the scenarios (2,2b,2c) to accomplish an automated assignment of a maneuver step for each maneuver type, wherein the different maneuver types with selectable maneuver steps are provided as a maneuver catalog, so that a maneuver matrix (M,M1,M2) with different maneuver type-dependent maneuver sequences for each maneuver type is generated by the assignments, and a comparison module (11) is provided, which is designed to automatically combine the same maneuver matrices (M, M1, M2) into a cluster (C1, C2, C3) as maneuvers, and to determine the size of the clusters (C1, C2, C3) as the frequency of identical or similar scenarios (2,2b,2c), and wherein a processor (12) is provided, which is designed to create a test environment for testing the autonomous driving functions (3) based on the size of the clusters (C1, C2, C3) and for testing the autonomous Driving functions (3) of the ego vehicle (4) are trained in the test environment. System (1) according to Claim 1 , characterized in that the processor (12) is further designed to carry out the testing by simulating the maneuver in the corresponding cluster (C1, C2, C3) using the autonomous driving functions (3). System (1) according to Claim 2 , characterized in that the processor (12) is further designed to carry out the testing by means of various driving parameters for the maneuver corresponding in the cluster (C1, C2, C3) by the autonomous driving functions (3) and to verify the scenario (2,2b,2c) presented in the cluster (C1, C2, C3) with the driving parameters used as valid if the maneuver sequence describing the cluster (C1, C2, C3) and the maneuver sequence obtained by testing by means of various driving parameters are the same. System (1) according to one of the preceding claims, characterized in that the module (9) is designed to recognize the individual maneuver steps for each maneuver type in a scenario (2,2b,2c), wherein the scenario (2,2b,2c) is assigned to an environment, and wherein the module (9) is designed to use the map data corresponding to the environment to recognize the individual maneuver steps. System (1) according to one of the preceding claims, characterized in that the comparison module (11) is designed to consider two maneuver matrices (M, M1, M2) as equal or similar based on the number of matches and/or mismatches of the individual maneuver sequences of the individual maneuver types in the maneuver matrix (M, M1, M2). System (1) according to one of the preceding claims, characterized in that the comparison module (11) is designed to create a parameter distribution for comparing parameters of the individual maneuvers in the maneuver matrices (M, M1, M2) belonging to a cluster (C1, C2, C3), wherein each of the maneuver matrices (M, M1, M2) has the same maneuver sequences. Method for creating a virtual test environment based on a recognized frequency of identical or similar scenarios (2,2b,2c) for testing the autonomous driving functions (3) for an at least partially autonomously operable ego vehicle (4), comprising the steps of: - generating different scenarios (2,2b,2c) from the perspective of the ego vehicle (4) with an ego trajectory (7) using a simulator (5) and/or providing different scenarios (2,2b,2c) from the perspective of the ego vehicle (4) with an ego trajectory (7) using real data, - carrying out an automated assignment of a maneuver step for each maneuver type for each time step of the ego trajectory (7) in the scenarios (2,2b,2c), wherein the different maneuver types with selectable maneuver steps are provided as a maneuver catalog, and carrying out a automated assignment of a maneuver step for each maneuver type, whereby the different maneuver types with selectable maneuver steps are assigned as maneuvers catalog are provided so that the assignments generate a maneuver matrix (M, M1, M2) with different maneuver type-dependent maneuver sequences for each maneuver type, and - automated grouping of the same maneuver matrices (M, M1, M2) into a cluster (C1, C2, C3) as maneuvers, and - determining the size of the clusters (C1, C2, C3) as the frequency of identical or similar scenarios (2,2b,2c), and - creating a test environment for testing the autonomous driving functions (3) based on the size of the clusters (C1, C2, C3), and - testing the autonomous driving functions (3) of the ego vehicle (4) in the test environment. Procedure according to Claim 7 , characterized in that the testing is carried out by simulating the maneuver in the corresponding cluster (C1, C2, C3) using the autonomous driving functions (3) present in the subsequent ego vehicle (4). Procedure according to Claim 8 , characterized in that the testing by means of different driving parameters for the maneuver corresponding in the cluster (C1, C2, C3) is carried out by the autonomous driving functions (3) and the scenario (2,2b,2c) presented in the cluster (C1, C2, C3) is verified as valid with the driving parameters used if the maneuver sequence describing the cluster (C1, C2, C3) and the maneuver sequence obtained by testing by means of different driving parameters are the same. Method according to one of the preceding Claims 7 until 9 , characterized in that for the recognition of the individual maneuver steps for each maneuver type in a scenario (2,2b,2c), wherein the scenario (2,2b,2c) is assigned to an environment, the map data corresponding to the environment are used. Method according to one of the preceding Claims 7 until 10 , characterized in that the beginning and the end of a scenario (2,2b,2c) are determined by the change of a maneuver step of a single, predetermined maneuver type. Method according to one of the preceding Claims 7 until 11 , characterized in that two maneuver matrices (M, M1, M2) are considered to be equal or similar based on the number of matches and/or mismatches of the individual maneuver sequences of the individual maneuver types in the maneuver matrix (M, M1, M2). Procedure according to Claim 12 , characterized in that a score is awarded at least for matches and two maneuver matrices (M, M1, M2) are considered equal depending on the score. Method according to one of the preceding Claims 7 until 13 , characterized in that each time step is designed as a predetermined time period. Method according to one of the preceding Claims 7 until 14 , characterized in that a parameter distribution is created to compare parameters of the individual maneuvers in the maneuver matrices (M, M1, M2) belonging to a cluster (C1, C2, C3), wherein each of the maneuver matrices (M, M1, M2) has the same maneuver sequences.

Images