DE102020210964A1 - Computer-implemented method for verifying an error rate of autonomously driving vehicles, computer systems and data carriers - Google Patents

Abstract

The invention relates to a computer-implemented method for verifying an error rate P(crashh) of autonomously driving vehicles, with the steps: - providing functional environment scenarios using a microsimulation or real test drives, - microscopic quantification of the functional environment scenarios using the criticality to identify critical functional environment scenarios, - Converting the critical functional environmental scenarios into at least critical logical environmental scenarios using a statistical experimental design, whereby initial parameters are defined by means of the statistical experimental design, on which a logical environmental scenario depends, - generating critical specific environmental scenarios using a nano-simulation, using the initial parameters the critical logical environment scenarios, - nanoscopic verification of the criticality of the critical functional environment scenarios based on the determination of the criticality of critical specific environmental scenarios. The invention also relates to a computer system and a data carrier.

Description

The invention relates to a computer-implemented method for verifying an error rate of autonomously driving vehicles. Furthermore, the invention relates to a computer system and a data carrier.

Fully autonomous vehicles (also known as AD systems) have a number of clear advantages over vehicles with manual operation. In this way, road safety can be improved through fast reaction times and the avoidance of human error. An optimal driving style can also reduce fuel consumption. Elderly road users can also remain mobile thanks to autonomously driving vehicles. Networking can also reduce accidents.

However, the development costs of AD systems are enormous. Economic aspects such as cost efficiency, pressure to succeed and time-to-market considerations are therefore playing an increasingly important role.

However, the legitimate influences on economic efficiency must be actively managed in order to guarantee a safe development and ultimately a safe product at the same time.

The safety rules are defined, for example, by SOTIF (Safety of the Intended Functionality), which contains guidelines for autonomously driving vehicles according to levels 0, 1 and 2 (and higher).

All these factors have an increased impact on the development phase of the AD system. The question of the safe operation of an AD prototype, which is still in an early development phase, in mixed traffic operation is therefore a very complex task. Safe operation is determined, among other things, by an error rate.

An error rate or failure rate is also referred to as a hazard rate in the event time analysis in statistics. From this, the probability can be determined that a certain event (e.g. death of a person, sale of goods, decay of a radioactive element) will occur at a fixed point in time.

In order to make an autonomous vehicle suitable for series use, it is usually tested in real traffic or in a simulation. This means that a large number of test kilometers are driven with real test vehicles or in the simulation. This leads to a high test effort in terms of time and costs. However, this poses a number of problems for further development.

the DE 10 2016 109 651 A1 discloses a method for validating at least one vehicle function of a virtual autonomous vehicle, comprising the following steps: starting a computer-aided simulation program for simulating the virtual autonomous vehicle and its autonomous movement in a virtual environment by specifying a virtual destination, generating simulated input parameters for at least a system component of a control system of the autonomous vehicle based on the simulation, which correspond to real input parameters of the at least one system component of the control system, generating output parameters of the at least one system component of the control system using the at least one system component of the control system based on the simulated input parameters and logging the behavior of the virtual autonomous vehicle.

the DE 10 2006 044 086 B4 discloses a system for virtual simulation of traffic situations, with a display unit for pictorial display of the virtual traffic situations and an input unit, and with at least one modeled reference vehicle and at least one other modeled road user. The reference vehicle has a modeled detection device for detecting the virtual environment of the reference vehicle, the behavior of the reference vehicle during a simulated traffic situation taking into account the information detected by the detection device at least at times, and the reference vehicle is based on a base model which is used during a simulation run coupled with behavioral patterns selected according to a stochastic model.

It is therefore an object of the invention to specify an improved method and an improved computer system for the further development of an autonomously driving vehicle. It is also an object to specify a data carrier.

The object is achieved by a method having the features of claim 1, a computer system having the features of claim 13 and a data carrier having the features of claim 14.

Further details of the invention and advantages of various embodiments emerge from the features of the dependent claims.

• - Bereitstellen von funktionalen Umfeldszenarien anhand einer Mikrosimulation oder realen Testfahrten,
• - Mikroskopische Quantifizierung der funktionalen Umfeldszenarien anhand einer Kritikalität, zum Identifizieren kritischer funktionaler Umfeldszenarien,
• - Umwandeln der kritischen funktionalen Umfeldszenarien in zumindest kritische logische Umfeldszenarien unter Verwendung einer statistischen Versuchsplanung, wobei mittels der statistischen Versuchsplanung initiale Parameter definiert werden, von dem ein logisches Umfeldszenario abhängig ist,
• - Generieren anhand einer Nanosimulation, unter Heranziehung der initialen Parameter, kritischer konkreter Umfeldszenarien aus den kritischen logischen Umfeldszenarien,
• - Nanoskopische Verifizierung der Kritikalität der kritischen funktionalen Umfeldszenarien anhand der Bestimmung der Kritikalität der kritischen konkreten Umfeldszenarien.

The object is achieved by a computer-implemented method for verifying an error rate of autonomously driving vehicles, comprising the steps:

• - Provision of functional environment scenarios based on a micro-simulation or real test drives,
• - Microscopic quantification of the functional environment scenarios based on a criticality to identify critical functional environment scenarios,
• - Conversion of the critical functional environment scenarios into at least critical logical environment scenarios using a statistical experimental design, with the statistical experimental design being used to define initial parameters on which a logical environmental scenario is dependent,
• - Generate based on a nano-simulation, using the initial parameters, critical concrete environmental scenarios from the critical logical environmental scenarios,
• - Nanoscopic verification of the criticality of the critical functional environment scenarios based on the determination of the criticality of the critical concrete environment scenarios.

A scene describes a static environment given by an unchanging structure. Scenes are street information such as intersections, roundabouts, bus stops, motorway exits and additional static environments such as buildings.

A situation is the description of the vehicle's intent for a particular scene.

Scenarios, which are referred to here as environment scenarios for better differentiation, mean the description of all dynamic objects in the static environment. Dynamic objects are, for example, pedestrians, traffic vehicles and the respective environmental conditions, which can lead to different reaction needs/possibilities of the dynamic objects. Such environmental scenarios depend on many different parameters that vary in time, frequency, speed, distance, etc.

Functional environment scenarios can be generated, for example, by real test drives with a vehicle in an autonomous (or non-autonomous and with a correspondingly designed sensor system in the vehicle) or by appropriate microsimulation.

Environmental scenarios can be divided into known secure and insecure environmental scenarios and unknown secure and insecure environmental scenarios. Unknown environmental scenarios are environmental scenarios that were not previously known, for example the existence of objects that have not yet occurred. Furthermore, an unknown parameterization, for example the speed of a road user, can lead to unknown environmental scenarios. The environmental scenarios depend on many different parameters that vary in time, frequency, speed, distance, etc.

Critical environmental scenarios are to be understood here as preferably critical, known, insecure environmental scenarios; however, the procedure is generally applicable to all.

In addition to the division into scene, situation, environment scenario, environment scenarios, a division into logical, concrete and functional scene, situation, environment scenario/environment scenarios can also be made. A functional environment scenario is a description in words (prose) without a declaration or definition of parameters that belong to the environment scenario. These are, for example, the description of a 3-lane motorway, a speed limit of 100 km/h due to traffic signs, traffic jams, etc.

A logical environment scenario is the functional description, with the declaration of parameters in a parameter area, to describe the environment scenario in a mathematical/physical way, for example lane width between 2.3 and 3.5 m, position of traffic signs between 0 and 200 m Etc.

A concrete environmental scenario is a logical but parameterized environmental scenario. For example, the width of the lanes can be 3.2 m and the position of the traffic signs can be 150 m.

To differentiate between safe and unsafe (critical) environmental scenarios, a measure is required with which the dangerousness of an environmental scenario can be assessed.

For this purpose, the already mentioned criticality is used. Criticality describes the probability that an accident will occur in the further course of an environmental scenario. If this probability is 100%, this corresponds to a contact between the vehicle and another object/item/vehicle. The criticality can be described multidirectionally, so that both the longitudinal and lateral criticality can be taken into account.

There are several methods to obtain a measure of criticality. These are divided into rule-based, potential-based and trajectory-based methods.

Such measures are, for example, Time to Collision (TTC), Deceleration to avoid the Crash (DRAC, deceleration rate to avoid the crash), Distance to Collision (DTC), Deceleration to avoid the Crash on Brake (DRAC2 , braking to avoid the crash).

A potential-based measure, for example the time to collision (Time to Collision (TTC)) and/or the Distance to Collision (DTC; collision distance) is preferably used as a measure for the criticality. This is preferably determined using a potential-based method. Time to Collision (TTC) is an important time-based safety indicator for detecting rear-end collisions in road safety assessments. This corresponds to a simple measure of criticality, which reduces the amount of data.

A macro simulation is understood here as the modeling of the traffic flow. Traffic flow is modeled using traffic flow theory, which describes the behavior of traffic volume, traffic speed and traffic density over time and space. Traffic flow means the flow or flex, i. H. the number of traffic elements (e.g. vehicles) that cross a certain traffic area or line (as the limiting case of the area) per unit of time. Macro simulations can be used to generate/determine the number of vehicles at a given time in a given place to use as input for the micro simulation.

A microsimulation can be understood as the modeling of traffic flows based on the description of the movement of each individual vehicle that makes up the traffic flow. This implies modeling the actions, such as acceleration, deceleration and lane changing, of each driver in response to the surrounding traffic with the specification of parameter ranges.

In terms of the macro simulation, which is similar to a hydrodynamic model, the micro simulation is similar to a multi-body model, where each has an effect on its neighboring bodies. The micro simulation receives the macro simulation as input data.

Nano simulation is the simulation of the vehicle behavior itself. While microsimulation describes the body's path, nanosimulation describes the body's behavior along this path and the possibilities of reacting and interacting with the environment.

Here, using a nano-simulation, critical concrete environmental scenarios are generated from the critical logical environmental scenarios, for which the initial parameters of each logical critical environmental scenario are used. This means that the various concrete environmental scenarios are generated from the behavior of the road users involved and the related reactions of the autonomous vehicle.

Failure rate / error rate is the frequency with which a technical system or component fails, expressed in failures per unit of time. In order to verify the mode of operation of an autonomously driving vehicle, the vehicle has to cope with a large number of different specific environmental scenarios during its operating time. With the method presented here, critical environmental scenarios in particular can be verified in concrete terms using a microscopic and nanoscopic simulation/environment.

This achieves a focus on the most relevant environmental scenarios and thus the possibility of significantly reducing important test routes. Furthermore, the unknown environment scenarios can be recognized in a simulation using the method.

wobei P(Scenario) die Auftretenswahrscheinlichkeit des kritischen Umfeldszenarios und P(Crash) die Wahrscheinlichkeit ist, in diesem einen Fehler (Crash) zu erfahren und wobei $P\left(\frac{\text{crash}}{\text{h}}\right)$

die Fehlerrate ist. Dies entspricht dem Gesetz der totalen Wahrscheinlichkeit zum Nachweis der gesamten Fehlerrate.The error rate (related to time, preferably an hour) is preferably formed from the sum of the error probabilities of all functional environment scenarios: $$P\left(\frac{\text{crash}}{\text{H}}\right)={\displaystyle \sum P\left(\text{scenario}\right)\ast P\left(CasH\right)}$$

where P(scenario) is the probability of occurrence of the critical environment scenario and P(crash) is the probability of experiencing an error (crash) in this and where $P\left(\frac{\text{crash}}{\text{H}}\right)$

is the error rate. This corresponds to the law of total probability for proving the total error rate.

The microsimulation preferably includes at least the traffic density as a parameter Classes of road users such as pedestrians, cyclists, cars, trucks, etc. and/or the speed of the respective road users and/or the behavior of the road users in relation to lateral changes and/or longitudinal changes, so that either time lines or complete trajectories of all relevant vehicles or the recording of the relevant environmental scenarios can be determined. This means that the traffic density and speed of all possible types of road users and behavior models such as reaction, perception and recognition times are generated.

Statistical data is required for a micro-simulation, which can be obtained in various ways. Starting from a macroscopic level, a traffic planning model is created. With the help of this traffic planning model, the traffic flow can be assumed for, for example, nodes in a coarse-meshed road network across the city. Further data can also be used for the microsimulation.

• - Identifizieren funktionaler kritischer Umfeldszenarien mit einem Schwellenwert der Kritikalität,
• - clustern / klassifizieren der identifizierten kritischen funktionalen Umfeldszenarien durch zuordnen zu einem bekannten kritischen funktionalen Umfeldszenario oder zuordnen zu einem neuen kritischen funktionalen Umfeldszenario,
• - Zählen der identifizierten kritischen funktionalen Umfeldszenarien.

In a further preferred embodiment, the microscopic quantification is carried out by the following steps:

• - Identify functional critical environment scenarios with a criticality threshold,
• - cluster / classify the identified critical functional environment scenarios by assigning them to a known critical functional environment scenario or assigning them to a new critical functional environment scenario,
• - Counting of the identified critical functional environment scenarios.

To identify critical functional environment scenarios, the trajectories are evaluated in each time step according to the criticality function. The resulting criticality is assigned to each time step and can be plotted accordingly over time. A critical environmental scenario begins when a previously defined threshold value is exceeded and ends when it falls below it or after a local maximum has been passed. For this purpose, a threshold value is set in advance, with which a critical environmental scenario can be identified.

This means that the criticality is determined for each time increment, so it is possible to generate a time series of criticality between the vehicle and each other road user or to get an overlay between the vehicle and the environment. Every time this time series exceeds a certain threshold, the constellation of the vehicle and the surrounding objects is interpreted as a critical functional environment scenario.

Each identified critical functional environment scenario must then be classified in order to assign it to one of the already known functional environment scenarios or to define a new functional environment scenario. When all are identified, they can be counted.

Since all identified, functional environment scenarios are assigned to their corresponding classifications, their frequency can be determined (how much is a scenario class / functional environment scenario filled?). The quantified probability of occurrence of the critical functional environment scenarios results from the quotient of the frequency and the simulated time.

The probability of occurrence P(scenario) of critical functional environment scenarios is preferably determined using the quotient of the frequency n (scenario) and the simulated time t(sim): $$\text{P}\left(\text{scenario}\right)=\frac{n\left(S\mathrm{e.g}ena\mathrm{right}iO\right)}{t\left(sim\right)}$$

Each critical functional environment scenario is preferably converted into a critical logical environment scenario using possible influencing parameters and boundary conditions of the critical functional environment scenario.

A design-of-experiments method is preferably used as static test planning in order to convert the critical functional environment scenarios into critical concrete environment scenarios. For this it is necessary to create an initial list of parameters on which a functional environment scenario depends.

By using a design-of-experiments method, the initial parameters can be determined more easily. The aim of a design-of-experiments method is to limit the number of experiments required to determine the influence of parameters on a quality characteristic under investigation to a minimum.

In a so-called fully factorial analysis, all parameters examined must theoretically be tried out completely in order to also include non-linear effects. Since in this so-called full factorial analysis the number of parameters increases exponentially, simplified assumptions and the use of sta tistic evaluation method significantly reduces the number of tests. The influence of the parameters or the parameters to be used can be determined more easily with a design-of-experiments method.

A nano-simulation is then carried out to generate concrete environmental scenarios from the critical logical environmental scenarios by using the initial parameters of each logical critical environmental scenario and generating the various concrete environmental scenarios from the behavior of the road users involved and the reactions of the autonomous vehicle related to this. In this context, the nano simulation is used to stimulate the device under test (DuT).

According to the requirements of the design-of-experiments method, a first, initial set of concrete environmental scenarios or initialized vectors of the scenario parameters is simulated in the nano-simulation. For a closed-loop simulation, the environment scenarios are always only initialized for their start parameters.

Each resulting trajectory then results from the behavior of the road users involved and the related reactions of the autonomous vehicle. The trajectories of all dynamic objects involved are required for the nano-simulation, with which the criticality of the specific environmental scenarios generated can be determined, as was the case after the micro-simulation.

Furthermore, an initial response surface is preferably determined on the basis of the identified critical environment scenarios and the assigned criticality.

In statistics, the response surface (method) examines the relationships between multiple explanatory variables and one or more response variables. The design of experiments is used here to define the test points in order to obtain a suitable response surface efficiently and quickly, which mathematically describes the connection between the parameters of the logical environment scenario and the criticality.

The response surface is preferably represented by a matrix enk, where k: number of relevant parameters and n: number of critical specific environmental scenarios (c1,..,cn): $$enk=\left(\begin{array}{ccc}e11& \cdots & e1k\\ ⋮& \ddots & ⋮\\ en1& \cdots & enk\end{array}\right),n=\left(\begin{array}{c}\mathrm{<figure-callout\; id="1"\; label="c"\; filenames="DE102020210964A1\_0022.png,DE102020210964A1\_0024.png"\; state="\{\{state\}\}">c</figure-callout>}1\\ ⋮\\ cn\end{array}\right)$$

Thus, with the criticality that can be assigned to each specific environmental scenario, a solution vector is obtained with which a system of equations can be solved in such a way that the criticality can be solved with this system of equations for later specific environmental scenarios.

Furthermore, the response surface is preferably designed to estimate the criticality of each specific environmental scenario in advance, that is to say a priori.

The quality of the replacement model depends on the number of all concrete environmental scenarios that have already been tested, but also on the quality of the mathematical description of the physical relationships and the quality of the criticality calculation.

A confidence interval around the response surface is preferably determined, taking into account the number of environmental scenarios that have already been tested. The a priori criticality is in the median of the confidence interval of the replacement model.

Furthermore, the object is achieved by a computer system with memory and processor for verifying an error rate of autonomously driving vehicles, the processor and the memory being designed in such a way that the steps of the method as described above are executed.

The object is also achieved by a data carrier comprising a digital data structure which, after being loaded into a computer memory, executes a method as described above.

Further features, properties and advantages of the present invention result from the following description with reference to the enclosed figures. Variations may be devised by those skilled in the art without departing from the scope of the invention as defined by the following claims.

• 1: das erfindungsgemäße Verfahren im Überblick,
• 2: eine Szene, Situation, Szenario (Umfeldszenario) schematisch,
• 3: eine Unterteilung der Umfeldszenarien funktional, logisch und konkret,
• 4: eine hierarchische Beschreibung von Szenen, Situationen und Umfeldszenarien,
• 5: eine Unterteilung in bekannte und unbekannte Umfeldszenarien,
• 6: eine Makrosimulation, Mikrosimulation und Nanosimulation schematisch,
• 7: die Zusammenhänge zwischen Makrosimulation, Mikrosimulation und Nanosimulation im Detail,
• 8: den Verlauf von Trajektorien jeweils zweier dynamischer Objekte, und den Verlauf der daraus resultierenden Kritikalität über der Zeit,
• 9: ein Design-of-Experiments schematisch,
• 10: das erfindungsgemäße Verfahren bildlich,
• 11: eine Response Surface schematisch.

The figures show schematically:

• 1 : the method according to the invention at a glance,
• 2 : a scene, situation, scenario (environmental scenario) schematically,
• 3 : a subdivision of the environmental scenarios functional, logical and concrete,
• 4 : a hierarchical description of scenes, situations and environmental scenarios,
• 5 : a subdivision into known and unknown environmental scenarios,
• 6 : a macrosimulation, microsimulation and nanosimulation schematic,
• 7 : the relationships between macrosimulation, microsimulation and nanosimulation in detail,
• 8th : the course of trajectories of two dynamic objects, and the course of the resulting criticality over time,
• 9 : a design-of-experiments schematic,
• 10 : the method according to the invention illustrated,
• 11 : a Response Surface schematic.

In a first step S1 in 1 functional environment scenarios are generated using a micro-simulation.

A distinction is made between a scene, a situation and a scenario, with the scenario/scenarios being referred to here as environment scenario/environment scenarios for better differentiation.

2 shows a scene, situation, scenario (environment scenario) schematically A scene describes a static environment that is given by an unchanging structure. Scenes are street information such as intersections, roundabouts, bus stops, freeway exits, as well as static surroundings such as buildings.

A situation is the description of the vehicle's intent for a particular scene.

Environment scenario means the description of all dynamic objects in the static environment. Dynamic objects are, for example, pedestrians, traffic vehicles and the respective environmental conditions, which can lead to different reaction needs/possibilities of the dynamic objects. Such environmental scenarios depend on many different parameters that vary in time, frequency, speed, distance, etc.

Functional environment scenarios can be generated, for example, by real test drives with a sensor system in an autonomous vehicle (or non-autonomous and appropriately configured sensor system in the vehicle) or, for example, by appropriate simulation.

Furthermore, the scenes, situations and environmental scenarios can also be divided into functional, logical and concrete scenes, situations and environmental scenarios. This subdivision is in 3 shown schematically. A functional environment scenario is a description in words (prose) without a declaration or definition of parameters that belong to the environment scenario. These are, for example, the description of a 3-lane motorway, a speed limit of 100 km/h due to traffic signs, traffic jams, or the environment such as rain, summer, etc.

A logical environment scenario is the functional description, with the declaration of parameters in a parameter area to describe the parameter in a mathematical/physical way, e.g. lane width between 2.3 and 3.5 m, position of traffic signs between 0 and 200 m or environment such as temperature between 10 and 40°C.

As a description, a concrete environment scenario corresponds to a logical one, but one with precise parameterization. For example, the lane width may be 3.2 m, the position of traffic signs 150 m, and the temperature 20 °C.

A hierarchical description of scenes, situations and environmental scenarios is given in the 4 given.

According to this comparison, the classifications in this hierarchical approach are used as follows: The concrete abstraction always describes specified test cases and real observations from simulation, test field, field test. The logical/functional descriptions are used to collect and classify the scene/situation/environmental scenarios.

Furthermore, the environmental scenarios or scenes and situations can also be divided into known safe and unsafe (critical) environmental scenarios and unknown safe and unsafe environmental scenarios or scenes and situations.

5 shows such a classification schematically. The division into known and unknown environmental scenarios is equivalent to the distinction between verification and validation. Number 1 represents the known safe environment scenarios, number 2 the unsafe known environment scenarios, number 3 the known unsafe and number 4 the unknown safe environment scenarios. In addition, the following distinctions are possible with regard to unknown environment scenarios: In an unknown functional environment scenario the type of environmental scenario is not yet known, for example the existence of objects that were not considered during development, such as an e-scooter.

In the case of unknown logical environmental scenarios, the existence of the functional environmental scenario is known, but not the parameterization of it, for example the size of the e-scooter driver or his speed. Unknown concrete environmental scenarios represent the concrete instance of an environmental scenario that was not previously specified. This means that the method is also able to discover the unknown concrete environmental scenarios in particular by skillfully varying the parameters in the simulation.

To differentiate between safe and unsafe environmental scenarios, a measure is required with which the dangerousness of an environmental scenario can be assessed.

The distinction between safe and critical unsafe environment scenarios is made with the help of criticality. Criticality describes the probability that an accident will occur in the further course of an environmental scenario. If this probability is 100%, this corresponds to the vehicle coming into contact with another object/item/vehicle. The criticality can be described multidirectionally, so that both longitudinal and lateral criticality can be taken into account.

There are several methods to obtain a measure of criticality. These are divided into rule-based, potential-based and trajectory-based methods.

A potential-based measure, for example the time to collision (Time to Collision (TTC)) and/or the Distance to Collision (DTC; collision distance) is preferably used as a measure for the criticality. This corresponds to a simple measure of criticality, which reduces the amount of data.

The criticality is determined in the microsimulation and in the nanosimulation (verification).

A microsimulation and a nanosimulation is in the 6 shown. First, the macro simulation is described here. The macro simulation is the modeling of the traffic flow. Traffic flow is modeled using traffic flow theory, which describes the behavior of traffic volume, traffic speed and traffic density over time and space.

Macroscopic models can be used to generate/determine the number of vehicles at a given time in a given place to use as input for the microscopic model. Under traffic flow, the river or flux, i. H. the number of traffic elements (e.g. vehicles) crossing a certain traffic area or traffic line per unit of time.

For this purpose, a continuum theory, which describes the flow of traffic on roads in analogy to hydrodynamics, is used. The model presentation is based on the assumption of a stationary traffic flow, in which changes between different traffic states that are stationary in themselves occur abruptly.

In the micro-simulation, the movement of each individual vehicle along a traffic route is analyzed and represented, for example, as a trajectory in a distance-time diagram.

Microsimulation can be understood as the modeling of traffic flows based on the description of the movement of each individual vehicle that makes up the traffic flow. This implies modeling the actions such as acceleration, deceleration and lane changing of each driver in response to the surrounding traffic with the specification of parameter ranges.

In terms of the macro simulation, which is similar to a hydrodynamic model, the micro simulation is similar to a multi-body model, where each has an effect on its neighboring bodies. The micro simulation receives the macro simulation as input data.

Nano simulation is the simulation of the vehicle behavior itself. While microsimulation describes the body's path, nanosimulation describes the body's behavior along this path and the possibilities of reacting and interacting with the environment. Detailed modeling of both vehicle behavior and driver behavior is carried out in nanoscopic models.

7 shows the relationships in more detail. The macro-simulation and the micro-simulation refer to the environment of the vehicle and the nano-simulation also to the driver and his behavior.

The micro-simulation includes at least the traffic density of the classes of road users and/or the speed of the respective road users and/or the behavior of the road users in relation to lateral changes and/or longitudinal changes as parameters, so that either time lines or complete trajectories of all relevant Vehicles or the recording of the relevant environmental scenarios can be determined.

The result of the micro-simulation is either time lines or complete trajectories of all relevant vehicles or the recording of the relevant environmental scenarios.

In the first step S1 ( 1 ) - as already mentioned - functional environment scenarios are generated using the micro-simulation.

In a second step S2 ( 1 ) a microscopic quantification of functional environment scenarios based on criticality is carried out to identify critical functional environment scenarios.

For this purpose, functional critical environmental scenarios are identified with a threshold value in relation to the criticality. For the identification of critical environmental scenarios, the trajectories, as transferred from the microsimulation, are evaluated in each time step according to the criticality functions. The resulting criticality is assigned to each time step and can be plotted accordingly over time. A critical environmental scenario starts when a defined threshold value is exceeded and ends when it falls below it or after a local maximum has been passed.

8th shows the course of trajectories of two dynamic objects each, and the course of the resulting criticality over time using the example of four selected environmental scenarios. The diagram at the top left shows the environment scenario: oncoming traffic when turning left, based on a first and a second trajectory, with the star marking indicating the point at which the criticality indicates a defined threshold value. The chart below shows this criticality. The diagram on the top right shows the environment scenario: the right of way at a T-junction is ignored based on a first and a second trajectory, with the star marking indicating the point at which the criticality indicates a defined threshold value. The chart below shows this criticality. The diagram below left shows the environment scenario: entering the same lane together, using a first and a second trajectory, with the star marking indicating the point at which the criticality indicates a defined threshold value. The chart below shows this criticality. The diagram below right shows the environment scenario: crossing pedestrian based on a first and a second trajectory, with the star marking indicating the point at which the criticality indicates a defined threshold.

Furthermore, the identified critical functional environment scenarios are clustered/classified and assigned to known critical functional environment scenarios or assigned to a new critical functional environment scenario. This means that each identified environment scenario is then classified in order to assign it to one of the known functional environment scenarios or to define a new functional environment scenario.

The identified critical functional environment scenarios are then counted.

In a third step S3 ( 1 ) the critical functional environment scenarios are converted into logical environment scenarios using a statistical design of experiments, whereby initial parameters are defined by means of the statistical design of experiments, on which a logical environment scenario is dependent.

A statistical design of experiments is primarily a design of experiments (DoE).

9 shows a design of experiments schematically.

Such design-of-experiments (DoE) is a practical and ubiquitous approach to exploring multifactorial possibilities. The design-of-experiment aims to predict the outcome by introducing a change in the preconditions represented by one or more independent variables, also known as “input variables”. A change in one or more independent variables is generally assumed to result in a change in one or more dependent variables, also known as "output" or "response" variables. The test plans (“designs”) are chosen in such a way that maximum information can be obtained with a minimum number of tests. The aim of DoE is to learn as much as possible about the relationships between influencing parameters and results with as little effort as possible.

The experimental design includes not only the selection of suitable independent and dependent control variables, but also the planning of the execution of the experiment under statistically optimal conditions.

A logical environment scenario must first be derived from a functional environment scenario. For this it is necessary to create an initial list of parameters on which a functional environment scenario depends. All that is meant for this is the declaration of the parameters, i.e. the naming and assignment of a type one unit necessary. Rough limits are preferably sufficient for the parameter range of a logical environment scenario, which is also required. The DoE is used here to define the test points.

In a fourth step S4 ( 1 ) critical concrete environmental scenarios are aerated from the critical logical environmental scenarios using the nanosimulation. by using the initial parameters of each logical critical environment scenario. This means that the various concrete environmental scenarios are generated taking into account the behavior of the road users involved and the related reactions of the autonomous vehicle.

If you imagine all parameters in the logical environment scenario as a multidimensional space, each concrete environment scenario derived from them can be described with an instantiated vector in this multidimensional space. With the assumption that a logical environment scenario would be described by 10 parameters, each of which should be examined by 10 points between the limits and each test point would be run three times for statistical certainty, then 30 billion (30 * 10) scenarios / test / Simulation runs are necessary in order to run in all points for determining a response surface in a fully factorial test plan. By using the DoE method, this number can be reduced many times over (by a factor of 1e3 to 1e4).

According to the specifications from the DoE, a first, initial set of concrete environmental scenarios is simulated in the nano simulation. For a closed-loop simulation, the environment scenarios are always only initialized for their start parameters. Each trajectory then results from the behavior of the road users involved and the related reactions of the autonomous vehicle.

ermittelt, welche sich zumindest auf die gesamte Betriebszeit des autonomen Einzelfahrzeugs, vorzugsweise sogar auf die Betriebszeit einer ganzen Flotte, bezieht.Furthermore, the autonomous vehicle has to cope with a large number of different functional environment scenarios during its operating time. An error rate is used for verification $P\left(\frac{\text{crash}}{\text{H}}\right)$

determined, which relates at least to the entire operating time of the autonomous individual vehicle, preferably even to the operating time of an entire fleet.

wird aus der Summe der Fehlerwahrscheinlichkeiten aller funktionalen Umfeldszenarien gebildet: $$P\left(\frac{\text{crash}}{\text{h}}\right)={\displaystyle \sum P\left(\text{Scenario}\right)\ast P\left(Cash\right)}$$

wobei P(Scenario) die Auftretenswahrscheinlichkeit des Umfeldszenarios ist und P(Crash) die Wahrscheinlichkeit ist, in diesem einen Fehler (Crash) zu erfahren.The error rate $P\left(\frac{\text{crash}}{\text{H}}\right)$

is formed from the sum of the error probabilities of all functional environment scenarios: $$P\left(\frac{\text{crash}}{\text{H}}\right)={\displaystyle \sum P\left(\text{scenario}\right)\ast P\left(CasH\right)}$$

where P(scenario) is the probability of occurrence of the environmental scenario and P(crash) is the probability of experiencing an error (crash) in this.

entspricht daher dem Gesetz der totalen Wahrscheinlichkeit.Evidence of the total error rate $P\left(\frac{\text{crash}}{\text{H}}\right)$

therefore corresponds to the law of total probability.

zusammen aus der Summe der Fehlerwahrscheinlichkeiten aller funktionalen Umfeldszenarien.An actual error rate sets accordingly $P\left(\frac{\text{crash}}{\text{H}}\right)$

together from the sum of the error probabilities of all functional environment scenarios.

This results in a two-pronged approach to error rate verification $P\left(\frac{\text{crash}}{\text{H}}\right).$

10 shows the method according to the invention schematically.

aufgelöst, so ergibt sich beispielsweise ausgehend von einer gesamt Fehlerrate $P\left(\frac{\text{crash}}{\text{h}}\right)$

von 10^-6, und der Annahme, dass ein Anwendungsfall aus 10 funktionalen Umfeldszenarien besteht (z.B. kreuzender Fußgänger; entgegenkommender Überholer, mitfahrender Überholer; etc.) und einer Auftretenswahrscheinlichkeit von 0.1% = 10, würde eine sich, nachzuweisende Fehlerrate von 10^-6 ergeben, wie nachfolgende Gleichung belegt: $$\frac{P\left(\frac{\text{crash}}{\text{h}}\right)}{P\left(\text{Scenario}\right)\ast 10}=P\left(Crash,i\right)$$

Is iterative for each functional environment scenario according to the remaining error rate to be proven $P\left(\frac{\text{crash}}{\text{H}}\right)$

resolved, this results, for example, from a total error rate $P\left(\frac{\text{crash}}{\text{H}}\right)$

of 10 ^-6 , and the assumption that an application consists of 10 functional environment scenarios (e.g. crossing pedestrians; oncoming overtakers, overtakers traveling along; etc.) and a probability of occurrence of 0.1% = 10, an error rate of 10 ^-6 result, as shown by the following equation: $$\frac{P\left(\frac{\text{crash}}{\text{H}}\right)}{P\left(\text{scenario}\right)\ast 10}=P\left(C\mathrm{right}asH,i\right)$$

In a sixth step S6 ( 1 ) a response surface is created. The response surface method explores the relationships between multiple explanatory variables and one or more response variables.

For all initially tested concrete environmental scenarios, which can each be represented as a vector, a response surface matrix enk with the dimensions k: number of relevant parameters and n: number of critical concrete environmental scenarios (c1,..,cn) results from: $$enk=\left(\begin{array}{ccc}e11& \cdots & e1k\\ ⋮& \ddots & ⋮\\ en1& \cdots & enk\end{array}\right),n=\left(\begin{array}{c}\mathrm{<figure-callout\; id="1"\; label="c"\; filenames="DE102020210964A1\_0022.png,DE102020210964A1\_0024.png"\; state="\{\{state\}\}">c</figure-callout>}1\\ ⋮\\ cn\end{array}\right)$$

With the criticality that can be assigned to each specific environmental scenario, one obtains a solution vector with which a system of equations can be solved in such a way that the criticality can be solved with this system of equations for later specific environmental scenarios.

11 shows such a response surface schematically. With an initial response surface, which serves as a mathematical substitute model, the criticality of each specific environmental scenario can be estimated in advance, i.e. a priori, as the test progresses. The quality of the replacement model depends on the number of all environmental scenarios that have already been tested, but also on the quality of the mathematical description of the physical relationships and the quality of the criticality calculation. From the error rate of FR = 10 ^-6 1/h to be verified mentioned in the previous section, a verification time of t = 3,000,000h results. With an average speed of v = 30 and a scenario length of s = 100m, the necessary number of environmental scenarios to be tested is n = t * = 900 million, which are determined with the help of microsimulation.

An a priori criticality can be assigned to each specific environment scenario with the aid of the mathematical substitute model using the method according to the invention. This results in the advantage of not having to test all 900 million environmental scenarios in the nano simulation, but only those that have already been assigned a certain criticality.

reference list

S1,..,S6

process steps

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102016109651 A1 [0009]
• DE 102006044086 B4 [0010]

Claims (14)

Computer-implemented method for verifying an error rate $P\left(\frac{\text{crash}}{\text{H}}\right)$

of autonomously driving vehicles, characterized by the steps: - providing functional environment scenarios using a micro-simulation or real test drives, - microscopic quantification of the functional environment scenarios using the criticality to identify critical functional environment scenarios, - converting the critical functional environment scenarios into at least critical logical environment scenarios using a statistical design of experiments, whereby initial parameters are defined by means of the statistical design of experiments, on which a logical environment scenario depends, - generation based on a nano-simulation, using the initial parameters, critical concrete environment scenarios from the critical logical environment scenarios, - nanoscopic verification of the criticality of the critical functional environment scenarios based on the determination of the criticality of the critical concrete environment scenarios. Computer-implemented method claim 1 , characterized in that the error rate $P\left(\frac{\text{crash}}{\text{H}}\right)$

is formed from the sum of the error probabilities of all functional environment scenarios: $$P\left(\frac{\text{crash}}{\text{H}}\right)={\displaystyle \sum P\left(\text{scenario}\right)\ast P\left(CasH\right)}$$

where P(scenario) is the probability of occurrence of the environmental scenario and P(crash) is the probability of experiencing an error (crash) in this. Computer-implemented method according to one of the preceding claims, characterized in that the criticality is determined using a potential-based method. Computer-implemented method claim 3 , characterized in that the time to collision (Time to Collision (TTC)) and/or the Distance to Collision (DTC; collision distance) is used as a measure of the criticality. Computer-implemented method according to one of the preceding claims, characterized in that the micro-simulation comprises at least the traffic density of the classes of road users and/or the speed of the respective road users and/or the behavior of the road users with regard to lateral changes and/or longitudinal changes as parameters, so that either time lines or complete trajectories of all relevant vehicles or the recording of the relevant environmental scenarios can be determined by means of the micro-simulation. Computer-implemented method according to one of the preceding claims, characterized in that the microscopic quantification is carried out by the following steps: - Identify functional critical environment scenarios with a threshold value of criticality, - Cluster / classify the identified critical functional environment scenarios by assigning to a known critical functional environment scenario or assign to a new critical functional environment scenario, - counting of the identified critical functional environment scenarios. Computer-implemented method claim 6 , characterized in that the probability of occurrence P(scenario) of critical functional environment scenarios is determined using the quotient of the frequency n (scenario) and the simulated time t(sim): $$\text{P}\left(\text{scenario}\right)=\frac{n\left(S\mathrm{e.g}ena\mathrm{right}iO\right)}{t\left(sim\right)}$$

Computer-implemented method claim 7 , characterized in that each critical functional environment scenario is converted into a critical logical environment scenario using possible influencing parameters and boundary conditions of the critical functional environment scenario. Computer-implemented method according to one of the preceding claims, characterized in that a design-of-experiments method is used as static test planning. Computer-implemented method claim 9 , characterized in that an initial response surface is determined on the basis of the identified critical environment scenarios and the assigned criticality. Computer-implemented method claim 10 , characterized in that the response surface is represented by a matrix (enk), where k: number of relevant parameters and n: number of critical specific environmental scenarios (c1,..,cn): $$enk=\left(\begin{array}{ccc}e11& \cdots & e1k\\ ⋮& \ddots & ⋮\\ en1& \cdots & enk\end{array}\right),n=\left(\begin{array}{c}c1\\ ⋮\\ cn\end{array}\right)$$

Computer-implemented method claim 10 or 11 , characterized in that the response surface is designed to estimate the criticality of each specific environmental scenario in advance. Computer system with memory and processor for verification of an error rate $P\left(\frac{\text{crash}}{\text{H}}\right)$

of autonomously driving vehicles, wherein the processor and the memory are designed in such a way to carry out the steps of the method according to any one of the preceding claims. A data carrier comprising a digital data structure which, after being loaded into a computer memory, executes a method according to one or more of the preceding method claims.

Images