EP4302196A1

EP4302196A1 - Method for testing a driver assistance system of a vehicle

Info

Publication number: EP4302196A1
Application number: EP22711437.8A
Authority: EP
Inventors: Tobias DÜSER
Original assignee: AVL List GmbH
Current assignee: AVL List GmbH
Priority date: 2021-03-01
Filing date: 2022-02-28
Publication date: 2024-01-10
Also published as: KR20230150350A; WO2022183228A1; JP2024507998A; AT524822A1; CN117242438A

Abstract

The invention relates to a system (10) for generating scenarios for testing a driver assistance system of a vehicle (1) and a corresponding method, the system comprising: means (11) for simulating a traffic situation (3) which includes the vehicle and at least one further road user (1, 4, 5a, 5b, 5c, 5d, 6), a first road user (4) being controllable by a first user (2); a first user interface (12) which, on the basis of the virtual traffic situation (3), outputs a virtual environment to the first user (2) via a first, in particular at least optical, user interface (12); and a second user interface (13) for capturing inputs from the first user (2) for controlling the at least one first road user (4) in the virtual environment; means (14) for operating the driver assistance system (2) in a virtual environment of the vehicle (1) on the basis of the simulated traffic situation (3); means (15) for capturing a scenario; and means (16) for determining the quality of the resulting scenario.

Description

Method for testing a driver assistance system of a vehicle

The invention relates to a computer-implemented method for testing at least one driver assistance system for a vehicle. Furthermore, the invention relates to a corresponding system for testing at least one driver assistance system.

The spread of driver assistance systems (Advanced Driver Assistance Systems

- ADAS), which in a further development autonomous driving (Autonomous Driving

- AD) are increasing both in the area of passenger cars and commercial vehicles. Driver assistance systems make an important contribution to increasing active traffic safety and serve to increase driving comfort.

In addition to the systems used in particular for driving safety, such as ABS (anti-lock braking system) and ESP (electronic stability program), a large number of driver assistance systems are offered in the area of passenger cars and commercial vehicles.

Driver assistance systems that are already being used to increase active road safety include a parking assistant, an adaptive cruise control system, also known as adaptive cruise control (ACC), which adaptively regulates a desired speed selected by the driver based on the distance from the vehicle in front. Another example of such driver assistance systems are ACC stop-and-go systems, which in addition to the ACC causes the vehicle to continue driving automatically in traffic jams or when vehicles are stationary, lane keeping or lane assist systems, which automatically keep the vehicle in its lane and pre-crash systems which, in the event of a collision, prepare or initiate braking, for example, in order to take the kinetic energy out of the vehicle, and initiate other measures if a collision is unavoidable.

These driver assistance systems both increase safety in traffic by warning the driver in critical situations and initiate independent intervention to avoid or reduce accidents, for example by activating an emergency braking function. In addition, driving comfort is enhanced by functions such as automatic parking, automatic lane keeping and automatic distance control increased.

The safety and comfort gain of a driver assistance system is only perceived positively by the vehicle occupants if the support from the driver assistance system is safe, reliable and - as far as possible - comfortable.

In addition, each driver assistance system, depending on its function, must manage scenarios that occur in traffic with maximum safety for the vehicle and without endangering other vehicles or other road users.

The respective degree of automation of vehicles is divided into so-called automation levels 1 to 5 (see, for example, the SAE J3016 standard). The present invention relates in particular to vehicles with driver assistance systems of automation level 3 to 5, which is generally considered to be autonomous driving.

The challenges for testing such systems are manifold. In particular, a balance must be found between the test effort and the test coverage. The main task when testing ADAS/AD functions is to demonstrate that the function of the driver assistance system is guaranteed in all conceivable situations, especially in critical driving situations. Such critical driving situations have a certain degree of danger, since no reaction or an incorrect reaction of the respective driver assistance system can lead to an accident.

The testing of driver assistance systems therefore requires a large number of driving situations, which can arise in different scenarios, to be taken into account. The range of possible scenarios is generally spanned by many dimensions (e.g. different road properties, behavior of other road users, weather conditions, etc.). From this almost infinite and multidimensional parameter space, it is particularly relevant for testing driver assistance systems to extract such parameter constellations for critical scenarios that can lead to unusual or dangerous driving situations. As shown in FIG. 1, such critical scenarios have a far lower probability of occurrence than usual scenarios.

Scientific publications assume that operating a vehicle in autonomous ferry operation is only statistically safer than a human-controlled vehicle if 275 million miles of accident-free driving time have been completed with the corresponding driver assistance system in order to validate the corresponding driver assistance system. This is practically impossible to achieve using real test drives, especially considering that the development cycles and quality standards required in the automotive industry already set a very tight time frame. It would also be unlikely that a sufficient number of critical scenarios or driving situations resulting from these scenarios would be included for the aforementioned reason.

It is known from the prior art to use real test driving data from a real fleet of test vehicles to validate and verify driver assistance systems and to extract scenarios from the recorded data. Furthermore, it is known to use full factorial test plans for validation and verification.

It is an object of the invention to be able to test driver assistance systems, in particular autonomous driving functions, in critical scenarios. In particular, it is an object of the invention to identify critical scenarios for driver assistance systems.

This object is solved by the teaching of the independent claims. Advantageous configurations can be found in the dependent claims.

A first aspect of the invention relates to a computer-implemented method for testing a driver assistance system of a vehicle, having the following work steps:

Simulating a virtual traffic situation that has the vehicle and at least one other road user, with a first road user being able to be controlled by a first user and with other road users who cannot be controlled by users being controlled in an automated manner, in particular by artificial intelligence or based on logic will; Outputting, on the basis of the virtual traffic situation, a virtual environment of the at least one first road user to the first user via a first, in particular at least visual, user interface;

Recording of inputs from the first user (2) to control the at least one first road user in the virtual environment of the first road user via a second user interface, with the simulation of the virtual traffic situation the recorded inputs of the first user and the resulting interaction of the at least one first road user with his virtual environment are taken into account;

operating the driver assistance system in a virtual environment of the vehicle based on the simulated traffic situation, the driver assistance system showing a driving behavior;

Capturing a scenario that arises from the driving behavior of the driver assistance system in the virtual environment of the vehicle;

Determining a quality of the resulting scenario as a function of a predefined criterion, in particular a dangerousness of the resulting scenario; and

Outputting the quality to the user via the first or a second user interface, in particular a display.

A second aspect of the invention relates to a system for generating scenarios for testing a driver assistance system of a vehicle, having:

Means for simulating a virtual traffic situation, which has the vehicle and at least one other road user, with a first road user being controllable by a first user and with other road users that cannot be controlled by users being automated, in particular by artificial intelligence or logic -based, to be controlled; a first, in particular at least optical, user interface for outputting, on the basis of the virtual traffic situation, a virtual environment of at least one first road user to the first user via a first, in particular at least optical, user interface; and a second user interface for recording inputs from the first user for controlling the at least one first road user in the virtual environment of the first road user via a second user interface, wherein when simulating the virtual traffic situation, the recorded inputs of the first user and the resulting interaction of the at least one first road user are taken into account with his virtual environment;

Means for operating the driver assistance system in a virtual environment of the vehicle on the basis of the simulated traffic situation, as a result of which the driver assistance system shows a driving behavior;

Means for detecting a scenario which arises as a result of the driving behavior of the driver assistance system in the virtual environment of the vehicle; and

Means for determining a quality of the resulting scenario as a function of a predefined criterion, in particular whether the resulting scenario is dangerous; wherein the first or the second user interface, in particular a display, is also set up to output the quality to the user.

Further aspects of the invention relate to a computer program and a computer-readable medium.

A user within the meaning of the invention is a natural person, i. H. a human.

A driver assistance system within the meaning of the invention is preferably set up to support a driver when driving or to at least partially guide a vehicle, in particular a driver assistance system of automation level 3 to 5, or more particularly an autonomous driving function.

Traffic user within the meaning of the invention is preferably any object that participates in traffic. In particular, a road user is a person, an animal or a vehicle.

For the purposes of the invention, extraction preferably means delimiting or isolating.

In particular, scenarios are delimited or isolated from the scenario data. In this case, data areas in the scenario data are preferably selected. Scenario data within the meaning of the invention are preferably characterized by the position and movement of road users and the position of static objects in relation to a scenario.

A scenario within the meaning of the invention is preferably formed from a chronological sequence of, in particular static, scenes. The scenes give example, the spatial arrangement of the at least one other object relative to the ego object, z. B. the constellation of road users. A scenario preferably considers dynamic and static content. A model for the systematic description of scenarios is preferably used here, more preferably the model of the PEGASUS project (https://www.pegasusprojekt.de) with the following six independent levels: 1st street (geometry,...) ; 2. Street furniture and rules (traffic signs,...); 3. Temporary changes and events (road construction,...); 4. Moving objects (traffic-related objects such as: vehicles, pedestrians,... that move relative to the vehicle to be tested); 5. Ambient conditions (light situation, road weather,...); 6. Digital information (V2X, digital data / map,...). A scenario can contain, in particular, a driving situation in which a driver assistance system at least partially controls the vehicle, which is called the ego vehicle and is equipped with the driver assistance system, e.g. B. performs at least one vehicle function of the ego vehicle autonomously.

A traffic situation within the meaning of the invention preferably describes the totality of all circumstances in traffic with road users in a defined spatial area and/or in a defined period of time or point in time. These circumstances are preferably taken into account by road users for the selection of suitable behavior patterns at a specific point in time. A traffic situation preferably includes all relevant conditions, possibilities and determinants of traffic. A traffic situation can, but does not have to, be represented from the point of view of a road user or object.

The simulated measured variables within the meaning of the invention are preferably selected from the following group: speed, in particular an initial speed, of a road user; a direction of movement, in particular a trajectory, of a road user; lighting conditions; Weather; road surface; Temperature; number and position of static and/or dynamic objects; one Speed and a direction of movement, in particular a trajectory, of the dynamic objects; condition of signaling installations, in particular light signaling installations; traffic signs; number of lanes; Acceleration or deceleration of road users or objects.

A predefined constellation of measured variables within the meaning of the invention is preferably a constellation of values of one or more measured variables, in particular over time.

For the purposes of the invention, label means preferably provided with a categorizing designation.

A danger of a scenario within the meaning of the invention is preferably the spatial or temporal proximity to a traffic situation without a possible accident-free outcome (under one's own efforts and taking into account the uncertainties mentioned). If an accident can no longer be avoided, the danger is at its maximum. Danger is preferably also referred to as criticality. If the driving behavior or driving ability of a driver assistance system is taken into account, the danger can characterize a probability of an accident and/or a calculated duration up to a collision time. A maximum level of danger is preferably given when the calculated duration is 0 seconds and/or the accident probability P=1. An increased probability of an accident can be triggered in particular by a driving manoeuvre, for example an evasive reaction or strong gradient changes when steering, braking or accelerating (e.g. a vehicle evades through a strong steering movement). An increased probability of an accident can also occur, in particular with regard to the other road users (who are guided based on logic or AI) and in a critical driving situation they have to leave their driving order or their actual trajectory (through an evasive maneuver). An increased probability of an accident can also arise, in particular, from external factors which affect the first road user or the other road users, for example if a driver is blinded.

A quality within the meaning of the invention preferably characterizes the simulated scenario. A grade is preferably a quality or condition and/or understood a relevance of the simulated scenario in relation to the dangerousness of a driving situation for a specific driver assistance system.

Relevance within the meaning of the invention is preferably understood to mean the frequency with which a scenario occurs in road traffic. For example, a backlit scenario is more relevant than a scenario in which an airplane lands on the street. The relevance preferably also depends on the region for which the road traffic is relevant. For example, there are scenarios that are relevant in Germany but not in China.

An area surrounding the vehicle within the meaning of the invention is preferably formed at least by the road users and other objects relevant to the vehicle guidance by the driver assistance system. In particular, the surroundings of the vehicle include scenery and dynamic elements. The scenery preferably includes all stationary elements.

A termination condition within the meaning of the invention is preferably defined objectively or can also be brought about by input from a user.

Means within the meaning of the invention can be hardware and/or software and in particular one, preferably with a memory and/or bus system data or signal connected, in particular digital, processing, in particular microprocessor units (CPU) and/or one or more Have programs or program modules. In this case, the CPU can be designed to process commands that are implemented as a program stored in a memory system, to detect input signals from a data bus and/or to emit output signals to a data bus. A storage system can have one or more, in particular different, storage media, in particular optical, magnetic, solid state and/or other non-volatile media. The program can be such that it embodies or is able to execute the methods described here, so that the CPU can execute the steps of such methods and then in particular can generate scenarios.

The invention is based on the approach of animate real people to generate scenarios, but no test drives in real traffic are required. According to the invention, the real driver moves a vehicle in a simulated traffic situation, in particular in a virtual environment or a virtual environment which is created by the simulated traffic situation. The invention makes the generation of scenarios accessible to a crowdsourcing approach. One or more users can now navigate a road user of their choice through virtual traffic situations on a simulator. Due to the almost infinite possibility of options when navigating the road user(s) and other aleatoric mechanisms when simulating the virtual traffic situation, as in real road traffic, various scenarios can arise in an almost infinite number. The invention preferably uses predefined criteria to determine whether known or new scenarios occur. For this purpose, the simulation process and in particular the simulation data generated by it are continuously analyzed and monitored.

With regard to the crowdsourcing approach, people's natural play instinct can be exploited. In this way, the method according to the invention or even a corresponding system can be made available to users. These users can then drive around in the simulated traffic "for fun". Alternatively, tasks could also be set for the users, for example that they should get from a location A to a location B as quickly as possible while adhering to traffic rules, or that they have to collect certain objects from different locations. Furthermore, the user can be distracted when navigating through the simulated traffic, e.g. B. by having to make certain voice inputs or the like.

The physics in the simulation preferably corresponds to reality in order to generate scenario data that is as realistic as possible. This applies in particular to the physical properties of road users and those of the environment. It is not possible to drive through objects or the like. Multiple users particularly preferably navigate multiple road users in the simulated traffic.

The quality or dangerousness of a scenario created by his activity is communicated to the user through feedback on the quality of the scenario. The user can then try to increase the quality by adapting his steering behavior of the road user he is steering. Preferably he can Simulation means, which simulate the scenario, prompt in this regard to repeat a scenario that has already been run through. In this way, he can continue to optimize his steering behavior until he achieves the best possible quality in relation to a specific scenario, ie the greatest danger, particularly when complying with the traffic regulations.

The resulting driving situation is preferably recorded as scenario data in order to be able to reproduce it later in a simulated scenario.

In an advantageous embodiment of the method, the scenario data generated in this way are already labeled, in particular objects of the virtual traffic situation are labeled. Information about the properties of objects is available in the simulation, so that the information can be assigned to the objects.

This is a particular advantage over data from a real test operation, in which all objects have to be labeled. This labeling is generally very complex, since it can only be carried out by humans.

In an advantageous embodiment of the method, the simulated scenario of the ascertained quality is changed until a termination condition is reached. In addition or as an alternative, the simulated scenario is preferably output until its quality reaches a termination condition. More preferably, the simulated scenario is only output when its quality reaches the termination condition. In this case, the test method is continued or repeated until the driving behavior of the user and/or the simulation of the scenario leads to a behavior of the driver assistance system that violates a predefined target value that serves as a termination condition. Such a termination condition can e.g. B. a duration up to a collision time of less than 0.25 seconds or a certain time budget, z. B. 600 hours maximum simulation time.

Assuming a sufficiently long time budget, a high probability can be achieved by means of the invention that the ADAS or AD system is working correctly.

In a further advantageous embodiment of the method, the change is made at least partially by the user via the first user interface or the second user interface using an editor, with operations of the User are preferably recorded and stored in a control table. Due to the possibility of changing the scenario, the user can not only influence the resulting driving situation through his driving behavior, but can also directly influence the simulated driving scenario. In this way, it has a further influence on the quality of the resulting driving situation. The user can thus optimize an existing driving scenario in such a way that driving situations with the highest possible quality result in interaction with his driving behavior.

In a further advantageous embodiment of the method, a speed, in particular an initial speed of the vehicle, and/or a trajectory of the vehicle is specified when simulating the scenario. These specifications are boundary conditions when testing the driver assistance system and can preferably also be changed by the user.

In a further advantageous embodiment of the method, only values of parameters of the simulated scenario are changed when the simulated scenario is changed. The scenario that is simulated in each case therefore remains of its type. New parameters, e.g. B. by adding more vehicles should not be added. This ensures that the user stays in the respectively simulated scenario to be tested and does not create a completely new scenario.

In a further advantageous embodiment of the method, the parameters of the scenario are selected from the following group, depending on the type of driver assistance system being tested:

Speed, in particular an initial speed, of a road user; a direction of movement, in particular a trajectory of a road user; lighting conditions; Weather; road surface, temperature; number and position of static and/or dynamic objects; a speed and a direction of movement, in particular a trajectory, of the dynamic objects; condition of signaling installations, in particular light signaling installations; traffic signs; number of lanes; Acceleration or deceleration of road users or objects.

By changing these parameters, the dangerousness of scenarios can be significantly influenced. In a further advantageous embodiment of the method, the quality is characterized by a reward for the user, in particular a fictitious one. In particular, depending on the quality achieved, the user is credited with a reward, for example in a virtual account. In this way it can be achieved that the user always tries to realize driving situations with a higher quality.

In a further advantageous embodiment of the method, the quality is higher the more dangerous the driving situation that has arisen is, in particular the shorter the calculated duration up to the time of the collision. This has the advantage that driving situations that are as critical as possible are identified, which, however, occur very rarely in normal road traffic. The number of test kilometers to be completed can be reduced to this.

In a further advantageous embodiment of the method, the simulated scenario is changed using evolutionary algorithms. Evolutionary algorithms are also known as genetic algorithms. When altered by such algorithms, different algorithms are crossed and mutated. The resulting algorithms become candidates for the next iteration step, i. H. the next variant of the simulated scenario.

In a further advantageous embodiment of the method, a utility function is approximated on the basis of the determined quality, which describes the quality value of a specific simulated scenario. In this way, a reward can be calculated for a user.

In a further advantageous embodiment of the method, the driver assistance system is simulated. This means that only the software or the actual code of the driver assistance system is taken into account or implemented when simulating the virtual traffic situation, according to the “software-in-the-loop” concept. Here, the testing of a driver assistance system can be carried out in a pure simulation. A stimulation or a provision of signals to a real driver assistance system can be omitted here.

In a further advantageous embodiment of the method, historical data from earlier test operations of a driver assistance system, in particular the driver assistance system, are taken into account during the initial simulation of the scenario. Historical data can be used to pre-train an algorithm used to simulate traffic scenarios. This can reduce the time it takes to find critical scenarios. Furthermore, algorithms can also be used which have been trained on another, in particular similar, ADAS or AD system. In particular, so-called regression tests can be carried out in this way in order to ensure that changes to parts of the software of the driver assistance system that have already been tested do not cause any new errors.

In a further advantageous embodiment of the method, when the driver assistance system is operated, data relating to the vehicle's surroundings are fed into the driver assistance system and/or the driver assistance system, in particular its sensors, are stimulated on the basis of the vehicle's surroundings. This configuration allows a real driver assistance system or even a real vehicle to be tested with such a real assistance system.

As a result, the driver assistance system, in particular its software or the entire hardware, can be tested on a test bench. In particular, a hardware-in-the-loop method can be used for this.

The features and advantages mentioned above in relation to the first aspect of the invention also apply correspondingly to the other aspects of the invention and vice versa.

Further features and advantages result from the following description of exemplary embodiments with reference to the figures. It shows at least partially schematically:

FIG. 1 shows a diagram of the probability of occurrence of scenarios as a function of their criticality and/or danger;

FIG. 2 shows a block diagram of an exemplary embodiment of a method for generating scenarios;

FIG. 3a shows a first example of a simulated virtual traffic situation;

FIG. 3b shows a second example of a simulated virtual traffic situation;

FIG. 4 shows a third example of a simulated virtual traffic situation; FIG. 5 shows an exemplary embodiment of a system for generating scenario data for testing a driver assistance system of a vehicle; and

FIG. 6 shows an exemplary embodiment of a means for operating a driver assistance system.

FIG. 1 shows the probability of occurrence of scenarios as a function of the criticality and/or danger of scenarios. The probability of occurrence is the probability in which scenarios occur in real road traffic.

It is noticeable in FIG. 1 that the majority of scenarios are of comparatively little complexity and/or danger, which also corresponds to the general life experience of a car driver. The range of these scenarios is denoted by “A” in FIG. On the other hand, scenarios of high complexity, the area of which is labeled “B” in FIG. 1, occur comparatively rarely. However, it is precisely those scenarios "B" with great complexity that are of great relevance for investigating the functionality of driver assistance systems.

In order to achieve a sufficient number and diversity of different scenarios with high complexity "B" during the test of a driver assistance system, a very large number of scenarios must therefore usually be run through, based on the distribution curve shown. An alternative approach is described below, in which those scenarios with great complexity are specifically created in order to test driver assistance systems and thus reduce the number of scenarios to be run through and the number of kilometers to be covered.

A method for generating a large number of different scenarios with the greatest possible complexity for testing driver assistance systems is described below with reference to FIGS. 2 to 3b.

In a first work step 101, a virtual traffic situation 3 is simulated, in which a driver assistance system 7 is to be tested. The virtual traffic situation 3 has a plurality of virtual road users 1, 4, 5a, 5b, 5c, 5d, 6, the vehicle 1 being controlled by the driver assistance system 7 to be tested. In this virtual traffic situation 3, at least one further first road user 4 of the plurality of road users 1, 4, 5a, 5b, 5c, 5d, 6 can be controlled by a first user 2 and those road users 5a, 5b, 5c, 5d, 6 that cannot be controlled by users are controlled automatically. Artificial intelligence or a logic-based controller is preferably used here. A plurality of road users, which are controlled by users, ie people, can preferably be located in the simulated virtual traffic situation 3 . A plurality of driver assistance systems 7 can also be tested simultaneously in a single vehicle or in multiple vehicles.

A traffic flow model, in particular PTV-Vissim® or Eclipse SUMO, in particular version 1.8.0, is preferably used to simulate the virtual traffic situation.

There are essentially two approaches to simulating the virtual traffic situation: The simulation is based on data obtained in a real test drive. In this case, the parameters of individual objects, e.g. B. the Ge speed of road users, can be changed or taken over as they were recorded during the real test drive. In an alternative embodiment of the method, the traffic situation 3 is created purely on the basis of mathematical algorithms. Both approaches can preferably also be mixed.

Such a simulated traffic situation 3 is shown, for example, in FIG. 3a. In the traffic situation 3 shown in FIG. 3a, a pedestrian 6 crosses a street. A motorcycle 4, which is controlled by the first user, approaches the pedestrian 6 on the pedestrian 6 lane facing. Other vehicles 5b, 5c, 5d are parked next to the lane, through which the pedestrian 6 is not visible or only poorly visible to the motorcycle 4 controlled by the first user 2. Another vehicle 5a is driving in the second lane for oncoming traffic at the level of pedestrian 6. A vehicle 1 is approaching behind the other vehicle 5a, the longitudinal and lateral control of which is carried out by a driver assistance system 7 . Whether this motorcyclist 4 is visible to the vehicle 1 controlled by the first user 2 is not clear from FIG. 3a.

The other vehicles 5a, 5b, 5c, 5d, the pedestrian 6 and the motorcyclist 4 form a virtual environment of the vehicle 1 controlled by the driver assistance system in the traffic situation 3. The other vehicles 5a, 5b, 5c, 5d, the pedestrian 6 a virtual environment of the motorcycle 4 controlled by the user in the traffic situation 3.

Depending on how the first user 2 reacts or acts in the initial scenario, which results from the traffic situation 3, i. H. which driving behavior the first user 2 shows in the virtual environment of the motorcycle 4 controlled by him, a scenario will result which is dangerous or less dangerous. In general, the respective scenario is all the more dangerous the more complex it is and is therefore more difficult for a driver assistance system 7 to process. If the first user 2 brings the motorcycle 4 to a standstill, for example, as indicated in FIG. 3a by the bar in front of the movement arrow of the motorcycle 4, the vehicle 1, which is controlled by the driver assistance system, can Overtake oncoming vehicle 5a in the lane undisturbed.

FIG. 3b shows the same virtual traffic situation 3 as FIG. 3a, in which the motorcycle controlled by the first user 2 is in the same initial traffic situation as in FIG. 3a. As indicated by the movement arrow, which emanates from the motorbike 4 controlled by the first user 2, the first user 2 continues to control it with undiminished speed.

There is a high probability that a scenario will develop in which the motorcycle 4 collides with the vehicle 1 controlled by the first driver assistance system 7 . This is also indicated in FIG. 3b. Such a driving situation or such a scenario corresponds to a very high degree of danger.

An initial speed and/or an initial trajectory of the vehicle 1 and/or the motorcycle 4 is preferably specified by simulating the traffic situation 3 .

In a second work step 102 the virtual traffic situation 3 is output to the first user 2 via a first user interface 12 .

Possible user interfaces 12 are shown in Fig. 4 as an example and preferably include optical user interfaces, in particular screens, audio user interfaces, in particular loudspeakers, and/or a user interface for stimulating the sense of balance of the first user 2. In a third work step 103, inputs by the first user 2 for controlling the motorcycle 4 in a virtual environment around the first road user 1 are recorded via a second user interface 13 .

The second user interfaces 13 are also shown in FIG. 4 . It is preferably the steering wheel, a gear shift, a handbrake, a brake pedal, a clutch and/or an accelerator pedal and other possible control instruments that are available to a driver in a vehicle.

Depending on which type of road user 1, 4, 5a, 5b, 5c, 5d, 6 the user 2 controls, however, other input means can also be present as the user interface 13, for example a type of joystick.

As already explained, the first road user 1 in FIGS. 3a and 3b is the motorcycle 4. When simulating the traffic situation 3 shown in FIGS. 3a and 3b, the recorded inputs of the first user 2 and the resulting interaction of the motorcycle 4, i.e. H. of the first road user, with its virtual environment.

The interaction in the traffic situations 3 shown in FIGS. 3a and 3b is, for example, how the first user 2 reacts to the initial scenario. Depending on the reaction of the first user 2 to this initial scenario, the other road users, in particular the oncoming vehicle 1 and the other oncoming vehicle 5a and the pedestrian 6, will also react. For example, it can be expected that the oncoming vehicle 1 will brake if the driver assistance system 7 detects that the vehicle 1 controlled by the first user 2 is not reducing its speed. These interactions in turn have an influence on the development of the virtual traffic situation 3.

The simulation of the traffic situation 3 is therefore a continuous process which, as indicated by an arrow in FIG. 2, constantly runs in a loop and generates simulation data in the process.

During the simulation, the objects that are part of the virtual traffic situation 3 are preferably labeled. This affects both static and dynamic objects. As a result, later data that can be obtained from the simulation data includes the so-called ground truth information. If the scenario data is used, for example, to test a driver assistance system 7, it can be understood which objects the driver assistance system 7 has correctly detected and which it has incorrectly detected. Such labels are, for example, tree, pedestrian, car, truck, etc.

More preferably, actions are set in the traffic situations 3, which encourage the first user to be active. For example, in traffic situation 3 in FIGS. 3a and 3b, this could be another vehicle that follows motorcycle 4 controlled by first user 2 and urges it to accelerate. Such an action can also be a surprising movement trajectory of the pedestrian 6, for example by starting to run.

In a fourth step 104, the driver assistance system 2 and with it the vehicle 1 controlled by it are operated in a virtual environment of the vehicle, which results from the position and orientation of the vehicle 1 in the simulated traffic situation 3, which results from the simulation and the Control of the motorcycle 4 by the first user 2 was created. As described further below, both the hardware of the driver assistance system 7 with or without sensors and only the software of the driver assistance system 7 can be operated in a tested manner. Missing hardware components can be simulated.

An example of such operation in the virtual environment is shown in FIG. The vehicle 1 with the function to be tested or the driver assistance system 2 to be tested is operated in the virtual road network, which contains at least one virtual traffic situation 3 . On its way from the starting point to the specified destination, the vehicle 1 must avoid a construction site that a user 2 has created to influence the traffic situation, control it through simulated traffic, master the traffic using a vehicle 4 with a real driver, etc. This results in, for example, the route marked with a dotted line. The road network with the traffic situations 2 provided is preferably a platform on which the developer, for example a vehicle manufacturer, can let your car drive, preferably anonymously. In other words, you can see which cars are driving autonomously, but not what type and/or manufacturer. In this case, preferably not only the function but also the driving dynamics are simulated. The users which Control road users manually, can then let off steam and try to bring the driver assistance system 7 of the vehicle 1 from the concept.

In a fifth work step 105, scenarios are recorded which arise as a result of the driving behavior of driver assistance system 7 in relation to vehicle 1 controlled by it in traffic situation 3 or in the area surrounding vehicle 1 .

Here, already known scenarios, which have already occurred earlier or are predefined as templates, can be checked, as well as scenarios that have not yet been predefined.

Both types of scenarios are preferably defined by predefined constellations of simulated measurement variables, which can be determined from the virtual traffic situation 3 . These predefined constellations either form the templates for scenarios or correspond to elementary maneuvers from which the occurrence of a scenario can be inferred. This could be, for example, a strong braking deceleration of the vehicle 1 in FIGS. 3a and 3b, which is used as a trigger condition for the occurrence of a scenario that has not yet been predefined.

In a sixth work step 106, a quality of the resulting scenario is preferably determined as a function of a predefined criterion, with the quality preferably being characterized by the dangerousness of one of the scenarios. The quality is preferably all the higher, the more dangerous the resulting scenario is. A degree of danger is preferably determined by so-called time-to-X metrics, such as those described in the publication “Metrics for assessing the criticality of traffic situations and scenarios”; P. Junietz et al., 11 . Workshop Driver Assistance Systems and Automated Driving”, FAS 2017. In particular, the following criteria can be used here: duration up to a point in time at which a collision occurs (time to collision), time to kickdown, time to steer, time to react, distance of closest encounter , Time to Closest Encounter, Worst Time to Collision. More preferably, the level of danger is characterized by a probability of an accident.

In a seventh step 107, the quality determined is preferably output to the user 2 via one of the aforementioned interfaces. This can be after each new scenario emerges or after going through each new one resulting scenarios happen. However, this can preferably also only take place after a number of scenarios have been run through.

More preferably, a reward, in particular a fictitious one, is credited to the first user 2 as a function of the quality of the scenario that has occurred. More preferably, the notional reward indicates the goodness.

In an eighth work step 108, inputs by the first user via the first user interface 12 or the second user interface 13 are preferably recorded. These additional entries are used to change the simulated traffic situation 3.

The first user 2 can preferably change, replace or remove objects in the simulation of the traffic situation 3 . In addition, he can add new objects. The first user 2 can thereby try to shape the traffic situation 3 in such a way that it becomes as complex as possible. In this way he can provoke errors in driver assistance system 7 when driving vehicle 1 . For example, the user 2 could provide that the trailer of a truck as another road user has the same color as the sky in order to make it more difficult for an optical sensor of a driver assistance system to perceive it. In this case, the user can preferably fall back on a large number of options for changing the traffic situation. A few examples are listed below, which can be used individually or together: Changing the textures of objects (e.g. traffic sign prints on trucks); Set up construction sites, e.g. with warning beacons in the middle of the street; let a dinosaur run across the street (e.g. relevant during the carnival season); consciously position objects in such a way that they cover each other; person dressed in white in front of white wall; injured person on the street; Martinshorn or horn (it can also be provided in the driver assistance system acoustic matic sensors).

A type of editor is preferably used here, with which the first user 2 can design the traffic situation 3 . More preferably, operations by the first user 2 are preferably recorded and stored in a control table. In this way it should be possible to determine which measures can be used to create traffic situations 3 which lead to particularly complex scenarios. In a ninth step 109, the simulated traffic situation 3 is preferably changed. This occurs either on the basis of the recorded inputs from the first user 2. However, algorithms for changing the simulated traffic situation 3 can preferably also be used here, in particular evolutionary algorithms. When changing, only existing parameters of the traffic situation 3 are preferably changed. For example, the color or speed of existing objects. However, the basic design of the traffic situation 3 is retained.

Possible parameters that can be influenced are: speed, in particular an initial speed, of a road user; a direction of movement, in particular a trajectory, of a road user; lighting conditions; Weather; road surface; Temperature; number and position of static and/or dynamic objects; state and appearance of static and/or dynamic objects; a speed and a direction of movement, in particular a trajectory, of the dynamic objects; condition of signaling installations, in particular light signaling installations; traffic signs; number of lanes; Acceleration or deceleration of road users or objects; Signs of soiling and/or aging of the road surface; geographic orientation of the traffic situation. In particular, colors, textures, shapes and clothing of objects and/or the position of the sun and the direction of incidence of the sunlight in the traffic situation 3 can be changed.

Even though the method 100 for testing a driver assistance system 7 has been described above in relation to a driver assistance system 7 to be tested in a vehicle 1 and a user 2 who controls a first road user 4, it is also possible for driver assistance systems of several vehicles be tested by the method 100 and/or that multiple traffic participants are controlled by users. For example, the pedestrian 6 could be controlled by a second user. This means that several vehicles can be controlled by driver assistance systems in the simulated traffic situation and/or that several other road users can be controlled by other users. 5 shows a system 10 for generating scenarios for testing a driver assistance system of a vehicle.

This system 10 preferably has means 11 for simulating a virtual traffic situation 3, which has a plurality of virtual road users. In order to make a road user 4 controllable by a first user 2 , the system also has at least one first user interface 12 and at least one second user interface 13 .

The at least one first user interface or user interfaces 12 serve to output a virtual environment of at least one first road user 1 to the first user 2. The virtual environment of the at least one first road user 4 is determined on the basis of the simulated virtual traffic situation 3. It is essentially a representation of the virtual traffic situation 3 in the initial scenario from the point of view of the first road user 4, which the first user 2 controls.

As shown in FIG. 5, these user interfaces 12 are optical user interfaces such as screens and audio interfaces such as loudspeakers and possibly devices with which the sense of balance of the respective user 2 can be influenced.

The second user interface or user interfaces 13 are set up to record inputs from the respective user 2 . As shown in FIG. 4, these are preferably different operating elements. As already explained above, these can be dependent on the respective road user 1 controlled by the user 2 . If the road user 1 controlled by the first user 2 is a vehicle, the user interfaces 12, 13 are preferably arranged in the area of a so-called seat box 19, which forms a simulator together with the user interfaces 12, 13, as shown in FIG.

Furthermore, the system 10 preferably has means 14 for operating the driver assistance system 2 in a virtual environment of the vehicle 1 on the basis of the simulated traffic situation 3 . The driver assistance system 2 drives the vehicle 1 through this traffic situation 3 and shows a certain driving behavior. Wei terhin the system 10 preferably means 15 for determining detection of a Scenarios, which is caused by the driving behavior of the driver assistance system 2 in the virtual environment of the vehicle 1. Further preferably, the system 10 preferably has means 16 for determining a quality of the simulated scenario of the 3 resulting scenario as a function of a predefined criterion in relation to the resulting driving situation, in particular a dangerousness of the resulting driving situation of the resulting scenario. The first or the second user interface 12, 13, in particular a display, are also set up to output the quality to the user. Furthermore, a data interface can be provided, which is set up to output the scenario data for further processing. The means 11, 14, 15, 16, 17, 18 are preferably part of a data processing device, which is preferably formed by a computer.

A means 14 for operating a driver assistance system 7 is shown again in detail in FIG.

Such a means 14 preferably has a device 20 which is set up to simulate a virtual environment of the vehicle 1 on the basis of the scenario data. Furthermore, the means 20 are set up to also render this environment.

An interface 21 is set up to output or emulate the virtual environment of a driver assistance system. Such an interface 21 can be a screen if the driver assistance system 7 has an optical camera.

In the example shown in FIG. 6, the sensor of driver assistance system 7 is a radar sensor, which emits a signal S. This signal S is detected by radar antennas 21, which form the interface.

On the basis of the detected signal and the simulated surroundings, the means 20 for simulating calculate a response signal S′, which in turn is output to the radar of the driver assistance system 7 via radar antennas. In this way, the function of the driver assistance system 7 can be tested. Depending on which components of a driver assistance system 7 are to be tested, the simulated virtual environment, as shown in FIG. 6, can be tested by emulating signals to the sensor of the driver assistance system 7. Alternatively, however, a signal S' can also be generated, which is fed directly into the data processing unit 7 of the driver assistance system, or else a signal S', which is only processed by the software of the driver assistance system 7.

It is pointed out that the exemplary embodiments are only examples that are not intended to restrict the scope, application and structure in any way. Rather, the above description gives the person skilled in the art a guideline for the implementation of at least one exemplary embodiment, with various changes, in particular with regard to the function and arrangement of the components described, being able to be made without departing from the scope of protection as can be seen from the Claims and these equivalent combinations of features results.

Claims

patent claims

1. Computer-implemented method (100) for testing a driver assistance system (2) of a vehicle (1), having the following work steps:

Simulating (101) a virtual traffic situation (3), which has the vehicle (1) and at least one other road user (4, 5a, 5b, 5c, 5d, 6), wherein a first road user (4) is controlled by a first user (2nd ) is controllable and other road users (5a, 5b, 5c, 5d, 6), which cannot be controlled by users, are controlled automatically, in particular by artificial intelligence or based on logic;

Outputting (102), on the basis of the virtual traffic situation (3), a virtual environment of the at least one first road user (4) to the first user (2) via a first, in particular at least optical, user interface (12);

Recording (103) of inputs from the first user (2) for controlling the at least one first road user (4) in the virtual environment of the first road user (4) via a second user interface (13), wherein when simulating the virtual traffic situation (3) the recorded inputs of the first user (2) and the resulting interaction of the at least one first road user (4) with his virtual environment are taken into account;

Operating (104) the driver assistance system (2) in a virtual environment of the vehicle (1) on the basis of the simulated traffic situation (3), the driver assistance system (2) showing a driving behavior;

Detecting (105) a scenario which arises as a result of the driving behavior of the driver assistance system (2) in the virtual environment of the vehicle (1); Determining (106) a quality of the resulting scenario depending on a predefined criterion, in particular a dangerousness of the resulting scenario; and

Outputting (107) the quality to the user via the first or a second user interface, in particular a display.

2. The method (100) according to claim 1, further comprising the step of: detecting (108) inputs from the first user (2) via the first user interface (12) or the second user interface (13) to change the simulated traffic situation ( 3), preferably by means of an editor, with operations by the first user (2) preferably being recorded and stored in a control table.

3. The method (100) according to claim 1, further comprising the step of: changing (109) the simulated traffic situation (3) on the basis of the determined quality, in particular until a termination condition is reached.

4. The method (100) according to any one of the preceding claims 1 to 3, wherein by simulating a speed, in particular an initial speed, of the vehicle (1) and/or the at least one other first road user (4) and/or a Trajectory, in particular an initial trajectory, of the vehicle (1) and/or the at least one other first road user (4) is specified.

5. Method (100) according to one of the preceding claims 2 to 4, wherein when changing the simulated traffic situation (3) only values of existing parameters of the simulated traffic situation (3) are changed.

6. The method (100) according to any one of the preceding claims 1 to 5, wherein parameters of the scenario (3), depending on the type of driver assistance system (2) to be tested, are selected from the following group:

Speed, in particular an initial speed, of a road user; a direction of movement, in particular a trajectory, one road user; lighting conditions; Weather; road surface; Temperature; number and position of static and/or dynamic objects; State and appearance of static and/or dynamic objects; a speed and a direction of movement, in particular a trajectory, of the dynamic objects; condition of signaling installations, in particular light signaling installations; traffic signs; number of lanes; acceleration or deceleration of road users or objects; Signs of soiling and/or aging of the road surface, geographic orientation of the traffic situation.

7. The method (100) according to any one of the preceding claims 1 to 6, wherein the quality is characterized by a, in particular fictitious, reward for the user.

8. The method (100) according to any one of the preceding claims 1 to 7, wherein the quality is higher, the more dangerous the respectively created scenario (3) is, in particular the shorter a calculated duration up to a collision time is.

9. The method (100) according to any one of the preceding claims 1 to 8, wherein the changing of the simulated scenario (3) is performed using evolutionary algorithms.

10. The method (100) according to any one of the preceding claims 1 to 9, wherein a utility function is approximated on the basis of the determined quality, which describes the quality value of a specific simulated scenario (3).

11. The method (100) according to any one of the preceding claims 1 to 10, wherein the driver assistance system (2) is simulated.

12. The method (100) according to any one of the preceding claims 1 to 11, wherein historical data from earlier test operations of a driver assistance system (2), in particular the driver assistance system (2), are taken into account during the initial simulation of the scenario.

13. The method (100) of one of the preceding claims 1 to 12, wherein when the driver assistance system (2) is operated, data relating to the environment (4) of the vehicle (1) is fed into the driver assistance system (2) and/or the driver assistance system (2), in particular its sensors, based on the environment (4) of the vehicle (1) are stimulated.

14. A computer program comprising instructions which, when executed by a computer, cause it to carry out the steps of a method according to any one of claims 1 to 13.

15. Computer-readable medium on which a computer program according to claim 14 is stored.

16. System (10) for generating scenarios for testing a driver assistance system of a vehicle (1), having:

Means (11) for simulating a virtual traffic situation (3), which has the vehicle and at least one other road user (1, 4, 5a, 5b, 5c, 5d, 6), wherein a first road user (4) is used by a first user (2) is controllable and wherein other road users (5a, 5b, 5c, 5d, 6), which cannot be controlled by users, are controlled automatically, in particular by artificial intelligence or logic-based; a first, in particular at least optical, user interface (12) for outputting, on the basis of the virtual traffic situation (3), a virtual environment of the at least one first road user (4) to the first user (2); and a second user interface (13) for detecting inputs from the first user (2) for controlling the at least one first road user (4) in the virtual environment of the first road user (4), with the virtual traffic situation (3) being simulated Input from the first user (2) and the resulting interaction of the at least one first road user (1) with his virtual environment who are taken into account;

Means (14) for operating the driver assistance system (2) in a virtual environment of the vehicle (1) on the basis of the simulated traffic situation (3), whereby the driver assistance system (2) shows a driving behavior;

Means (15) for detecting a scenario which is caused by the driving behavior of the driver assistance system (2) in the virtual environment of the vehicle (1); and

Means (16) for determining a quality of the resulting scenario as a function of a predefined criterion, in particular a dangerousness of the resulting scenario; wherein the first or the second user interface (12, 13), in particular a display, is also set up to output the quality to the user (2).