DE102022212760A1 - Film-based scenario generation for driving function tests - Google Patents

Abstract

The invention relates to a method for testing and/or validating an automated driving function of a vehicle, comprising the steps: automated analysis (S1) of a database with entertainment films and/or television programs for scenes with vehicles in traffic, mapping (S2) the traffic situation depicted in the scenes found in a virtual environment and mapping the movements of the road users depicted in the scenes in traffic in simulatable trajectories, storing (S3) the depicted virtual environment and the depicted trajectories for simulating the respective scenes in a test data set, and carrying out (S4) a validation test on the basis of the stored virtual environment and trajectories in the test data set.

Description

The invention relates to a method for testing and/or validating an automated driving function of a vehicle.

Automated driving functions, in particular driving control functions, can on the one hand support a human driver of a vehicle in the sense of a driver assistance system, but in the future they will become significantly more important for highly automated and autonomous vehicles. Depending on the level of authority, such automated driving control functions intervene more or less in the driving control of a vehicle and take over the vehicle control completely, especially in fully automated vehicles. This leads to demanding safety requirements, since vehicles participating in road traffic pose a fundamental risk to other road users if they behave incorrectly. Driving control functions receive data from corresponding sensor units in order to use perception processes to record the environment, in particular other road users and objects in the surroundings, including traffic signs. Since highly complex software components are therefore necessary to execute the automated driving control functions in order to take over the vehicle control and make corresponding decisions in road traffic, the proof of the required safety is correspondingly difficult and extensive. In order to test a sufficient number of real scenarios for the development and final validation of automated driving control systems in practice, however, an unacceptably large number of kilometers driven and a corresponding number of hours of driving by a vehicle equipped with an automated driving control system would be necessary. For example, covering billions of kilometers and the correspondingly delayed evaluation of the scenarios experienced there is disproportionately long for the development and validation of an automatic driving control system and cannot be carried out in practice. A purely distance-based, statistical proof of the safety of the driving function before such a vehicle is launched on the market is therefore technically not feasible. This and more is also explained in the following publication: "W. Wachenfeld and H. Winner, “The release of autonomous vehicles,” in Autonomous Driving: Technical, Legal and Social Aspects, M. Maurer, JC Gerdes, B. Lenz, and H. Winner, Eds. Springer, 2016, pp. 425-449 ". As well as in the publication of Sven Hallerbach, Ulrich Eberle, and Frank Koester, from 2022: “Simulation-Enabled Methods for Development, Testing, and Validation of Cooperative and Automated Vehicles”, Zenodo, https://doi.org/10.5281/zenodo.6542050 Future homologation, e.g. according to NCAP, will also require an increasingly complex test procedure that can only be carried out successfully with the help of realistic simulation environments.

In addition, it is not possible to test potentially critical scenarios in real traffic due to the risk to third parties. However, it is precisely potentially critical scenarios that provide the data that must be used to develop an automatic driving control system for a vehicle, as these are precisely the ones that lead to accidents. These critical scenarios are also important for the validation of a fully developed automatic driving control system. In the development and validation of automatic driving control systems, simulation-based methods are therefore used and tested in particular for the operating areas that are critical for the automated driving control function, such as when encountering the end of a traffic jam or in busy urban traffic areas. For example, through scenario-based development and testing, it is possible to estimate how safely the later real vehicle will operate in the open world (possibly within a limited operating area) by using the digital twin of the automated vehicle during early development phases.

However, the simulation-based development and testing of automated vehicles with an automatic driving control system, in particular the validation of a driving control function of an automatic driving control system for the automatic operation of a vehicle, is only possible with the help of simulation if the respective simulation modules can be proven to produce realistic results. This also applies, among other things, to the modeling of traffic dynamics, i.e. the simulation of the behavior of other road users. A realistic depiction of the environmental dynamics is therefore a key component for finding potentially critical scenarios for an automated vehicle.

For the reliable use of these simulation-based methods, it is a necessary prerequisite to model all entities relevant to the respective scenario with sufficient validity. In the context of simulation-based validation of automated driving functions, the realistic representation of the overall system, consisting of the digital twin of the vehicle and the environment, is becoming increasingly important, as the automated vehicle assumes responsibility for the entire driving task in a defined operating area. The valid representation of all relevant input variables for the environment detection of the ego vehicle (system under test) is essential, whereby the representation of realistic traffic dynamics (position, behavior and resulting movement of the surrounding road users) is of outstanding importance for the decision-making of the ego vehicle, for example. For the (simulative) validation of automated driving functions, a scenario database that is very broad is also necessary so that the vehicle does not encounter untested situations in real operation. Due to the natural rarity of such edge case scenarios, classic methods are not sufficient to discover them.

Due to the rarity of these edge case scenarios, synthetic video data is also used in this context. The tools used to create this synthetic video data come partly from industrial sectors outside the automotive industry, mainly the aerospace industry, as well as the "visual special effects" and "3D graphics engine" sectors.

The object of the invention is to improve the design and validation of an automatic driving control system of a vehicle, in particular for relevant and potentially critical traffic situations.

The invention results from the features of the independent claims. Advantageous further developments and embodiments are the subject of the dependent claims.

• - automatisiertes Analysieren einer Datenbank mit Unterhaltungsfilmen und/oder Fernsehsendungen auf Szenen mit Fahrzeugen im Straßenverkehr,
• - Abbilden der in den aufgefundenen Szenen dargestellten Verkehrssituation in eine virtuelle Umgebung und Abbilden der Bewegungen der in den Szenen dargestellten Verkehrsteilnehmer im Straßenverkehr in simulierbare Trajektorien, und Vervollständigen der in den Szenen dargestellten Trajektorien, wenn ermittelt wird, dass die in den Szenen dargestellten Trajektorien Sprünge über nicht dargestellte Bewegungsbahnen zwischen zwei Zeitpunkten aufweisen,
• - Abspeichern der abgebildeten virtuellen Umgebung und der abgebildeten Trajektorien zur Nachbildung der jeweiligen Szenen in einen Testdatensatz, und
• - Testen der automatisierten Fahrfunktion auf Basis der abgespeicherten virtuellen Umgebung und Trajektorien im Testdatensatz.

A first aspect of the invention relates to a method for testing and/or validating an automated driving function of a vehicle, comprising the steps:

• - automated analysis of a database of entertainment films and/or television programmes for scenes with vehicles in traffic,
• - mapping the traffic situation depicted in the scenes found into a virtual environment and mapping the movements of the road users depicted in the scenes in road traffic into simulatable trajectories, and completing the trajectories depicted in the scenes if it is determined that the trajectories depicted in the scenes have jumps over unrepresented movement paths between two points in time,
• - Saving the depicted virtual environment and the depicted trajectories for reproducing the respective scenes in a test data set, and
• - Testing the automated driving function based on the saved virtual environment and trajectories in the test data set.

The traffic situation includes in particular the prevailing traffic infrastructure shown in the scenes, i.e. for example a road layout, traffic lights, traffic signs, bridges, railway crossings, etc.;

The automated analysis of the database with the large number of entertainment films and/or television programs such as documentaries for scenes with vehicles is preferably carried out by automated image recognition. An artificial neural network that has been trained to recognize vehicles can be used for this. In particular, by parallelizing computing cores, large amounts of video material can be checked in a relatively short time. Although the automated analysis can in principle run completely automatically, additional control by a person can be used. In documentaries and entertainment films such as movies, scenes with vehicles often have an ulterior motive for a precarious traffic situation, which can be recognized through human interpretation, and if necessary, a focus can be placed on this precarious constellation using the information already available in the scenes or additional synthetic data. An automated examination of the scenes found can also be carried out to determine whether they can be simulated.

While the scenes are automatically checked for the presence of vehicles, especially cars and trucks, and only scenes with these vehicles are used, all road users who appear in these scenes are preferably analyzed and mapped for the test data set. Although scenes with only pedestrians appearing in them are hardly interesting, the pedestrians in the scenes with vehicles can contribute to a critical traffic situation and are therefore preferably mapped in the test data set.

Road user types (cars, trucks, cyclists, motorcyclists, etc.) are preferably determined using artificial intelligence and assigned to the respective trajectories. The vehicles can then be replaced by those available in the traffic simulation. In addition, any obstacles that occur are replaced with similar 3D assets or reproduced using automatic modeling methods, whereby textures can be taken from the respective ones found. Examples These include rockfalls, avalanches, etc.; the mapped trajectories are then allocated to all road users, including moving objects, and adapted to the generated traffic environment.

The intended use of the stored test data set is to use it to test or validate automated driving functions such as a lane keeping assistant, a maneuver decision module, a brake assistant, or a module for fully automated or fully autonomous vehicle control. This can be purely simulation-based, but also at the level of HIL, SIL, PIL - up to physical tests.

It is an advantageous effect of the invention that existing video material from entertainment films and/or television programs such as documentaries about traffic accidents can be used to recreate artificial scenarios for testing or validating an automated driving function. Precarious traffic situations that are rarely seen in everyday life are particularly often depicted in fictional works such as movies. This special property is advantageously exploited by the method according to the invention in order to recreate precisely those situations that rarely occur in everyday life, but must be given special consideration when developing an automated driving function, since it is precisely these rare cases that make the design of an automated driving function challenging and, on the other hand, can prove the reliability of the automated driving function even in extreme cases.

According to an advantageous embodiment, when mapping the found scenes into the mapped trajectories, it is checked whether the trajectories are plausible and/or physically possible, and if the trajectories are deemed implausible or not physically possible, the mapped trajectories are corrected into plausible and physically possible trajectories or the trajectories are discarded.

According to a further advantageous embodiment, a comparison of the generated virtual environment and the generated trajectories with a scenery database is carried out and a check for duplicates of the generated virtual environment and the generated trajectories with the scenery database is carried out within certain similarity limits, wherein the generated virtual environment and generated trajectories are only stored to the extent that they are outside the similarity to data already stored in the scenery database.

According to a further advantageous embodiment, the generated virtual environment and the generated trajectories are checked for relevance for the synthesis of a situation that is potentially critical for an automated driving function of a vehicle, wherein the generated virtual environment and the generated trajectories are only stored to the extent that a potentially critical situation is confirmed.

To test the relevance of whether a critical scenario occurs, all vehicles involved in a scene can act as ego vehicles.

According to a further advantageous embodiment, the generated virtual environment is complemented by a scenery generation, so that a potentially critical environment is always created for the automated driving function of a vehicle.

According to a further advantageous embodiment, for mapping the movements shown in the respective scene into simulatable trajectories and for mapping the respective scene onto the generated virtual environment, individual scenes are combined into scenes that are as continuous as possible by combining cutting sequences and camera angles.

According to a further advantageous embodiment, the trajectories are completed by generating completely synthetic trajectory sections for the trajectory sections missing in the scenes.

According to a further advantageous embodiment, the traffic situation shown in the scenes is supplemented by synthetic data and/or the traffic situation shown in one of the scenes is combined with the traffic situation shown in another of the scenes.

According to a further advantageous embodiment, the vehicle with the automated driving function to be tested and/or validated alternately takes on the roles of the vehicles from the scenes in a simulation.

For example, if the main character in the film is distracted and drives through an intersection at high speed and collides with another vehicle, the main character is the Challenger vehicle, while the vehicle at the intersection is the ego vehicle - and the simulated roles can be swapped.

According to a further advantageous embodiment, machine learning based clustering is used for the assignment of the elements of the generated virtual environment and generated trajectories.

Further advantages, features and details emerge from the following description, in which - if necessary with reference to the drawing - at least one embodiment is described in detail. Identical, similar and/or functionally identical parts are provided with the same reference numerals.

• 1: Ein Verfahren zum Testen und/oder Validieren einer automatisierten Fahrfunktion gemäß einem Ausführungsbeispiel der Erfindung.
• 2: Symbolisch die automatisierte Beschaffung der Szenariodaten aus einer Film-Datenquelle für das Ausführungsbeispiel der 1.

Show it:

• 1 : A method for testing and/or validating an automated driving function according to an embodiment of the invention.
• 2 : Symbolically, the automated acquisition of scenario data from a film data source for the implementation example of the 1 .

The representations in the figures are schematic and not to scale.

1 shows a method for testing and/or validating an automated driving function of a vehicle. For this purpose, an artificial neural network is provided which has been pre-trained with image data and is suitable for the automated detection of traffic events involving vehicles. This artificial neural network is repeatedly applied with input data from action films. The execution of the artificial neural network thus filters out the time stamps of the film at which scenes with vehicles in traffic begin and end. The films are cut accordingly and related scenes are analyzed for common and complementary data. Thus, an automatic analysis S1 is carried out to extract scenes with vehicles in traffic from the films. Once all relevant information about the respective scenes has been extracted from the films, this information is abstracted and both the traffic situation depicted in the scenes found are mapped into a virtual environment and the movements of the road users in traffic depicted in the scenes are mapped into simulatable trajectories S2. This image corresponds to a digitization of mathematically processable information in order to be able to use it for later simulation, for example, while the information about the environment and movements of the road users comes from the films. Such films often contain repetitions or even cuts, with the latter case missing trajectories of road users. However, such missing trajectories can be generated by appropriate interpolation, extrapolation, or other data reconstruction methods and therefore follow the boundary conditions as they still exist in the scenes at the last or first points in time before or after the missing points in time. The data obtained about the virtual environment and the trajectories are then saved S3. This creates a data set that is particularly suitable for sensitivity analyses, adjustments to the vehicle's driving function, cyclic validations, etc., since the data set can be modified as desired and is available in any number for simulation and other test and validation procedures, since it can be processed in a computing unit. This is followed by the use of the test data set for actual testing and/or validation S4 of the driving function of the vehicle, preferably in a simulation, whereby physically real elements can also be used in the test setup, for example as in the Hardware-In-The-Loop (abbreviated “HIL”) method.

2 symbolizes the data transfer from films such as classic movies, which are analyzed by artificial neural networks that perform automatic image analysis. In particular, films whose entertainment value is based on content with potentially critical scenes involving road users are particularly suitable for automatic data analysis as described in the process steps of 1 explained. In particular, machine analysis allows an almost complete (ideally even completely automatic) mechanism for computer-aided analysis and scenario generation, especially for later simulations. In particular, methods such as machine learning-based clustering are used for the assignment of scenario classes, the identification of possible parameter ranges in order to create logical scenarios, or the conversion of the data obtained into typical standards for scenario databases.

Although the invention has been illustrated and explained in detail by preferred embodiments, the invention is not limited by the disclosed examples and other variations can be derived from them by the person skilled in the art without departing from the scope of the invention. It is therefore clear that a multitude of possible variations exist. It is also clear that embodiments mentioned as examples really only represent examples that are not to be understood in any way as a limitation of the scope of protection, the possible applications or the configuration of the invention. Rather, the preceding description and the description of the figures enable the person skilled in the art to implement the exemplary embodiments in concrete terms, whereby the person skilled in the art, with knowledge of the disclosed inventive concept, can make a variety of changes, for example, with regard to the function or arrangement of individual elements mentioned in an exemplary embodiment, without departing from the scope of protection defined by the claims and their legal equivalents, such as further explanations in the description.

List of reference symbols

S1

Analyze

S2

Depict

S3

Save

S4

Testing

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA accepts no liability for any errors or omissions.

Cited non-patent literature

• "W. Wachenfeld and H. Winner, “The release of autonomous vehicles,” in Autonomous Driving: Technical, Legal and Social Aspects, M. Maurer, J. C. Gerdes, B. Lenz, and H. Winner, Eds. Springer, 2016, pp. 425-449 [0002]
• Sven Hallerbach, Ulrich Eberle, and Frank Koester, from 2022: “Simulation-Enabled Methods for Development, Testing, and Validation of Cooperative and Automated Vehicles”, Zenodo, https://doi.org/10.5281/zenodo.6542050 [0002]

Claims (10)

Method for testing and/or validating an automated driving function of a vehicle, comprising the steps: - automated analysis (S1) of a database with entertainment films and/or television programs for scenes with vehicles in traffic, - mapping (S2) the traffic situation shown in the scenes found into a virtual environment and mapping the movements of the road users in traffic shown in the scenes into simulatable trajectories, and completing the trajectories shown in the scenes if it is determined that the trajectories shown in the scenes have jumps over non-shown movement paths between two points in time, - saving (S3) the mapped virtual environment and the mapped trajectories for simulating the respective scenes in a test data set, and - testing (S4) the automated driving function on the basis of the saved virtual environment and trajectories in the test data set. Procedure according to Claim 1 , whereby when mapping the found scenes into the mapped trajectories it is checked whether the trajectories are plausible and/or physically possible, and if the trajectories are deemed implausible or not physically possible, the mapped trajectories are corrected into plausible and physically possible trajectories or the trajectories are discarded. Method according to one of the preceding claims, wherein a comparison of the generated virtual environment and the generated trajectories is carried out with a scenery database and a check is carried out for duplicates of the generated virtual environment and the generated trajectories with the scenery database within certain similarity limits, wherein the generated virtual environment and generated trajectories are only stored to the extent that they are outside the similarity to data already stored in the scenery database. Method according to one of the preceding claims, wherein the generated virtual environment and the generated trajectories are checked for relevance for the synthesis of a situation that is potentially critical for an automated driving function of a vehicle, wherein the generated virtual environment and the generated trajectories are only stored to the extent that a potentially critical situation is confirmed. Method according to one of the preceding claims, wherein the generated virtual environment is complemented by a scenery generation, so that a potentially critical environment for the automated driving function of a vehicle is always created. Method according to one of the preceding claims, wherein for mapping the movements shown in the respective scene into simulatable trajectories and for mapping the respective scene onto the generated virtual environment, individual scenes are combined by combining cutting sequences and camera angles to form scenes that are as continuous as possible. Method according to one of the preceding claims, wherein the completion of the trajectories is carried out by generating completely synthetic trajectory sections for the trajectory sections missing in the scenes. Method according to one of the preceding claims, wherein the traffic situation shown in the scenes is supplemented by synthetic data and/or the traffic situation shown in one of the scenes is combined with the traffic situation shown in another of the scenes. Method according to one of the preceding claims, wherein the vehicle with the automated driving function to be tested and/or validated alternately takes on the roles of the vehicles from the scenes in a simulation. Method according to one of the preceding claims, wherein machine learning-based clustering is used for the assignment of the elements of the generated virtual environment and generated trajectories.

Images