DE102021111724B4 - Method and computer program for evaluating a software version of a driver assistance system - Google Patents

Abstract

Method (2) for evaluating a software status of a driver assistance system, comprising:
- Providing (21) a recorded data sequence to a simulation environment, wherein the recorded data sequence comprises sensor data, odometry data and vehicle dynamics data from a real vehicle journey;
- Transforming the sensor data and/or the vehicle dynamics data of the recorded data sequence for the purpose of simulating a vehicle journey of an ego vehicle, the transformation taking place with regard to a speed which the ego vehicle should have during the simulated vehicle journey;
- Using the simulation environment, simulating (22) the vehicle journey of the ego vehicle depending on the recorded data sequence and the transformed sensor and/or vehicle dynamics data and on the basis of a software version of the driver assistance system, with a simulated data sequence being determined;
- Automatic checking (23) of the simulated data sequence to determine whether the simulated vehicle journey violates a performance criterion;
• if the checking shows that the simulated vehicle journey violates the performance criterion, automatic outputting (24a) of an error message;
• if the check does not reveal any violation of the performance criterion, providing (24b) the software status for operating a vehicle.

Description

The invention relates to a method and a computer program for evaluating a software version of a driver assistance system.

The invention can be used in particular as part of the development of a driver assistance system (FAS) which enables at least partially automated driving, preferably up to fully automated driving (VAF), of the vehicle.

In the context of the document, the term "automated driving" means driving with automated longitudinal and/or lateral guidance. Automated driving can, for example, involve driving on the freeway for a longer period of time or driving for a limited period of time when parking. The term "automated driving" includes automated driving with any degree of automation. Exemplary degrees of automation are assisted, partially automated, highly automated, fully automated and autonomous driving (with an increasing degree of automation in each case). The five levels of automation mentioned above correspond to SAE levels 1 to 5 of the SAE J3016 standard (SAE - Society of Automotive Engineering). With assisted driving (SAE Level 1), the system performs longitudinal or lateral guidance in certain driving situations. With semi-automated driving (SAE Level 2), the system takes over longitudinal and lateral guidance in certain driving situations, whereby the driver has to constantly monitor the system, as with assisted driving. With highly automated driving (SAE Level 3), the system takes over longitudinal and lateral guidance in certain driving situations without the driver having to constantly monitor the system; however, the driver must be able to take control of the vehicle within a certain period of time when requested by the system. With fully automated driving (SAE Level 4), the system takes over control of the vehicle in certain driving situations, even if the driver does not respond to a request to intervene, meaning that the driver is no longer a fallback option. With autonomous driving (SAE Level 5), the system can carry out all aspects of the dynamic driving task under any road and environmental conditions, which can also be controlled by a human driver. SAE Level 5 thus corresponds to driverless driving, in which the system can automatically handle all situations like a human driver throughout the journey; a driver is generally no longer required.

The function of modern ADAS, which enables automated driving, consists of a complex system consisting of various components. Therefore, changes made in the software in a component during the development stage can lead to a degradation of the function as a whole, without this being noticeable in any case on the basis of tests at the component level alone.

To solve this problem, the overall function can be tested in a simulation or by real test drives on the road. However, both approaches have their specific disadvantages: Real test drives involve a lot of effort and are difficult to scale, while tests using simulation only offer an approximation of real sensor data and therefore cannot take all error cases into account.

An approach that combines sensor data from a real journey with a simulation was developed in the German patent application DE 10 2016 205 806 A1 suggested. In a method described therein for optimizing a controller in a vehicle, relative measurement data recorded in a specified period of time are converted into a specified coordinate system and integrated into a simulation environment, so that the function is simulated on the basis of this processed measurement data.

Bach et al. have presented a method for reusing recorded test data in the context of virtual verification of controller-based ADAS. The method is based on an identification of causal dependencies between different input data streams (cf. BACH, J. [et al.]: Control based driving assistant functions' test using recorded in field data. In: 7th Conference on Driver Assistance Systems, 2015, https:/ /mediatum.ub.tm.de/node?id=1285215).

A further development of the method by Bach et al. In addition to the reuse of recorded driving data in a closed-loop simulation, it also provides for automated assessments to evaluate software increments (cf. BACH, Johannes: Reactive-Replay Approach for Verification and Validation of Closed-Loop Control Systems in Early Development. In: SAE technical papers, Vol.2017-01-1671, 2017, pages 1-8 - ISSN 0148-7191

There is still a need for technical solutions for reliable and efficient verification and validation of ADAS functions, especially with a view to preventing degradation of an overall ADAS function during the development of automated driving.

Proceeding from this, it is an object of the present invention to specify a method for evaluating a software version of an ADAS, which enables reliable and efficient (further) development of the ADAS.

The object is solved by a method and a computer program according to the independent patent claims. Advantageous embodiments are specified in the dependent claims.

It is pointed out that additional features of a patent claim dependent on an independent patent claim without the features of the independent patent claim or only in combination with a subset of the features of the independent patent claim can form a separate invention independent of the combination of all features of the independent patent claim, which can be made the subject of an independent patent claim, a divisional application or a subsequent application. This applies equally to the technical teachings explained in the description, which can form an invention independent of the features of the independent patent claims.

A first aspect of the invention relates to a method for evaluating a software version of a driver assistance system (DAS). Evaluating the software status can be understood to mean, for example, checking, testing, validating and/or verifying the software status.

The ADAS can be set up to perform functions as part of at least partially automated driving of a motor vehicle, such as an ACC function (i.e. a combined speed and distance control).

The term motor vehicle is understood to mean, in particular, a land vehicle that is moved by machine power without being tied to railroad tracks. A motor vehicle in this sense can, for example, be designed as a motor vehicle, motorcycle or tractor.

One step of the method is providing a recorded data sequence to a simulation environment, the recorded data sequence comprising sensor data, odometry data and vehicle dynamics data from a real vehicle journey (in particular from a specific section of the real vehicle journey).

For example, the data sequence can have been recorded during a real test drive of a test vehicle or during a real drive of a customer vehicle on the road. Such recorded data sequences, which characterize a chronological sequence of relevant data during a real vehicle journey (or a section thereof), are also referred to as "traces".

In one embodiment variant of the method, the method can also include the recording of such a data sequence during a real vehicle journey as a preceding step. Provision can be made for the data sequence to be recorded automatically when a specific trigger condition is met, the trigger condition characterizing, for example, specific properties of a vehicle environment and/or specific vehicle dynamic states.

For example, environmental sensor data (e.g. from camera and radar), odometry data and vehicle dynamics data can be recorded during a test drive with a test vehicle in traffic. A scenario is preferably selected that is of interest for the development of a function of automated driving, such as following in the context of ACC, in which the speed of the test vehicle is limited by that of a vehicle in front.

The data sequence is preferably recorded under similar conditions (e.g. in terms of speed, distance to a vehicle in front, etc.) that are relevant to the ADAS function to be evaluated (such as an ACC following route). For example, while driving, the test driver can ensure that the distance to the vehicle in front, the speed, the position in the lane, etc., meet the requirements for the automated driving function. The trip can thus be a best-case trip, so to speak, in which the test vehicle implements a subsequent trip that would meet the requirements of the automated function. The following drive can be carried out manually by the test driver during the test drive, i.e. e.g. without activated ACC function. In other words, the experienced test driver can control the test vehicle during the recording as one would wish for the automated driving function (e.g. ACC) and thus create a reference for the development of a new software version of the automated driving function.

The sensor data contained in the recorded data sequence can in particular include data containing information about the vehicle's surroundings. These may have been detected by means of an environment sensor system of the vehicle, such as by means of a (particularly their optical) camera, a radar sensor, a lidar sensor, an ultrasonic sensor and/or an acoustic sensor for detecting audible sound.

The odometry data contained in the recorded data sequence can in particular include position and/or orientation-related information based on recorded movement parameters of the vehicle, such as wheel revolutions. For example, the odometry data can indicate a relative position and/or orientation of the vehicle with respect to a (past) reference position or reference orientation. In addition, relevant data about environmental conditions, such as current road conditions (e.g. in the form of friction coefficients), can also be derived from the odometry data.

The vehicle-dynamic data contained in the recorded data sequence can in particular indicate a respective acceleration and/or a speed of the vehicle at different points in time during the real vehicle journey.

The simulation environment can be realized by means of a computing device which is arranged outside of the vehicle which has carried out the real vehicle journey. For example, it can be a comparatively powerful computing device in a development laboratory. In this case, the recorded data sequence can be transmitted from the vehicle to the computing device, e.g. using a portable data storage medium or via a wired or wireless data connection (e.g. mobile radio, WLAN or Bluetooth).

In another embodiment, the simulation environment can be implemented by means of a computing device that is arranged in the vehicle that carries out the real vehicle journey. In this case, the computing device can be connected directly to a data bus of the vehicle, for example, and obtain the relevant data of the data sequence from there. As an alternative or in addition, the recorded data sequence can also be transmitted from a data memory to the computing device in a wired or wireless manner within a vehicle.

In a further step, the simulation environment is used to simulate a vehicle journey depending on the recorded data sequence and on the basis of a software version of the ADAS (to be checked), with a simulated data sequence being determined.

In other words, the ADAS or a specific function of the ADAS can be reprocessed with any software version in the simulation environment on the basis of the recorded data sequence. The recorded vehicle dynamics and odometry data (in particular data of the recorded data sequence, which correspond to a start time of the recording, for example) can be used to initially stimulate a vehicle model realized in the simulation environment. The vehicle model (or a virtual ego vehicle simulated by it) can then “drive” independently within the framework of a closed-loop simulation in the reprocessed vehicle environment.

A closed-loop simulation is to be understood in particular as a simulation in which, after the initial stimulation of the vehicle model and as a function of the recorded sensor data, a first target trajectory is planned for a first time step and processed by the vehicle model. A result of the processing of the first target trajectory in the vehicle model, which in particular can include simulated odometry and vehicle dynamics data, is used in turn - together with the recorded sensor data - as the basis for planning a second target trajectory for a second time step, which in turn is in the vehicle model is processed. This loop can be executed iteratively for a number of time steps, resulting in the virtual ego vehicle being able to drive over a longer period of time. The simulated data sequence (trace) can be determined and recorded, which can in particular include the simulated vehicle dynamics data.

The invention is based on the idea that such a closed-loop reprocessing based on real sensor data can close the initially described gap between complex real test drives on the one hand and imperfect simulated surroundings sensor data on the other. This opens up the possibility of being able to test the overall function of an ADAS on the basis of real sensor data, without having to rely on test drives for each new software version.

The software status of the FAS, which influences the simulated behavior of the FAS, can be implemented as part of the vehicle model or at least conceptually understood as part of the vehicle model according to some embodiments, such as in an embodiment described in more detail below with reference to the figures. It is understood that FAS functions in other embodiments but also, so to speak, a software module, wel ches can be understood as the actual vehicle model module, can be implemented around where there is a functional interaction between the ADAS functions and the more narrowly understood vehicle model module.

The software version used for the simulation can basically be any software version that is of interest, for example, in the context of the development of ADAS (e.g. as an interim version or as a version intended for a final release). In particular, it may be a modified or expanded (developed) software version of the ADAS compared to an earlier software version. The modification or extension can, for example, directly affect only one component or some components of the entire ADAS.

For example, the software version (compared to an earlier software version) may have an innovation or modification in a component that affects the processing of environmental sensor data, such as a new filter setting in a sensor fusion component. In this case, the new software version can lead to objects that are generated as output from the sensor fusion component according to the new software version differing from objects that the sensor fusion component has output according to the previous software version (i.e. that e.g. less or more detected in the vehicle environment objects can be output than in the previous software version). During the simulated journey, the virtual ego vehicle then reacts, among other things, to the objects generated by the new version of the sensor fusion component.

As an alternative or in addition to changes in the area of processing the environmental sensor data, innovations or modifications in the software version can also affect, for example, a driving strategy or trajectory planning within the ADAS.

A further step of the method is an automatic check of the simulated data sequence to determine whether the simulated vehicle journey violates a (e.g. predetermined) performance criterion. The checking can be carried out, for example, using one or more software modules set up for this purpose, which can be referred to as evaluation modules or evaluators. The evaluators each implement specific evaluation functions for an evaluation of the simulated data sequence with regard to a respective performance criterion.

For example, the automatic checking can include a comparison of the simulated data sequence with the recorded data sequence or a comparison of the corresponding simulated vehicle journey with the real vehicle journey. The simulated data sequence (or a specific section thereof) can therefore be compared with the recorded data sequence (or a specific section thereof) as part of the check. Any relevant deviation found can be assessed as a violation of the performance criterion. The performance criterion can therefore relate, for example, to the question of whether and, if so, to what extent data from the first data sequence differ from data from the second data sequence. On the basis of such deviations - or more generally on the basis of detected violations of the performance criterion - errors and/or degradations of the ADAS software status examined can be detected, for example.

In the event that the check reveals that the simulated vehicle journey violates the performance criterion, an error message is automatically output. In this context, an error message is to be understood, in very general terms, as information that a human user can perceive that the performance criterion has been violated.

The error message preferably also includes more precise information about the breach of the performance criterion, e.g. in the form of a marking at a relevant point in the simulated data sequence and/or at a corresponding point in the recorded data sequence. The error message can therefore include, for example, an automatically generated marker that indicates which part of the simulated data sequence the violation of the performance criterion is based on.

For example, differences between the real journey (i.e. the recorded data sequence) and the simulated journey (i.e. the simulated data sequence) that come to light and are relevant with regard to the performance criterion can be automatically written out as errors. This enables automatic detection of errors and functional degradations, which makes it easier for the ADAS developers to discover and specifically correct a deterioration in a software status due to specific extensions or changes (which may only affect a specific component).

In the example of a new filter setting of a sensor fusion component mentioned above, for example, it can happen that the new filter settings, a vehicle in front that is actually present during the real journey disappears in the reprocessing. As a result, the simulated behavior of the vehicle model differs from real driving, since the speed is no longer limited by the incorrectly filtered out vehicle in front. With the help of evaluators, this misconduct is automatically recognized and a defect is generated.

If, on the other hand, the automatic check does not result in a violation of the performance criterion, the checked software status is made available for operating a vehicle.

In particular, an automated software gate can be provided, which releases the software version for use in a vehicle only under the condition that the performance criterion has not been violated when the simulated data sequence is checked. For example, it can be ensured in this way that a software version that contains an incorrect change in a component does not even receive the status of a new (overall) software version of the ADAS during the development of the ADAS or that the incorrect change does not even appear in a current (overall) software version of the ADAS is allowed in. As a result, deterioration of a software status due to incorrect changes during the development of an ADAS can be avoided in a reliable and efficient manner.

• Zunächst wird mittels eines Trajektorienplanungsmoduls und in Abhängigkeit von der aufgezeichneten Datensequenz (insbesondere in Abhängigkeit von Odometriedaten und Fahrzeugdynamikdaten aus der aufgezeichneten Datensequenz) eine erste Soll-Trajektorie für einen ersten Zeitschritt der Simulation geplant.

According to one embodiment, the simulating includes the following process:

• First, a first target trajectory for a first time step of the simulation is planned using a trajectory planning module and depending on the recorded data sequence (in particular depending on odometry data and vehicle dynamics data from the recorded data sequence).

According to some embodiments, the trajectory planning module can, in addition to the actual trajectory planning, carry out functions of a higher level, which are often referred to as driving strategy or “motion planning”. Furthermore, the trajectory planning module can also include functions of a trajectory follow-up controller (subsequent to the actual trajectory planning). According to some statements, the term trajectory planning module should therefore be understood functionally broadly in this sense.

Then, using a vehicle model module and depending on the first target trajectory for the first time step, first simulated odometry data and first simulated vehicle dynamics data of a simulated ego vehicle are generated, the first simulated odometry and vehicle dynamics data e.g. for one or more points in time within the first time step may apply.

A second target trajectory for a second time step (following the first time step) is then planned by means of the trajectory planning module and as a function of the first simulated odometry data and the first simulated vehicle dynamics data.

Depending on the target trajectory for the second time step, second simulated odometry data and second simulated vehicle dynamics data for the second time step are then generated by means of the vehicle model module.

This loop can be recursively executed repeatedly for a number of time steps. The simulated odometry and/or vehicle dynamics data can be determined and recorded as the simulated data sequence (or as a part thereof).

In one embodiment variant, the simulation includes processing of sensor data and odometry data from the recorded data sequence using an environment model module, environment data being generated. The environmental data are taken into account when planning the target trajectories. In other words, based on the recorded sensor data and odometry data, a suitable environment model or environment model can be generated for the simulation, with the environment model being able to contain information about objects and about the course of a road or lane. The environment model preferably also includes a prediction (e.g. relating to objects and the course of the road or lane) with a view to a future relevant to the simulation.

It is also within the scope of the invention for the sensor data and/or the vehicle dynamics data of the recorded data sequence to be transformed for the purpose of simulation and/or for the purpose of automatic checking, ie for example a comparison with the simulated data sequence. This can, for example, concern a transformation of the recorded sensor data into a suitable coordinate system of the simulated ego vehicle. Such a transformation has been described in the German patent application mentioned above DE 10 2016 205 806 A1 described.

In the method proposed here, a transformation of the sensor data and / or the vehicle dynamics data with respect to a Running speed of the simulated ego vehicle, for example, specifically to produce suitable conditions for checking the software version or certain functions of the software version by means of the simulation. For example, the recording may have taken place at a driving speed of 100 km/h, but a simulation may be desired for a maximum of 80 km/h (e.g. according to an activation condition for the relevant

ADAS function, which is only approved for speeds up to 80 km/h). In such a case, the environment model can use a transformation to adapt the objects accordingly and, for example, slow them down so that they can be used for reprocessing under the desired conditions.

It is also conceivable that the recorded odometry data is transformed or modified for the purposes of the simulation, e.g. to simulate different road conditions, such as a different coefficient of friction, for the virtual drive than for the recorded drive.

If checking the simulated data sequence has not resulted in a violation of the performance criterion, the method can also include carrying out another real vehicle journey, in which a (real) driver assistance system is operated on the basis of the software status provided. According to some embodiment variants, it is also conceivable, for example, that a different sensor system or a sensor that is installed at a different location than during the original vehicle drive is used in the further test drive.

Optionally, another data sequence can then be recorded during the further real vehicle journey. This can be followed by carrying out the method steps described above again according to the first aspect of the invention on the basis of the further recorded data sequence and a further software version, which can differ from the software version used first.

When the method steps are carried out again (i.e. when reprocessing again on the basis of the further recorded data sequence), a different performance criterion can be used than in the previous run. For example, the performance criterion used on a subsequent pass may be more ambitious than the performance criterion used on a previous pass. In other words, the software gate for allowed changes to the software level of the ADAS can become increasingly strict. For example, a performance criterion that is initially used can only evaluate whether incorrect braking processes (to "false positives", i.e. objects that are actually not present) or the like occur. A more stringent performance criterion used later can, for example, also evaluate whether the virtual ego vehicle always drives sufficiently close to a center line. Even more advanced performance criteria, which can be evaluated, for example, by a so-called rescue lane evaluator, can relate to functions such as desired behavior of the ego vehicle in a traffic jam (creating a rescue lane).

As a result, the proposed method can be classified in an iterative process in which the software status of the ADAS is further developed and improved as continuously as possible.

A second aspect of the invention is a computer program for evaluating a software version of a driver assistance system. The computer program can be set up to carry out method steps according to the first aspect of the invention. Therefore, embodiments of the computer program according to the invention can correspond to the advantageous embodiments of the method according to the invention described above and below and vice versa. In particular, a simulation environment of the type described above in connection with the method can be implemented using the computer program when it is executed on a computing device.

• - Eine aufgezeichnete Datensequenz zu empfangen, wobei die aufgezeichnete Datensequenz Sensordaten, Odometriedaten und Fahrzeugdynamikdaten von einer realen Fahrzeugfahrt umfasst;
• - eine Fahrzeugfahrt in Abhängigkeit von der aufgezeichneten Datensequenz und auf Basis eines Softwarestands des Fahrerassistenzsystems zu simulieren, wobei eine simulierte Datensequenz ermittelt wird;
• - die simulierten Datensequenz daraufhin zu überprüfen, ob die simulierte Fahrzeugfahrt ein Performance-Kriterium verletzt;
  • • eine Fehlermeldung auszugeben, falls das Überprüfen ergibt, dass die simulierte Fahrzeugfahrt das Performance-Kriterium verletzt;
  • • den Softwarestands zum Betreiben eines Fahrzeugs bereitzustellen, falls das Überprüfen keine Verletzung des Performance-Kriteriums ergibt.

In line with this, the computer program according to the second aspect of the invention comprises instructions which cause a computing device on which the computer program is executable to:

• - receive a recorded data sequence, the recorded data sequence comprising sensor data, odometry data and vehicle dynamics data from a real vehicle journey;
• - to simulate a vehicle journey as a function of the recorded data sequence and based on a software status of the driver assistance system, with a simulated data sequence being determined;
• - check the simulated data sequence to determine whether the simulated vehicle journey violates a performance criterion;
  • • issue an error message if the checking reveals that the simulated vehicle journey violates the performance criterion;
  • • Provide the software status for operating a vehicle if the check does not reveal a violation of the performance criterion.

According to some embodiments, the computer program can comprise a number of parts, each of which can be executed on different computing devices (such as, for example, a number of processors) that may be spatially remote from one another.

According to one embodiment, the computer program comprises in particular the following software modules: an environment model module for generating environment data as a function of the recorded data sequence, in particular as a function of sensor data and odometry data from the recorded data sequence; a trajectory planning module for planning a target trajectory depending on the recorded data sequence and the environment data (and possibly in later iterations of the simulation depending on already simulated odometry data and simulated vehicle dynamics data); a vehicle model module for generating simulated odometry data and simulated vehicle dynamics data as a function of the target trajectory; and an evaluation module for checking whether the simulated data sequence comprising the simulated odometry data and the simulated vehicle dynamics data violates the performance criterion.

A third aspect of the invention is a computer-readable (storage) medium storing a computer program according to the second aspect of the invention.

• 1 veranschaulicht schematisch ein Computerprogramm zum Evaluieren eines Softwarestands eines Fahrerassistenzsystems, wobei das Computerprogramm mehrere unterschiedliche Softwaremodule umfasst.
• 2 zeigt ein schematisches Ablaufdiagramm eines Verfahrens zum Evaluieren eines Softwarestands eines Fahrerassistenzsystems.

The invention will now be explained in more detail using exemplary embodiments and with reference to the accompanying drawings. The features and feature combinations mentioned above or below in the description and/or shown alone in the drawings can be used not only in the combination specified, but also in other combinations or on their own, without departing from the scope of the invention.

• 1 schematically illustrates a computer program for evaluating a software status of a driver assistance system, the computer program comprising a number of different software modules.
• 2 shows a schematic flowchart of a method for evaluating a software version of a driver assistance system.

The following are shown in the block diagram according to 2 Steps 21-26 shown of an embodiment of the method 2 according to the invention are explained, whereby at the same time reference is made to the in 1 illustrated hardware and software components that implement a suitable simulation environment.

In a step of method 2 (see block 21 in 2 ) a recorded data sequence is provided to a simulation environment. The recorded data sequence includes sensor data, odometry data and vehicle dynamics data from a real vehicle journey.

The simulation environment is created using an in 1 Computing device 1 shown schematically realized, which is arranged outside of the vehicle that has carried out the real vehicle journey. This can be, for example, a high-performance computing device 1 in a development laboratory of a vehicle manufacturer.

The computing device 1 retrieves the recorded data sequence (or specific data from it) from a data memory 3 . For example, the data memory 3 is a portable data memory that is used to transmit the recorded data sequence from the real vehicle to the computing device 1 .

In a further step (see block 22 in 2 ) a vehicle journey is simulated using the simulation environment depending on the recorded data sequence and on the basis of a software version of an ADAS. A simulated data sequence is determined in the process.

The step 22 of simulating comprises several sub-steps, which are carried out by means of different, in 1 schematically illustrated software modules 11-14 of the computer program and which are explained in detail below.

In 1 are data that are transmitted between individual software components or between the data memory and certain software components, symbolized by arrows. Solid arrows represent recorded data, ie data from the data sequence recorded and stored in the data memory 3 . Dashed arrows represent data generated as part of the simulation.

An environment model module 11 retrieves sensor data and odometry data from the recorded data sequence from the data memory 3 and processes them in order to generate environment data for the simulation. Objects and a road geometry are generated from the merged sensor data. The environment data can, for example, initially be related to a coordinate system whose origin is on the rear axle of the real ego vehicle, during which the data sequence was recorded.

The environment data are transmitted to a trajectory planning module 12 . As already mentioned above, the term trajectory planning module 12 should be understood functionally broadly. In this sense, the trajectory planning module 12 can also be referred to more generally as a motion planning module.

The environment data transmitted from the environment model module 11 to the trajectory planning module 12 can, for example, first be subjected to a coordinate transformation into a coordinate system suitable for the simulation in a submodule 121 of the trajectory planning module 12, which can be referred to as a transformation module 121. The transformation module 121 can also process odometry data from the recorded data sequence, which is provided, for example, by a vehicle model module 13 of the computer program, which in turn has retrieved the odometry data from the data memory 3. The coordinate transformation can, for example, ensure that objects and the road geometry are correctly seen from the perspective of a simulated (virtual) ego vehicle and not from the perspective of the real vehicle, during which the data sequence was recorded.

The correspondingly transformed object and road geometry data are provided to a further sub-module 122 of the trajectory planning module 12, which can be referred to as an optimization module or also as a QP planner (where QP stands for quadratic programming). In the QP planner 122, several possible trajectories of the virtual ego vehicle are generated for a first time step, e.g. The transformed environment data are taken into account.

In addition, the planning of the possible trajectories for the first time step takes into account odometry data and vehicle dynamics data that are provided by the vehicle model module 13 to the QP planner 122 . The vehicle model module 13 first retrieves odometry data and vehicle dynamics data from the recorded data sequence from the data memory 3 . From this, the vehicle model module 13 generates vehicle dynamics data that may have been adjusted for the virtual ego vehicle whose journey is to be simulated and provides this (in coordinates that are related to the virtual ego vehicle) to the QP planner 122 as a basis for trajectory planning .

The vehicle model module 13 can include an activation function which queries whether certain conditions are met as a prerequisite for activating an ADAS function to be checked. A targeted transformation with regard to a speed of the simulated ego vehicle is carried out in order, for example, to create suitable conditions for checking the software status or certain functions of the software status using the simulation. For example, the recording may have taken place at a driving speed of 100 km/h, but a simulation for a maximum of 80 km/h is desired (e.g. according to an activation condition for the relevant ADAS function, which is only released for speeds up to 80 km/h is). In such a case, the vehicle model module 13 can, for example, adapt the vehicle dynamics data to be used (initially) for the simulation in such a way that the conditions for activating the relevant ADAS function are given. At the same time, the environment model module 11 can adapt the objects accordingly through a suitable transformation and, for example, slow them down so that they can be used for reprocessing under the desired conditions.

In the exemplary embodiment shown, a trajectory selection module 123 is connected downstream of the QP planner 122 . The trajectory selection module 123 receives the multiple possible trajectories in coordinates that relate to the virtual ego vehicle and selects a trajectory that is most suitable (e.g. with a view to a driving strategy) as the target trajectory for the first time step.

The target trajectory is transmitted from the trajectory selection module 123 to the vehicle model module 13 and processed there. In this case, first simulated odometry data and first simulated vehicle dynamics data of the virtual ego vehicle are generated in the vehicle model module 13 as a function of the target trajectory for the first time step. The first simulated odometry and vehicle dynamics data can apply, for example, to one or more points in time within the first time step.

A second target trajectory for a second time step (following the first time step) is then planned by means of the trajectory planning module 12 and as a function of the first simulated odometry data and the first simulated vehicle dynamics data.

Depending on the target trajectory for the second time step, second simulated odometry data and second simulated vehicle dynamics data for the second time step are then generated by means of the vehicle model module 13, etc.

This loop can be executed iteratively (“closed loop”) for a number of time steps. The simulated odometry and/or vehicle dynamics data can be used as a simulated data set sequence (ie a simulated trace) or as a part thereof and recorded.

The simulated vehicle dynamics and odometry data and, optionally, the environment data used in the simulation can be generated, for example, from a (in 1 not shown) visualization module are processed, which generates a clear visualization of the virtual vehicle journey to make it easier for the developers of the ADAS to analyze the software status.

In a further step of the method (see block 23 in 2 ) the simulated data sequence is automatically checked to determine whether the simulated vehicle journey violates a performance criterion.

Checking 23 is performed automatically by one or more evaluation modules 14, each of which includes specific evaluation functions for evaluating the simulated data sequence with regard to a respective performance criterion.

Checking 23 includes a comparison of the simulated data sequence with the recorded data sequence. In particular, the simulated odometry and/or vehicle dynamics data (such as accelerations) that the evaluation module 14 in the exemplary embodiment according to 1 receives from the vehicle model module 13, with the recorded odometry and/or vehicle dynamics data (such as accelerations), which the evaluation module 14 in the exemplary embodiment according to 1 retrieves from the data memory 3, are compared.

The simulated data sequence (or part of it) is thus compared with the recorded data sequence (or part of it), and any relevant deviation found (e.g. between recorded and simulated accelerations) can be evaluated by the evaluation module 14 as a violation of the performance criterion become. The performance criterion can, for example, relate to the question of whether and, if so, to what extent data from the first data sequence differ from data from the second data sequence. On the basis of such deviations or, more generally, on the basis of identified violations of the performance criterion, errors or degradations in the ADAS software version being examined can be uncovered.

If checking 23 shows that the simulated vehicle journey violates a performance criterion, an error message is automatically output (procedural step according to block 24a in 2 ). The error message can include detailed information about the breach of the performance criterion, for example in the form of a marking generated by the evaluation module 14 at a relevant point in the simulated data sequence and/or at a corresponding point in the recorded data sequence. The error message can thus include, for example, an automatically generated marker which indicates which section of the simulated data sequence is associated with the violation of the performance criterion. For example, differences between the real journey (ie the recorded data sequence) and the simulated journey (ie the simulated data sequence) that come to light and are relevant with regard to the performance criterion can be written out automatically as errors.

For example, if the software status to be checked includes a new filter setting for a sensor fusion component, it can happen that a vehicle in front that is actually present during the real journey disappears in the reprocessing. As a result, the simulated behavior of the vehicle model differs from the real journey, since the speed is no longer limited by the incorrectly filtered out vehicle in front. With the help of evaluators, this misconduct is automatically recognized and a defect is generated. Conversely, as a result of a new filter setting for a sensor fusion component, it could also happen that a "ghost object" is detected, i.e. e.g. that a vehicle in front that is actually not present or is at least not driving in the lane of the ego vehicle is incorrectly evaluated as a target object, so that it Virtual vehicle driving leads to an unnecessary braking process.

The new filter setting of a sensor fusion component mentioned above is just one example of a large variety of possible changes in a software version, which can lead to errors that can be discovered using the method described. In principle, these can be changes in any component of the ADAS, from the sensors to the environment model to trajectory planning.

If, on the other hand, the check does not result in a violation of a performance criterion, the checked software version is made available for operating a vehicle (method step according to block 24b in 2 ). In particular, an automated software gate can be provided, which releases the software version for use in a vehicle only under the condition that the performance criterion has not been violated when the simulated data sequence is checked. In this way it can be ensured that a software version that contains a faulty contains a change in a component, does not even receive the status of a new (overall) software version of the ADAS as part of the development of the ADAS or that the faulty change is not even allowed into a current (overall) software version of the ADAS.

If checking the simulated data sequence has not revealed any violation of a performance criterion, another real vehicle journey can be carried out, for example, in which a (real) driver assistance system is operated on the basis of the software version provided (see block 25 in 2 ). During the further real vehicle journey, another data sequence can be recorded (see block 26 in 2 ). This can be followed by renewed reprocessing of the data sequence recorded in the simulation environment and of a further software version that differs from the software version used first.

When reprocessing again, the same or a different performance criterion can be used than used in the previous pass. For example, the performance criterion used on a subsequent pass may be more ambitious than the performance criterion used on a previous pass. The method described can therefore be part of an iterative process in which the software status of the ADAS is further developed and continuously improved as far as possible.

Reference List

1

computing device

11

environment model module

12

trajectory planning module

121

transformation module

122

QP planner

123

trajectory selection module

13

vehicle model module

14

evaluation module

2

Proceedings

21 - 26

process steps

3

data storage

Claims (9)

Method (2) for evaluating a software status of a driver assistance system, comprising: - Providing (21) a recorded data sequence to a simulation environment, wherein the recorded data sequence comprises sensor data, odometry data and vehicle dynamics data from a real vehicle journey; - Transforming the sensor data and/or the vehicle dynamics data of the recorded data sequence for the purpose of simulating a vehicle journey of an ego vehicle, the transformation taking place with regard to a speed which the ego vehicle should have during the simulated vehicle journey; - Using the simulation environment, simulating (22) the vehicle journey of the ego vehicle depending on the recorded data sequence and the transformed sensor and/or vehicle dynamics data and on the basis of a software version of the driver assistance system, with a simulated data sequence being determined; - Automatic checking (23) of the simulated data sequence to determine whether the simulated vehicle journey violates a performance criterion; • if the checking shows that the simulated vehicle journey violates the performance criterion, automatic outputting (24a) of an error message; • if the check does not reveal any violation of the performance criterion, providing (24b) the software status for operating a vehicle. Method (2) according to claim 1 , wherein the simulating (22) comprises: - planning, by means of a trajectory planning module (12) and depending on the recorded data sequence, a first target trajectory for a first time step; - Generating, by means of a vehicle model module (13) and depending on the first target trajectory for the first time step, of first simulated odometry data and first simulated vehicle dynamics data of the simulated ego vehicle; - Planning, by means of the trajectory planning module (12) and depending on the first simulated odometry data and the first simulated vehicle dynamics data, a second target trajectory for a second time step; - Generating, by means of the vehicle model module (13) and depending on the target trajectory for the second time step, of second simulated odometry data and second simulated vehicle dynamics data for the second time step. Method (2) according to claim 2 , further comprising: - processing of sensor data and odometry data from the recorded data sequence by means of an environment model module (11), environment data being generated; - Considering the environment data when planning the target trajectories. Method (2) according to one of the preceding claims, wherein the automatic checking (23) comprises a comparison of the simulated data sequence with the recorded data sequence. Method (2) according to any one of the preceding claims, further comprising: - If the check has not resulted in a violation of the performance criterion, performing (25) another real vehicle journey, with a driver assistance system being operated on the basis of the software status provided. Method (2) according to claim 5 , further comprising: - recording (26) a further data sequence in the further real vehicle journey; and - carrying out the method steps according to again claim 1 on the basis of the further data sequence and a further software status. Method (2) according to claim 6 , wherein when carrying out the method steps again according to claim 1 another performance criterion is used. Computer program (1) for evaluating a software status of a driver assistance system, the computer program comprising instructions which cause a computing device (1) on which the computer program can be executed: - receive a recorded data sequence, the recorded data sequence comprising sensor data, odometry data and vehicle dynamics data from a real vehicle journey; - to transform the sensor data and/or the vehicle dynamics data of the recorded data sequence for the purpose of simulating a vehicle journey of an ego vehicle, the transformation taking place with regard to a speed which the ego vehicle should have during the simulated vehicle journey; - to simulate the vehicle journey of the ego vehicle as a function of the recorded data sequence and the transformed sensor and/or vehicle dynamics data and on the basis of a software version of the driver assistance system, with a simulated data sequence being determined; - check the simulated data sequence to determine whether the simulated vehicle journey violates a performance criterion; • issue an error message if the checking reveals that the simulated vehicle journey violates the performance criterion; • Provide the software status for operating a vehicle if the check does not reveal a violation of the performance criterion. Computer program (1) after claim 8 , comprising: - an environment model module (11) for generating environment data as a function of the recorded data sequence; - A trajectory planning module (12) for planning a target trajectory depending on the recorded data sequence and the environment data; - a vehicle model module (13) for generating simulated odometry data and simulated vehicle dynamics data as a function of the desired trajectory; and - an evaluation module (14) for checking whether the simulated data sequence, which comprises the simulated odometry data and the simulated vehicle dynamics data, violates the performance criterion.

Images