DE102019124504A1 - Method and device for simulating and evaluating a sensor system for a vehicle as well as method and device for designing a sensor system for environment detection for a vehicle - Google Patents

Abstract

A method for simulating and evaluating a sensor system for a vehicle and a corresponding device are specified. The sensor system (10) has one or more environment sensors (1, 2, 3, n). The method comprises the steps: A1) providing a sensor model (SM) for each of the environment sensors (1-n) of the sensor system (10), which is representative of the physical properties of the respective environment sensor (1-n) when mapping a real environment of the vehicle (100) by means of real measurement by the respective environment sensor (1-n), B1) providing an environment model (UM) which is representative of a virtual environment of a virtual vehicle (100), C1) determining virtual sensor data (SD) for each of the environment sensors (1-n) of the sensor system (100) depending on the environment model (UM) and the respective sensor model (SM), D1) evaluating the virtual sensor data (SD) depending on the environment model (UM) for the design of a sensor system for the detection of the surroundings for a vehicle and a corresponding device.

Description

The invention relates to a method for simulating and evaluating a sensor system for a vehicle and a corresponding device. The invention also relates to a method for designing a sensor system for detecting surroundings for a vehicle and a corresponding device. Finally, a computer program and a computer-readable storage medium are specified.

With the increasing degree of automation of (motor) vehicles, the requirements for sensor systems in vehicles for detecting the surroundings also grow. Such sensor systems include one or more environment sensors in order to detect obstacles in the environment of the respective vehicle or the environment itself at an early stage, safely and precisely. Numerous such environmental sensors already exist on the market and the underlying sensor technologies are constantly being further developed. A particular challenge in this context is to evaluate the interaction of individual environment sensors in a sensor system without having to undertake complex test drives.

One object on which the invention is based is to create a method and a corresponding device that enables a uniform, cost-effective evaluation of various sensor systems. In particular, the aim is to help configure the sensor system in such a way that safe operation of the vehicle is guaranteed.

The object is achieved by the independent patent claims. Advantageous refinements are characterized in the subclaims.

According to a first aspect, the invention relates to a method for simulating a sensor system for a vehicle. The sensor system has one or more environment sensors. The environment sensors are, for example, a camera and / or a radar and / or a lidar and / or an ultrasonic sensor.

• In einem Schritt A1) wird jeweils ein Sensormodell für jeden der Umfeldsensoren des Sensorsystems bereitgestellt. Das Sensormodell ist jeweils repräsentativ für physikalische Eigenschaften des jeweiligen Umfeldsensors bei Abbildung eines realen Umfelds des Fahrzeugs mittels realer Messung durch den jeweiligen Umfeldsensor. In anderen Worten beschreibt das jeweilige Sensormodell eine Datenstruktur, die eine Übersetzung von einer realen Gegebenheit auf eine Messung der realen Gegebenheit durch den jeweiligen Umfeldsensors auf Basis physikalischer Eigenschaften und/oder vorgegebenen Testmessungen des jeweiligen Umfeldsensors modelliert.

The process consists of the following steps:

• In a step A1), a sensor model is provided for each of the environment sensors of the sensor system. The sensor model is in each case representative of the physical properties of the respective environment sensor when mapping a real environment of the vehicle by means of real measurement by the respective environment sensor. In other words, the respective sensor model describes a data structure that models a translation from a real situation to a measurement of the real situation by the respective environment sensor on the basis of physical properties and / or predetermined test measurements of the respective environment sensor.

In a step B1) an environment model is provided. The environment model is representative of a virtual environment of a virtual vehicle. The environment model particularly denotes a data structure that models a largely exact replica of a real environment of a real vehicle. For example, the real environment can be measured to generate such an environment model and constructed as a virtual polygon model. However, any environment with various polygon elements can also be virtually designed and generated. Polygon models of the environment offer the advantage that they can provide the sensor models with important physical information (shape contour, surface properties such as reflection / absorption). Alternatively, the environment model can also be implemented in a simplified manner in the form of object lists, which enables simulation on a "high-level" level.

In a step C1), virtual sensor data are determined for each of the environment sensors of the sensor system as a function of the environment model and the respective sensor model. In other words, in step C1) virtual measurements are carried out in the virtual environment represented by the environment model. For this purpose, points of intersection of a number of beams can be determined starting from the respective environment sensor with virtual objects of the environment model. In the case of a “high-level” simulation, the virtual sensor data can alternatively also be implemented in the form of object lists. The virtual sensor data represent, in particular, a virtual image of the virtual environment by the respective environment sensor.

In a step D1), the virtual sensor data are evaluated as a function of the environment model. Such an evaluation is referred to here and below as a comparison of the virtual environment detected by the environment sensors and the virtual environment represented by the environment model. Depending on the comparison, one or more evaluation key figures (so-called “key performance indicators”, KPIs) can be assigned, with the virtual environment recorded by the environment sensors ideally matching the virtual environment represented by the environment model.

In an advantageous manner, the sensor system can be uniformly evaluated in an early development phase without having to carry out complex test drives, which is particularly important enables an efficient comparison of several sensor systems. The method according to the invention contributes to establishing an optimal sensor system for a respective vehicle type on the basis of cost-benefit analyzes, even with a high level of complexity with regard to a number of conceivable different sensor systems. This makes it easier for automobile manufacturers in particular to define suitable sensor systems that take into account or meet all (safety) requirements. In addition to the performance of the individual environment sensors and a preferably complete field of vision coverage around the vehicle (i.e. 360 ° around the vehicle in the near and far range), other factors such as total costs of the respective sensor system and integration of the individual environment sensors or the entire sensor system in the vehicle, in particular Arrangement and alignment of the individual environment sensors are taken into account with regard to a design of the vehicle. In addition, a redundant structure of the sensor system and disturbing environmental influences such as weather and light conditions as well as pollution can be taken into account. In summary, this can also help to configure the sensor system in such a way that safe operation of the vehicle is guaranteed.

In an advantageous embodiment according to the first aspect, in step A1) configuration parameters are provided for the respective environment sensor, which are representative of a configuration of the respective environment sensor. The configuration parameters can in particular include one or more of the following parameters: a position of the environment sensor in or on the vehicle, an alignment of the environment sensor with respect to a predefined reference coordinate system of the vehicle.

In addition, in step C1) the virtual sensor data of the respective environment sensor are determined as a function of the configuration parameters. In this way, for example, different installation positions of different environment sensors can advantageously be taken into account.

In an advantageous embodiment according to the first aspect, at least one important area around the virtual vehicle is specified. Here and in the following, a so-called “region of interest”, ROI, is referred to as an important area. The important area describes, for example, an area around the vehicle that should preferably be monitored by the sensor system. Such areas can arise, for example, from regulations and boundary conditions in this context. For example, a minimum curve radius of a road section can be regulated by law at a given maximum speed, so that a necessary visual range of the course of the road in the direction of travel and at an angle to the direction of travel can be determined geometrically.

In addition, in step D1) the virtual sensor data are weighted with regard to the important area.

Individual important areas can differ in their weighting, for example depending on the probability of an accident and the extent to be expected in the event of an accident. The weighting can include a relative speed of the vehicle to other road users and / or a distance between the vehicle and other road users and / or a type of other road user and an associated criticality in the event of an accident.

In an advantageous embodiment according to the first aspect, a sensor data fusion is determined in step D1) as a function of the virtual sensor data of all environment sensors. The sensor data fusion is evaluated depending on the environment model. In particular, the sensor data fusion is determined after step C1) and before the environment model is evaluated in step D1). In other words, when determining the sensor data fusion, the information from all sensors is combined and an image of the environment model is generated from the perspective of the entire sensor setup. When performing such a data fusion, algorithms (typically Kalman filters) can be used, which take into account the performance of the respective sensors when creating this overall image. For example, a sensor with lower accuracy is assigned less weighting than higher-performance sensors.

In an advantageous embodiment according to the first aspect, the environment model includes environment data that are representative of a large number of three-dimensional coordinates of virtual object boundaries in the virtual environment of the virtual vehicle. The environment data can also be referred to as “Ground Truth” data (“GT” for short).

The virtual sensor data of a respective environment sensor are each representative of a large number of three-dimensional coordinates of a virtual measurement of the respective environment sensor in the virtual environment of the virtual vehicle.

The determined sensor data fusion includes fusion data that are representative of a large number of three-dimensional coordinates of virtual object boundaries in the virtual environment of the virtual vehicle. In other words, the merger data represent results of a merger that has been carried out Sensor data and can also be referred to as "tracks".

In step D1), the merged data with the surrounding data are evaluated by means of evaluation indicators. In other words, in step D1), an interaction of all environmental sensors (which contribute to the sensor data fusion) is evaluated. In particular, one or more metrics that relate the merged data to the surrounding data come into question as evaluation indicators.

• - Hausdorff,
• - optimal subpattern assignment, OSPA,
• - Positions-/Geschwindigkeits-/Beschleunigungsfehler,
• - normalized estimation error squared, NEES,
• - multiple object tracker accuracy, MOTA,
• - multiple object tracker performance, MOTP,
• - Detektionsrate, oder
• - Falsch-Alarm-Rate.

In an advantageous embodiment according to the first aspect, the merged data are evaluated in step D1) with the environment data using one or a combination of several of the following statistical evaluation indicators:

• - Hausdorff,
• - optimal subpattern assignment, OSPA,
• - position / speed / acceleration errors,
• - normalized estimation error squared, NEES,
• - multiple object tracker accuracy, MOTA,
• - multiple object tracker performance, MOTP,
• - detection rate, or
• - False alarm rate.

As an alternative to the aforementioned metrics, any other metrics, in particular other statistical evaluation indicators, can also be used.

The Hausdorff metric evaluates the greatest distance between a track and an associated GT value (outlier).

In the OSPA metric, a global optimum with regard to correspondence (subpattern) is sought between GT and tracks and the averaged distance between tracks and associated GT values is determined. If there are more GT values than tracks (or vice versa), this flows negatively into the OSPA metric so that false positives or false negatives can be taken into account.

For the position / speed / acceleration error, a distance between tracks and GT values (in meters) is determined so that the corresponding error indicates the accuracy of the sensor data.

With the NEES metric, a distance between tracks and GT values is determined in units of uncertainty. The NEES metric can also be an indicator of whether an uncertainty such as a measurement error in an environment sensor or process noise in the sensor model was assumed to be too large or too small.

With the MOTA metric, deviations between tracks and the associated GT values are added up and divided by the number of all assignments (averaged errors).

With the MOTP metric, false positives, false negatives and false assignments are added up and divided by a number of objects in the virtual environment of the vehicle.

In the detection rate, a number of tracks is divided by a number of GTs within the cones of vision of the environment sensors.

With the false alarm rate, a number of tracks that could not be assigned a GT value is determined divided by the total number of assignments.

• - Objekt-Detektionszeit,
• - Erfassungsrate,
• - Sensor-aufgelöste Erfassungsrate.

In an advantageous embodiment according to the first aspect, the merged data are evaluated in step D1) with the environment data on the basis of one or a combination of several of the following global assessment indicators:

• - object detection time,
• - acquisition rate,
• - Sensor-resolved acquisition rate.

As an alternative to the aforementioned metrics, any other metrics, in particular other global evaluation indicators, can also be used.

The global evaluation indicators are used in particular to differentiate between different sensor systems.

The object detection time determines when an object was recognized by the respective sensor system. In particular, it can be checked in this context when a track was created. In addition, the distance to the object is determined. It is also determined whether a braking distance is higher than the determined distance to the object, especially at a predetermined speed, so that an accident would occur or whether there is enough time to be able to prevent an accident by braking.

In the detection rate, a number of tracks divided by a number of all GTs is determined, in particular regardless of whether or not the GT values are within a cone of vision of one of the surroundings sensors.

With the sensor-resolved detection rate, an overview is determined of which environmental sensor contributed to which track in order to highlight redundancy in the sensor system.

According to a second aspect, the invention relates to a method for designing a sensor system for detecting surroundings for a vehicle. All of the features described in connection with the first aspect also apply to the second aspect and vice versa.

The method according to the second aspect comprises the following steps: In a step A2) a multiplicity of surroundings sensors are provided. Furthermore, a sensor model is generated for each of the environment sensors. The sensor model is in each case representative of the physical properties of the respective environment sensor when mapping a real environment of the vehicle by means of real measurement by the respective environment sensor. The multitude of environment sensors has, in particular, several different types of environment sensors. The sensor model is generated using measurement protocols, for example, in the case of prototypes as part of so-called “proof-of-concepts”, POCs. In particular, so-called “corner cases” can be checked in the measurement protocols in order to systematically test key criteria of the respective environment sensor. Alternatively or additionally, a data sheet from the manufacturer or an already existing sensor model can be used when generating the sensor model.

At least one environment model is generated in a step B2). The respective environment model is representative of a virtual environment of a virtual vehicle. When generating the respective environment model, a scenario in particular can be mapped using scenario parameters that specifically addresses relevant key criteria of the sensor system (e.g. corner cases). A scenario can relate to the timely detection of the end of a traffic jam or an obstacle in a curve.

In a step C2), several different virtual sensor systems are generated. A virtual sensor system has one or more environment sensors of the multiplicity of environment sensors. In other words, a subset is selected from the plurality of environment sensors provided, some of which have different types of environment sensors, in order to be able to evaluate the interaction of these environment sensors.

In a step D2), a simulation according to the first aspect of one of the several different virtual sensor systems is carried out depending on the respective sensor model of the environment sensors of the corresponding sensor system and the at least one environment model. In other words, in step A1) the sensor model generated for each of the environment sensors of the sensor system to be simulated is provided, in step B1) one of the environment models is provided and the evaluation is determined in step D1).

In a step E2), an evaluation of the respective sensor system is output or stored as a function of the simulation.

In an advantageous manner, the several different virtual sensor systems can thus be assessed in a uniform, reproducible evaluation process that delivers objective and comparable results, for example in the form of KPIs. The method contributes to a time-saving and cost-effective solution for assessing different sensor system variants for a preselection or a determination of a sensor system to be finally installed in the vehicle. In this case, an analysis with regard to various installation positions, combinations and sensor technologies of the various environment sensors can advantageously be implemented, in particular with regard to the interactions in the interaction of various environment sensors.

In an advantageous embodiment according to the second aspect, a selection of the at least one environment sensor is made depending on the evaluation of the respective sensor system. Alternatively or additionally, depending on the evaluation of the respective sensor system, the environment sensor is placed in the vehicle. Alternatively or additionally, depending on the evaluation of the respective sensor system, the surroundings sensor is aligned with respect to the vehicle. In other words, the simulation result can be used specifically to integrate the environment sensors of the real sensor system in the real vehicle in the best possible way in order to contribute to the safe operation of the vehicle.

In an advantageous embodiment according to the second aspect, the respective sensor model is determined in step A2) as a function of a predetermined set of sensor parameters that are representative of the physical properties of the respective environment sensor when mapping a real environment of the vehicle by means of real measurements by the respective environment sensor . The sensor parameters can in particular include one or more of the following parameters: a measurement accuracy of the environment sensor, a detection probability of the environment sensor, a range of the environment sensor, a cone of vision of the environment sensor (so-called “field of view”, FOV). The sensor parameters correspond, for example, to findings from test measurements and / or data sheet information. The sensor parameters are variable and are set in particular in such a way that they represent the environment sensor to be viewed as well as possible and thus form the set of sensor parameters specified for the corresponding environment sensor. To set the sensor parameters, information can be taken from sensor data sheets and measurement protocols or derived from test measurements with the individual environment sensor. Alternatively, it is possible to use a sensor model provided by the manufacturer of the environment sensor, which represents the corresponding environment sensor with a high degree of correspondence. If such a sensor model from the respective sensor manufacturer is used, then there is no need to set the sensor parameters. If sensor models are not available from the sensor manufacturer at the time of the sensor setup design, custom sensor models can be created by setting the sensor parameters and new sensor properties can be tested that are not yet implemented in available sensors in order to derive requirements for future sensor technologies.

Following step E2), the method is iteratively continued in step A2). Depending on the evaluation, at least one of the sensor parameters and / or one of the configuration parameters is or are varied. The process is iterated until the evaluation of the respective sensor system reaches a high point or all sensor parameters of the set have been varied. Such an iterative process is also referred to below as “closed-loop simulation”. By feedback from the evaluation, in particular KPIs, the influencing variables on which the sensor model or the virtual sensor data and thus also the corresponding sensor system are based can be varied and optimized. For example, the specified set of sensor parameters can be changed in each iteration until all possible combinations have been varied. Alternatively, modeling strategies from the so-called “Design of Experiments”, DoE, can also be used. As soon as it becomes apparent that the variation of an influencing variable after several runs does not lead to a change in the evaluation, the next influencing variable is switched on, so that only relevant influencing variables have to be varied in the direction of the desired optimum.

In an advantageous embodiment according to the second aspect, the multiple different virtual sensor systems generated in step C2) serve as the initial population of a genetic algorithm. The several different virtual sensor systems each consist of one or more configuration areas. Here and in the following, a configuration area is understood to be an exchangeable part of a virtual sensor system which can be replaced by a part of another virtual sensor system. By way of example, such a part comprises one or more specific environment sensors together with the corresponding configuration such as position, orientation and type.

The genetic algorithm comprises the following steps: In a step A3), a cost parameter of the respective sensor system is determined for each of the several different virtual sensor systems depending on a number of environment sensors of the respective sensor system and a respective sensor type of the environment sensors of the respective sensor system. In addition to the acquisition or production costs of the respective environment sensor, an expense associated with installing the corresponding environment sensor in the vehicle can also be considered as a cost parameter.

In a step B3), a utility parameter of the respective sensor system is determined as a function of the evaluation of the respective sensor system. The benefit value can be derived from the KPIs, for example. In addition, the aforementioned important areas can be taken into account, for example.

In a step C3), a ratio is determined from the utility parameter to the cost parameter and a check is made to determine whether the ratio exceeds a predetermined threshold value. In the event that the predefined threshold value is exceeded, the corresponding sensor system is output or stored.

Otherwise, in a step D3), different configuration areas of at least two of the sensor systems are selected depending on the respective ratio. This can also be referred to as the “select” step of the genetic algorithm. In particular, the configuration areas of the two to three sensor systems with the best determined ratio are selected here.

In a step E3), the selected configuration areas are randomly combined to form a plurality of virtual sensor systems as a new population of the genetic algorithm, and the genetic algorithm is iteratively continued with the new population in step A3). This can also be referred to as the “crossover” step of the genetic algorithm. In step A3), the newly created sensor systems can in turn be assigned cost parameters until the determined ratio of one of the sensor systems in step C3) exceeds the predefined threshold value. In this way, an optimization of the influencing variables of the environment sensors with regard to the important areas can advantageously be implemented and, in particular, analyzed, such as the Costs of a sensor system are in relation to its benefits.

In an advantageous embodiment according to the second aspect, after step D3), in a step F3), a random variation of at least one of the selected configuration areas is carried out as a mutation in the genetic algorithm. In this way, the diversity of the sensor systems produced can advantageously be increased.

In an advantageous embodiment according to the second aspect, the at least one environment model is determined in step B2) as a function of a predetermined set of scenario parameters. The scenario parameters are representative of a predefined driving maneuver of the virtual vehicle. Scenario parameters include, for example, a speed of the virtual vehicle or a curve radius of the virtually traveled route.

The method is iteratively continued after step E2) in step B2). Depending on the evaluation, at least one of the scenario parameters is varied. The method is iterated until the evaluation of the respective sensor system reaches a low point or all scenario parameters of the set have been varied. As already explained, such an iterative process is also referred to as “closed-loop simulation”. By feedback from the evaluation, in particular KPIs, the influencing variables on which the environment model is based and thus also the corresponding sensor system can be varied and optimized. For example, the specified set of scenario parameters can be changed in each iteration until all possible combinations have been varied. Alternatively, modeling strategies from the so-called “Design of Experiments”, DoE, can also be used. As soon as it becomes apparent that the variation of an influencing variable after several runs does not lead to a change in the evaluation, the next influencing variable is switched on, so that only relevant influencing variables have to be varied in the direction of the desired optimum. By varying the scenario parameters until the evaluation, in particular the corresponding KPIs, reaches or reaches a low point, a negative optimization can be implemented through which a worst-case scenario including the performance of the respective sensor system can be determined automatically.

According to a third aspect, the invention relates to a device for simulating a sensor system for a vehicle. The sensor system has one or more environment sensors. The device is set up to carry out the method according to the first aspect. The device is in particular a processing unit such as a computer.

According to a fourth aspect, the invention relates to a device for designing a sensor system for detecting surroundings for a vehicle. The device is set up to carry out the method according to the second aspect. The device is in particular a processing unit such as a computer.

According to a fifth aspect, the invention relates to a computer program comprising instructions which, when the computer program is executed by a computer, cause the computer to execute the method according to the first or second aspect.

According to a sixth aspect, the invention relates to a computer-readable storage medium on which the computer program according to the fifth aspect is stored.

Embodiments of the invention are explained in more detail below with reference to the schematic drawings.

• 1 ein Fahrzeug mit einem Sensorsystem zur Umfelddetektion,
• 2 ein Schema zur Bewertung eines Sensorsystems,
• 3 eine Datenverarbeitungskette zur Umfelddetektion im automatisierten Fahrzeug,
• 4 ein weiteres Schema zur Bewertung eines Sensorsystems,
• 5 eine schematische Darstellung eines Verfahrens zur Simulation eines Sensorsystems,
• 6 beispielhafter Vergleich statistischer und globaler Bewertungskennzahlen,
• 7 beispielhafte Illustration globaler Bewertungskennzahlen,
• 8 eine schematische Darstellung eines weiteren Verfahrens zur Simulation eines Sensorsystems,
• 9 beispielhaftes Schema zur Optimierung eines Sensorsystems mittels Rückkopplung,
• 10 eine beispielhafte gewichtete Evaluierung anhand wichtiger Bereiche,
• 11 ein Ablaufdiagramm eines beispielhaften Verfahrens zur Erzeugung eines Umfeldmodells,
• 12 ein Ablaufdiagramm eines beispielhaften Verfahrens zur Simulation eines Sensorsystems für ein Fahrzeug, und
• 13 ein Ablaufdiagramm eines beispielhaften Verfahrens zum Entwurf eines Sensorsystems zur Umfelddetektion für ein Fahrzeug.

Show it:

• 1 a vehicle with a sensor system for detecting the surroundings,
• 2 a scheme for evaluating a sensor system,
• 3 a data processing chain for the detection of the surroundings in the automated vehicle,
• 4th another scheme for evaluating a sensor system,
• 5 a schematic representation of a method for simulating a sensor system,
• 6th exemplary comparison of statistical and global valuation indicators,
• 7th exemplary illustration of global valuation indicators,
• 8th a schematic representation of a further method for simulating a sensor system,
• 9 exemplary scheme for optimizing a sensor system by means of feedback,
• 10 an exemplary weighted evaluation based on important areas,
• 11 a flowchart of an exemplary method for generating an environment model,
• 12th a flowchart of an exemplary method for simulating a sensor system for a vehicle, and
• 13th a flowchart of an exemplary method for designing a sensor system for environment detection for a vehicle.

Elements of the same construction or function are provided with the same reference symbols in all the figures.

Based 1 is vehicle 100 with a sensor system 10 shown for environment detection. The sensor system 10 has a multiplicity of surroundings sensors 1-n, each of which is set up as objects 30th , 31 in the vicinity of the vehicle 100 capture. The sensor system 10 has a specific structure according to which a number as well as respective types and installation positions and orientations of the individual environment sensors 1-n result. This structure is also referred to as “setup” or “sensor setup” in the following. The environment sensor is an example 1 in the setup shown around a camera, the objects 30th , 31 detected in the field of view and outputs. With the objects 30th , 31 In particular, it concerns (dynamic) objects like other road users. With the environment sensor 2 it is, for example, a radar sensor. The environment sensor 3 is designed as a lidar sensor, for example. With the environment sensor n it is an example of an ultrasonic sensor. The sensor system 10 can be a control unit 11 be assigned, the signal technology coupled with the environment sensors 1-n and for sensor data fusion SF is set up.

With the increasing degree of automation of vehicles 100 The requirements placed on the environment sensors 1-n for the environment of the vehicle also increase 100 to be recognized early, safely and precisely. When developing future vehicle concepts, the configuration of the sensor setup therefore plays a central role. There are numerous environmental sensors 1-n on the market and new sensor technologies are constantly being added. Automobile manufacturers, however, are faced with the challenge of defining a suitable sensor arrangement that meets all (safety) requirements: In addition to individual sensor performance and field of view coverage (360 °, near and far range), other factors such as total setup costs, vehicle integration and design should be considered must be taken into account. In addition, the sensor setup should be redundant and disruptive environmental influences (weather and light conditions, pollution) should be taken into account. This results in a high level of complexity in terms of possible sensor setup configurations. So far, there has been no standardized evaluation method to uniformly assess different setup variants and to determine an optimal sensor setup for a respective vehicle type on the basis of cost-benefit analyzes. At an early stage of development, it is practically impossible to examine every conceivable sensor combination with test drives.

In 2 a sensor setup evaluation is outlined. The evaluation of the individual sensors 1-n is based on the one hand on that of the respective sensor manufacturers H specifications provided in the data sheet D. and optionally provided measurement protocols M. This information about the sensor 1-n represents a first reference point for making a preselection. However, since the information was obtained under different and ideal measurement conditions depending on the manufacturer, it is also necessary for the automobile manufacturers to check these values uniformly O common. For this purpose, the respective sensor 1-n in its current state of development 1 * -n * (partly prototype status) is loaned for test purposes, mostly in the context of proof-of-concepts (POCs), and its own measurement protocols M * made to check and supplement the information in the data sheet. A list of “corner cases” can be used to systematically test the key criteria. Based on the knowledge gained, the sensors 1-n, 1 * -n * can be characterized (characterization C. of the individual sensor 1-n, 1 * -n *) and evaluated (sensor evaluation B. ) will.

The sensor behavior SV in the network as well as the overall setup performance are determined by test drives T determined. Typically, the test drives take place T However, at a later point in time during vehicle development (for the purpose of fine adjustment / optimization, when the final setup has already been determined), as the overall functional chain (cf. 3 - Environment sensors 1-n detect the environment of the vehicle 100 . After a preprocessing step P A sensor data fusion takes place from the environment data output by the environment sensors 1-n SF , based on an environment model U can be determined that is representative of objects in the vicinity of the vehicle 100 from the point of view of the sensor system 10 . The environment model U can then use FAS functions F. are provided to implement the function actuators A. of the vehicle 100 control) is required. Since the development of the detection function and fusion algorithms are based on the respective sensor setup, these are not yet available at the time of sensor selection and setup design. Due to the high time and cost factor of test drives and their data evaluation as well as the lack of the required overall functional chain at the time of the early development stage, test drives T cannot be used to evaluate any number of different setup variants. Test drives can only represent an extension of the corner case list in the context of the individual sensor evaluation to include the better assess the dynamic behavior of the sensor.

When evaluating the sensor B. and sensor selection can be a benchmark of individual sensors of the same type (via a characterization C. of the individual sensor), a benchmark of various sensor technologies and sensor setup configurations (installation position), a cost-benefit analysis, an evaluation of a sensor performance in the data processing chain or in the overall system as well as a selection based on redundancy.

There is currently no methodical procedure for evaluating various sensor setup configurations at the time the vehicle concept is being drawn up. Up until now, the setups for the driver assistance systems (ADAS) were, for example, a camera, a radar and four ultrasonic sensors at the front and rear of the vehicle 10 still manageable. When designing the setup, cones of vision of the sensors (Field of Views, FOVs) are sketched in side views and the top view of the vehicle silhouette and arranged and aligned in such a way that the field of view overlooked by the ADAS is covered as well as possible. However, as the degree of automation increases, so do the setup requirements, as there are significantly larger areas around the vehicle 100 must be recorded. A vehicle sketch with drawn-in FOVs can quickly become confusing and lose its meaningfulness with regard to the setup evaluation. Also a 3D view of the FOVs around the vehicle 100 within a computer-aided design (CAD) environment (such as CATIA) only provides a visual impression. In order to evaluate and compare different setup variants, a simulation-based methodology is required which, based on evaluation criteria and key performance indicators (KPIs), allows both qualitative and quantitative statements to be made. However, there is currently no such simulation infrastructure. Existing simulation tools address the development and optimization of detection and fusion algorithms as well as the ADAS, but not the evaluation of the sensor setup configuration.

One goal is therefore to create a consistent and objective evaluation and selection process for sensor setups. The high complexity resulting from increasing (safety) requirements for the setup, the multitude of possible setup variations and the diversity for different vehicle types / derivatives must be countered. The aim is to create an alternative to setup considerations based on sketches / 3D views of the sensor viewing areas, which become confusing as the number of sensors increases and do not allow a meaningful evaluation with regard to the detection performance. Such setup considerations discuss the sensor arrangement and integration on the respective vehicle type without quantitative assessment and comparability of various possible solutions. Even just looking at the vehicle 100 should be avoided without taking into account the surrounding situations / driving scenarios. In particular, time-consuming and costly test drives and subsequent data preparation / evaluation are to be reduced, also in view of the fact that recognition and fusion algorithms are already required for data preparation in order to evaluate the setup performance in the context of test drives, which were still required at the time of the setup design does not exist. Similarly, new types of sensors are often not (meaningfully) ready for test drives (prototypes) at an early stage of development. In addition, it is desirable to take into account some important driving scenarios and driving maneuvers that cannot be reproduced by test drives, for example dangerous situations. The risk that some setup configurations are not investigated in more detail due to the time and expense required for test drives should be reduced. Ultimately, it is important to create a simulation infrastructure for sensor setup evaluation that addresses the sensor setup design instead of just developing and optimizing the subsequent data processing.

A three-stage process is therefore proposed for the design and evaluation of sensor setups and new sensor technologies (see 4th ). Between the evaluation of the individual sensors as part of the benchmarking (stage I) and test drives to evaluate the setup performance in the overall system under real conditions (stage III), a simulation-based evaluation of the sensor setup should take place as an intermediate step (stage II). Steps I and III of this three-step process correspond to the explanations above 2 and 3 . In the step I. sensor specifications are to be identified. The performance of an individual sensor is determined using data sheets, measurement protocols and corner case tests. Intermediate level II represents a new development tool with the help of which different setup variants / sensor combinations can be configured, tested and evaluated in a time- and cost-efficient manner in order to describe the sensor setup performance at this early point in time in a qualitative and quantitative manner. The sensor setup is to be simulated, with a configuration of the sensor setup and a sensor parameter setting based on information obtained from step I. , a simulation of the setup performance and an analysis of cause and effect relationships can be carried out. Finally, in step III, test drives can be carried out in order to validate the sensor setup performance in the overall system under real ambient conditions. This preferably only takes place for the setup variants that have shown promising results within the simulation in step II. Step III serves primarily to optimize and fine-tune the target setup.

The one in the crotch I. Sensor properties determined on the basis of data sheets, measurement protocols and corner case tests serve as input for the parameter settings of generic sensor models / virtual sensors of the simulation infrastructure in the step II . In this concept phase, there are typically no sensor models from the sensor manufacturer that could be incorporated into the simulation workflow. Therefore, the use of generic sensor models and the filling of setting parameters based on the findings from test measurements / data sheet information is indicated. Various sensor setups can be designed within the simulation environment and the respective sensor properties can be set individually. The respective sensor setup performance can then be determined for any virtual test drive and clearly displayed in the form of evaluation indicators. In this way, a fast, systematic and objective analysis of the different setup variants is possible, which provides a sound basis for making decisions about the final setup. In step III Only the few setups that have shown promising results in the simulation or only the setup finally determined on this basis are evaluated in real test drives in order to record the setup performance in the overall vehicle system and under real ambient conditions.

Based on 5 and 8th a detailed simulation process is outlined in each case, which covers the entire process chain from data acquisition (ground truth and sensor data), data processing (data fusion), data evaluation (metrics) to data analysis (KPIs, cost-benefit analysis) to the derivation of knowledge ( Setup performance).

It is provided that the following simulation workflow runs on a “high level”, ie the ground truth data UD and the simulated sensor data SD are available in the form of object lists. In principle, it would also be conceivable to set up the simulation workflow with physically based sensor models and environment simulation. However, for the creation and parameterization of physical sensor models, there is usually a lack of in-depth knowledge about the functioning of the sensors 1-n to be evaluated, which the sensor manufacturers H is withheld. Furthermore, there is a lack of recognition functions for the performance evaluation that provide usable data for the fusion from the “raw sensor data” (classified objects, as in the object list format). Finally, there is also a lack of an established environment simulation tool that delivers the necessary parameters such as material properties in sufficient quality.

The simulation can be used in two different modes, an open-loop mode and a closed-loop mode. In a first variant, the simulation is operated in open-loop mode ( 5 ).

The simulation workflow is divided into two parts: In the first part, an environment simulation US virtual driving scenarios SZ (Vehicle environment, course of the road, road users and the driving maneuver of the ego vehicle) designed (driving scenarios generation SG ) on which the setup evaluation should be based. Contrary to the procedure for the functional validation / validation of a complete vehicle shortly before the start of production, in which several hundreds or thousands of different scenarios and millions of kilometers are simulated, a limited number of selected scenarios should be used in the evaluation of the setup configuration in the early concept phase which specifically address the points relevant to the setup assessment (e.g. corner cases).

The virtual driving scenarios generated in the first part SZ serve as the ground truth of an environment model AROUND with environment data UD as ground truth data. These simulated ground truth data of the scenarios are transferred to the second part of the simulation workflow. The second part addresses the sensor simulation SS and the analysis part AP . The second part runs as an example within the integrated development environment Microsoft Visual Studio. The incoming ground truth data UD from the previous environment simulation US can either in live mode L. processed or in a recording mode R. recorded and retrieved later. The latter enables the persistent storage of many scenarios in a scenario catalog K . The saved scenarios can then be played back for the setup evaluation; the first part of the workflow (virtual environment simulation US ) is then no longer necessary for working with the sensor configuration and evaluation simulation in open-loop mode. The second part is thus an independent framework that leads to a flexible work tool.

• Ein erstes Modul betrifft die Simulation der Sensordaten SD aller Sensoren 1-n eines Setups 10: Hier wird das virtuelle Sensorsetup konfiguriert und Sensormodelle SM der einzelnen Sensoren 1-n erstellt. Über eine grafische Benutzeroberfläche (GUI) können zudem Eigenschaften SE der einzelnen Sensoren des Setups (entsprechend der gewonnenen Informationen aus Schritt I des Drei-Stufen-Modells in 4) eingestellt werden. Insbesondere können hierbei sowohl die im allgemeinen Teil der Beschreibung genannten Sensorparameter als auch die im allgemeinen Teil der Beschreibung genannten Konfigurationsparameter als Eigenschaften SE variiert werden. Bei den Sensormodellen SM handelt es sich um generische probabilistische Sensormodelle, die für jeden virtuellen Sensor des Setups entsprechend der jeweiligen sensorspezifischen Werte SK (u.a. Sensorposition am Fahrzeug, Messgenauigkeit, Detektionswahrscheinlichkeit) eingestellt werden, um den realen Sensor 1-n möglichst präzise in der Simulation zu repräsentieren. Bei Bedarf ist es jedoch auch möglich, über ein Software Development Kit (SDK) / ProgrammCode neue Sensormodelle SM hinzuzufügen oder die bestehenden Sensormodelle SM anzupassen, um das Tool für neuartige Sensortechnologien zu erweitern. An dieser Stelle besteht die Möglichkeit, die Parameter der Sensorsetup-Konfiguration über ein Optimierungstool automatisiert zu optimieren wie anhand 8 und dem Closed-Loop Modus geschildert. Zusammengefasst werden in dem ersten Modul die Ground Truth Daten UD entsprechend den Einstellungen der probabilistischen Sensormodelle SM modifiziert und als simulierte Sensordaten SD ausgegeben.

The sensor simulation SS and analysis part AP comprises three modules:

• A first module concerns the simulation of the sensor data SD of all sensors 1-n of a setup 10: The virtual sensor setup and sensor models are configured here SM of the individual sensors 1-n created. Via a graphic User interface (GUI) can also have properties SE of the individual sensors of the setup (according to the information obtained from step I. of the three-step model in 4th ) can be set. In particular, both the sensor parameters mentioned in the general part of the description and the configuration parameters mentioned in the general part of the description can be used as properties SE can be varied. With the sensor models SM These are generic probabilistic sensor models that correspond to the respective sensor-specific values for each virtual sensor in the setup SK (including sensor position on the vehicle, measurement accuracy, detection probability) can be set in order to represent the real sensor 1-n as precisely as possible in the simulation. If necessary, however, it is also possible to use a software development kit (SDK) / program code to add new sensor models SM or add the existing sensor models SM adapt to expand the tool for novel sensor technologies. At this point there is the option of automatically optimizing the parameters of the sensor setup configuration using an optimization tool, such as using 8th and the closed-loop mode. The ground truth data are summarized in the first module UD according to the settings of the probabilistic sensor models SM modified and as simulated sensor data SD issued.

The simulated sensor data are used in a second module SD in a data fusion module SF , which is based on extended Kalman filters, for example, and merged into the subsequent environment model U fed. The data fusion or the underlying algorithms such as the Kalman filters, when merging the various sensor information, can give greater weight to those sensors which also have a higher measurement accuracy. This information about the respective sensor measurement accuracy can be taken from the settings in the sensor models.

The simulated and merged sensor data are then processed in a third module FD (Tracks) with the Ground Truth (GT) data UD compared on the basis of various metrics (evaluation E). This part gives a result report for the tested setup P that lists the KPIs for each scenario.

1. 1. Statistische KPIs, beispielsweise die folgenden Metriken:
   • - Hausdorff Metrik: größter Abstand zwischen einem Track und dem zugehörigen GT-Wert (Ausreißer).
   • - Optimal SubPattern Assignment (OSPA): Zwischen GT und Tracks wird ein globales Optimum hinsichtlich Übereinstimmung (subpattern) gesucht und der gemittelte Abstand zwischen Tracks und den zugehörigen GT-Werten ermittelt. Gibt es mehr GT-Werte als Tracks (oder umgekehrt), fließt dies negativ in die Metrik ein, um False-Positives / False Negatives zu berücksichtigen.
   • - Position/Velocity/Acceleration Error: Abstand zwischen Tracks und GT-Werten (in Metern). Gibt die Genauigkeit der Sensordaten an.
   • - Normalized Estimation Error Squared (NEES): Abstand zwischen Tracks und GT-Werten in Einheiten der Unsicherheit. Diese Metrik kann auch ein Indikator dafür sein, ob die Unsicherheit (Sensor-Messfehler, Prozessrauschen) als zu groß oder zu klein angenommen wurde (im Sensormodell).
   • - Multiple Object Tracker Accuracy (MOTA): Abweichungen zwischen Tracks und zugehörigen GT-Werten aufsummiert und geteilt durch die Anzahl aller Zuordnungen (gemittelter Fehler).
   • - Multiple Object Tracker Performance (MOTP): False-Positives, False-Negatives und Falsch-Zuordnungen aufsummiert und geteilt durch die Objektanzahl.
   • - Detektionsrate: Anzahl der Tracks geteilt durch die Anzahl der GTs (innerhalb von Sensor-FOVs).
   • - Falsch-Alarm-Rate: Anzahl an Tracks, die keinem GT-Wert zugeordnet werden konnten, geteilt durch die Gesamtanzahl an Zuordnungen.
2. 2. Globale KPIs (zur Differenzierung verschiedener Setup-Konfigurationen), beispielsweise die folgenden Metriken:
   • - Objekt-Detektionszeit: Wann wurde ein Objekt vom Sensorsystem erkannt (Track angelegt); Welcher Abstand liegt hier zum Objekt vor? Ist der Bremsweg (bei gegebener Geschwindigkeit) höher als der vorhandene Abstand zum Objekt, sodass es zu einem Unfall kommen würde oder reicht die Zeit aus, um noch bremsen zu können?
   • - Erfassungsrate: Anzahl der Tracks geteilt durch die Anzahl aller GTs (unabhängig davon, ob die GT-Werte innerhalb eines Sensor-FOVs liegen oder nicht).
   • - Sensor-aufgelöste Erfassungsrate: Übersicht, welcher Sensor zu welchem Track beigetragen hat, um Redundanz aufzuzeigen.

The KPIs can be divided into two types:

1. 1. Statistical KPIs, such as the following metrics:
   • - Hausdorff metric: largest distance between a track and the associated GT value (outlier).
   • - Optimal SubPattern Assignment (OSPA): A global optimum with regard to correspondence (subpattern) is sought between GT and tracks and the averaged distance between tracks and the associated GT values are determined. If there are more GT values than tracks (or vice versa), this flows negatively into the metric in order to take false positives / false negatives into account.
   • - Position / Velocity / Acceleration Error: Distance between tracks and GT values (in meters). Indicates the accuracy of the sensor data.
   • - Normalized Estimation Error Squared (NEES): Distance between tracks and GT values in units of uncertainty. This metric can also be an indicator of whether the uncertainty (sensor measurement error, process noise) was assumed to be too large or too small (in the sensor model).
   • - Multiple Object Tracker Accuracy (MOTA): Deviations between tracks and associated GT values summed up and divided by the number of all assignments (averaged errors).
   • - Multiple Object Tracker Performance (MOTP): false positives, false negatives and false assignments summed up and divided by the number of objects.
   • - Detection rate: number of tracks divided by the number of GTs (within sensor FOVs).
   • - False alarm rate: Number of tracks that could not be assigned a GT value, divided by the total number of assignments.
2. 2. Global KPIs (to differentiate between different setup configurations), for example the following metrics:
   • - Object detection time: When was an object recognized by the sensor system (track created); What is the distance to the object here? Is the braking distance (at a given speed) greater than the existing distance to the object, so that an accident would occur, or is there enough time to be able to brake?
   • - Acquisition rate: number of tracks divided by the number of all GTs (regardless of whether the GT values are within a sensor FOV or not).
   • - Sensor-resolved acquisition rate: overview of which sensor contributed to which track in order to show redundancy.

Based on these results, the performance of the tested sensor setups can be assessed quantitatively and objectively. The KPI reports P and then possible cost-benefit analyzes K / N offer a well-founded decision-making basis for determining the optimal sensor setup, which will be carried out later by means of test drives T can be evaluated and optimized in more depth.

Based on 6th and 7th Various KPIs are shown as examples for better understanding.

6th shows a first setup on the left, a second setup in the middle and on the right. If one were only to look at “statistical KPIs” that only focus on the accuracy between tracks and GT values, no statement about the overall setup performance would be possible. In the first setup, the statistical KPIs could turn out to be mediocre because some sensors are less accurate. The second setup, on the other hand, could have very good statistical KPIs due to higher quality sensors. However, the second setup only covers a small area around the vehicle 100 from. Without “global KPIs”, however, no statement can be made on this (middle). In order to take into account the "environment coverage" and thus differences between setups, global KPIs can therefore be used that take into account how many of the GT values were not recorded by the setup (right).

• Links (1.) ist die Erkennungszeit „Zuerst gesehen“ und/oder Abstandsrate „Bremsweg / Objektabstand“ verschiedener Objekte (Objekt IDs #1-#4) dargestellt. Es wird geprüft, wann ein Objekt zum ersten Mal gesehen wird, insbesondere um eine „Unfallprognose“ unter Berücksichtigung des Bremsweges in die Evaluierung einfließen lassen zu können.

7th also illustrates various global KPIs:

• On the left (1.) the detection time “first seen” and / or the distance rate “braking distance / object distance” of various objects (object IDs # 1- # 4) is shown. It is checked when an object is seen for the first time, in particular in order to be able to incorporate an "accident prognosis" into the evaluation, taking the braking distance into account.

In the middle on the left (2.) the detection periods “in total” are shown. It is checked which objects are recognized at each time step.

In the middle on the right, the acquisition periods “per sensor” are shown for a lidar (3a), a radar (3b) and a camera (3c). It is checked which objects are detected by which sensor at any point in time.

On the right (4th) the global recognition rate is shown in order to take into account coverage of the surrounding area.

In a second variant, the simulation is operated in closed-loop mode ( 8th ). This has essentially the same open loop portion OIL like that based on 5 described open-loop mode, which by a feedback CL is supplemented.

In a first exemplary embodiment according to the second variant embodiment, the feedback CL used for scenario variation.

Using an optimization tool OP For parameterization, the individual variables of a scenario (such as the speed of the ego vehicle or curve radius) can be varied in order to automatically generate several variants of a scenario. The parameterization can change the variables step by step (eg ego speed between 30 km / h and 100 km / h in steps of 10 km / h). In this way, the scenario catalog K Conveniently set up and the critical scenario variants / borderline cases are revealed in the later analysis.

It is also possible to send the KPIs directly as feedback to an optimization tool OP to be returned for the scenario creation, and thus to create a closed simulation chain (closed loop) ( 9 ). The optimization tool OP varies the scenario parameters (such as ego speed) until the KPIs have reached a low point. After generating relevant scenarios (in step SG ) becomes a scenario set SZ on the sensor simulation SS and analysis part AP which outputs KPIs accordingly. In the test step NO it is checked whether a respective scenario has become more critical. This is followed by a change step VSP the scenario parameters changed gradually, for example the parameters ego speed between 0 km / h and 100 km / h, object speed between 0 km / h and 100 km / h and the curve radius between 50 m and 100 m flow as a new sub-variant SZ * turn each scenario into the generation SG relevant scenarios SZ a. This “negative optimization” can be used to automatically determine the worst-case scenario, including the setup performance determined there.

Changing the scenario parameters within the optimization tool OP can be done through combinatorics or optimization strategies by changing one parameter in each run and leaving the other parameters unchanged until all combinations have been played through (full raster), so that a large number of scenarios Variants arise that can subsequently be classified according to their “criticality” (combinatorics). Alternatively, modeling strategies from the optimization tools from the Design of Experiments, DoE, can be used. As soon as it becomes apparent that increasing / decreasing a parameter after several runs has not contributed to an increase in the criticality of a scenario (KPI deterioration), the process moves on to the next parameter, so that not all combinations are played through, but only the relevant ones in the direction the sought-after optimum (optimization strategies).

In an optional embodiment, the so-called "Regions of Interest" ( ROI ) To be defined. The ROIs describe the areas around a vehicle 100 which should be covered by the setup being designed. The ROIs are derived from various regularities and the resulting relationships. Based 10 an exemplary scheme for deriving the ROIs is shown. First a situation or a scenario is selected ( R1 ), which is relevant for the sensors, such as early detection of the end of a traffic jam or an obstacle in a curve. Boundary conditions for urban, rural or motorway segments are then applied as parameters in order to determine ROIs ( R2 ) such as legal regulations (curve radii, maximum speed) and / or the distance and relative speed between the ego vehicle and the obstacle. In the case under consideration, the end of a traffic jam in a curve serves as parameters which minimum curve radius is legally regulated for city, country or motorway segments at a given speed and which braking distance results at the given speed. In addition, the required viewing range can be derived from geometrical considerations at the front and side (FOV) and the "ROI" can thus be defined. The ROI areas can differ in their importance (depending on the probability / frequency of an accident and the extent to be expected in the event of an accident) and can therefore be weighted with a critical factor ( R3 ), e.g. on the basis of the extent or consequences of an accident (accident statistics, safety requirements). The weighting includes, for example, the relative speed and the distance between the ego vehicle and other road users, as well as a factor that takes into account the criticality in the event of a collision (pedestrian is more critical than another vehicle). The weighted ROI * s can be displayed as a “heat map” in the form of weighted points around the ego vehicle. Using this representation, specific driving scenarios can be designed, which provide a sound basis for evaluating the sensor setup performance. In addition, the heat map can be used to evaluate the variation in the sensor model properties and setup configurations (how much was covered by the very important areas, how much was covered by the moderately important areas, and how much was the less important areas), as explained in more detail below .

In a second embodiment according to the second embodiment variant, the feedback CL used to vary the setup configuration and sensor properties.

Similar to the scenario variation according to the first exemplary embodiment, the setup configurations and the individual parameters of the sensor models can also be used SM (Range, accuracy, FOV) can be varied automatically by the KPI feedback loop and optimized in the closed-loop process. It is conceivable to carry out an optimization with regard to the KPIs or an optimization with regard to the heat map.

The optimization with regard to the KPIs can be carried out in the same way as the scenarios. The parameter set is exemplified according to the scheme 9 changed in each run until all combinations have been played through or an optimum has been found according to modeling strategies from the Design of Experiments (DoE) area. The different setup variants can then be sorted according to their performance using the KPIs and the optimal setup can be easily selected. It is important here that an “all-encompassing scenario” is created for this procedure, on which the KPIs are optimized. The underlying scenario should therefore cover all points to be considered for the setup assessment and also take into account the situation (important / frequent / more relevant situations should occur more often than less important situations).

When optimizing with regard to the heat map, a heat map can be created as described above that shows the areas around the vehicle 100 weighted. This information can help in the optimization process that more importance is given to the more important areas during setup creation and optimization. A cost-benefit analysis can be used to analyze how the costs of a setup (number of sensors and costs of the respective sensor types) are in relation to the benefits (which areas are covered by these sensors). For example, a genetic algorithm can be used for this optimization process, which is shown schematically on the basis of the 11 is shown.

With the help of a genetic algorithm, an optimal setup can be found automatically. To do this, an initial set is first created different setups defined ( C2 ), which serves as the starting point for the optimization. In addition, a cost function is defined that evaluates the setup set ( A3 , B3 ). In addition to the costs, the cost function can take into account the KPIs (feedback from the simulation workflow according to closed loop mode) and / or the heat map (weighted ROIs G) as values for optimization. It is also defined ( G3 ), from when the setup is optimal (ratio of costs to environmental coverage of important areas), so that it can be checked when the genetic algorithm can be terminated ( C3 ).

From the previous (in the first run from the initial) setup set (population), different areas are selected from the 2-3 setups that are best according to the cost function ( D3 ) ("Selection") and in the next step ( E3 ) ("Crossover") randomly reassembled to form a new setup. Some areas can also be varied randomly ( F3 ) to increase diversity (“mutation”).

The newly created setup is again evaluated using the cost function and the cycle begins again until, after several runs, the setup meets the requirements of the cost function.

Using the flowcharts of the 12th and 13th are the following program steps when simulating a sensor system 10 ( 12th ) or when designing a sensor system 10 ( 13th ) for the implementation of the above explanations. All the features disclosed in the general part of the description can be applied to the following program steps and vice versa.

The program for simulating a sensor system according to 12th is in one step A1 ) started, in each of which a sensor model SM for each of the environment sensors 1-n of the sensor system 10 provided. In a subsequent step B1 ) becomes an environmental model AROUND provided. This will be done in one step C1 ) virtual sensor data SD for each of the environment sensors 1-n of the sensor system 10 depending on the environment model AROUND and the respective sensor model SM determined. Finally, the virtual sensor data SD merged and dependent on the environment model AROUND evaluated. The program according to 12th can be especially useful when designing a sensor system 10 in the program according to 13th can be used.

The program according to 13th is in one step A2 ) started in which a sensor model SM is generated for each of a plurality of environment sensors 1-n. In a subsequent step B2 ) becomes at least one environment model AROUND generated. This will be done in one step C2 ) several different virtual sensor systems 10 generated, each having one or more of the environment sensors 1-n. Then in one step D2 ) a simulation is carried out in each case according to the program 12th each one of the sensor systems 10 is used as a basis and the respective sensor models of the environment sensors 1-n of the corresponding sensor system 10 and at least one environment model AROUND to be provided. Finally, an evaluation of the respective sensor system is output or saved as a function of the simulation.

• Es kann eine virtuelle Testsuite für die Auslegung und Bewertung von Sensorsetups für beliebige Fahrszenarien/Fahrmanöver erzielt werden. Hierbei kann ein einheitlicher, reproduzierbarer Bewertungsprozess, der objektive und vergleichbare Ergebnisse in Form von KPIs liefert, erreicht werden. Dies bietet eine zeitsparende und kostengünstige Lösung zur Beurteilung verschiedener Setup-Varianten für die Vorauswahl / Festlegung auf ein finales Setup. Hierdurch wird ein wichtiges Werkzeug in der frühen Phase der Fahrzeugkonzept-Entwicklung angegeben, das Entwickler beim Setup-Entwurf unterstützt und als fundierte Entscheidungsgrundlage dient. Eine Setup-Analyse hinsichtlich verschiedener Sensor-Einbaupositionen, Sensor-Kombinationen und verschiedener Sensoren/Sensortechnologien wird ermöglicht. Darüber hinaus bietet sich die Möglichkeit einer Performance-Analyse für zu untersuchende Setups (Open-Loop Simulation) oder eine automatisierte Setup-Optimierung für gegebene Fahrzeugmodelle (Closed-Loop Simulation). Es wird zur Aufdeckung von Wirkzusammenhängen beim Zusammenspiel verschiedener Sensoren beigetragen. Eine einfache Nutzung der Simulationsumgebung über eine nutzerfreundliche GUI zum Konfigurieren des virtuellen Sensorsetups und schnelles Anpassen/Einstellen der Sensorparameter wird ermöglicht; gleichzeitig besteht aber auch die Möglichkeit, über das Software Development Kit die Sensormodelle in ihrer Funktionsweise im Programmcode anzupassen oder neue Sensormodelle selbst zu erstellen. Dies kann entscheidend für die Evaluation zukünftiger Sensortechnologien sein, die Anpassungen der bisherigen Sensormodelle erfordern. KPIs ermöglichen eine quantitative und objektive Bewertung der verschiedenen Sensorsetups. Es kann eine übersichtliche Ergebnisübersicht über die jeweilige Setup-Performance in diversen Fahrszenarien erzielt werden. Durch eine implementierte, standardisierte OSI-Schnittstelle kann eine hohe Kompatibilität mit diversen Umgebungssimulationstools erreicht werden. Schließlich kann die Simulation als kompaktes und flexibles Simulationswerkzeug (lauffähig auf einem Notebook) implementiert werden.

The described simulation-based evaluation of sensor setup configurations for vehicle environment detection offers the following advantages:

• A virtual test suite for the design and evaluation of sensor setups for any driving scenario / driving maneuver can be achieved. A uniform, reproducible evaluation process that delivers objective and comparable results in the form of KPIs can be achieved here. This offers a time-saving and cost-effective solution for assessing different setup variants for the pre-selection / determination of a final setup. This provides an important tool in the early phase of vehicle concept development that supports developers in the setup design and serves as a well-founded basis for decision-making. A setup analysis with regard to different sensor installation positions, sensor combinations and different sensors / sensor technologies is made possible. In addition, there is the option of performing a performance analysis for setups to be examined (open-loop simulation) or automated setup optimization for given vehicle models (closed-loop simulation). It contributes to the discovery of causal relationships in the interaction of different sensors. A simple use of the simulation environment via a user-friendly GUI for configuring the virtual sensor setup and quick adjustment / setting of the sensor parameters is made possible; At the same time, however, it is also possible to use the software development kit to adapt the functionality of the sensor models in the program code or to create new sensor models yourself. This can be decisive for the evaluation of future sensor technologies that require adjustments to the previous sensor models. KPIs enable a quantitative and objective evaluation of the various sensor setups. A clear overview of the results of the respective setup performance can be achieved in various driving scenarios. An implemented, standardized OSI interface enables high compatibility with various environment simulation tools to be achieved. Finally, the simulation can be implemented as a compact and flexible simulation tool (executable on a notebook).

List of reference symbols

100

vehicle

10

Sensor system

30, 31

object

1, 1 *

Camera (prototype *)

2, 2 *

Radar (prototype *)

3, 3 *

Lidar (prototype *)

n, n *

Ultrasonic sensor (prototype *)

11

Control unit

SF

Sensor data fusion

H

Sensor creator

D.

data sheet

M, M *

Measurement protocol (automobile manufacturer *)

O

Automobile manufacturer

C.

Sensor characterization

B.

Sensor evaluation

SV

Sensor behavior in the network

T

test drive

P.

Preprocessing step

U

Environment model

F.

FAS function

A.

Actuator

I-III

steps

US

Environment simulation

SZ, SZ *

virtual driving scenarios (optimized *)

SG

Driving scenario generation

AROUND

Environment model

UD

Environment data

SS

Sensor simulation

AP

Analysis part

L.

Live mode

R.

Recording mode

K

Scenario catalog

SE

Sensor properties

SM, SM *

Sensor model (optimized *)

SK

sensor-specific values

SD

simulated sensor data

FD

Fusion data

P.

Results report

K / N

Cost-benefit analysis

OIL

Open loop portion

CL

Feedback

OP

Optimization tool

NO

Test step

VSP

Change step

R1-R3

ROI determination step

ROI, ROI *

important area (weighted *)

KPI

Valuation indicator

A1-G3

Program steps

Claims (15)

A method for simulating a sensor system (10) for a vehicle (100), the sensor system (10) having one or more environment sensors (1, 2, 3, n), and the method comprising the following steps: A1) Providing a sensor model (SM) for each of the environment sensors (1-n) of the sensor system (10), which is representative of the physical properties of the respective environment sensor (1-n) when mapping a real environment of the vehicle (100) by means of real measurement by the respective environment sensor (1-n), B1) providing an environment model (UM) which is representative of a virtual environment of a virtual vehicle (100), C1) determining virtual sensor data (SD) for each of the environment sensors (1-n) of the sensor system (100) depending on the environment model (UM) and the respective sensor model (SM), D1) Evaluation of the virtual sensor data (SD) as a function of the environment model (UM). Procedure according to Claim 1 , whereby in step A1) configuration parameters are provided for the respective environment sensor (1-n) which are representative of a configuration of the respective environment sensor (1-n), and in step C1) the virtual sensor data (SD) of the respective environment sensor (1-n) can be determined depending on the configuration parameters. Method according to one of the preceding claims, in which at least one important area (ROI *) around the virtual vehicle (100) is specified, and the virtual sensor data (SD) are weighted with respect to the important area (ROI *) in step D1). Method according to one of the preceding claims, wherein in step D1) a sensor data fusion (SF) is determined as a function of the virtual sensor data (SD) of all environment sensors (1-n) and is evaluated as a function of the environment model (UM). Procedure according to Claim 4 wherein - the environment model (UM) comprises environment data (UD) which are representative of a large number of three-dimensional coordinates of virtual object boundaries in the virtual environment of the virtual vehicle (100), - the virtual sensor data (SD) of a respective environment sensor (1-n) Representative are a large number of three-dimensional coordinates of a virtual measurement of the respective environment sensor (1-n) in the virtual environment of the virtual vehicle (100), - the sensor data fusion (SF) includes fusion data (FD) which are representative of a large number of three-dimensional coordinates of virtual object boundaries in the virtual environment of the virtual vehicle (100), and in step D1) the fusion data (FD) with the environment data (UD) are evaluated by means of evaluation indicators (KPI). Procedure according to Claim 5 , whereby in step D1) the fusion data (FD) with the environment data (UD) are evaluated on the basis of one or a combination of several of the following static evaluation indicators (KPI): - Hausdorff, - optimal subpattern assignment, OSPA, - position / speed - / acceleration error, - normalized estimation error squared, NEES, - multiple object tracker accuracy, MOTA, - multiple object tracker performance, MOTP, - detection rate, or - false alarm rate. Procedure according to Claim 5 or 6th , wherein in step D1) the fusion data (FD) with the environment data (UD) are evaluated using one or a combination of several of the following global assessment indicators (KPI): object detection time, detection rate, sensor-resolved detection rate. Method for designing a sensor system (10) for detecting the surroundings for a vehicle (100) with the following steps: A2) providing a multiplicity of surroundings sensors (1-n) and generating a sensor model (SM) for each of the surroundings sensors (1-n), which each is representative of the physical properties of the respective environment sensor (1-n) when mapping a real environment of the vehicle (100) by means of real measurement by the respective environment sensor (1-n), B2) generating at least one environment model (UM), each representative is for a virtual environment of a virtual vehicle (100), C2) generating several different virtual sensor systems (10), a virtual sensor system (10) each having one or more environment sensors (1-n) of the plurality of environment sensors (1-n) , D2) each carrying out a simulation of one of the several different virtual sensor systems (10) according to one of the Claims 1 to 7th depending on the respective sensor model (SM) of the environment sensors (1-n) of the corresponding sensor system (10) and the at least one environment model (UM), and E2) output or storage of an evaluation (E) of the respective sensor system (10) depending on the Simulation. Procedure according to Claim 8 , in which, depending on the evaluation (E) of the respective sensor system (10), the at least one environment sensor (1-n) is selected and / or the environment sensor (1-n) is placed in the vehicle (100) and / or a Alignment of the environment sensor (1-n) with respect to the vehicle (100) is carried out. Method according to one of the preceding Claims 8 or 9 In which - the respective sensor model (SM) is determined in step A2) depending on a predetermined set of sensor parameters that are representative of the physical properties of the respective environment sensor when mapping a real environment of the vehicle (100) by means of real measurement by the respective environment sensor (1-n), and - the method is continued iteratively following step E2) in step A2), - depending on the evaluation (E) at least one of the sensor parameters and / or one of the configuration parameters is varied or and - the method is iterated until the evaluation (E) of the respective sensor system (10) reaches a high point or all sensor parameters of the set have been varied. Method according to one of the preceding Claims 8 or 9 , in which the several different virtual sensor systems (10) generated in step C2) serve as the initial population of a genetic algorithm and each consist of one or more configuration areas, with the genetic algorithm A3) for each of the several different virtual sensor systems (10) depending on a number of environment sensors (1-n) of the respective sensor system (10) and a respective sensor type of the environment sensors (1-n) of the respective sensor system (10), a cost parameter of the respective sensor system (10) is determined, B3) depending on the evaluation (E) of the respective sensor system (10), a utility parameter of the respective sensor system (10) is determined, C3) a ratio (N / K) is determined from the utility parameter to the cost parameter, and it is checked whether that Ratio (N / K) exceeds a predetermined threshold value, with the corresponding sensor system (10) being output or stored in the event that the predetermined threshold value is exceeded, and otherwise D3) at least two different configuration areas depending on the respective ratio (N / K) of the sensor systems (10) are selected, E3) the selected configuration areas are randomly combined to form several virtual sensor systems (10) as a new population of the genetic algorithm, and the genetic algorithm is iteratively continued with the new population in step A3). Procedure according to Claim 11 , wherein after step D3) F3) a random variation of at least one of the selected configuration areas is carried out as a mutation in the genetic algorithm. Method according to one of the preceding Claims 7 to 12th in which the at least one environment model (UM) is determined in step B2) as a function of a predefined set of scenario parameters that are representative of a predefined driving maneuver of the virtual vehicle (100), and the method following step E2 ) is continued iteratively in step B2), - at least one of the scenario parameters is varied depending on the evaluation (E), and - the method is iterated until the evaluation of the respective sensor system (10) reaches a low point or all scenario parameters of the set were varied. Device for simulating a sensor system (10) for a vehicle (100), wherein the sensor system (10) has one or more environment sensors (1-n), and the device is set up, the method according to one of the Claims 1 to 7th to execute. Apparatus for designing a sensor system (10) for detecting the surroundings of a vehicle (100), wherein the apparatus is set up to carry out the method according to one of the Claims 8 to 13th to execute.

Images