DE102022118631A1 - Method, system and computer program product for validating a driver assistance system (ADAS) and/or an automated driving system (ADS) - Google Patents

Abstract

The invention relates to methods for validating a driver assistance system (ADAS) and/or an automated driving system (ADS) for a variety of scenarios, wherein a scenario represents a traffic situation in a temporal sequence, comprising: - Acquiring real sensor data (250) of an environment moving object (10) by means of sensors (22) of a sensor device (20) of the moving object (10); - generating at least one scenario from the real sensor data (250); - determining a specific configuration of a parameterizable virtual model of the sensor device (20 ) by selecting parameters such as the number, the sensor types and the installation positions of the sensors (22); - Generating a simulation environment (340), the simulation environment (340) containing at least the generated scenario, a virtual model of the object (10) and the specified Configuration of the parameterizable virtual model of the sensor device (20) includes; - simulating virtual sensor data (350) for the generated simulation environment (340); - transmitting the virtual sensor data (350) to an ADAS/ADS module (30) of the moving object (10);- Validating, testing and/or training at least one function of the ADAS/ADS system.

Description

The invention relates to a method, a system and a computer program product for validating a driver assistance system (ADAS) and/or an automated driving system (ADS).

Modern vehicles are equipped with a variety of driving assistance systems or automated driving assistance functions to support the driver while driving and increase his safety. Driving assistance systems support, for example, speed and distance control as well as lane keeping and lane changing functions. A specific maximum speed can be set, which will not be exceeded as long as the speed limit function is activated. Radar and camera systems are used for distance control, in which a certain distance is set, especially to a vehicle in front. This allows the distance to vehicles in front, but also to vehicles in the side area, to be monitored. This leads to improved driving comfort and greater safety, especially when driving on the motorway and during overtaking maneuvers.

However, this trend towards driver assistance systems (Advanced Driver Assistance System, ADAS) and automated driving systems (ADS) in motor vehicles, but also in aircraft or watercraft, requires extensive security strategies, since responsibility for driving the vehicle no longer lies entirely with the driver , but active functions are taken over by computer units in the vehicle. Therefore, it must be ensured that autonomously moving objects have a very low error rate in driving behavior. The detection and classification of objects and the interpretation of traffic scenarios in the area around a vehicle are important prerequisites for the safe functionality of ADAS/ADS systems. This requires targeted testing and training of driver assistance systems and automated driving systems both in extreme and exceptional situations (corner cases) as well as in everyday situations.

As the performance of ADAS/ADS systems increases, the number of driving scenarios that the systems have to handle in road traffic also increases. To ensure safe, comfortable and efficient behavior of ADAS/ADS systems, individual functions and the entire system go through a validation process during vehicle development.

However, such validation represents a major challenge when integrating modern ADAS/ADS systems into a vehicle due to a functional specification deficit of driving assistance systems. While a requirements-based test process has been established for conventional systems in the automotive industry, in which test cases are implemented based on test specifications, this is currently missing for ADAS/ADA systems because, in contrast to conventional systems, a significantly larger number of influencing variables have to be taken into account, such as in particular the driving environment, which is recorded using sensors.

Due to the challenges in validating ADAS/ADS systems, new approaches are required in addition to the known methods in order to carry out the vehicle development process with reasonable effort and costs. It is therefore desirable to increasingly use virtual methods to simulate the validation of ADAS/ADS systems and to design these virtual simulation methods in a way that enables objective comparability of different ADAS/ADS systems with different functionalities in terms of performance and safety is possible. Data simulations can be used, for example, to simulate a section of highway in the real world that is suitable for training data-driven AI models. In addition, data annotation can be automatically carried out on the data generated by simulation.

The sensors of the moving object or vehicle to perceive its surroundings play a crucial role in the development and functionality of ADAS/ADS systems. However, in the early development phase of an ADAS/ADS system, for example for a vehicle model, the sensor types to be used and their sensor positioning on the vehicle have not yet been determined or are only determined with a high tolerance value. Existing sensor systems for other vehicle models can be used, but their use is limited for validating an ADAS/ADS system for a new vehicle model, for example due to the field of view depending on the sensor type and the installation position of the sensor. In addition, attaching, adjusting and wiring a sensor system to a test vehicle requires a lot of effort. Furthermore, in addition to a virtual simulation of sensor sets in conjunction with ADAS/ADS systems, an evaluation by a real driver is required, particularly with regard to the longitudinal and lateral dynamic driving behavior of a vehicle. Since real driving maneuvers with a driver are time-consuming and cost-intensive, it is desirable to do this driving maneuvers with a vehicle that already has optimal sensors for the ADAS/ADS system being tested.

The US 2021/0350046 A1 discloses a method for testing multimodal transport services, for example an aircraft or a ground vehicle, using simulations. The system determines scenario data, including a scenario definition, and creates a simulation environment based on the scenario data.

The DE 102020/202650 A1 discloses a method for route calculation for an autonomous motor vehicle, wherein a simulation environment is provided that has map data of a real area of operation.

The DE 102021/100696 A1 discloses a method for crowdsourcing and analyzing real-time streaming raw signal data from a fleet of connected motor vehicles with sensor systems. This can, for example, supplement the sensor capabilities of the sensor system or provide for the targeted use of one or more sensors in a single vehicle.

The DE 112020/000413 T5 discloses a method for recognizing landmarks in an autonomous driving environment. Simulated virtual environmental sensor data can be used to train an artificial intelligence.

The US 2020/0209858 A1 discloses a method for generating environmental parameters using a model with artificial intelligence (AI) algorithms. The AI model can be trained with real and/or virtual environmental sensor data.

An object of the present invention is therefore to provide options for validating a driver assistance system (ADAS) and/or an automated driving system (ADS) so that the validation process requires less time and can be carried out with improved efficiency in order to ensure the safety of ADAS/ ADS systems and to be able to save resources and costs.

This object is achieved according to the invention with regard to a method by the features of patent claim 1, with regard to a system by the features of patent claim 11, and with regard to a computer program product by the features of patent claim 15. The further claims relate to preferred embodiments of the invention.

• - Erfassen von realen Sensordaten einer Umgebung eines sich bewegenden Objekts mittels Sensoren einer Sensoreinrichtung des sich bewegenden Objekts;
• - Übermitteln der realen Sensordaten an ein Szenarienmodul;
• - Generieren zumindest eines Szenarios aus den realen Sensordaten für das Verkehrsgeschehen in der Umgebung des sich bewegenden Objekts mittels einer Softwareapplikation des Szenarienmoduls;
• - Übermitteln des generierten Szenarios an ein Simulationsmodul;
• - Festlegen einer bestimmten Konfiguration eines parametrisierbaren virtuellen Modells der Sensoreinrichtung durch Auswählen von Parametern wie der Anzahl, der Sensortypen und der Einbaupositionen der Sensoren;
• - Generieren einer Simulationsumgebung, wobei die Simulationsumgebung zumindest das generierte Szenario, ein virtuelles Modell des Objekts und die festgelegte Konfiguration des parametrisierbaren virtuellen Modells der Sensoreinrichtung umfasst;
• - Simulieren von virtuellen Sensordaten für die generierte Simulationsumgebung;
• - Übermitteln der virtuellen Sensordaten an ein ADAS/ADS-Modul des sich bewegenden Objekts, wobei das ADAS/ADS-Modul zumindest ein ADAS/ADS-System umfasst;
• - Validieren, Testen und/oder Trainieren von zumindest einer Funktion des ADAS/ADS-Systems.

According to a first aspect, the invention relates to a method for validating a driver assistance system (ADAS) and/or an automated driving system (ADS) for a plurality of scenarios, wherein a scenario represents a traffic event in a temporal sequence, comprising:

• - Acquiring real sensor data of an environment of a moving object using sensors of a sensor device of the moving object;
• - Transmitting the real sensor data to a scenario module;
• - Generating at least one scenario from the real sensor data for traffic events in the area surrounding the moving object using a software application of the scenario module;
• - Transmitting the generated scenario to a simulation module;
• - Determining a specific configuration of a parameterizable virtual model of the sensor device by selecting parameters such as the number, the sensor types and the installation positions of the sensors;
• - Generating a simulation environment, the simulation environment comprising at least the generated scenario, a virtual model of the object and the specified configuration of the parameterizable virtual model of the sensor device;
• - Simulating virtual sensor data for the generated simulation environment;
• - transmitting the virtual sensor data to an ADAS/ADS module of the moving object, the ADAS/ADS module comprising at least one ADAS/ADS system;
• - Validate, test and/or train at least one function of the ADAS/ADS system.

In a further development, it is planned to measure the behavior of the respective function of the ADAS/ADS system in relation to the virtual sensor data using sensors such as, in particular, acceleration sensors, speed sensors, capacitive sensors, inductive sensors, speed sensors, torque sensors, voltage sensors and to transmit the measurement data from the sensors Pass on the evaluation module.

In an advantageous embodiment it is provided that the evaluation module includes a software application that derives test results from the measurement data and evaluates them.

In a further embodiment, it is provided that the software application includes optimization algorithms with which an optimal configuration of the sensor device for a large number of scenarios can be calculated from the test results for the various configurations of the sensor device.

In particular, it is provided that the sensors of the sensor device include one or more radar systems with one or more radar sensors, one or more LIDAR systems for optical distance and speed measurement, one or more image-recording 2D/3D cameras in the visible range and/or in the IR Range and/or in the UV range, and/or GPS systems.

Advantageously, the software application of the scenario module includes algorithms of artificial intelligence and machine learning, in particular deep learning with, for example, at least one convolutional neural network and/or at least one reinforcement learning agent (LV) for generation of scenarios from the recorded real sensor data.

In a further embodiment it is provided that the software application of the simulation module uses algorithms of artificial intelligence and machine learning, in particular deep learning with, for example, at least one convolutional neural network and/or at least one reinforcement learning agent , LV), for generating the simulation environment and the virtual sensor data.

In a further development it is provided that the virtual model of the object and/or the parameterizable virtual model of the sensor device is/are designed as a digital twin.

In particular, the scenario module and the simulation module are integrated in a cloud computing infrastructure.

Advantageously, a 5G mobile connection or 6G mobile connection is provided for the communication of the sensor device with the scenario module or the cloud computing infrastructure for data transmission in real time.

According to a second aspect, the invention provides a system for validating a driver assistance system (ADAS) and/or an automated driving system (ADS) for a variety of scenarios. A scenario represents a traffic event in a temporal sequence. The system includes a moving object with a sensor device and an ADAS/ADS module, a scenario module and a simulation module. The sensor device is designed to detect real sensor data of an environment of the moving object using sensors and to transmit the real sensor data to the scenario module, wherein the scenario module is designed to contain at least one scenario from the real sensor data for the traffic situation in the area of the moving object to be generated using a software application and to transmit the generated scenario to a simulation module. The simulation module is designed to determine a specific configuration of a parameterizable virtual model of the sensor device by selecting parameters such as the number, the sensor types and the installation positions of the sensors and to generate a simulation environment, the simulation environment containing at least the generated scenario, a virtual model of the object and includes the specified configuration of the parameterizable virtual model of the sensor device. Furthermore, the simulation module is designed to simulate virtual sensor data for the generated simulation environment and to transmit the virtual sensor data to the ADAS/ADS module of the moving object. The ADAS/ADS module includes at least one ADAS/ADS system and is designed to validate, test and/or train at least one function of the ADAS/ADS system.

In a further development it is provided that the behavior of the respective function of the ADAS/ADS system is measured in relation to the virtual data by sensors such as in particular acceleration sensors, speed sensors, capacitive sensors, inductive sensors, speed sensors, torque sensors, voltage sensors and the measurement data of the sensors passed on to an evaluation module.

In an advantageous embodiment, it is provided that the evaluation module comprises a software application which derives test results from the measurement data and evaluates them, the software application comprising optimization algorithms with which an optimal configuration of the sensor device for a high number of test results for the different configurations of the sensor device of scenarios can be calculated.

In a further embodiment it is provided that the sensors of the sensor device have one or more radar systems with one or more radar sensors, one or more LIDAR systems for optical distance and speed ness measurement, one or more image-recording 2D/3D cameras in the visible range and/or in the IR range and/or in the UV range, and/or GPS systems.

According to a third aspect, the invention relates to a computer program product comprising an executable program code which, when executed, carries out the method according to the first aspect.

The invention is explained in more detail below using an exemplary embodiment shown in the drawing.

• 1 ein Blockdiagramm zur Erläuterung eines Ausführungsbeispiels eines erfindungsgemäßen Systems;
• 2 ein Flussdiagramm zur Erläuterung der einzelnen Verfahrensschritte eines erfindungsgemäßen Verfahrens;
• 3 ein Computerprogrammprodukt gemäß einer Ausführungsform des dritten Aspekts der Erfindung.

This shows:

• 1 a block diagram to explain an exemplary embodiment of a system according to the invention;
• 2 a flowchart to explain the individual method steps of a method according to the invention;
• 3 a computer program product according to an embodiment of the third aspect of the invention.

Additional features, aspects and advantages of the invention or its embodiments will become apparent from the detailed description taken in conjunction with the claims.

Simulated traffic scenarios created through programming are increasingly being used to test, train and validate driver assistance systems (ADAS) and automated driving systems (ADS). In the context of the invention, a scenario is a traffic event in a temporal sequence. An example of a scenario is driving on a highway bridge, turning in a turning lane, driving through a tunnel, turning into a roundabout or stopping in front of a pedestrian crossing.

In order to be able to use driver assistance systems (ADAS) and automated driving systems (ADS) in a motor vehicle, they must be validated for reliable use. The aim of validation is to demonstrate the reliability and robustness of a driver assistance system (ADAS) or an automated driving system (ADS) for a variety of traffic situations and traffic scenarios and then to grant approval.

1 shows a system 100 according to the invention for validating a driver assistance system (ADAS) and/or an automated driving system (ADS). The system 100 includes a moving object 10, which is designed in particular as a motor vehicle, and several modules, which can include both integrated or assigned processors and/or memory units. In particular, the moving object can be an autonomous motor vehicle, an agricultural vehicle such as a combine harvester, a robot in production or in service and care facilities, or a watercraft or a flying object such as an airplane or a drone.

In connection with the invention, a “module” can therefore be understood to mean, for example, a processor and/or a memory unit for storing program instructions. For example, the module is specifically set up to execute the program commands in order to implement or realize the method according to the invention or a step of the method according to the invention.

In connection with the invention, a “processor” can be understood to mean, for example, a machine or an electronic circuit or a powerful computer. A processor can in particular be a main processor (Central Processing Unit, CPU), a microprocessor or a microcontroller, for example an application-specific integrated circuit or a digital signal processor, possibly in combination with a memory unit for storing program instructions, etc . A processor can also be understood as a virtualized processor, a virtual machine or a soft CPU. For example, it can also be a programmable processor that is equipped with configuration steps for carrying out the said method according to the invention or is configured with configuration steps in such a way that the programmable processor has the features of the method, the component, the modules, or other aspects and/or other aspects according to the invention. or partial aspects of the invention are realized. In addition, highly parallel computing units and powerful graphics modules can be provided.

In connection with the invention, a “memory unit” or “memory module” and the like can, for example, be a volatile memory in the form of random access memory (RAM) or a permanent memory such as a hard drive or a data carrier or, for example. B. a replaceable memory module can be understood. The storage module can also be a cloud-based storage solution.

The moving object 10 includes a sensor device 20 with sensors 22 that are real Sensor data 250 from the surroundings of the object 10 such as road markings, vehicles, people, guardrails, etc. are recorded and transmitted to a scenario module 200. In addition to the sensor device 20, the object 10 has an ADAS/ADS module 30, which includes software applications and storage units for ADAS/ADS systems.

In connection with the invention, sensor data 250 means both raw data and already processed data from the measurement results of the sensors 22 and, if necessary, other data sources.

The sensors 22 of the sensor device 20 can in particular have one or more radar systems with one or more radar sensors, one or more LIDAR systems (Light Detection and Ranging) for optical distance and speed measurement, one or more image-recording 2D/3D cameras visible range, but also in the IR and UV range, and/or GPS systems.

In particular, the 2D/3D image-recording camera is designed as an RGB camera in the visible range with the primary colors blue, green and red. However, a UV camera in the ultraviolet range and/or an IR camera in the infrared range can also be provided. The cameras, which differ in their recording spectrum, can therefore model different lighting conditions in the recording area. Furthermore, it is envisaged that a 3D camera is designed as a stereo camera.

The recording frequency of the sensor device 20 is designed in particular for fast speeds of the object 10 and can record sensor data 250 with a high image recording frequency. Furthermore, the sensor device 20 can be equipped with a microphone for detecting acoustic signals. This allows tire rolling noise or engine noise to be recorded.

In addition, it can be provided that the sensor device 20 automatically starts the image recording process when there is a significant change in area in the recording area of the sensor device 20, for example when a clear change in a traffic situation is recognizable. This enables a selective data collection process and only relevant sensor data 250 is processed by the scenario module 200. This allows computing capacity to be used more efficiently.

In particular, it is intended to use a weatherproof action camera as the camera type for one or more cameras, which can be arranged in particular in the outside area of the object 10. An action camera has wide-angle fisheye lenses, making it possible to achieve a visible radius of approximately 180°. This allows the road ahead to be comprehensively mapped. Action cameras can usually record videos in Full HD (1,920 x 1,080 pixels), but action cameras can also be used in Ultra HD or 4K (at least 3,840 x 2,160 pixels), which results in a significant increase in image quality. The image capture rate is typically 60 frames per second in 4K and up to 240 per second in Full HD. An integrated image stabilizer can also be provided. In addition, action cameras are often equipped with an integrated microphone. Differential signal processing methods can also be used to specifically block out background noise.

The attachment position of a camera on the object 10 determines which recording area can be recorded by the camera. In particular, it can be provided that the recording areas of two or more cameras overlap in order, for example, to generate a panoramic display as part of further image processing. This allows the spatial surroundings of a moving object 10 to be comprehensively recorded. In addition to the desired recording area, however, in the case of an object 10 such as a vehicle, the technically possible attachment positions and questions relating to sensible integration into the design of a body must also be taken into account.

Radar sensors can be used for longer distances of up to 250 meters and have the advantage of being independent of weather and lighting conditions. The performance of a radar depends on many factors such as the selected hardware components, software processing and the radar echo. For example, radar accuracy is less precise with a lower signal-to-noise ratio than with a high signal-to-noise ratio. In addition, the installation position is crucial for the high performance of a radar sensor, as effects such as multipath propagation and distortion caused by covers affect the detection accuracy. For example, if the azimuth measurement is inaccurate, the position of a target object appears offset from the actual position. An azimuth measurement error of 1° for a target object one hundred meters away results in a lateral offset of 1.75 m compared to the actual position. This can result in a driving maneuver in an autonomous vehicle be solved that does not correspond to the real traffic situation. It is therefore important that the angle measurement error of a radar sensor used is very small.

Typically, radar sensors are installed in the bumper of a vehicle. However, the transmission and absorption properties of the material used can lead to distortions in the radar signal. Therefore, in addition to the spatial installation location, the material surroundings of a radar sensor, such as a specific plastic or metal material and a specific paint finish, are also important. The geometric shape of a casing can also play a role. In particular, the area of brand emblems is often chosen as the installation location for long-range radars. However, since these are often made of metal, their electromagnetic properties can influence the performance of a radar sensor.

In addition to image-recording cameras and radar sensors, LIDAR sensors are an important type of sensor for perceiving the environment for highly automated and autonomous driving. As with cameras and radar sensors, the environment can be recorded and distances to objects can be measured. In particular, 3D LIDAR sensors can record detailed information about an object through a high sampling rate. Compared to radar sensors, LIDAR sensors are characterized by a higher spatial and depth resolution. When it comes to LIDAR sensors, a distinction is made between a mechanical scanning LIDAR with mechanically rotating components for scanning a laser beam and an SSL LIDAR (Solid State Lidar) without moving components. An SLL LIDAR system typically consists of a laser source or laser diode, optical elements such as lenses and diffusers, beam control elements, photodetectors and signal processing units. The recording range of SLL-LIDAR is smaller, but the cost is lower and the reliability is higher. On The laser beam in SLL-LIDAR is controlled using microelectromechanical systems (MEMS). In addition, SLL LIDAR systems differentiate between flash LIDAR with illumination of the entire recording area for short and medium-range detection and MEMS-based scanning LIDAR with micromirrors for beam deflection for long-range or wide-angle detection. Flash LIDAR systems are limited in range and detect objects in the immediate vicinity of a vehicle, while scanning LIDAR systems are suitable for distance and speed measurements of more distant objects. The range, resolution, field of view and reliable detection of reflected light are crucial parameters for the performance of a LIDAR system.

Furthermore, a GPS connection is advantageously provided in order to determine the geographical location of the object 10 and to assign this to the recorded sensor data 250.

The real sensor data 250 of the environment of the object 10 recorded by the sensor device 20 are passed on to a scenario module 200 in order to derive a scenario from the real sensor data 250. The data recorded by the sensors 800 is forwarded to the scenario module 200 via communication connections. In particular, a wireless communication connection is provided, which can be designed, for example, as a mobile radio connection and/or a near-field communication connection such as ^Bluetooth® , Ethernet, NFC (near field communication) or Wi-Fi®.

Since a fast calculation result is sought in order to be able to quickly test and train driving assistance systems and automated driving functions, the calculation speed is crucial for the large number of calculation operations in the scenario module 200. In order to enable the computing operations to be carried out in real time, the use of a cloud computing infrastructure 400 is advantageous. This ensures fast calculations, as cloud-based solutions offer the advantage of high and therefore fast computing performance. For the communication of the sensor device 20 with the scenario module 200 or the cloud computing infrastructure 400, a 5G mobile phone connection or 6G mobile phone connection is used in particular, since data transmission can take place in real time in this way. For this purpose, the sensor device 20 is equipped with the corresponding mobile radio modules.

5G is the fifth generation mobile communications standard and, compared to the 4G mobile communications standard, is characterized by higher data rates of up to 10 Gbit/sec, the use of higher frequency ranges such as 2100, 2600 or 3600 megahertz, increased frequency capacity and thus increased data throughput and real-time data transmission because up to a million devices per square kilometer can be addressed simultaneously. The latency times range from a few milliseconds to less than 1 ms, so that real-time transmission of data and calculation results is possible. Therefore, the sensor data 250 recorded by the sensor device 20 can be forwarded to the scenario module 200 in real time.

By integrating the scenario module 200 in a cloud computing infrastructure 400 in conjunction with a 5G mobile radio connection, processing of the sensor data 250 recorded by the sensor device 20 can be ensured in real time. In order to protect the connection to the cloud computing infrastructure 400 using a mobile radio connection, cryptographic encryption methods are provided in particular.

The scenario module 200 has a software application 220 that creates a scenario from the real sensor data 250. In the context of the invention, a scenario is a traffic event in a temporal sequence. An example of a scenario is driving on a highway bridge, turning in a turning lane, driving through a tunnel, turning into a roundabout or stopping in front of a pedestrian crossing. In addition, specific visibility conditions, for example due to twilight or high levels of sunlight, as well as environmental conditions such as the weather and season, traffic volume and certain geographical topographical conditions can influence a scenario. For example, an overtaking operation can be described as a scenario in which a first vehicle is initially behind another vehicle, then changes lanes to the other lane and increases speed in order to overtake the other vehicle. The time of year can also play a role, as an overtaking maneuver looks different in heavy rain and slippery conditions than in sunshine.

For example, a scenario can be defined by various parameters and associated parameter values. The parameter values determine the value range of a parameter. The parameters include, for example, a moving object such as a motor vehicle, an immovable object such as a building, a road configuration such as a highway, a speed, a street sign, a traffic light, a tunnel, a roundabout, a turning lane, an acceleration, a direction, an angle , a radius, a location, a traffic volume, a topographical structure such as a slope, a time, a temperature, a precipitation value, a weather condition, a season.

The software application 220 includes in particular artificial intelligence and machine image analysis algorithms in order to select and classify the real sensor data 250. Advantageously, the software application 220 uses algorithms from the field of machine learning, preferably deep learning with, for example, at least one convolutional neural network and/or at least one reinforcement learning agent (LV) to create scenarios the recorded sensor data 250.

A neural network consists of neurons arranged in several layers and connected to each other in different ways. A neuron is able to receive information from outside or from another neuron at its input, evaluate the information in a certain way and forward it in a modified form to another neuron at the neuron output or output it as the end result. Hidden neurons are located between the input neurons and output neurons. Depending on the network type, there may be multiple layers of hidden neurons. They ensure the forwarding and processing of the information. Output neurons ultimately deliver a result and output it to the outside world. The different arrangement and connection of the neurons creates different types of neural networks such as feedforward networks, recurrent networks or convolutional neural networks. The networks can be trained using unsupervised or supervised learning

The Convolutional Neural Network (CNN) has several convolution layers and is very well suited for machine learning and applications with artificial intelligence (AI) in the area of image and speech recognition. The way a convolutional neural network works is to a certain extent modeled on biological processes and the structure is comparable to the visual cortex of the brain. Conventional neural networks consist of fully or partially connected neurons in several levels and these structures reach their limits when processing images, as there would have to be a number of inputs corresponding to the number of pixels. The convolutional neural network is made up of different layers and is basically a partially locally linked feedforward neural network. The individual layers of the CNN are the convolutional layer, the pooling layer and the fully connected layer. The convolutional layer is the actual convolution layer and is able to recognize and extract individual features in the input data. In image processing, these can be features such as lines, edges or certain shapes. The input data is processed in the form of tensors such as a matrix or vectors. The pooling layer, also called the subsampling layer, compresses and reduces the resolution of the recognized features through appropriate filter functions. The reduced data volume increases the calculation speed.

The Convolutional Neural Network (CNN) is therefore suitable for machine learning and artificial intelligence applications with large amounts of input data, such as in image recognition. The network works reliably and is insensitive to distortions or other optical changes. The CNN can process images captured under different lighting conditions and from different perspectives. It still recognizes the typical features of an image. Since the CNN includes several local partially connected layers, it has a significantly lower storage space requirement than fully connected neural networks, as the convolutional layers significantly reduce the storage requirements. This also shortens the training time of a CNN, especially when using modern graphics processors.

The scenario created by the software application 220 is forwarded to a simulation module 300. The simulation module 300 includes a software application 320 that generates a simulation environment 340 for the transmitted scenario. In the context of the invention, a simulation environment 340 refers to a virtual environment that includes the respective generated scenario, a virtual model of the object 10 and a parameterizable virtual model of the sensor device 20. The virtual model of the object 10 is designed in particular as a digital twin and is able to carry out the same movements as the real object 10. A digital twin is a virtual representation of a real physical product or process and includes geometric, kinematic and other process information. In addition, the virtual model of the object 10 is linked to the parameterizable virtual model of the sensor device 20. The parameterizable virtual model of the sensor device 20 is preferably also designed as a digital twin. The parameters of the parameterizable virtual model of the sensor device 20 relate in particular to the selected number, the sensor types and the respective positioning of the sensors 22 on the object 10. By changing one or more parameters, different configurations of the sensor device 20 can thus be created.

The data or scripts for the virtual model of the object 10 and the parameterizable virtual model of the sensor device 20 can be stored in a database 370, from which they are retrieved by the simulation module 300. In particular, virtual models or digital twins can be stored for different objects 10, in particular for different vehicle models. In addition, the parameters for different sensor types of the sensors 22 are stored in the database 370. Furthermore, 370 historical data in the form of images, graphics, time series, parameters, etc. can be stored in the database. Target variables and target values can define a security standard. “Database” means both a storage algorithm and the hardware in the form of a storage unit. In particular, the database 370 can also be integrated into the cloud computing infrastructure 400.

After the simulation environment 340 has been generated, the software application 320 extracts virtual sensor data 350 for this simulation environment 340. This virtual sensor data 350 simulates the real sensor data that would be recorded from the environment of the moving object 10 if it had been recorded with sensors 22 in accordance with the selected sensor configuration of the sensor device 20.

It is therefore an inverse process to the creation of the simulation environment 340 from the real sensor data 250. Since the simulation environment 340 was generated from the scenario created using the real sensor data 250 and a virtual model of the object 10 as well as a selected configuration of the sensor device 20, The virtual sensor data 350 can differ from the real sensor data 250.

The software application 320 also includes, in particular, artificial intelligence and machine image analysis algorithms to create the simulation environment 340. Advantageously, the software application 320 uses algorithms from the field of machine learning, preferably deep learning with at least one convolutional neural network and/or at least one reinforcement learning agent (LV). In particular, the neural networks are arranged in a reverse order, since the virtual sensor data 350 are now extracted from the simulation environment 340.

This virtually generated sensor data 350 is passed from the simulation module 300 to the ADAS/ADS module 30 to test, train and validate the various functions of one or more ADAS/ADS systems. For the communication of the simulation module 300 with the ADAS/ADS module 30, in particular a 5G mobile radio connection or 6G mobile radio connection is used, since this is the case Way data transmission can take place in real time. Due to the real-time calculation, the current scenario has hardly changed, so that one or more functions of the ADAS/ADS system can be tested or validated for this scenario.

The behavior of the respective function of the ADAS/ADS system can be measured by sensors 24 such as, in particular, acceleration sensors, speed sensors, capacitive sensors, inductive sensors, speed sensors, torque sensors, voltage sensors. These measurement data for the respective configuration of the sensor device 20 can be passed on to an evaluation module 500. The evaluation module 500 includes a software application 550 that derives test results from the measurement data and evaluates them.

In addition, optimization algorithms can be provided with which an optimal configuration of the sensor device 20, which is suitable for a large number of scenarios, can be calculated from the test results for the different configurations of the sensor device 20. Optimization algorithms that can be used are algorithms for estimating distributions such as Bayesian optimization, evolutionary algorithms such as genetic algorithms and evolutionary strategies with operators such as mutation, selection, recombination, or numerical methods such as the Nelder-Meat method, ant algorithms (ant colony optimization), particle swarm optimization, or heuristic approximation methods such as simulated annealing can be used. A variety of optimization methods in the area of machine learning are therefore conceivable, which can be used by the evaluation module 500 to find a suitable configuration for the sensor device 20.

• In einem Schritt S10 werden reale Sensordaten 250 einer Umgebung eines sich bewegenden Objekts 10 mittels Sensoren 22 einer Sensoreinrichtung 20 des sich bewegenden Objekts 10 erfasst.
• In einem Schritt S20 werden die realen Sensordaten 250 an ein Szenarienmodul 200 übermittelt.
• In einem Schritt S30 wird zumindest ein Szenario aus den realen Sensordaten 250 für das Verkehrsgeschehen in der Umgebung des sich bewegenden Objekts 10 mittels einer Softwareapplikation 220 des Szenarienmoduls 200 generiert.
• In einem Schritt S40 wird das generierte Szenario an ein Simulationsmodul 300; übermittelt.
• In einem Schritt S50 wird eine bestimmte Konfiguration eines parametrisierbaren virtuellen Modells der Sensoreinrichtung 20 durch Auswählen von Parametern wie der Anzahl, der Sensortypen und der Einbaupositionen der Sensoren 22 festgelegt.
• In einem Schritt S60 wird eine Simulationsumgebung 340 generiert, wobei die Simulationsumgebung 340 zumindest das generierte Szenario, ein virtuelles Modell des Objekts 10 und die festgelegte Konfiguration des parametrisierbaren virtuellen Modells der Sensoreinrichtung 20 umfasst.
• In einem Schritt S70 werden virtuelle Sensordaten 350 für die generierte Simulationsumgebung 340 simuliert.
• In einem Schritt S80 werden die virtuellen Sensordaten 350 an ein ADAS/ADS-Modul 30 des sich bewegenden Objekts 10 übermittelt, wobei das ADAS/ADS-Modul 30 zumindest ein ADAS/ADS-System umfasst.
• In einem Schritt S90 wird zumindest eine Funktion des ADAS/ADS-Systems validiert, getestet oder trainiert.

A method for validating a driver assistance system (ADAS) and/or an automated driving system (ADS) for a variety of scenarios includes the following procedural steps:

• In a step S10, real sensor data 250 of an environment of a moving object 10 is recorded by means of sensors 22 of a sensor device 20 of the moving object 10.
• In a step S20, the real sensor data 250 is transmitted to a scenario module 200.
• In a step S30, at least one scenario is generated from the real sensor data 250 for the traffic situation in the surroundings of the moving object 10 using a software application 220 of the scenario module 200.
• In a step S40, the generated scenario is sent to a simulation module 300; transmitted.
• In a step S50, a specific configuration of a parameterizable virtual model of the sensor device 20 is determined by selecting parameters such as the number, the sensor types and the installation positions of the sensors 22.
• In a step S60, a simulation environment 340 is generated, the simulation environment 340 comprising at least the generated scenario, a virtual model of the object 10 and the specified configuration of the parameterizable virtual model of the sensor device 20.
• In a step S70, virtual sensor data 350 is simulated for the generated simulation environment 340.
• In a step S80, the virtual sensor data 350 is transmitted to an ADAS/ADS module 30 of the moving object 10, the ADAS/ADS module 30 comprising at least one ADAS/ADS system.
• In a step S90, at least one function of the ADAS/ADS system is validated, tested or trained.

By generating a simulation environment 340 according to the invention, which is based on a real environment, virtual sensor data 350 can be generated in real time. By varying the parameters of the sensor device 20, it is possible to examine how different configurations of the sensor device 20, which differ in terms of the number, the sensor types and the positioning of the sensors 22 and thus each generate a different spectrum of sensor signals in relation to a specific environment , affect the behavior of one or more functions of an ADAS/ADS system. The actual construction of a large number of prototypes of sensor devices 20 with different configurations can therefore be dispensed with, since the behavior of an ADAS/ADS system for a specific configuration of a sensor device 20 can be tested and validated using the method according to the invention. This makes significant cost and time savings possible.

Reference symbols

10

object

20

Sensor device

22

Sensors

24

Sensors

30

ADAS/ADS module

100

system

200

Scenario module

220

Software application

250

real sensor data

300

Simulation module

320

Software application

340

Simulation environment

350

virtual sensor data

370

Database

400

Cloud computing infrastructure

500

Assessment module

550

Software application

900

Computer program product

950

Program code

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• US 2021/0350046 A1 [0008]
• DE 102020/202650 A1 [0009]
• DE 102021/100696 A1 [0010]
• DE 112020/000413 T5 [0011]
• US 2020/0209858 A1 [0012]

Claims (15)

Method for validating a driver assistance system (ADAS) and/or an automated driving system (ADS) for a variety of scenarios, wherein a scenario represents a traffic event in a temporal sequence, comprising: - Detecting (S10) real sensor data (250) of an environment of a moving object (10) using sensors (22) of a sensor device (20) of the moving object (10); - Transmit (S20) the real sensor data (250) to a scenario module (200); - Generating (S30) at least one scenario from the real sensor data (250) for the traffic events in the area around the moving object (10) using a software application (220) of the scenario module (200); - Transmitting (S40) the generated scenario to a simulation module (300); - Determining (S50) a specific configuration of a parameterizable virtual model of the sensor device (20) by selecting parameters such as the number, the sensor types and the installation positions of the sensors (22); - Generating (S60) a simulation environment (340), the simulation environment (340) comprising at least the generated scenario, a virtual model of the object (10) and the specified configuration of the parameterizable virtual model of the sensor device (20); - Simulating (S70) virtual sensor data (350) for the generated simulation environment (340); - Transmitting (S80) the virtual sensor data (350) to an ADAS/ADS module (30) of the moving object (10), the ADAS/ADS module (30) comprising at least one ADAS/ADS system; - Validating, testing and/or training (S90) of at least one function of the ADAS/ADS system. Procedure according to Claim 1 , wherein the behavior of the respective function of the ADAS/ADS system based on the virtual data (350) is measured by sensors (24) such as in particular acceleration sensors, speed sensors, capacitive sensors, inductive sensors, speed sensors, torque sensors, voltage sensors and the measurement data of the sensors (24) are passed on to an evaluation module (500). Procedure according to Claim 2 , wherein the evaluation module (500) comprises a software application (550) which derives test results from the measurement data and evaluates them. Procedure according to Claim 3 , wherein the software application (550) comprises optimization algorithms with which an optimal configuration of the sensor device (20) can be calculated for a large number of scenarios from the test results for the various configurations of the sensor device (20). Procedure according to one of the Claims 1 until 4 , wherein the sensors (22) of the sensor device (20) include one or more radar systems with one or more radar sensors, one or more LIDAR systems for optical distance and speed measurement, one or more image-recording 2D/3D cameras in the visible range and/or in the IR range and/or in the UV range, and/or GPS systems. Procedure according to one of the Claims 1 until 5 , wherein the software application (220) of the scenario module (200) uses algorithms of artificial intelligence and machine learning, in particular deep learning with, for example, at least one convolutional neural network and / or at least one reinforcement learning agent , LV), for generating scenarios from the recorded real sensor data (250). Procedure according to one of the Claims 1 until 6 , wherein the software application (320) of the simulation module (300) uses algorithms of artificial intelligence and machine learning, in particular deep learning with, for example, at least one convolutional neural network and / or at least one reinforcement learning agent , LV), for generating the simulation environment (340) and the virtual sensor data (350). Procedure according to one of the Claims 1 until 7 , wherein the virtual model of the object (10) and / or the parameterizable virtual model of the sensor device (20) is / are designed as a digital twin. Procedure according to one of the Claims 1 until 8th , wherein the scenario module (200) and the simulation module (300) are integrated in a cloud computing infrastructure (400). Procedure according to one of the Claims 1 until 9 , wherein a 5G mobile phone connection or 6G mobile phone connection is provided for the communication of the sensor device (20) with the scenario module (200) or the cloud computing infrastructure (400) for data transmission in real time. System (100) for validating a driver assistance system (ADAS) and/or an automated driving system (ADS) for a variety of scenarios, a scenario being a traffic situation hen in a temporal sequence, comprising a moving object (10) with a sensor device (20) and an ADAS/ADS module (30), a scenario module (200) and a simulation module (300), wherein the sensor device (20) is designed to record real sensor data (250) of an environment of the moving object (10) using sensors (22) and to send the real sensor data (250) to the scenario module (200); to transmit; wherein the scenarios (200) are designed to generate at least one scenario from the real sensor data (250) for the traffic events in the surroundings of the moving object (10) using a software application (220) and the generated scenario is sent to a simulation module (300) to transmit; wherein the simulation module (300) is designed to determine a specific configuration of a parameterizable virtual model of the sensor device (20) by selecting parameters such as the number, the sensor types and the installation positions of the sensors (22) and to generate a simulation environment (340), whereby the simulation environment (340) comprises at least the generated scenario, a virtual model of the object (10) and the fixed configuration of the parameterizable virtual model of the sensor device (20), and to simulate virtual sensor data (350) for the generated simulation environment (340) and the to transmit virtual sensor data (350) to the ADAS/ADS module (30) of the moving object (10), the ADAS/ADS module (30) having at least one ADAS/ADS system and being designed to have at least one function to validate, test and/or train the ADAS/ADS system. System (100) after Claim 11 , wherein the behavior of the respective function of the ADAS/ADS system based on the virtual data (350) is measured by sensors (24) such as in particular acceleration sensors, speed sensors, capacitive sensors, inductive sensors, speed sensors, torque sensors, voltage sensors and the measurement data of the sensors (24) are passed on to an evaluation module (500). System (100) after Claim 11 or 12 , wherein the evaluation module (500) comprises a software application (550) which derives test results from the measurement data and evaluates them, and wherein the software application (550) comprises optimization algorithms with which an optimal one is derived from the test results for the various configurations of the sensor device (20). Configuration of the sensor device (20) can be calculated for a large number of scenarios. System (100) according to one of the Claims 11 until 13 , wherein the sensors (22) of the sensor device (20) include one or more radar systems with one or more radar sensors, one or more LIDAR systems for optical distance and speed measurement, one or more image-recording 2D/3D cameras in the visible range and/or in the IR range and/or in the UV range, and/or GPS systems. Computer program product (900), comprising an executable program code (950) which is configured so that, when executed, it implements the method according to one of Claims 1 until 10 executes.

Images