DE202022106107U1 - System for testing level 3 automated driving systems (ADS) - Google Patents

Abstract

A system for testing level 3 automated driving systems (ADS), the system comprising:
a simulator configured to validate the L3 ADS exam framework and examine autonomous driving capabilities;
a bank of cameras connected to the simulator, the bank of cameras configured to capture images of the vehicles and roads;
a semantic segmentation unit connected to the array of cameras to assign a label to each pixel in the image to classify the captured images; and
a test case generation unit connected to the semantic segmentation unit, the test case generation unit being configured to:
Analysis of the factors influencing autonomous driving functions, including the vehicle's ego speed, and their role in conducting the test;
in a second step, to create parameters from the determined criteria;
executing a test run using a set of instructions and a variety of simulation tools when simulating the test situations; and
Evaluate the test, including evaluating the simulation against multiple criticality criteria under driving conditions, including time to collision.

Description

FIELD OF THE INVENTION

The present invention relates generally to the field of a level 3 automated driving system (ADS) testing system.

BACKGROUND OF THE INVENTION

These days, the automotive industry is taking a quantum leap toward a future in which the driver will play an ever-decreasing role in operating their vehicle, which turns out to be completely banned. At the point where we move from driver assistance to automated driving, the basic test is managing an open environment; that is, autonomous systems should have the ability to deal with whatever circumstances happen in their environment on their own, without assistance from a human driver. Accordingly, the verification of AI-based video perception, which aims to distinguish objects in a camera image, is also becoming more and more amazing. Numerous investigations are being carried out in the industry to develop strategies and proofs of concept (POCs) for the verification of AI-based systems. One of the essential capabilities required to confirm such an AI-based video perception is the ability to generate synthetic images with different parameters to simulate different use cases.

Automated driving innovation has recently been a focus of research and application due to its commitment to improving driving safety, efficiency and comfort. Most accidents caused by human error can be avoided by using advanced driver assistance systems (ADAS). The goal of ADAS is to reduce the risk of accidents and the severity of accidents that cannot be avoided, thereby preventing death and injury. A study has shown that 94% of accidents are due to human error, while the rest are caused by climatic and mechanical influences. The ability to avoid car accidents has fueled the spread of ADAS in cars. ADAS applications that assist drivers with safety-critical functions to reduce car accidents and save lives include "automatic emergency braking, pedestrian detection, all-around visibility, parking assistance, driver drowsiness detection and gaze detection."

To detect environmental factors, self-driving vehicles use a range of sensors, including thermal imaging cameras, radar, LIDAR, sonar, GPS, odometers and inertial sensors.” Advanced control systems interpret the sensor information to evaluate acceptable maneuvers as well as obstacles and necessary signage. An automated driving system performs all real-world driving functions required to drive a ground-based vehicle without real-time human driver or sensor intervention. Broadly speaking, an automated driving system is an integrated bundle of individual automated systems working together. Also, automated driving is divided into three subsystems: environmental factor detection, decision making, and motion control.

In March 2016, a self-driving Google vehicle, now known as Waymo, crashed into the side of a bus as it tried to change lanes to get out from behind an obstacle in its way. This example shows how difficult it is to predict all vehicle behaviors in advance. A Level 3 autonomous vehicle from Uber struck and killed a person on foot in Arizona in March 2018. Tesla vehicles in Level 2 autonomous mode have been linked to two other fatal crashes in Florida and California. At the time of these numerous fatal accidents, a human driver was in full control of the vehicle. These incidents underscore the importance of coordinating control system robustness and conducting exploratory testing that covers as many unsurprising situations as possible.

At the point when the transition from driver assistance to automated driving occurs, the fundamental test is open-ended situation management, and this means that autonomous systems should have the ability to deal with any conditions that arise in their current situation , without depending on a human driver. It is of fundamental importance that such systems are fully tested before being used in series production.

One method of studying safety is to test autonomous vehicles in a detectable rush-hour traffic jam that resembles the real world as closely as possible. The downside is that real traffic is unpredictable and randomly moving, requiring a large amount of data to ensure the legitimacy of the test results ten. Due to the high cost, it is only offered to large companies or institutions.

Another critical test is thinking about what can happen during automated driving, leading to suggestions for safety design of tasks ranging from planning to confirmation. Due to the complexity and uncertainty of the driving climate, as well as the complexity of the driving process itself, the number of circumstances that an automated driving system (ADS) or an advanced driver assistance system (ADAS) can encounter is almost unlimited.

Virtual testing can be used to solve such problems. Model-in-the-Loop (MIL), Software-in-the-Loop (SIL), Hardware-in-the-Loop (HIL), Vehicles-in-the-Loop (VIL) and other strategies can be employed to test and approve frameworks in different stages of improvement.

This has sparked a great deal of interest in virtual assessment and verification, which is about 'safe checking' and exploration, as well as possible practical confirmation of the evidence provided. However, this raises the associated questions: which tests to select (i.e., which ones are critical), how to encourage them, and how to think about safety investigation coverage and compliance, given a real simulation climate (a test at themselves).

Existing technologies cannot solve the problem of testing level 3 automated driving systems. Therefore, there is an urgent need to develop a system for testing level 3 automated driving systems (ADS).

SUMMARY OF THE INVENTION

The present invention discloses a system for testing level 3 automated driving systems (ADS). The system includes: a simulator configured to validate the framework for testing L3-ADS and examining autonomous driving capabilities; an array of cameras connected to the simulator, the array of cameras configured to capture images of vehicles and roads; a semantic segmentation unit connected to the array of sensors to assign a label to each pixel in the image to classify the captured images; and a test case generation unit connected to the semantic segmentation unit, wherein the test case generation unit is configured to: analyze the factors that affect the autonomous driving functions, including the ego speed of the vehicle, and that play a role in the test to be performed; generating parameters from the identified criteria in a second stage; executing a test run using a set of instructions and a variety of simulation tools when simulating the test situations; and evaluating the test including evaluating the simulation using multiple criticality criteria in driving conditions including time to collision.

The aim of the present invention is to develop a system for testing level 3 automated driving systems (ADS).

Another objective of the present invention is to analyze the factors affecting autonomous driving functions, including the vehicle's ego speed and the player's role.

In order to further clarify the advantages and features of the present invention, a more detailed description of the invention will be given by reference to specific embodiments thereof illustrated in the accompanying figures. It is understood that these figures represent only typical embodiments of the invention and therefore should not be considered as limiting its scope. The invention will be described and illustrated with additional specificity and detail with the accompanying figures.

character list

• 1 ein Blockdiagramm eines Systems zur Prüfung von automatisierten Fahrsystemen der Stufe 3 (ADS) zeigt; und
• 2 einen technischen Aufbau zur Prüfung der Wahrnehmung mit synthetischen Daten zeigt.

These and other features, aspects and advantages of the present invention will be better understood when the following detailed description is read with reference to the accompanying figures, in which like characters represent like parts throughout the figures, wherein:

• 1 Figure 3 shows a block diagram of a system for testing level 3 automated driving systems (ADS); and
• 2 shows a technical setup for testing perception with synthetic data.

Those skilled in the art will understand that the elements in the figures are presented for simplicity and are not necessarily drawn to scale. For example, the flow charts illustrate the method in terms of the most important steps that help to better understand aspects of the present invention. In addition, it may be that one or several components of the device are represented in the figures by conventional symbols, and that the figures show only the specific details that are relevant for understanding the embodiments of the present invention, so as not to overload the figures with details that would be difficult for those skilled in the art familiar to the present specification are readily apparent.

DETAILED DESCRIPTION

For the purposes of promoting an understanding of the invention, reference will now be made to the embodiment illustrated in the figures and specific language will be used to describe the same. It should be understood, however, that no limitation on the scope of the invention is intended thereby, contemplating such changes and further modifications to the illustrated system, and such further applications of the inventive principles illustrated therein, as would normally occur to one skilled in the art to which the invention pertains.

It will be understood by those skilled in the art that the foregoing general description and the following detailed description are exemplary and explanatory of the invention and are not intended to be limiting.

When this specification refers to "an aspect," "another aspect," or the like, it means that a particular feature, structure, or characteristic described in connection with the embodiment is present in at least one embodiment of the present invention. Therefore, the phrases "in one embodiment," "in another embodiment," and similar phrases throughout this specification may or may not all refer to the same embodiment.

The terms "comprises," "including," or other variations thereof are intended to cover non-exclusive inclusion, such that a method or method that includes a list of steps includes not only those steps, but may also include other steps that are not expressly stated or pertaining to any such process or method. Likewise, any device or subsystem or element or structure or component preceded by "comprises...a" does not, without further limitation, exclude the existence of other devices or other subsystem or other element or other structure or other component or additional device or additional subsystems or additional elements or additional structures or additional components.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one skilled in the art to which this invention pertains. The system and examples given herein are intended to be illustrative only and not limiting.

Embodiments of the present invention are described in detail below with reference to the attached figures.

1 FIG. 12 shows a block diagram of a system (100) for testing level 3 automated driving systems (ADS). The system (100) comprises: a simulator (102) configured to validate the L3 ADS testing framework and examine autonomous driving capabilities; an array of cameras (104) connected to the simulator (102), the array of cameras (104) configured to capture images of the vehicles and the roads; a semantic segmentation unit (106) connected to the array of cameras (104) for assigning a label to each pixel in the image to classify the captured images; and a test case generation unit (108) connected to the semantic segmentation unit (106), the test case generation unit (108) being configured to: analyze the factors affecting the autonomous driving functions, including the ego speed of the vehicle, and that play a role in the test to be performed; generating parameters from the identified criteria in a second stage; executing a test run using a set of instructions and a variety of simulation tools when simulating the test situations; and Evaluating the test including evaluating the simulation using multiple criticality criteria in driving conditions including time to collision.

In another embodiment, the simulator is based on Car Learning to Act (CARLA), which includes large city formats, a large number of vehicle models, structures, pedestrians, street signs, and so on.

In another embodiment, CARLA is configured to focus on presenting three pathways to autonomous driving, including an example particular pipeline, end-to-end imitation learning, and end-to-end model reinforcement learning.

In another embodiment, the semantic segmentation unit treats multiple objects of the same class as a single unit.

In another embodiment, a set of test cases is created after evaluating the tests performed by the test case creation unit.

In another embodiment, the test case generation unit is further configured to explore the test evaluation, either exploring the environment further or applying existing knowledge in the exploration phase.

In another embodiment, the test case generation unit is also configured to change parameters using an e-greedy function to choose a random action that offers the best reward.

In another embodiment, selecting an action results in the value of a parameter being changed by increasing or decreasing, after changing the parameter a new test case is created with the new parameters and the cycle repeats.

2 illustrates a technical setup for testing perception with synthetic data. Vision is an important perceptual modality in automated driving and robotics that must deal with an open context. It is not possible to model every possible visual scene. Therefore, the most modern approach to computer vision is machine learning, a technique that automatically creates generalizations from examples. Perception is a function that takes in raw input signals from a sensor and computes semantic information as an output. As an example, consider an image processing function that takes an image as input and outputs an interpretation of the image content such as object detection, semantic segmentation, or 3D depth information (see 2 ).

In another embodiment, CARLA was created from the ground up to support the development, training, and validation of autonomous driving systems. In addition to the open source code and protocols, CARLA offers open, advanced resources (big city designs, buildings, vehicles) that have been developed for this purpose and can be used without restrictions. The simulation phase offers customizable details of sensor suites, weather conditions, full control of all static and dynamic actors, map generation and much more.

In another embodiment, Unreal Engine 4 (UE4) is a real-time 3D rendering engine and CARLA is a wrapper for UE4 that meets the following requirements: Generation of ground truth output, e.g. B.: Semantic segmentation, object boxes, RGB input image generation, programmatically controllable - necessary for test generation, contains free content (landscapes, objects, cars, behaviors), and is a popular academic tool supported by the following companies: Intel , Toyota, GM.

In another embodiment, semantic segmentation is the process of assigning a label to each pixel of the image. This is in stark contrast to classification, which assigns a single label to the entire image. In semantic segmentation, multiple objects of the same class are treated as a single entity.

In a further embodiment, an approach is proposed that generates critical concrete scenarios and test cases for automated driving functions using the example of an overtaking assistant on the basis of reinforcement learning optimization. A Q-learning approach is used, which automates the parameter generation for the test cases.

In another embodiment, the test case creation concepts consists of 7 steps, including analysis, test case creation, test run execution, test evaluation, exploration, parameter change, and saving of critical test cases. The first phase is the analysis, in which the factors that influence the autonomous driving functions, such as the vehicle's ego speed, are identified and that play a role in the test to be performed. In the second phase, the test case generation, parameters are generated from the identified criteria. The test is carried out purely simulatively using software-in-the-loop tests with simulation tools in the third step, in which the test situations are simulated. The fourth phase is the test evaluation, in which the simulation is evaluated against several criticality criteria under driving conditions, such as e.g. B. time to collision, is evaluated. In the second step, new test cases can be created after the evaluation. After the test evaluation, the environment is either further explored or the existing knowledge is applied in the exploration phase. This is done by the amplification algorithm in conjunction with the test evaluation. The sixth stage is parameter modification, which employs an E-Greedy algorithm to choose a random action that offers the best reward. Choosing an action changes the value of a parameter by increasing or decreasing it. After After the parameter change, a new test case is created with the new parameters and the cycle repeats. In addition, critical tests are saved in the Save Critical TestCases phase.

In another embodiment, the critical scenarios are described as potentially harmful scenarios for system design, security analysis, verification, or validation. It is of vital importance to discover unknown critical scenarios and then make them safe. A critical scenario in an ODD has two main components: initiating conditions and safety-critical operational events. Therefore, an unknown critical scenario could result from an unknown initiating circumstance or an unknown safety-critical operational situation.

Also, safety validation of SAE Level 3-5 systems in complex environments, including road and climate conditions, requires entirely new systems and cycles. Scenario-based strategies for testing and approving automated driving systems (ADS) under virtual test conditions are gaining momentum and are fast becoming a significant part of ADS verification and validation. The increased complexity of the systems and the costs of real tests lead to a dramatic development of the test effort. By using scenario and simulation-based approaches, this effort in terms of costs and time can be significantly reduced. Determining a suitable simulation framework and selecting suitable scenarios for the framework under test are the most tedious tasks. A data-driven strategy is used to study the security of the automated vehicles (AVs) to be broadcast, with scenarios having a significant impact. Through virtual and real safety validation, selected test conditions are used to quantitatively examine an AV. Certifiable driving data is used to describe various circumstances along the way, including the movements of road users, normal road and infrastructure layouts, and climate and light conditions. On the first level of reflection, the scenarios are presented subjectively, while on the second level of reflection, the scenarios are described quantitatively with the help of boundaries and models. The scenarios should cover a wide range of scenarios that an AV could experience in demonstrable real traffic. Accordingly, various circumstances are conceivable.

In another embodiment, the camera uses "semantic segmentation" to classify each object in the view by displaying it in a different color depending on the object class. So e.g. B. Pedestrians displayed in a different color than vehicles. This is done by pre-tagging each item in the scene (either at the beginning of the game or at spawns). This camera classifies any object in view by displaying it in a different color according to its label (e.g. pedestrians in a different color than vehicles). At the start of the simulation, each component of the scene is tagged. This happens when an actor is created. The objects are classified according to their common file path in the task. For example, meshes dropped into Unreal/CarlaUE4/Content/Static/Pedestrians are called Pedestrians.

In another embodiment, the scenarios presented in this report are intended to cover most of the scenarios that a vehicle may encounter in traffic. These scenario categories can then be used to construct suitable test cases for AV evaluation. In most situations, there are at least two other actors besides the ego vehicle. Many more interactions are possible when more than one additional actor is considered. Indeed, the number of actors involved multiplies the possibilities immensely. Some critical test situations are identified and a synthetic scene is generated within a defined ODD using the CARLA simulator.

Another aspect is that the transition to a higher degree of automation of driving brings with it a multitude of unexpected problems and occurrences. In order to overcome these difficulties, the number of functions in today's vehicles is increasing in terms of the complexity of the data processing and the equipment required. Testing the implemented solutions in reality is extremely expensive and takes up a large part of the day. Because of this, virtual simulation technologies for automotive testing have received a lot of positive press. These tools are used by OEMs to create a closed cycle of discover, calculate and act. The creation software is tried with simulated recognition information and the simulated action results are generated in terms of the software commands. This helps OEMs fine-tune vehicle blueprints before delivering prototypes. Early improvement results in lower costs and shorter lead times. CARLA, an open source simulator, is used to create synthetic images. The purpose of this project is to create a POC validation framework for using a CARLA simulator to test L3 ADS.

The figures and the preceding description give examples of embodiments. Those skilled in the art will understand that one or more of the elements described may well be combined into a single functional element. Alternatively, certain elements can be broken down into multiple functional elements. Elements from one embodiment may be added to another embodiment. For example, the order of the processes described herein may be changed and is not limited to the manner described herein. Additionally, the actions of a flowchart need not be performed in the order shown; Also, not all actions have to be carried out. Also, the actions that are not dependent on other actions can be performed in parallel with the other actions. The scope of the embodiments is in no way limited by these specific examples. Numerous variations are possible, regardless of whether they are explicitly mentioned in the description or not, e.g. B. Differences in structure, dimensions and use of materials. The scope of the embodiments is at least as broad as indicated in the following claims.

Advantages, other benefits, and solutions to problems have been described above with respect to particular embodiments. However, the benefits, advantages, problem solutions, and any component that may cause an advantage, benefit, or solution to occur or become more pronounced are not to be construed as a critical, required, or essential feature or component of any or all claims.

Reference List

100

A system for testing level 3 automated driving systems (ADS)

102

simulator

104

array of cameras

106

Semantic Segmentation Unit

108

test case generation unit

202

Image = pixel data of a camera

204

Framework for Computer Vision

206

Semantic Segmentation

208

Unreal engine 4/ CARLA (3D graphics)

210

synthetic image

212

basic truth

Claims (8)

A system for testing level 3 automated driving systems (ADS), the system comprising: a simulator configured to validate the L3 ADS exam framework and examine autonomous driving capabilities; a bank of cameras connected to the simulator, the bank of cameras configured to capture images of the vehicles and roads; a semantic segmentation unit connected to the array of cameras to assign a label to each pixel in the image to classify the captured images; and a test case generation unit connected to the semantic segmentation unit, the test case generation unit being configured to: Analysis of the factors influencing autonomous driving functions, including the vehicle's ego speed, and their role in conducting the test; in a second step, to create parameters from the determined criteria; executing a test run using a set of instructions and a variety of simulation tools when simulating the test situations; and Evaluate the test, including evaluating the simulation against multiple criticality criteria under driving conditions, including time to collision. system after claim 1 , where the simulator is based on Car Learning to Act (CARLA), which includes metropolitan formats, a large number of vehicle models, structures, pedestrians, street signs and so on.. system after claim 2 , with CARLA configured to focus on presenting three pathways to autonomous driving, including an exemplary particular pipeline, end-to-end imitation learning, and end-to-end model reinforcement learning. system after claim 1 , where the semantic segmentation unit treats multiple objects of the same class as a single unit. system after claim 1 , wherein a set of test cases is created after evaluating the tests performed by the test case generation unit. system after claim 1 , wherein the test case generation unit is further configured to explore the test evaluation, the environment is either further explored or existing knowledge is applied in the exploration phase. system after claim 1 , wherein the test case generation unit is further configured to change parameters using an e-greedy function to choose a random action that offers the best reward. system after claim 1 , where selecting an action involves changing the value of a parameter by increasing or decreasing it, following the parameter change a new test case is created using the new parameters and the cycle repeats again.

Images