CN114428998A - A method and system for integrated simulation test and evaluation of automatic driving system - Google Patents

Abstract

The invention relates to the technical field of automatic driving system testing, in particular to an integrated simulation testing and evaluating method and system of an automatic driving system. The invention integrates the automatic driving system test based on the scene and the automatic driving system test based on the traffic flow, fully exerts the advantages of the two test methods, improves the test efficiency, evaluates the automatic driving system from two aspects of the safety of the vehicle and the influence on the traffic safety, has more comprehensive evaluation angle and more accurate evaluation result.

Description

A method and system for integrated simulation test and evaluation of automatic driving system

technical field

The invention relates to the technical field of automatic driving system testing, in particular to a method and system for integrated simulation testing and evaluation of automatic driving systems.

Background technique

Traffic accidents bring serious harm to human beings. As the most random factor in the traffic environment, the driver is also the main cause of traffic accidents. For the problem of traffic accidents, self-driving cars are recognized as a way to avoid or mitigate human errors. Effective way of traffic accident. The automatic driving system is the software used to realize the driving control function on the self-driving car. In order to ensure the accuracy and safety of the driving of the self-driving car, it is necessary to test and evaluate the self-driving system on the self-driving car before it goes on the road. Especially for high-level self-driving cars, that is, self-driving cars of SAE L3 level and above, the driver no longer always has the right to control the motion of the vehicle during operation, and the automatic driving system becomes the main body of driving environment monitoring and vehicle operation control.

At present, the commonly used simulation test methods for autonomous driving systems mainly start from standard regulations, design scene fragments in advance, and then evaluate the safety of autonomous driving systems based on passability. However, because the randomness of the traffic environment and the interaction between traffic participants are ignored in this test method, it is very different from the real environment when the vehicle is driving, and it is difficult to meet the requirements of test scene coverage, and the existing methods cannot test and evaluate. The impact of the automatic driving system on the traffic environment, such as traffic efficiency and traffic conflict safety, leads to inaccurate evaluation results of the automatic driving system.

SUMMARY OF THE INVENTION

The present invention is intended to provide an integrated simulation testing and evaluation method for an automatic driving system, so as to improve the accuracy of the testing and evaluation of the automatic driving system.

The integrated simulation test and evaluation method of the autonomous driving system in this solution includes the following steps:

Step 1, establish a vehicle dynamics model of the autonomous vehicle and store it in the dynamic model library, establish a scene model and store it in the test scene library, establish a test case and store it in the test case library, establish a traffic flow model and store it in in the traffic flow library;

Also includes the following steps:

Step 2, establishing a mapping relationship between the traffic flow model and the scene model according to the preset map information in the traffic flow model and the scene model;

Step 3, obtaining test requirements, establishing a simulation computing center for simulation testing, and scheduling the dynamic model library, test scenario library, test case library and traffic flow library according to the acquired test requirements during simulation;

Step 4, after scheduling, simulate according to the simulation configuration data in the test requirements, store the simulation data, and collect the simulation results in the simulation process;

Step 5: Carry out a simulation evaluation of the automatic driving system according to the used test scene library or traffic flow library, and generate an evaluation report.

The beneficial effects of this program are:

By adding test scenarios, test cases, and traffic flows in the simulation process, the specific interaction scenarios of the autonomous vehicle in the actual driving process are added to the simulation, and the automatic driving system test based on the scene and the automatic driving system test based on the traffic flow are carried out. Integration, give full play to the advantages of the two test methods, improve the test efficiency, and make the simulation results more accurate. At the same time, the automatic driving system is evaluated from the perspectives of the vehicle's own safety and the impact on traffic safety. The evaluation angle is more comprehensive and the evaluation results are more accurate. precise.

Further, in the step 1, the test scene elements, the complexity of the test scene and the passability conditions of the test scene are combined to form a test case, and the test case library includes an ODD-based test case library and a functional safety-based test case. Use case library, test case library based on expected functional safety, and test case library based on expert experience, the complexity is the weighted value of scene element complexity and driving task complexity, and the scene element complexity is based on the test scene. The types of static elements and dynamic elements are quantified, and the driving task complexity is quantified according to the types of driving tasks in the test scene per unit time.

The beneficial effects are: test cases are formed from multiple aspects, and multiple test cases under different conditions are formed, so that the test scene is more complete, closer to the actual driving situation, and the accuracy of the simulation test is improved.

Further, in the step 4, when using the test scene library, use the test function, test target, the number of test scenes and the distribution characteristics of the test scene as simulation data, when using the traffic flow library, use the map file, traffic flow density as the simulation data. , vehicle distribution, and driver style distribution as simulation data.

The beneficial effects are: when using different libraries for simulation, according to different data as simulation data, when evaluating in different aspects, the evaluation results are more accurate.

Further, in the step 5, when the test scenario library is used for simulation, the complexity, coverage and passability of the test case are scored according to the first model to score the safety level of the automatic driving system, and the first model is:

其中：dc为测试用例的覆盖度，C为单个测试场景的复杂度，T为单个测试用例的通过性；

Among them: dc is the coverage of test cases, C is the complexity of a single test scenario, and T is the passability of a single test case;

The coverage of the test case is:

Among them: m _odd is the total number of test cases in the test case library based on ODD, m _f is the total number of test cases in the test case library based on functional safety, m _s is the total number of test cases in the test case library based on expected functional safety, m _e is the total number of test cases in the test case library based on expert experience; n _odd is the total number of real test cases derived from the ODD-based test case library in the test cases actually tested, and n _f is the source in the test cases actually tested The total number of real test cases in the test case library based on functional safety, n _s is the total number of real test cases derived from the test case library based on expected functional safety in the test cases actually tested, and n _e is the test cases in the actual test The total number of real test cases derived from the test case library based on expert experience.

The beneficial effects are as follows: the first model is used to score the safety level of the automatic driving system, and the safety of the automatic driving vehicle is quantitatively evaluated, thereby improving the accuracy of the vehicle safety evaluation.

Further, in the step 5, when the traffic flow library is used for simulation, the state information, trajectory information and traffic flow density information of the traffic participants are used to simulate and evaluate the traffic conflict of the automatic driving system.

The beneficial effects are: from the traffic angle of the actual driving of the vehicle, the safety of the automatic driving system is evaluated, the integrity of the evaluation angle is improved, and the vehicle safety evaluation is made more accurate.

Further, in the step 5, the traffic conflict is represented by the substitute variables of headway, vehicle distance and collision time. When there is a traffic conflict, the occurrence frequency of the traffic conflict is collected for simulation evaluation, and the occurrence frequency is the number of conflicts and the mileage. The ratio of the number of traffic conflicts is extracted, and the location and time of the traffic conflict are extracted, and the time distribution of the traffic conflict and the spatial distribution of the traffic conflict are formed by the location and time.

The beneficial effects are: obtaining traffic conflicts through corresponding parameters, and performing simulation evaluation according to the traffic conflicts, thereby improving the accuracy and comprehensiveness of the simulation evaluation.

Further, step 6 is also included. When the test scene library and the traffic flow library are used for simulation at the same time, weighted comprehensive evaluation is performed on the simulation results in the two cases to obtain a comprehensive performance safety score value of the safety of the autonomous vehicle.

The beneficial effects are: comprehensive evaluation of the simulation from two perspectives of scene and traffic flow, so that the overall evaluation of the automatic driving system is more reliable.

The integrated simulation test and evaluation system of automatic driving system, including simulation interaction module, modeling module, database module, processing module, scheduling module, sensor module and calculation module;

The simulation interaction module is used to obtain the test requirements and send them to the processing module;

a modeling module for establishing a vehicle dynamics model of the autonomous vehicle, the modeling module for establishing a scene model, the modeling module for establishing a test case, and the modeling module for establishing a traffic flow model;

Database module, including dynamic model library, test scene library, test case library and traffic flow library for storing the required test;

The processing module is used to acquire the vehicle dynamics model and store it in the dynamic model library in the database module, the processing module is used to acquire the scene model and store it in the test scene library in the database module, and the processing module acquires the test The use case is stored in the test case library of the database module, and the processing module obtains the traffic flow model and stores it in the traffic flow library of the database module;

The scheduling module is used to schedule the dynamic model library, the test scene library, the test case library and the traffic flow library according to the test requirements;

The sensor module is used to detect the simulation results in the simulation test process;

The calculation module is used for receiving the simulation result obtained by the processing module from the sensor module, and calculates the simulation evaluation according to the simulation result, and generates an evaluation report.

Description of drawings

Fig. 1 is the principle block diagram of the integrated simulation test and evaluation system of the automatic driving system in the first embodiment of the present invention;

FIG. 2 is a flowchart of a method for integrated simulation testing and evaluation of an automatic driving system in Embodiment 2 of the present invention;

FIG. 3 is a schematic block diagram of the calculation of the complexity of the test scene in the integrated simulation test and evaluation method of the automatic driving system according to the second embodiment of the present invention;

FIG. 4 is a schematic block diagram of test case generation in the integrated simulation test and evaluation method for an automatic driving system in Embodiment 2 of the present invention;

5 is a schematic diagram of the data mapping relationship between the models of the computing center in the automatic driving system integrated simulation test and evaluation method in Embodiment 2 of the present invention;

FIG. 6 is a schematic diagram of a test execution and evaluation process simulated under a scenario model in the automatic driving system integrated simulation test and evaluation method according to Embodiment 2 of the present invention;

FIG. 7 is a schematic diagram of a test execution and evaluation process simulated under a traffic flow model in the automatic driving system integrated simulation test and evaluation method according to Embodiment 2 of the present invention.

Detailed ways

The following is further described in detail through specific embodiments.

Example 1

The integrated simulation test and evaluation system of the autonomous driving system, as shown in Figure 1, includes a simulation interaction module, a modeling module, a database module, a processing module, a scheduling module, a sensor module and a computing module.

The simulation interaction module is used to obtain the test requirements and send them to the processing module. The test requirements are represented by the simulation configuration parameters that need to be set during the test process. The simulation interaction module can obtain the test requirements through the existing keyboard, touch screen and other input peripherals to The existing display shows the simulation configuration parameters for the desired settings.

The modeling module is used to establish the vehicle dynamics model of the autonomous vehicle, the modeling module is used to establish the scene model, the modeling module is used to establish the test case, and the modeling module is used to establish the traffic flow model. The software installed on the PC host or notebook computer is used to establish various models, and the established software and existing modeling methods are used to establish various models.

The database module can be built with existing database software, such as Oracle software. The database module includes a dynamic model library, a test scenario library, a test case library and a traffic flow library for storing the required testing.

The processing module is used to acquire the vehicle dynamics model and store it in the dynamic model library in the database module, the processing module is used to acquire the scene model and store it in the test scene library in the database module, and the processing module acquires the test case and stores it in the database In the test case library of the module, the processing module obtains the traffic flow model and stores it in the traffic flow library of the database module; the processing module establishes a mapping relationship between the traffic flow model and the scene model according to the map information, and the map information is the establishment of the traffic flow model and the scene model. For example, the processing module will have the same static map scene model file and traffic flow model file, and establish the mapping relationship with the same file name prefix.

The scheduling module is used to schedule the dynamic model library, the test scenario library, the test case library and the traffic flow library according to the test requirements. Both are required to perform model library scheduling.

The sensor module is used to detect the simulation results in the simulation test process. The simulation results include the corresponding parameters simulated under the test scene and traffic flow conditions. The sensor module can use the sensors required in the existing automatic driving system, such as ranging sensors and images. sensors, etc.

The calculation module is used to receive the simulation result obtained by the processing module from the sensor module, calculate the simulation evaluation according to the simulation result, generate an evaluation report, and calculate the simulation evaluation according to the parameters in the obtained simulation result according to the preset calculation formula, such as the method in the second embodiment The evaluation report includes the scores or graphs of the simulation tests for the vehicle dynamics model under different conditions.

In the system of this embodiment, the vehicle is modeled through the modeling module, and the modeled vehicle model is subjected to simulation tests of test scenarios and traffic flow under different test cases, and is automatically dispatched after mapping during the simulation process. The state of the autonomous vehicle driving under different conditions is simulated and tested with different library maps, so that the test conditions are closer to the actual driving situation, the test simulation is closer to the actual driving environment, the accuracy of the test simulation is improved, and the automatic detection can be detected in advance. Problems with the driving system.

Embodiment 2

The automatic driving system integrated simulation test and evaluation method, using the automatic driving system integrated simulation test and evaluation system of the first embodiment, as shown in Figure 2, includes the following steps:

Step 1. The vehicle dynamics model of the autonomous vehicle is established through the modeling module and stored in the dynamics model library. The vehicle dynamics model of the autonomous vehicle is established according to the actual vehicle to be tested. The establishment of the vehicle dynamics module uses The existing vehicle dynamics software and the existing technology are carried out, and are not repeated here.

Establish a scene model and store it in the test scene library. Based on the design operation domain (ODD), functional safety, expected functional safety and expert experience stipulated by the existing standards of the automatic driving system, the scene model is established respectively. The format of the scene model storage is open-x Format, the scene model is built using the existing VTD software, such as the scene model of the urban expressway exit scene, and the test scene library includes the scene sub-library based on ODD, the scene sub-library based on functional safety, and the scene sub-library based on predetermined functional safety. And a sub-library of scenarios based on expert experience.

As shown in Figure 4, test cases are established and stored in the test case library, and the test scene elements, the complexity of the test scene and the passability conditions of the test scene are combined to form a test case. The test case library includes ODD-based test cases. Library, test case library based on functional safety, test case library based on expected functional safety, and test case library based on expert experience, as shown in Figure 3, the complexity is the weighted value of scene element complexity and driving task complexity , for example, the weighted value is the sum of the scene element complexity A multiplied by the weight w1 and the driving task complexity multiplied by the weight w2. The weights w1 and w2 are set according to the simulation test requirements, such as w1=4 and w2=5, the scene elements are complex The degree of complexity is quantified according to the types of components of static elements and dynamic elements in the test scene, such as the sum of the types of components of static elements and those of dynamic elements, and the complexity of driving tasks is quantified according to the types of driving tasks in the test scene per unit time. For example, the number P of driving task types in T unit time is the driving task complexity.

Establish a traffic flow model and store it in the traffic flow database. The establishment of the traffic flow model is based on the design and operation domain, functional safety, expected functional safety and expert experience of the autonomous driving system. The traffic flow model can be carried out with the existing VISSIM software. Established, the traffic flow model is the average traffic volume Q on a certain route, the average vehicle speed V, the average density K, the trajectory information of traffic participants, and the state information of traffic participants. Models and test cases are created and stored in the database module.

Step 2, through the processing module, according to the preset map information in the traffic flow model and the scene model, the map information is uniformly preset when the traffic flow model and the scene model are established, as shown in FIG. For example, the scene model file and the traffic flow model file with the same static map are used to establish the mapping relationship with the same file name prefix.

Step 3: Obtain test requirements through the simulation interaction module, establish a simulation computing center for simulation testing, and perform simulation tests on the dynamic model library, test scene library, test case library and traffic flow library through the scheduling module during simulation according to the acquired test requirements. Scheduling, that is, according to whether the test requirements are scenario-based testing, traffic flow-based testing, or both scenarios and traffic flow, the model library in the simulation software is scheduled. For example, the test requirements are for Type A autonomous vehicles. During the scene-based test, the vehicle dynamics model of the corresponding model is retrieved from the dynamics model library, the vehicle dynamics software and the scene simulation software are opened, and the API interface communication connection between the vehicle dynamics software and the scene simulation software is established.

Step 4: After scheduling, the processing module performs simulation according to the simulation configuration data in the test requirements, stores the simulation data, and collects the simulation results during the simulation process. The simulation results include the corresponding parameters obtained by simulation under the test scene and traffic flow conditions. . When using the test scene library, use the test function, test target, number of test scenes and distribution characteristics of test scenes as simulation data; when using the traffic flow library, use the map file, traffic flow density, vehicle distribution, and driver style as the simulation data. The distribution is simulation data, the traffic flow density is the number of vehicles passing through per unit time, the vehicle distribution is the proportion of cars, trucks, and two-wheelers in the total, the driver style distribution includes aggressive, normal, conservative, map files Map information in traffic flow models and scene models.

Step 5, the calculation module performs the simulation evaluation of the automatic driving system according to the used test scene library or traffic flow library, and generates an evaluation report, as shown in Figure 6, when the test scene library is used for simulation, the complexity of the test case is calculated. , coverage and passability to score the safety level of the automatic driving system according to the first model, the first model is:

其中：dc为测试用例的覆盖度，C为单个测试场景的复杂度，T为单个测试用例的通过性；

Among them: dc is the coverage of test cases, C is the complexity of a single test scenario, and T is the passability of a single test case;

The coverage of the test case is:

Among them: m _odd is the total number of test cases in the test case library based on ODD, m _f is the total number of test cases in the test case library based on functional safety, m _s is the total number of test cases in the test case library based on expected functional safety, m _e is the total number of test cases in the test case library based on expert experience; n _odd is the total number of real test cases derived from the ODD-based test case library in the test cases actually tested, and n _f is the source in the test cases actually tested The total number of real test cases in the test case library based on functional safety, n _s is the total number of real test cases derived from the test case library based on expected functional safety in the test cases actually tested, and n _e is the test cases in the actual test The total number of real test cases derived from the test case library based on expert experience, n _odd , n _f , _ns and _ne are from the simulation configuration parameters in the test requirements.

As shown in Figure 7, when the traffic flow library is used for simulation, the state information, trajectory information and traffic flow density information of traffic participants are used to simulate and evaluate the traffic conflict of the automatic driving system to replace the variable headway, vehicle headway, and traffic flow. The distance and collision time represent traffic conflicts. When there are traffic conflicts, the frequency of traffic conflicts is collected for simulation evaluation. The frequency is the ratio of the number of conflicts to the number of miles traveled (CR), and the location and time of the traffic conflict are extracted. , and the time distribution of traffic conflicts and the spatial distribution of traffic conflicts are formed by location and time. The time distribution is a two-dimensional map: the abscissa of the time coordinate is the time period within a day, and the ordinate is the conflict frequency; the spatial distribution is a three-dimensional map: The horizontal and vertical coordinates are the global geographic coordinates, and the frequency of traffic conflicts is displayed in the form of a heat map.

Step 6: When the test scene library and the traffic flow library are used for simulation at the same time, the weighted comprehensive evaluation is performed on the simulation results in the two cases, and the comprehensive performance safety score value of the safety of the autonomous vehicle is obtained, which is expressed as:

Safety _all = CR×K ₁ +Safety×K ₂ , where CR is the ratio of the number of conflicts to the mileage obtained based on the traffic flow simulation test, K1 is the weight value, and the specific value of K1 is set according to actual needs, Safety is the number of security levels obtained based on the scene simulation test, K2 is the corresponding weight value, and the specific value of K2 is set according to actual needs.

Because under conventional thinking, self-driving cars are considered to have a high safety factor because they can avoid human subjective negligence or mistakes. Therefore, when evaluating the self-driving system, it generally meets the requirements of the corresponding standards according to the self-driving system, that is, It is considered that the safety of the automatic driving system meets the requirements. In the method of this embodiment, by setting test cases, test scenarios and traffic flow, it is used for the performance simulation evaluation of the automatic driving system on the self-driving car, the environment and traffic flow in actual driving can be considered, and the influence of the actual driving environment and traffic flow can be taken into account, and it can be used from multiple dimensions The test is carried out on the test, and the evaluation is made according to the results after the test, which improves the test accuracy and the evaluation accuracy.

The above descriptions are only embodiments of the present invention, and common knowledge such as well-known specific structures and characteristics in the solution are not described too much here. It should be pointed out that for those skilled in the art, some modifications and improvements can be made without departing from the structure of the present invention. These should also be regarded as the protection scope of the present invention, and these will not affect the implementation of the present invention. Effectiveness and utility of patents. The scope of protection claimed in this application shall be based on the content of the claims, and the descriptions of the specific implementation manners in the description can be used to interpret the content of the claims.

Claims (8)

1. An integrated simulation test and evaluation method for an automatic driving system, comprising the following steps: Step 1, establish a vehicle dynamics model of the autonomous vehicle and store it in the dynamic model library, establish a scene model and store it in the test scene library, establish a test case and store it in the test case library, establish a traffic flow model and store it in In the traffic flow library; it is characterized in that: it also comprises the following steps: Step 2, establishing a mapping relationship between the traffic flow model and the scene model according to the preset map information in the traffic flow model and the scene model; Step 3, obtaining test requirements, establishing a simulation computing center for simulation testing, and scheduling the dynamic model library, test scenario library, test case library and traffic flow library according to the acquired test requirements during simulation; Step 4, after scheduling, simulate according to the simulation configuration data in the test requirements, store the simulation data, and collect the simulation results in the simulation process; Step 5: Carry out a simulation evaluation of the automatic driving system according to the used test scene library or traffic flow library, and generate an evaluation report. 2. The automatic driving system integrated simulation test and evaluation method according to claim 1, characterized in that: in the step 1, the elements of the test scene, the complexity of the test scene and the passability conditions of the test scene are combined into an integrated form Test cases, the test case library includes an ODD-based test case library, a functional safety-based test case library, an expected functional safety-based test case library, and an expert experience-based test case library, and the complexity is The weighted value of the complexity of the scene elements and the complexity of the driving task, the complexity of the scene elements is quantified according to the types of the components of the static elements and the dynamic elements in the test scene, and the complexity of the driving tasks is based on the types of driving tasks in the test scene per unit time. quantify. 3. The automatic driving system integrated simulation test and evaluation method according to claim 2, characterized in that: in the step 4, when using the test scene library, the test function, test target, the number of test scenes and the test The distribution characteristics of the scene are simulation data. When using the traffic flow library, map files, traffic flow density, vehicle distribution, and driver style distribution are used as simulation data. 4. The automatic driving system integrated simulation test and evaluation method according to claim 3, characterized in that: in the step 5, when using the test scene library for simulation, the complexity, coverage and pass of the test case are compared. The safety level of the automatic driving system is scored according to the first model, and the first model is:

Among them: dc is the coverage of test cases, C is the complexity of a single test scenario, and T is the passability of a single test case; The coverage of the test case is:

Among them: m _odd is the total number of test cases in the test case library based on ODD, m _f is the total number of test cases in the test case library based on functional safety, m _s is the total number of test cases in the test case library based on expected functional safety, m _e is the total number of test cases in the test case library based on expert experience; n _odd is the total number of real test cases derived from the ODD-based test case library in the test cases actually tested, and n _f is the source in the test cases actually tested The total number of real test cases in the test case library based on functional safety, n _s is the total number of real test cases derived from the test case library based on expected functional safety in the test cases actually tested, and n _e is the test cases in the actual test The total number of real test cases derived from the test case library based on expert experience. 5. The automatic driving system integrated simulation test and evaluation method according to claim 4, characterized in that: in the step 5, when using the traffic flow library for simulation, the state information, trajectory information and Traffic flow density information is used to simulate and evaluate traffic conflicts in autonomous driving systems. 6. The integrated simulation test and evaluation method for an automatic driving system according to claim 5, characterized in that: in the step 5, the traffic conflict is represented by the alternative variables headway distance, vehicle distance and collision time, and when there is a traffic conflict When the traffic conflict occurs, the frequency of occurrence of traffic conflict is collected for simulation evaluation, and the occurrence frequency is the ratio of the number of conflicts to the number of miles traveled. spatial distribution. 7. The automatic driving system integrated simulation test and evaluation method according to claim 6, further comprising step 6, when simultaneously using the test scene library and the traffic flow library for simulation, the simulation in both cases is performed. The results are weighted and comprehensively evaluated, and the comprehensive performance safety score value of the safety of the autonomous vehicle is obtained. 8. An integrated simulation testing and evaluation system for an autonomous driving system, characterized in that it includes a simulation interaction module, a modeling module, a database module, a processing module, a scheduling module, a sensor module and a computing module; The simulation interaction module is used to obtain the test requirements and send them to the processing module; a modeling module for establishing a vehicle dynamics model of the autonomous vehicle, the modeling module for establishing a scene model, the modeling module for establishing a test case, and the modeling module for establishing a traffic flow model; Database module, including dynamic model library, test scene library, test case library and traffic flow library for storing the required test; The processing module is used to acquire the vehicle dynamics model and store it in the dynamic model library in the database module, the processing module is used to acquire the scene model and store it in the test scene library in the database module, and the processing module acquires the test The use case is stored in the test case library of the database module, and the processing module obtains the traffic flow model and stores it in the traffic flow library of the database module; The scheduling module is used to schedule the dynamic model library, the test scene library, the test case library and the traffic flow library according to the test requirements; The sensor module is used to detect the simulation results in the simulation test process; The calculation module is used for receiving the simulation result obtained by the processing module from the sensor module, and calculates the simulation evaluation according to the simulation result, and generates an evaluation report.

Images