CN112835790A

CN112835790A - Test method, device, equipment and medium for automatic driving software

Info

Publication number: CN112835790A
Application number: CN202110088625.8A
Authority: CN
Inventors: 任静; 马广志
Original assignee: Beijing Co Wheels Technology Co Ltd
Current assignee: Beijing Co Wheels Technology Co Ltd
Priority date: 2021-01-22
Filing date: 2021-01-22
Publication date: 2021-05-25

Abstract

The disclosure relates to a method, an apparatus, a device and a medium for testing automatic driving software. The test method of the automatic driving software comprises the following steps: acquiring problem data in the running process of an automatic driving vehicle, wherein the problem data comprises running data of the vehicle in an emergency control intervention state; generating a test case for testing the automatic driving software according to the problem data; and testing the automatic driving software through the test case to obtain a test result. According to the embodiment of the disclosure, real vehicle test data can be introduced into a simulation test of the automatic driving software, namely, a test case for testing the automatic driving software is generated by utilizing problem data in the automatic driving process of the vehicle, the automatic driving software is tested through the test case, and the automatic driving software can be comprehensively tested.

Description

Test method, device, equipment and medium for automatic driving software

Technical Field

The present disclosure relates to the field of automatic driving technologies, and in particular, to a method, an apparatus, a device, and a medium for testing automatic driving software.

Background

With the rapid development of vehicle technology and electronic technology, autonomous vehicles are increasingly emerging in people's lives. Autonomous vehicles typically implement autonomous driving of the vehicle through autonomous driving software.

In order to ensure the safety of automatic driving, the automatic driving software is often required to be tested in the development and use processes of the automatic driving software. Currently, there are three main ways to test the autopilot software: simulation test, closed field test and open environment test. Because the test cost of the closed field test and the open environment test is relatively high, the problem is mainly found by the automatic driving software through a simulation test.

However, the automatic driving environment of the vehicle is very complex, the simulation scenes which can be designed by testers are limited, and the automatic driving software cannot be comprehensively tested.

Disclosure of Invention

In order to solve the technical problems described above or at least partially solve the technical problems, the present disclosure provides a method, an apparatus, a device, and a medium for testing autopilot software.

In a first aspect, the present disclosure provides a method for testing automatic driving software, including:

acquiring problem data in the running process of an automatic driving vehicle, wherein the problem data comprises running data of the vehicle in an emergency control intervention state;

generating a test case for testing the automatic driving software according to the problem data;

and testing the automatic driving software through the test case to obtain a test result.

In some embodiments, the driving data comprises at least one of dynamics data of the vehicle, environmental data of an environment in which the vehicle is located, and behavior data of a driver of the vehicle.

In some embodiments, the obtaining of problem data during driving of the autonomous vehicle includes:

and receiving the problem data sent by the automatic driving vehicle in real time.

In some embodiments, the issue data includes historical travel data of the autonomous vehicle that meets issue data screening criteria.

In some embodiments, prior to said obtaining problem data during travel of the autonomous vehicle, the method further comprises:

receiving historical driving data sent by the autonomous vehicle;

wherein, the acquiring of the problem data in the driving process of the automatic driving vehicle comprises the following steps:

and screening out the problem data which meets the problem data screening condition from the historical driving data.

In some embodiments, generating a test case for testing the autopilot software based on the issue data includes:

generating a parameter combination according to a preset parameter combination strategy and each parameter in the problem data;

and respectively setting parameter threshold values for each parameter in the parameter combination to obtain the test case, wherein the parameter threshold values comprise at least one threshold value determined according to the performance of the automatic driving software.

In some embodiments, after generating a test case for testing the autopilot software based on the issue data, the method further comprises:

marking the problem data according to the scene element labels corresponding to the parameters in the problem data to obtain target scene element labels corresponding to the problem data;

constructing a test scene according to the test case and the target scene element label;

wherein, through the test case, the automatic driving software is tested to obtain a test result, and the test result comprises:

and under the test scene, testing the automatic driving software through the test case to obtain a test result.

In some embodiments, the scene element tags include a primary scene element tag including an element type tag and a secondary scene element tag including an element attribute tag.

In some embodiments, before generating the test case for testing the autopilot software based on the issue data, the method further comprises:

determining the historical problem data amount with the target scene element label;

generating a test case for testing the automatic driving software according to the problem data, wherein the test case comprises:

and under the condition that the historical problem data amount is equal to a preset amount, generating the test case according to the problem data.

inputting the problem data into a preset data recognition model to obtain a data recognition result;

and under the condition that the data identification result indicates that the problem data is non-misjudged data, generating the test case according to the problem data.

In some embodiments, after the inputting the question data into a preset data recognition model to obtain a data recognition result, the method further includes:

and deleting the problem data under the condition that the data identification result indicates that the problem data is misjudged data.

In some embodiments, after the automatic driving software is tested through the test case to obtain a test result, the method further includes:

carrying out diagnosis analysis on the test result to obtain diagnosis information;

and generating a test report according to the test result and the diagnosis information.

In a second aspect, the present disclosure provides a device for testing autopilot software, the device comprising:

the system comprises a first acquisition module, a second acquisition module and a third acquisition module, wherein the first acquisition module is configured to acquire problem data in the driving process of the automatic driving vehicle, and the problem data comprises driving data of the vehicle in an emergency control intervention state;

the first generation module is configured to generate a test case for testing the automatic driving software according to the problem data;

and the software testing module is configured to test the automatic driving software through the test case to obtain a test result.

In some embodiments, the first obtaining module is specifically configured to:

In some embodiments, the apparatus further comprises:

a data receiving module configured to receive historical driving data transmitted by the autonomous vehicle;

wherein the first obtaining module is specifically configured to:

In some embodiments, the first generating module is specifically configured to:

In some embodiments, the apparatus further comprises:

the data marking module is configured to mark the problem data according to the scene element labels corresponding to the parameters in the problem data to obtain target scene element labels corresponding to the problem data;

the scene construction module is configured to construct a test scene according to the test case and the target scene element label;

wherein the software testing module is specifically configured to:

In some embodiments, the apparatus further comprises:

a data volume determination module configured to determine a historical problem data volume having the target scene element label;

wherein the first generating module is specifically configured to:

In some embodiments, the apparatus further comprises:

the data identification module is configured to input the problem data into a preset data identification model to obtain a data identification result;

wherein the first generating module is specifically configured to:

In some embodiments, the apparatus further comprises:

and the data deleting module is configured to delete the problem data under the condition that the data identification result indicates that the problem data is misjudged data.

In some embodiments, the apparatus further comprises:

the diagnosis analysis module is configured to perform diagnosis analysis on the test result to obtain diagnosis information;

and the second generation module is configured to generate a test report according to the test result and the diagnosis information.

In a third aspect, the present disclosure provides a test apparatus for autopilot software, comprising:

a processor;

a memory for storing executable instructions;

the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method for testing the automatic driving software according to the first aspect.

In a fourth aspect, the present disclosure provides a computer-readable storage medium storing a computer program which, when executed by a processor, causes the processor to implement the method of testing autopilot software according to the first aspect.

Compared with the prior art, the technical scheme provided by the embodiment of the disclosure has the following advantages:

according to the method, the device, the equipment and the medium for testing the automatic driving software, real vehicle test data can be introduced into a simulation test of the automatic driving software, namely, a test case for testing the automatic driving software is generated by utilizing problem data in the automatic driving process of a vehicle, the automatic driving software is tested through the test case, and the automatic driving software can be comprehensively tested.

Drawings

The above and other features, advantages and aspects of various embodiments of the present disclosure will become more apparent by referring to the following detailed description when taken in conjunction with the accompanying drawings. Throughout the drawings, the same or similar reference numbers refer to the same or similar elements. It should be understood that the drawings are schematic and that elements and features are not necessarily drawn to scale.

Fig. 1 is a schematic flow chart of a testing method of automatic driving software according to an embodiment of the present disclosure;

FIG. 2 is a schematic flow chart illustrating another method for testing autopilot software according to an exemplary embodiment of the present disclosure;

fig. 3 is a schematic structural diagram of a testing apparatus for automatic driving software according to an embodiment of the present disclosure;

fig. 4 is a schematic structural diagram of a test device for autopilot software according to an embodiment of the present disclosure.

Detailed Description

Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. While certain embodiments of the present disclosure are shown in the drawings, it is to be understood that the present disclosure may be embodied in various forms and should not be construed as limited to the embodiments set forth herein, but rather are provided for a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the disclosure are for illustration purposes only and are not intended to limit the scope of the disclosure.

It should be understood that the various steps recited in the method embodiments of the present disclosure may be performed in a different order, and/or performed in parallel. Moreover, method embodiments may include additional steps and/or omit performing the illustrated steps. The scope of the present disclosure is not limited in this respect.

The term "include" and variations thereof as used herein are open-ended, i.e., "including but not limited to". The term "based on" is "based, at least in part, on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Relevant definitions for other terms will be given in the following description.

It should be noted that the terms "first", "second", and the like in the present disclosure are only used for distinguishing different devices, modules or units, and are not used for limiting the order or interdependence relationship of the functions performed by the devices, modules or units.

It is noted that references to "a", "an", and "the" modifications in this disclosure are intended to be illustrative rather than limiting, and that those skilled in the art will recognize that "one or more" may be used unless the context clearly dictates otherwise.

The test method of the automatic driving software provided by the embodiment of the disclosure can be applied to test the automatic driving software, specifically, real vehicle test data can be introduced into a simulation test, the real vehicle test can be performed again after the simulation test, and the real vehicle test data can be introduced into the simulation test again, so that a closed-loop test of the automatic driving software can be formed, and the problem that the automatic driving software cannot be tested in all directions due to limited simulation scenes which can be designed by a tester is solved; meanwhile, tests in various complex automatic driving environments can be achieved, and cost for finding real vehicle test problems is reduced.

The following describes an exemplary method, apparatus, device, and medium for testing the autopilot software according to the embodiments of the present disclosure with reference to fig. 1 to 4.

Fig. 1 shows a flow chart of a test method of the automatic driving software. As shown in fig. 1, the test method of the automatic driving software includes the following steps.

And S110, acquiring problem data in the running process of the automatic driving vehicle.

The problem data may include driving data of the vehicle in an emergency control intervention state, that is, the problem data may be real vehicle test data.

Wherein an emergency control intervention (disengaged or intervening) state may also be understood as an autonomous vehicle being disengaged from an autonomous mode and entering a human control state. Under the artificial control state, the automatic driving vehicle does not travel based on automatic driving software, including the control of vehicle states such as forward traveling, steering and speed regulation, but realizes vehicle control based on the operation of a driver (such as a tester).

For example, the timing of the autonomous vehicle to exit the autonomous mode may include: the autopilot technology detects a fault and fails, or the tester exits the autopilot mode based on safety criteria, or exits the autopilot mode in consideration of vehicle safety, on-board personnel safety, public safety, and the like. For example, when the automatic driving software cannot acquire the pose information, the speed information, the acceleration information and other vehicle running related state information of the vehicle, the tester controls the vehicle to run; or, in the driving process of the automatic driving vehicle, if a driver finds that people, animals or other vehicles suddenly penetrate the front of the vehicle, the vehicle can be immediately and autonomously controlled to decelerate or stop; or in other emergency control intervention states.

For example, when the autonomous vehicle travels in the autonomous driving state, if the driver finds that another vehicle suddenly penetrates in front of the autonomous vehicle, the driver may immediately control the vehicle to decelerate or stop.

The driving data is data related to a vehicle state within a driving time period corresponding to the emergency control intervention, and may include environmental data outside the vehicle, behavior and action data of a driver of the vehicle, dynamics data of the vehicle, and the like, which are described in detail below.

In the step, problem data sent by the automatic driving vehicle can be acquired in real time; or after the automatic driving vehicle completes a single test, the problem data in the single test process is acquired at one time, or after the vehicle which is on line runs for a period of time, the problem data in the period of time is acquired at one time. The data acquisition mode may be, for example, an artificial copy mode, or a wireless transmission mode after the problem data is packaged, and the embodiment of the present disclosure is not limited.

And S120, generating a test case for testing the automatic driving software according to the problem data.

The Test Case (Test Case) is used for describing a Test task of the automatic driving software and embodying a Test scheme, a method, a technology and a strategy; the contents of the test case comprise a test target, a test environment, input data, a test step, an expected result, a test script and the like, and finally form a document.

In the step, a test case for testing the automatic driving software can be generated based on the driving data of the vehicle in the emergency control intervention state, so that the real vehicle test data is introduced into the test process of the automatic driving software, and the automatic driving scene can be restored in an all-round manner.

And S130, testing the automatic driving software through the test case to obtain a test result.

The test result can be understood as an error alert or a message alert that the software is not problematic for the test of the automatic driving software. For example, the error notification may include a problem in a certain control module (e.g., a driver module) in the automatic driving software, or an error in running of a code segment, or a missing software parameter, or an unreasonable setting of a software parameter range, and may include other software error notification results, which is not limited by the embodiments of the disclosure.

In the step, the automatic driving software is tested by using the test case generated based on the problem data in the step, and the test result of the automatic driving software including the software error reminding result is generated.

Optionally, the automatic driving software may be tested offline by using the test case generated based on the problem data in the foregoing steps, so as to reproduce the problem in the automatic driving process, and generate the test result of the automatic driving software including the software error reminding result.

The off-line test may be to test the on-line or off-line automatic driving software by a test device (i.e., a test device of the automatic driving software, also referred to as a device for short) independent of the automatic driving vehicle.

According to the test method of the automatic driving software, provided by the embodiment of the disclosure, real vehicle test data can be introduced into a simulation test of the automatic driving software, namely, a test case for testing the automatic driving software is generated by utilizing problem data in the automatic driving process of the vehicle, and the automatic driving software is tested through the test case, so that the automatic driving software can be tested in all directions.

The following exemplifies the traveling data.

The dynamic data of the vehicle may include vertical dynamic data, longitudinal dynamic data and transverse dynamic data. For example, the vertical dynamic data may include data related to vertical motion of vehicle shock absorbers, springs, rubber metal parts, and tires, and the longitudinal dynamic data may include data related to driving force, resistance, speed, acceleration, etc. related to acceleration and braking; the lateral dynamics data may include body roll steering angle, steering flexibility, cornering stiffness, aligning moment, etc.

The environmental data of the environment of the vehicle may include road conditions, weather, surrounding pedestrians, dynamic objects, static objects, and other relevant data. For example, the road condition-related data may include technical conditions of road subgrade, road surface, structure, and accessories, and data of how dense vehicles are in the road, i.e., road congestion. The weather related data can comprise weather types, air temperatures and other data which influence the driving or traveling experience of the vehicle; the surrounding pedestrian related data comprises data such as the relative position of a pedestrian and a vehicle, the number of pedestrians, the pose of the pedestrian, the traveling direction and the traveling speed; the dynamic target related data can comprise data related to vehicles, pedestrians and other targets with changed relative ground positions around the vehicle, such as data of the traveling direction, the traveling speed and the like of the target; static object-related data may include data related to fixed position buildings around the vehicle, greens, stationary vehicles, people, or other objects that are not in a fixed position relative to the ground, and may include, for example, data such as the location of the object, its distance from the vehicle, attitude, contour, etc. In this section, the definition of the dynamic target or the static target is defined based on whether the position of the target changes within a preset distance before and after the vehicle passes through the target; or defining according to whether the position of the target changes within a preset time after the vehicle detects a certain target; or by other criteria known to those skilled in the art, and are not limited by the embodiments of the present disclosure.

The behavior data of the driver may include data related to interaction between the driver and the vehicle, for example, the data related to behavior of the driver for controlling steering, acceleration, deceleration, and adjusting ride comfort and stability of the vehicle may include, for example, an angle of turning a steering wheel, a speed of acceleration, a speed of braking, and a corresponding key of a human-computer interface that is clicked (pressed or slid) and the like, which is not limited in the embodiments of the present disclosure.

The aforementioned problem data in S110 includes the aforementioned traveling data of the vehicle in the emergency control intervention state.

The manner in which the problem data is obtained is explained below by way of example.

In some embodiments, based on fig. 1, S110 may include:

and receiving problem data sent by the automatic driving vehicle in real time.

Judging whether the vehicle is in an emergency control intervention state or not in the running process of the automatic driving vehicle; if so, transmitting the driving data, namely problem data, in the emergency control intervention time period to the testing device in real time; correspondingly, the testing device receives problem data sent by the automatic driving vehicle in real time.

It should be noted that the judgment of the emergency control intervention state needs to be implemented by combining the intervention time of the emergency control intervention and the driving data in the preset time period before and after the intervention time, so as to avoid erroneous judgment, and meanwhile, the construction of a complete test scene is facilitated, which is exemplarily described later.

For example, in connection with the scenario shown above, the issue data may include the driving data of the moment when the driver autonomously controls the vehicle, the driving data of a period of time that continues forward from the moment, and the driving data of a period of time that continues backward from the moment, to obtain a complete emergency control intervention scenario.

In some embodiments, the issue data includes historical travel data of the autonomous vehicle that meets the issue data screening criteria.

The historical driving data comprises driving data in an emergency control intervention state and driving data in an automatic driving state. The driving data in the automatic driving state is the driving data of the vehicle when the vehicle drives under the control of the automatic driving software, and the vehicle can normally drive at the moment without the intervention of a driver in the automatic driving state so as to meet the driving requirement.

The problem data screening condition may be a screening condition configured by a tester in advance according to a test requirement, and the embodiment of the present disclosure is not limited.

By setting the problem data screening condition, the driving data in the emergency control intervention state can be distinguished from the driving data in the automatic driving state, so that the problem data can be screened from the historical driving data, and the automatic driving software is tested only by using the problem data, so that the test data amount in the automatic driving software testing process is reduced, and the test efficiency is improved.

For example, the issue data filtering condition may include a judgment of a vehicle control state. Judging whether the historical driving data is the driving data of the vehicle in the emergency control intervention state; if so, the historical driving data is problem data, otherwise, the corresponding historical driving data is not problem data.

The problem data screening condition may include a determination of a vehicle control state, among others. For example, in the vehicle automatic driving process, the historical travel data corresponding to the vehicle deceleration caused by the intervention of the driver due to the penetration of another vehicle ahead of the current vehicle is the travel data of the vehicle in the emergency control intervention state, which is the problem data. In this case, for example, the question data filtering condition may include: the speed of the vehicle at the current moment is less than or equal to a preset speed threshold, and the difference between the speed at the previous moment and the speed at the current moment is greater than or equal to a preset speed difference. If it is detected that the speed of the vehicle meets the problem data screening condition, the corresponding historical driving data can be determined to be problem data. Or the question data screening conditions may include: the steering angle of a steering wheel of the vehicle is greater than or equal to a preset angle threshold, and the difference between the angle at the previous moment and the angle at the current moment is greater than or equal to a preset angle difference. If it is detected that the steering angle of the steering wheel of the vehicle satisfies the problem screening condition, the corresponding historical driving data can be determined as problem data.

The preset speed threshold, the preset speed difference, the preset angle threshold, the preset angle difference and the like can be any values set according to the needs of the user, and are not limited herein.

In other embodiments, the determination of the vehicle control state may also be implemented by using a combination of two or more parameters to implement the screening of the problem data, which is not limited herein.

In some embodiments, the filtering of historical travel data may be performed within an autonomous vehicle. That is, the autonomous driving vehicle screens out problem data that meets the problem data screening condition among the historical driving data during a certain period of time, and transmits the screened problem data to the testing device.

In some embodiments, the screening of historical driving data may also be performed in the testing device. That is, on the basis of fig. 1, before S110, the method further includes:

historical travel data transmitted by the autonomous vehicle is received.

The automatic driving vehicle sends the historical driving data to the testing device, specifically, the historical driving data can be sent in real time, can be sent at intervals of preset time periods (for example, 30 minutes, 1 hour or other time lengths), and can also send the historical driving data obtained by a single test at one time; the sending method may be a wireless transmission method, a wired transmission method, or a storage medium, and the historical driving data is transmitted from the vehicle to the testing device.

On this basis, S110 may include:

and screening out problem data which meets the problem data screening condition from the historical driving data.

Specifically, whether historical driving data meet problem data screening conditions or not is judged; if yes, the data is used as problem data; otherwise it will not be the problem data.

Therefore, problem data meeting the problem data screening conditions are determined through the problem data screening conditions based on the historical driving data.

The following exemplarily illustrates a test case generation manner.

In some embodiments, on the basis of fig. 1, S120 may include the following steps.

The method comprises the following steps: and generating a parameter combination according to a preset parameter combination strategy and each parameter in the problem data.

The problem data comprises various different types of data, such as vehicle dynamics data, environment data, behavior data of drivers and the like; each category of data may be characterized by at least one parameter. For example, the parameters of the vehicle dynamics data may include parameters such as driving force, resistance, speed, acceleration, body attitude, smoothness, stability, etc., the environmental data may include parameters such as road conditions, weather, positions and speeds of surrounding static and dynamic targets, and the behavior data of the driver may include motion-related parameters such as turning a steering wheel, stepping on a throttle, stepping on a brake, and setting buttons in a motor-human-machine interface.

The preset parameter combination strategy can be a combination of different parameters determined through parameter statistics based on the parameter change condition when the vehicle is converted from the automatic driving state to the emergency control intervention state.

For example, for a scenario in which a driver manually turns a steering wheel, steps on an accelerator, steps on a brake, and the like, and the vehicle is converted from an automatic driving state to an emergency control intervention state, the preset parameter combination strategy may include vehicle dynamic related parameters, such as: steering wheel rotation angle, speed, acceleration, position of dynamic or static objects in the surrounding environment, speed, acceleration, and distance from the vehicle.

In other embodiments, when the specific scenario of the other emergency control intervention state is targeted, the preset parameter combination policy may also be another parameter combination policy corresponding to the specific scenario, and the embodiment of the present disclosure is not limited.

In this step, a parameter combination can be obtained by combining each parameter in the problem data based on a preset parameter combination strategy.

Step two: and respectively setting parameter threshold values for each parameter in the parameter combination to obtain the test case.

Wherein the parameter threshold comprises at least one threshold determined based on performance of the automated driving software. For example, only the upper parameter value, only the lower parameter value, both the upper parameter value and the lower parameter value, or a plurality of consecutive or non-consecutive parameter intervals may be set for the same parameter.

The parameter threshold value is related to the specific scene of the emergency control intervention state and can be set according to the empirical value and the performance of the automatic driving software.

In combination with the above, for example, in a scene where the driver sees that another vehicle penetrates into the lane of the host vehicle, the parameter combination includes the real-time position and speed of the host vehicle, and the real-time direction and speed of the other vehicle, taking the scene of avoiding the host vehicle as an example. Wherein, corresponding to the real-time position and speed under the automatic driving state, a plurality of speed intervals can be set to meet different advancing requirements; corresponding to the real-time position and speed in the emergency control intervention state, a plurality of position and speed intervals can be set by combining the real-time positions and speeds of other vehicles so as to meet the avoidance requirement and simultaneously meet the traveling requirement of the current vehicle; corresponding to the position and the speed of the driver during the intervention, the relative position and the relative speed of the two vehicles can be counted to obtain a speed upper limit value corresponding to each speed interval, and the speed upper limit value is used as a speed threshold value for the manual intervention and is also called a speed adjusting threshold value.

Therefore, based on each parameter in the parameter combination and the parameter threshold value thereof, a test case can be obtained, so that the test of the automatic driving software can be realized in the following.

Illustratively, the format of the test case may be a data file format, such as an EXCEL form, an extensible markup language XML text, or other format data file.

On the basis of the above embodiment, the method may further include constructing (or constructing) a test scenario based on the problem data; and under the constructed test scene, the automatic driving software is tested through the test case so as to realize the restoration of various complex application scenes and the software test under different application scenes, which is concretely as follows.

In some embodiments, on the basis of fig. 1, after S120, the method further comprises the following steps.

The method comprises the following steps: marking the problem data according to the scene element labels corresponding to the parameters in the problem data to obtain the target scene element labels corresponding to the problem data.

The scene element label is a label set for distinguishing different scenes, each different scene can be associated with the emergency control intervention state, and the association relationship can be one-to-one, one-to-many, many-to-one or many-to-many association correspondence relationship.

Illustratively, the scene element tags may include an environmental element tag, a vehicle dynamics tag, and a user behavior tag, which are detailed below.

In this step, marking is performed on the problem data, that is, adding scene element labels to the problem data, and corresponding target scene element labels may be added to the problem data associated with the same target scene, so as to implement association of the problem data corresponding to the same target scene, so as to implement construction of a test scene in the subsequent step.

Step two: and constructing a test scene according to the test case and the target scene element label.

Specifically, a test scene is constructed according to the test case for the marked problem data.

The problem data is real vehicle test data, and the test scene constructed based on the problem data is a reduction of the scene in the real vehicle test process, or at least a part of the reduction. Based on the method, the complicated automatic driving environment can be restored, the problem of limited scenes caused by the design of simulation scenes by testers is avoided, and the method is favorable for realizing the all-round test of automatic driving software.

Correspondingly, S130 may include:

and under a test scene, testing the automatic driving software through the test case to obtain a test result.

The test scene in the step is a test scene constructed based on real vehicle test data in the step, and the actual scene in the automatic driving process can be restored; the test case is obtained based on real vehicle test data in the previous step, and the representation of a scene and an emergency control intervention state in the automatic driving process can also be realized; on the basis, the automatic driving software is tested, real vehicle test data can be automatically introduced into a simulation test process, a test scene is improved, and test efficiency is improved.

Specifically, the scene element labels are divided into different hierarchies to identify the data in different hierarchies.

The first-level scene element labels (i.e., first-level labels) include element type labels, which are used to distinguish the feature levels of the problem data, and may include, for example, weather labels, road type labels, road condition labels, dynamic object labels, first-level static object labels, and the like.

The secondary scene element labels are labels for further refining the problem data based on the first-layer labels, and comprise element attribute labels. For example, based on weather tags, their corresponding element attribute tags may include clear, rainy, cloudy, snowy, haze, and the like; based on the road type tags, their corresponding element attribute tags may include highways, urban roads, rural roads, and the like.

In some embodiments, the scene element tags may also include only primary scene element tags to enable classification, statistics, etc. by element category of the issue data.

In other embodiments, the scene element labels may further include more levels of scene element labels, and the scene element labels are used to gradually refine the scene element labels to further refine and classify the problem data, which is not limited in this disclosure.

On the basis of the above embodiment, optionally, in the method, a test case may also be generated based on the problem data in a non-real-time manner. For example, the test equipment may store the obtained problem data to obtain historical problem data, when new problem data is obtained each time, the historical problem data in a scene corresponding to the problem data may be counted, and when the amount of the historical problem data corresponding to a target scene reaches a preset amount, a test case is generated based on the problem data, so that the data accuracy is improved, and thus the test accuracy of the automatic driving software is improved, specifically as follows:

in some embodiments, on the basis of fig. 1, before S120, the method further comprises the following steps.

Specifically, adding a scene element label to the problem data may add a corresponding target scene element label to the problem data associated with the same target scene, thereby facilitating the implementation of association statistics of the problem data corresponding to the same target scene, and facilitating the implementation of statistics of historical problem data amount in subsequent steps.

Step two: the amount of historical problem data with the target scene element label is determined.

The problem data with a certain target scene element label is accumulated and counted, so that the historical problem data quantity of the target scene element label can be determined.

Specifically, when the data amount of the historical problem is counted, the counting is performed based on the scene element label with the finest classification. For example, if the scene element labels only include the primary scene element labels, the problem data is counted with the primary scene element labels as the standard, and one or more problem data having the identical primary scene element labels are accumulated to count the problem data having the target scene element labels. If the scene element labels comprise primary scene element labels and secondary scene element labels, counting the problem data by taking the secondary scene element labels as the standard, and accumulating one or more problem data with the completely same secondary scene element labels to realize counting the problem data with the target scene element labels.

Based on this, in some examples, S120 may include:

and generating a test case according to the problem data under the condition that the historical problem data amount is equal to the preset amount.

The preset number can be used for defining whether the emergency control intervention state corresponding to the problem data is sporadic or is usually generated. Specifically, when the data volume of the historical problems is less than the preset number, the corresponding emergency control intervention state is sporadic; when the amount of the historical problem data is equal to (or greater than) the preset amount, the corresponding emergency control intervention state is usually generated, and the automatic driving software needs to be tested and adjusted based on the historical problem data reaching the preset amount.

Based on the data, judging whether the data volume of the historical problems reaches a preset number; if yes, generating a test case according to the problem data; otherwise, the test case corresponding to the part of the problem data does not need to be generated.

In other examples, S120 may further include:

and under the condition that the historical problem data amount is equal to a preset amount, respectively generating a test case of the problem data pair and a test case corresponding to the historical problem data according to the problem data and the historical problem data.

The method for generating the test case corresponding to the historical problem data is similar to the method for generating the test case corresponding to the problem data, and details are not repeated herein.

Therefore, the effectiveness of problem data is improved, the data processing amount in the software testing process is reduced, and the testing efficiency and the testing accuracy are improved.

On the basis of the above embodiment, in the method, the problem data may be identified and screened to eliminate the influence of the misjudgment data on the test result, which is specifically as follows.

In some embodiments, on the basis of fig. 1, before S120, the method further comprises:

and inputting the problem data into a preset data recognition model to obtain a data recognition result.

The data identification model may be a classification model for identifying problem data as misjudged data or non-misjudged data.

Before the step, the method also comprises the following steps: a data recognition model is generated.

In particular, the classification model may be trained using sample data. The sample data may include positive and negative sample data; wherein, the positive sample data can be misjudged data, and the marking value can be 1; the negative sample data may be non-false positive data, and the flag value may be 0. After each sample data is input into the classification model, the classification model can output the predicted value of each sample data, and then the model parameters of the classification model are adjusted by using a back propagation method according to the marking value and the predicted value of each sample data until the accuracy of the classification model reaches a preset accuracy threshold value, so that the trained data recognition model is obtained.

Specifically, the sample data is also referred to as training data, which may employ historical travel data obtained during a previous travel of the vehicle that has been brought online; when the amount of the training data is enough, the accuracy of the recognition result of the data recognition model is improved.

Further, after the problem data is input into a preset data identification model, a predicted value corresponding to the problem data may be obtained, the predicted value corresponding to the problem data may be compared with a predetermined threshold (i.e., a preset accuracy threshold), and when the predicted value corresponding to the problem data is greater than or equal to the predetermined threshold, it may be determined that the data identification result is that the problem data is misjudged data, otherwise, it may be determined that the data identification result is that the problem data is non-misjudged data.

In some embodiments, the predetermined threshold may be 0.5, and in other embodiments, the predetermined threshold may also be a value between other (0,1) values set as needed, which is not limited herein.

In this step, when new problem data is received, a preset data identification model can be used to identify whether the new problem data is misjudged data, and if the new problem data is not misjudged data, a test case is generated in the subsequent steps according to the problem data.

That is, S120 may specifically include:

and under the condition that the data identification result indicates that the problem data is non-misjudged data, generating a test case according to the problem data.

The scene corresponding to the non-misjudgment data can include the data which avoids the occurrence of accidents, improves the driving experience or generates positive effects.

In the step, the test case is generated based on the non-misjudged data, so that the accuracy of the test case can be improved, the subsequent data processing amount is reduced, and the test efficiency and the test result accuracy are improved.

In some embodiments, after inputting the question data into a preset data recognition model and obtaining the data recognition result, the method further includes:

and deleting the problem data when the data identification result indicates that the problem data is misjudged data.

The misjudgment data may be data that has no positive effect on the implementation of the automatic driving process of the vehicle or the trip experience. The corresponding scene may include a scene of a vehicle state change caused by the driver autonomously changing the destination or the travel route.

For example, when the autonomous vehicle travels to a destination according to a predetermined navigation route, the driver wants to change to another destination, and the driver manually controls the autonomous vehicle to change the predetermined navigation route; correspondingly, the vehicle is converted from the automatic driving state into the emergency control intervention state, but the conversion of the state is not caused by the problem of the automatic driving software, but is caused by the active control of the driver based on the self idea, and the test and the improvement of the automatic driving software have no beneficial effect, so that the test and the improvement are misjudgment data.

In the step, the misjudged data are deleted, so that the subsequent data processing amount is reduced, and the software testing efficiency is improved.

In some embodiments, on the basis of fig. 1, after S130, the method further comprises the following steps.

The method comprises the following steps: and carrying out diagnosis analysis on the test result to obtain diagnosis information.

In the step, the test result obtained in the step is diagnosed and analyzed to determine the reason causing the test result, and then the diagnosis information is obtained.

Illustratively, the diagnostic information may include: logic problems, statement problems, parameter problems in the autopilot software; or generating a test case based on the problem data, and constructing data problems, step problems and the like in the test scene process so as to realize the omnibearing diagnosis of the test process.

Illustratively, the diagnostic analysis may be performed by a tester or may be performed based on a pre-trained diagnostic analysis model. Specifically, the diagnostic analysis model may be trained based on sample data, in which the input data may be a test result and the output data may be a cause of the test result.

Step two: and generating a test report according to the test result and the diagnosis information.

Specifically, the test result is associated with the diagnostic information to obtain a test report. Wherein, the test report displayed to the user can be generated based on the preset report format. The preset report format may include a text type format, a distribution format of contents of each part in the text, and other report related formats, and the embodiments of the present disclosure are not limited thereto.

And subsequently, adjusting the automatic driving software and optimizing the software testing process.

FIG. 2 illustrates another autopilot software testing method of an embodiment of the present disclosure, including the following steps.

And S200, starting.

Specifically, the test method for starting the automatic driving software may be, for example, a program corresponding to the test method.

S201, receiving problem data.

In one example, the vehicle end may screen the driving data of the vehicle in the emergency control intervention state from the historical driving data based on the problem data screening condition, that is, the problem data meeting the problem data screening condition is obtained and sent to the testing device. Correspondingly, the test device may receive issue data sent by the autonomous vehicle.

In another example, the vehicle end may monitor whether the vehicle is in an emergency control intervention state in real time, and if the vehicle is monitored to be in the emergency control intervention state, may acquire driving data of the vehicle in the emergency control intervention state, that is, obtain problem data, and send the problem data to the testing device. Correspondingly, the test device may receive issue data sent by the autonomous vehicle.

S202, judging whether the problem data are misjudgment data or not.

Specifically, the problem data is input into a preset data identification model to obtain a data identification result, and whether the problem data is misjudgment data or not is judged according to the data identification result.

Based on this, if the determination result is yes (Y), S2031 is executed; if the determination result is no (N), S203 is executed.

And S2031, deleting the misjudgment data.

Specifically, rejecting misjudged data may enable the testing device to perform subsequent steps using only non-misjudged data. Therefore, complete and accurate test cases and test scenes can be generated subsequently, and the software testing efficiency and accuracy can be improved.

Subsequent steps may include generating test cases (i.e., S203-S208), building test scenarios (i.e., S209), and conducting software tests (i.e., S210-S212).

And S203, generating a parameter combination.

Specifically, according to each parameter in the preset parameter combination strategy and the question data, corresponding to different scenes, a parameter combination associated with the scene is generated based on each parameter in the preset parameter combination strategy and the question data associated with the scene.

And S204, setting a parameter threshold value.

Specifically, a parameter upper limit value and/or a parameter lower limit value is set for each parameter in the parameter combination based on the empirical value, the performance of the automatic driving software, and the performance of the vehicle, respectively.

And S205, marking problem data.

Specifically, according to scene element labels corresponding to parameters in the problem data, the scene element labels are added to the problem data to obtain target scene element labels corresponding to the problem data, so that a test scene is constructed in the subsequent steps and the historical problem data volume is counted.

And S206, determining the historical problem data volume with the target scene element label.

Specifically, the data volume of the historical problem data having the same target scene element label as the problem data is counted to obtain the historical problem data volume.

And S207, judging whether the data volume of the historical problems reaches a preset number.

Specifically, as the historical problem data is accumulated, in the case where the amount of the historical problem data reaches (including is equal to or greater than) a predetermined amount, the test case may be generated according to the problem data and the parameter threshold thereof, and the historical problem data and the parameter threshold thereof.

If the judgment result of the step is yes (Y), continuing to execute the subsequent steps; otherwise (N), the determination is performed again based on the data amount of the historical problems and the preset number, that is, the loop execution is returned to S207.

And S208, generating a test case.

Specifically, under the condition that the amount of the historical problem data reaches the preset amount, the test case is generated based on the problem data and the parameter threshold thereof, and the historical problem data and the parameter threshold thereof.

And S209, constructing a test scene.

Specifically, a test scene is constructed based on the target scene element labels corresponding to the test cases and the problem data obtained in the previous steps, so that the actual scene of the automatic driving environment is restored.

And S210, performing off-line test to obtain a test result.

Specifically, under the test scene obtained in the previous step, the automatic driving software is subjected to offline test through the test case obtained in the previous step, and a test result is obtained.

And S211, diagnosing and analyzing to obtain diagnosis information.

Specifically, the test results are analyzed to find the cause of the problem occurring in the test results, i.e., the diagnostic information is obtained.

In some embodiments, this step may be omitted when the test results do not include autopilot software or software test related issues; or the output result of the step is null, the result is good and has no cause, or other presenting modes with no problem of the surface test result.

And S212, generating a test report.

Specifically, a test report including the test result and the diagnostic information is generated according to the test result and the diagnostic information, thereby completing a complete test of the automatic driving software.

And S213, ending.

Specifically, the program execution ends.

Subsequently, the program corresponding to the test method of the automatic driving software and the running environment of the program, such as a background server, may be closed.

According to the test method of the automatic driving software, provided by the embodiment of the disclosure, real vehicle test data can be automatically introduced into a simulation test, the real vehicle test is performed again after the simulation test, an automatic driving test closed loop is created, a test scene is improved, and the automation rate of the test is improved; secondly, by setting different scene element labels, the omnibearing coverage of the automatic driving scene can be perfected, and the omnibearing test of a more complex automatic driving environment is realized; meanwhile, a test case is generated by combining the problem data when the historical problem data amount is equal to the preset amount, the software test is completed, the screening and the misjudgment elimination of the problem data can be completed in a data driving mode, the data processing amount can be reduced, the test efficiency can be improved, and the test accuracy can be improved.

On the basis of the above embodiments, the present disclosure provides a testing device for automatic driving software, which can be used to execute the steps of the testing method for automatic driving software in any of the above embodiments.

In some embodiments, FIG. 3 illustrates a test setup for autopilot software. As shown in fig. 3, the apparatus 30 includes: a first obtaining module 301, configured to obtain problem data during the driving process of the autonomous vehicle, where the problem data includes driving data of the vehicle in an emergency control intervention state; a first generating module 302 configured to generate a test case for testing the autopilot software according to the problem data; the software testing module 303 is configured to test the automatic driving software through the test case to obtain a test result.

In the test device for the automatic driving software, provided by the embodiment of the disclosure, real vehicle test data can be introduced into a simulation test of the automatic driving software, namely, a test case for testing the automatic driving software is generated by utilizing problem data in the automatic driving process of a vehicle, and the automatic driving software is tested through the test case, optionally, an off-line test is performed, so that the automatic driving software can be tested in all directions.

Therefore, the running state of the vehicle can be restored in all directions, and the test accuracy is improved.

In some embodiments, the first obtaining module is specifically configured to: and receiving problem data sent by the automatic driving vehicle in real time.

Accordingly, the data screening process may be performed at the vehicle end (i.e., in an autonomous vehicle). That is, the vehicle transmits only its driving data in the emergency control intervention state to the first acquisition module, and does not transmit its driving data in the autonomous driving state.

Therefore, the data processing capacity of the testing device can be reduced, and the testing efficiency can be improved.

In some embodiments, the issue data includes historical travel data of the autonomous vehicle that meets the issue data screening criteria. Based on this, the apparatus further comprises:

a data receiving module configured to receive historical driving data transmitted by an autonomous vehicle;

the first obtaining module is specifically configured to:

Specifically, the data screening process is completed in the test device. The vehicle sends data generated in the driving process, namely historical driving data to the data receiving module; correspondingly, the data receiving module receives historical driving data, and the first obtaining module screens out problem data based on the historical driving data.

Therefore, the data processing amount of the vehicle end can be reduced, and the vehicle end structure can be simplified.

In some embodiments, the first generation module is specifically configured to:

and respectively setting parameter threshold values for each parameter in the parameter combination to obtain a test case, wherein the parameter threshold values comprise at least one threshold value determined according to the performance of the automatic driving software.

Therefore, the parameters in the problem data can be combined, and the parameter threshold value of each parameter is set to obtain the test case.

In some embodiments, the apparatus further comprises:

based on this, the software testing module is specifically configured to:

Therefore, a test scene can be constructed based on the problem data and the test cases correspondingly generated by the problem data and by combining the scene element labels, so that the scene reduction of the automatic driving environment is realized, the automatic driving software is tested by the test cases under various complex scenes, and the corresponding test results are obtained.

The element classes can be divided through the primary scene element labels, and different scenes can be distinguished through the secondary scene element labels. With the arrangement, the test scene can be constructed conveniently, and statistics of problem data associated with the same test scene can be realized.

In some embodiments, the apparatus further comprises:

a data volume determination module configured to determine a historical problem data volume having a target scene element label;

the first generation module is specifically configured to:

Therefore, the problem data with the historical problem data amount smaller than the preset amount can be eliminated, so that the data processing amount is reduced, and the test efficiency is improved.

In some embodiments, the apparatus further comprises:

the first generation module is specifically configured to:

In some embodiments, the apparatus further comprises:

Therefore, misjudged data can be eliminated, and only non-misjudged data is utilized to generate a test case, so that the accuracy of problem data can be improved, the subsequent data processing amount is reduced, the test efficiency is improved, and the accuracy of a test result is improved.

In some embodiments, the apparatus further comprises:

the diagnostic analysis module is configured to perform diagnostic analysis on the test result to obtain diagnostic information;

Therefore, the reason corresponding to the test result can be determined, and the diagnosis information can be obtained; and generating a test report comprising the test result and the diagnosis information, and providing support for subsequent optimization of the automatic driving software and optimization of the test process.

It should be noted that the testing device 30 of the autopilot software shown in fig. 3 may execute each step in the above-described method embodiment, and implement each process and effect in the above-described method embodiment, which is not described herein again.

On the basis of the above embodiment, the embodiment of the present disclosure further provides a testing device for automatic driving software, which can be used to implement any one of the above testing methods for automatic driving software.

Exemplarily, fig. 4 is a schematic structural diagram of a testing device of autopilot software according to an embodiment of the present disclosure. Referring to fig. 4, the test apparatus 40 of the automatic driving software includes: a processor 401 and a memory 402 for storing executable instructions; the processor 401 is configured to read executable instructions from the memory 402 and execute the executable instructions to implement any one of the above-described methods for testing the automatic driving software.

Specifically, the processor 401 may include a Central Processing Unit (CPU), or an Application Specific Integrated Circuit (ASIC), or may be configured to implement one or more Integrated circuits of the embodiments of the present Application.

Memory 402 may include a mass storage for information or instructions. By way of example, and not limitation, memory 402 may include a Hard Disk Drive (HDD), a floppy Disk Drive, flash memory, an optical Disk, a magneto-optical Disk, tape, or a Universal Serial Bus (USB) Drive or a combination of two or more of these. Memory 402 may include removable or non-removable (or fixed) media, where appropriate. Memory 402 may be internal or external to the integrated gateway device, where appropriate. In a particular embodiment, the memory 402 is a non-volatile solid-state memory. In a particular embodiment, the Memory 402 includes Read-Only Memory (ROM). The ROM may be mask-programmed ROM, Programmable ROM (PROM), Erasable PROM (Electrically Erasable PROM, EPROM), Electrically Erasable PROM (Electrically Erasable PROM, EEPROM), Electrically Alterable ROM (Electrically Alterable ROM, EAROM), or flash memory, or a combination of two or more of these, where appropriate.

The processor 401 reads and executes the computer program instructions stored in the memory 402 to execute the steps of the multimedia file playing method provided by the embodiment of the present disclosure.

In one example, the multimedia playing device 40 may further include a transceiver 403 and a bus 404. As shown in fig. 4, the processor 401, the memory 402 and the transceiver 403 are connected via a bus 404 to complete communication therebetween.

Bus 404 comprises hardware, software, or both. By way of example, and not limitation, a BUS may include an Accelerated Graphics Port (AGP) or other Graphics BUS, an Enhanced Industry Standard Architecture (EISA) BUS, a Front-Side BUS (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) BUS, an InfiniBand interconnect, a Low Pin Count (LPC) BUS, a memory Bus, a Micro Channel Architecture (MCA) Bus, a Peripheral Component Interconnect (PCI) Bus, a PCI-Express (PCI-X) Bus, a Serial Advanced Technology Attachment (SATA) Bus, a Video Electronics Standards Association Local Bus (VLB) Bus, or other suitable Bus, or a combination of two or more of these. Bus 404 may include one or more buses, where appropriate. Although specific buses are described and shown in the embodiments of the application, any suitable buses or interconnects are contemplated by the application.

On the basis of the above embodiment, the embodiment of the present disclosure further provides a computer-readable storage medium, where a computer program is stored, and when the computer program is executed by a processor, the processor is enabled to implement any one of the above-mentioned methods for testing the automatic driving software.

Illustratively, in conjunction with fig. 4, a storage medium comprising instructions, such as memory 402 comprising instructions, executable by processor 401 to perform the method for testing autopilot software provided by embodiments of the present disclosure.

Alternatively, the storage medium may be a non-transitory computer readable storage medium, for example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a Compact Disc read only Memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

The foregoing description is only exemplary of the preferred embodiments of the disclosure and is illustrative of the principles of the technology employed. It will be appreciated by those skilled in the art that the scope of the disclosure herein is not limited to the particular combination of features described above, but also encompasses other embodiments in which any combination of the features described above or their equivalents does not depart from the spirit of the disclosure. For example, the above features and (but not limited to) the features disclosed in this disclosure having similar functions are replaced with each other to form the technical solution.

Further, while operations are depicted in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Likewise, while several specific implementation details are included in the above discussion, these should not be construed as limitations on the scope of the disclosure. Certain features that are described in the context of separate embodiments can also be implemented in combination in a single embodiment. Conversely, various features that are described in the context of a single embodiment can also be implemented in multiple embodiments separately or in any suitable subcombination.

Although the subject matter has been described in language specific to structural features and/or methodological acts, it is to be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A test method of automatic driving software is characterized by comprising the following steps:

2. The method of claim 1, wherein the driving data comprises at least one of dynamics data of the vehicle, environmental data of an environment in which the vehicle is located, and behavior data of a driver of the vehicle.

3. The method of claim 1, wherein the obtaining problem data during travel of the autonomous vehicle comprises:

4. The method of claim 1, wherein the issue data includes historical travel data of the autonomous vehicle that meets issue data screening criteria.

5. The method of claim 4, wherein prior to said obtaining problem data during travel of an autonomous vehicle, the method further comprises:

receiving historical driving data sent by the autonomous vehicle;

6. The method of claim 1, wherein generating a test case for testing the autopilot software based on the issue data comprises:

7. The method of claim 1, wherein after the generating a test case for testing the autopilot software based on the issue data, the method further comprises:

8. The method of claim 7, wherein the scene element tags comprise primary scene element tags comprising element type tags and secondary scene element tags comprising element attribute tags.

9. The method of claim 1, wherein prior to said generating a test case for testing the autopilot software based on the issue data, the method further comprises:

10. The method of claim 1, wherein prior to said generating a test case for testing the autopilot software based on the issue data, the method further comprises:

11. The method according to claim 10, wherein after inputting the question data into a preset data recognition model to obtain a data recognition result, the method further comprises:

12. The method of claim 1, wherein after said testing the autopilot software with the test case to obtain a test result, the method further comprises:

13. An autopilot software testing apparatus, comprising:

14. An autopilot software testing apparatus, comprising:

a processor;

a memory for storing executable instructions;

wherein the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method for testing the autopilot software of any one of claims 1-12.

15. A computer-readable storage medium, characterized in that the storage medium stores a computer program which, when executed by a processor, causes the processor to carry out a method of testing autopilot software according to any one of claims 1-12.