CN114935918A - Performance evaluation method, device and equipment of automatic driving algorithm and storage medium - Google Patents
Performance evaluation method, device and equipment of automatic driving algorithm and storage medium Download PDFInfo
- Publication number
- CN114935918A CN114935918A CN202210319751.4A CN202210319751A CN114935918A CN 114935918 A CN114935918 A CN 114935918A CN 202210319751 A CN202210319751 A CN 202210319751A CN 114935918 A CN114935918 A CN 114935918A
- Authority
- CN
- China
- Prior art keywords
- scene
- algorithm
- evaluation
- data
- automatic driving
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B23/00—Testing or monitoring of control systems or parts thereof
- G05B23/02—Electric testing or monitoring
- G05B23/0205—Electric testing or monitoring by means of a monitoring system capable of detecting and responding to faults
- G05B23/0218—Electric testing or monitoring by means of a monitoring system capable of detecting and responding to faults characterised by the fault detection method dealing with either existing or incipient faults
- G05B23/0243—Electric testing or monitoring by means of a monitoring system capable of detecting and responding to faults characterised by the fault detection method dealing with either existing or incipient faults model based detection method, e.g. first-principles knowledge model
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05B—CONTROL OR REGULATING SYSTEMS IN GENERAL; FUNCTIONAL ELEMENTS OF SUCH SYSTEMS; MONITORING OR TESTING ARRANGEMENTS FOR SUCH SYSTEMS OR ELEMENTS
- G05B2219/00—Program-control systems
- G05B2219/20—Pc systems
- G05B2219/24—Pc safety
- G05B2219/24065—Real time diagnostics
Landscapes
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- Automation & Control Theory (AREA)
- Testing Of Devices, Machine Parts, Or Other Structures Thereof (AREA)
Abstract
The invention relates to the technical field of automatic driving tests, and discloses a performance evaluation method, a device, equipment and a storage medium of an automatic driving algorithm, which are used for improving the performance evaluation accuracy, shortening the algorithm iteration cycle and improving the development efficiency. The method comprises the steps of obtaining simulation test data of each automatic driving algorithm in an automatic driving system, analyzing the simulation test data to obtain abnormal indexes and corresponding scene data, determining the types of the abnormal indexes, matching corresponding decision trees from a pre-constructed index diagnosis logic table based on the types, evaluating the corresponding scene data according to evaluation content corresponding to each node in the decision trees, and generating a simulation test report of the automatic driving algorithm based on the evaluation result.
Description
Technical Field
The invention relates to the technical field of automatic driving tests, in particular to a performance evaluation method, a performance evaluation device, performance evaluation equipment and a storage medium of an automatic driving algorithm.
Background
With the development of artificial intelligence technology, especially the research of automatic driving direction of automobiles, automatic control is performed on each device in the automobile by developing an automatic control program, and in the process of applying the automatic driving technology, scenes to be processed are very complex and diverse, so that simulation tests are performed in a large number of test scenes, and the behavior expression of a new algorithm and an original algorithm in the test scenes of a main automobile is compared, so as to evaluate the change of the performance of the new algorithm.
At present, the performance of the main vehicle in a test scene, such as safety, comfort, lane change completion and the like, is evaluated through various evaluation indexes. During comparison, a scene with important changes of evaluation indexes is focused, a scene with really good/bad behavior of the main vehicle is found through diagnosis, and the finding and labeling of the scene are mainly realized manually, so that the realization mode is low in efficiency.
Disclosure of Invention
The invention provides a performance evaluation method, a performance evaluation device, performance evaluation equipment and a storage medium of an automatic driving algorithm, and mainly aims to solve the problems that the diagnosis evaluation time is too long and the accuracy is low, and the iterative development of the algorithm is not facilitated due to the fact that a large amount of manual access is needed in the existing algorithm simulation result diagnosis process.
The invention provides a performance evaluation method of an automatic driving algorithm, which comprises the following steps:
acquiring simulation test data of each automatic driving algorithm in an automatic driving system, and analyzing the simulation test data to obtain abnormal indexes and corresponding scene data;
determining the category of the abnormal index, and matching a corresponding decision tree from a pre-constructed index diagnosis logic table based on the category, wherein the decision tree comprises at least one node, and each node corresponds to different evaluation contents of the index;
and evaluating corresponding scene data according to the evaluation content corresponding to each node in the decision tree, and generating a simulation test report of the automatic driving algorithm based on the evaluation result.
Optionally, the obtaining simulation test data of each automatic driving algorithm in the automatic driving system, and analyzing the simulation test data to obtain an abnormal index and corresponding scene data includes:
determining an automatic driving algorithm to be evaluated;
respectively acquiring corresponding real driving data and simulation test data from an automatic driving system and an algorithm simulation platform based on the automatic driving algorithm;
comparing the real driving data with the simulation test data, and extracting corresponding scene data from the simulation test data based on the comparison result;
and analyzing the indexes of the scene data to obtain abnormal indexes.
Optionally, the analyzing the indexes of the scene data to obtain abnormal indexes includes:
identifying a main vehicle running track and control data in the running process in the scene data;
matching the main vehicle running track and the control data with preset scene definition information, and determining a scene type based on a matching result;
according to the scene type, performance parameters related to the scene are extracted from the control data, and an abnormality index is determined based on the performance parameters.
Optionally, the determining the category of the abnormal indicator, and matching a corresponding decision tree from a pre-constructed indicator diagnosis logic table based on the category includes:
determining the category of the abnormal index in the scene under the scene type according to the corresponding relation between the scene type and each index;
determining corresponding scene-related problems according to the scene types, and determining corresponding labeling information based on the scene-related problems, wherein the scene-related problems are evaluation items used for evaluating the control performance of the automatic driving algorithm in the corresponding scene;
and matching a corresponding decision tree from a pre-constructed index diagnosis logic table based on the labeling information, wherein the index diagnosis logic table comprises decision trees for evaluating different indexes.
Optionally, the evaluating the corresponding scene data according to the evaluation content corresponding to each node in the decision tree, and generating the simulation test report of the automatic driving algorithm based on the evaluation result includes:
analyzing the evaluation items of all trunks in the decision tree and the evaluation contents of all nodes under the evaluation items;
extracting vehicle control parameters in the control data based on the evaluation item;
taking the evaluation content of each node as an evaluation marking rule, and sequentially evaluating and marking the corresponding vehicle control parameters to obtain a marking result;
and uploading the labeling result to the algorithm simulation platform for diagnosis and evaluation, and generating a corresponding simulation test report.
Optionally, the uploading the labeling result to the algorithm simulation platform for diagnosis and evaluation to generate a corresponding simulation test report includes:
determining the type of the corresponding algorithm fault problem based on the labeling result;
carrying out simulation diagnosis by using a scene model corresponding to the type of the algorithm fault problem through the algorithm simulation platform to obtain a diagnosis result;
and generating a corresponding simulation test report according to the diagnosis result and a report format corresponding to the type of the algorithm fault problem.
Optionally, before obtaining simulation test data of each automatic driving algorithm in the automatic driving system and analyzing the simulation test data to obtain an abnormal index and corresponding scene data, the method further includes:
extracting an automatic driving algorithm in an automatic driving system, and determining a scene corresponding to the algorithm;
determining test content according to the scene, and constructing an accident problem based on the test content;
analyzing scene-related problems corresponding to the accident problems, and configuring evaluation contents;
and constructing nodes and leaf nodes by using a tree structure according to the evaluation content corresponding to each accident problem to generate a corresponding decision tree, wherein leaf nodes in the decision tree are connected with the nodes, and all the nodes are connected to form the decision tree.
A second aspect of the present invention provides a performance evaluation device of an automatic driving algorithm, including:
the system comprises an acquisition module, a display module and a display module, wherein the acquisition module is used for acquiring simulation test data of each automatic driving algorithm in an automatic driving system and analyzing the simulation test data to obtain abnormal indexes and corresponding scene data;
the matching module is used for determining the category of the abnormal index and matching a corresponding decision tree from a pre-constructed index diagnosis logic table based on the category, wherein the decision tree comprises at least one node, and each node corresponds to different evaluation contents of the index;
and the evaluation module is used for evaluating the corresponding scene data according to the evaluation content corresponding to each node in the decision tree and generating a simulation test report of the automatic driving algorithm based on the evaluation result.
Optionally, the collecting module includes:
a determination unit for determining an automated driving algorithm to be evaluated;
the acquisition unit is used for acquiring corresponding real driving data and simulation test data from an automatic driving system and an algorithm simulation platform respectively based on the automatic driving algorithm;
the extraction unit is used for comparing the real driving data with the simulation test data and extracting corresponding scene data from the simulation test data based on the comparison result;
and the first analysis unit is used for analyzing the indexes of the scene data to obtain abnormal indexes.
Optionally, the first parsing unit is specifically configured to:
identifying a main vehicle running track and control data in the running process in the scene data;
matching the main vehicle running track and the control data with preset scene definition information, and determining a scene type based on a matching result;
according to the scene type, performance parameters related to the scene are extracted from the control data, and an abnormality index is determined based on the performance parameters.
Optionally, the matching module includes:
the type query unit is used for determining the category of the abnormal index in the scene under the scene type according to the corresponding relation between the scene type and each index;
the labeling unit is used for determining corresponding scene-related questions according to the scene types and determining corresponding labeling information based on the scene-related questions, wherein the scene-related questions are evaluation items used for evaluating the control performance of the automatic driving algorithm in the corresponding scenes;
and the matching unit is used for matching a corresponding decision tree from a pre-constructed index diagnosis logic table based on the labeling information, wherein the index diagnosis logic table comprises the decision trees for evaluating different indexes.
Optionally, the evaluation module includes:
the second analysis unit is used for analyzing the evaluation items of all the trunks in the decision tree and the evaluation contents of all the nodes under the evaluation items;
a parameter extraction unit configured to extract a vehicle control parameter in the control data based on the evaluation item;
the evaluation unit is used for taking the evaluation content of each node as an evaluation marking rule, sequentially evaluating and marking the corresponding vehicle control parameters to obtain a marking result;
and the report generation unit is used for uploading the labeling result to the algorithm simulation platform for diagnosis and evaluation and generating a corresponding simulation test report.
Optionally, the report generating unit is specifically configured to:
determining the type of the corresponding algorithm fault problem based on the labeling result;
carrying out simulation diagnosis by using a scene model corresponding to the type of the algorithm fault problem through the algorithm simulation platform to obtain a diagnosis result;
and generating a corresponding simulation test report according to the diagnosis result and a report format corresponding to the type of the algorithm fault problem.
Optionally, the performance evaluation apparatus for an automatic driving algorithm further includes a decision tree construction module, which is specifically configured to:
extracting an automatic driving algorithm in an automatic driving system, and determining a scene corresponding to the algorithm;
determining test content according to the scene, and constructing an accident problem based on the test content;
analyzing scene-related problems corresponding to the accident problems, and configuring evaluation contents;
and constructing nodes and leaf nodes by using a tree structure according to the evaluation content corresponding to each accident problem to generate a corresponding decision tree, wherein leaf nodes in the decision tree are connected with the nodes, and all the nodes are connected to form the decision tree.
A third aspect of the present invention provides a computer apparatus comprising: a memory and at least one processor, the memory having stored therein a computer program; the at least one processor invokes the computer program in the memory to cause the computer device to perform the steps of the performance evaluation method of the autonomous driving algorithm described above.
A fourth aspect of the present invention provides a computer-readable storage medium having stored therein a computer program which, when run on a computer, causes the computer to execute the steps of the above-described performance evaluation method of an automated driving algorithm.
According to the technical scheme provided by the invention, the simulation test data of each automatic driving algorithm in the automatic driving system is obtained and analyzed to obtain the abnormal index and the corresponding scene data, the category of the abnormal index is determined, the corresponding decision tree is matched from the pre-constructed index diagnosis logic table based on the category, the corresponding scene data is evaluated according to the evaluation content corresponding to each node in the decision tree, and the simulation test report of the automatic driving algorithm is generated based on the evaluation result, so that the unmanned vehicle algorithm and the evaluation index need to be deeply known, a test engineer needs to spend a large amount of time in the task, and the cost is high. Under the condition that the number of test engineers is limited, if a large number of version comparison diagnosis requirements cannot be completed in time, the iteration cycle of the algorithm is delayed, and the development efficiency is influenced. Meanwhile, the decision tree mode avoids the problem that misdiagnosis is easy to occur due to the thought subjective judgment.
Drawings
FIG. 1 is a schematic diagram of an embodiment of a method for evaluating performance of an autopilot algorithm in accordance with an embodiment of the present invention;
FIG. 2 is a schematic diagram of another embodiment of a performance evaluation method of an autonomous driving algorithm in an embodiment of the invention;
FIG. 3 is a flow chart of comparative evaluation in an embodiment of the present invention;
FIG. 4 is a feedback flow diagram of a simulation test report according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of an embodiment of a performance evaluation device of an automatic driving algorithm according to an embodiment of the present invention;
FIG. 6 is a schematic diagram of another embodiment of the performance evaluation device of the automatic driving algorithm in the embodiment of the invention;
FIG. 7 is a diagram of an embodiment of a computer device in an embodiment of the invention.
Detailed Description
The embodiment of the invention provides a performance evaluation method, a performance evaluation device, performance evaluation equipment and a storage medium of an automatic driving algorithm. Under the condition that the number of test engineers is limited, if a large number of version comparison diagnosis requirements cannot be completed in time, the iteration cycle of the algorithm is delayed, and the development efficiency is influenced. Meanwhile, the decision tree mode avoids the problem that misdiagnosis is easy to occur due to the thought subjective judgment.
The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be implemented in other sequences than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.
It is to be understood that the execution subject of the present invention may be a performance evaluation device of an automatic driving algorithm, and may also be a terminal or a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.
For ease of understanding, a detailed flow of an embodiment of the present invention is described below, and referring to fig. 1, an embodiment of a performance evaluation method of an automatic driving algorithm in an embodiment of the present invention includes:
101. acquiring simulation test data of each automatic driving algorithm in the automatic driving system, and analyzing the simulation test data to obtain abnormal indexes and corresponding scene data;
in this step, each index in the automatic driving scene is an important parameter for evaluating the superiority and inferiority of an automatic driving technique such as an automatic driving algorithm or an automatic driving system. In the automatic driving process, driving accidents can be caused by control abnormity of indexes such as disorderly steering wheel hitting, continuous snubbing, over-quick speed increasing and the like in an automatic control algorithm, indexes in different scenes are extracted by simulating the algorithm through indexes such as accidents occurring in different predefined driving scenes or algorithm interruption, stopping and ending, and abnormal indexes in the indexes are analyzed.
In this embodiment, when extracting the abnormal index and the scene data, the abnormal index and the scene data are specifically extracted by comparing and analyzing the real driving data and the simulated scene data, where the real driving data is mainly read from an automatic driving system of a vehicle, then the read real driving data is segmented according to a scene based on a scene model, and after the segmentation is completed, the real driving data is arranged in a time sequence to analyze code segments in an automatic driving algorithm, where the code segments should be understood as different control algorithms in automatic driving, and the real scene data obtained after the segmentation corresponds to the control algorithms and is labeled with corresponding scene types. And calling a driving simulation program based on a control algorithm to carry out simulation so as to obtain simulation test data.
And then analyzing the corresponding simulation test data and the real scene data respectively, extracting key information from the simulation test data and the real scene data, comparing the key information of the simulation test data and the real scene data one by one, screening out inconsistent abnormal indexes based on comparison results, extracting corresponding data from the simulation test data and the real scene data aiming at the abnormal indexes, and combining the extracted data of the two sides to obtain the scene data corresponding to the abnormal indexes.
102. Determining the category of the abnormal index, and matching a corresponding decision tree from a pre-constructed index diagnosis logic table based on the category;
in this step, each decision tree corresponds to one index, the index diagnosis logic table includes decision trees of multiple indexes in multiple scenes, the decision trees can be understood as a rule representing index diagnosis logic and diagnosis content, each decision tree includes at least one node, and each node corresponds to different evaluation content of the index. The types of the indexes can be understood as fault types occurring in vehicle driving, specifically, the corresponding scene types are determined by analyzing scene data, a plurality of indexes and fault types corresponding to each index are analyzed based on the scene types, wherein the fault types comprise types of scene unreal problems, evaluation index algorithm problems, scene incomparable problems, planning algorithm problems and the like. And matching the corresponding abnormal indexes based on the fault categories to obtain the categories of the abnormal indexes, and then inquiring a diagnosis logic table of the indexes based on the categories to inquire out a matched decision tree.
In practical application, all decision trees belonging to the scene information are found out from the index diagnosis logic table based on the scene information corresponding to the scene data, corresponding decision trees are selected from all the found decision trees based on the matching information corresponding to the abnormal indexes, the root nodes and the child nodes of the decision trees are extracted, and an evaluation rule sequence is constructed, wherein the evaluation rule sequence is obtained by sequentially ordering all the child nodes by taking the root nodes as first elements and the first elements as starting points.
103. And evaluating the corresponding scene data according to the evaluation content corresponding to each node in the decision tree, and generating a simulation test report of the automatic driving algorithm based on the evaluation result.
In the step, the evaluation content of each node is obtained by analyzing a knowledge graph, and the evaluation content is obtained by firstly identifying each root node in the decision tree, taking the root node as a starting point, sequentially extracting branch points on a trunk where the root node is located in an upward analysis mode, then sequentially extracting sub-nodes on branches corresponding to the branch points according to upward sequencing of the branch points, and extracting the content on each sub-node, wherein the evaluation content is a problem related to a scene and a corresponding standard or condition provided according to the evaluation index.
And then evaluating corresponding parameters in the scene data based on the extracted problems, standards or conditions, outputting a result decision tree based on the evaluation result, and generating a simulation test report of the automatic driving algorithm according to the result decision tree.
In this embodiment, the result decision tree is obtained by replacing the evaluation content on the child node with the evaluation result after the evaluation of the corresponding parameter by the evaluation content on the child node is completed.
Further, before the evaluation of the scene data based on the decision tree, the method further comprises analyzing the scene data based on the problems related to the scenes on the child nodes in the decision tree, and constructing a data structure tree corresponding to the decision tree. And matching and corresponding each child node of the decision tree with each child node of the data structure tree on the basis of the data structure tree, calling the standard or condition on each child node in the decision tree to evaluate the corresponding child node data in the data structure tree on the basis of the matched result, and generating a simulation test report on the basis of the evaluation result after the data structure tree falls down.
In the embodiment of the invention, in order to improve the performance evaluation accuracy of the algorithm, the evaluation indexes in the algorithm are used for constructing the evaluated decision tree, the index data in the acquired simulation and the scene data for executing the algorithm are evaluated based on the decision tree, so that the indexes are automatically evaluated. Meanwhile, the decision tree mode avoids the problem that misdiagnosis is easy to occur due to the thought subjective judgment.
Referring to fig. 2, another embodiment of the method for evaluating performance of an automatic driving algorithm according to the embodiment of the present invention includes:
201. determining an automatic driving algorithm to be evaluated, and acquiring corresponding simulation test data for analysis to obtain abnormal indexes and corresponding scene data;
further, before step 201, the method further includes: collecting control algorithms in the current vehicle to be evaluated, classifying the control algorithms according to scenes to obtain an algorithm set to be evaluated, collecting data based on the algorithm set, and executing step 201.
In this embodiment, after determining the algorithm to be evaluated, data is prepared by the following steps:
determining an automatic driving algorithm to be evaluated;
respectively acquiring corresponding real driving data and simulation test data from an automatic driving system and an algorithm simulation platform based on the automatic driving algorithm;
comparing the real driving data with the simulation test data, and extracting corresponding scene data from the simulation test data based on the comparison result;
and analyzing the indexes of the scene data to obtain abnormal indexes.
In this embodiment, the method is mainly applied to the comparative evaluation of the new version and the old version of the automatic driving algorithm to determine the stability of the new version of the algorithm. When data of an algorithm is analyzed to obtain abnormal indexes and scene data, the method specifically comprises the steps of obtaining simulation test data and real driving data of a new version algorithm and an old version algorithm according to an automatic driving algorithm, analyzing indexes of the data of the new version and the old version, comparing the data of the new version and the data of the old version based on the index data of the simulation scene and the real scene, extracting the index data of the new version and the old version and the abnormal indexes to compare after the abnormality exists, determining the data of the new version and the old version as abnormal indexes when front and back comparison results are abnormal, and extracting the scene data of the new version and the old version corresponding to the abnormal indexes.
In this embodiment, performing index analysis on scene data to obtain an abnormal index may be implemented in the following manner:
identifying a main vehicle running track and control data in a running process in the scene data;
matching the main vehicle running track and the control data with preset scene definition information, and determining a scene type based on a matching result;
according to the scene type, performance parameters related to the scene are extracted from the control data, and an abnormality index is determined based on the performance parameters.
In practical application, different indexes are set in different scenes, corresponding indexes are selected based on scene types, index data in control data are sequentially extracted, namely performance parameters related to the scenes are extracted, and then the performance parameters are compared between new versions and old versions to determine abnormal indexes.
202. Determining the category of the abnormal index in the scene under the scene type according to the corresponding relation between the scene type and each index;
it should be noted that each scene includes multiple types of indicators, and each type of indicator corresponds to a fault problem, for example, a scene unreal problem, an evaluation indicator algorithm problem, a scene incomparable problem, a planning algorithm problem, and other categories; each category contains a plurality of indicators, such as security, comfort, lane-change capability, etc., where each indicator corresponds to a decision tree that is in effect a scoring algorithm model for that indicator.
Specifically, the correspondence between the scene type and each index is actually a correspondence between the scene type and an index type, an index type corresponding to the scene data is searched, and then matching is performed one by one based on the indexes in the index type and the abnormal indexes, so as to obtain the type of the abnormal index. In practical application, the matching may also be performed based on a matching code, for example, a first random number associated between the abnormal indicator and the scene type is calculated by using a random number generation algorithm, then a second random number associated between the scene type and the category of the indicator is calculated based on the random number generation algorithm, the first random number and the second random number are compared, and if the comparison is consistent, the category of the abnormal indicator is determined. In order to ensure the comparability of the two, firstly, a random number generation rule is required to be configured as an equality rule between a scene type and an index and between the scene type and a category, then two equality rules are learned based on a random number generation algorithm, and finally, when the method is applied, the learned random number generation algorithm is used for calculating the category of the abnormal index.
203, determining corresponding scene-related problems according to the scene types, and determining corresponding labeling information based on the scene-related problems;
in the step, the determination of the labeling information is specifically realized through a labeling model, problem features in the scene-related problems are extracted through the labeling model, and category labels are calculated based on the problem features and a classification algorithm, so that the labeling information is obtained.
204. Matching a corresponding decision tree from a pre-constructed index diagnosis logic table based on the labeling information;
in this step, the scene-related problem is an evaluation item for evaluating the control performance of the automatic driving algorithm in a corresponding scene; the index diagnosis logic table comprises decision trees for evaluating different indexes.
In this embodiment, the decision tree may be understood as a logic tree constructed based on an evaluation algorithm of an automatic driving algorithm, and is specifically implemented as follows:
extracting an automatic driving algorithm in an automatic driving system, and determining a scene corresponding to the algorithm;
determining test content according to the scene, and constructing an accident problem based on the test content;
analyzing scene-related problems corresponding to the accident problems, and configuring evaluation contents;
and constructing nodes and leaf nodes by using a tree structure according to the evaluation content corresponding to each accident problem to generate a corresponding decision tree, wherein leaf nodes in the decision tree are connected with the nodes, and all the nodes are connected to form the decision tree.
In practical application, a marking information field is added in the standard diagnosis logic table, after marking information is determined, a decision tree with field information equal to the marking information is matched from the index diagnosis logic table based on the marking information, and then the decision tree screened out through the marking information is screened based on categories to obtain a decision tree corresponding to an abnormal index.
Analyzing the evaluation items of all the trunks in the decision tree and the evaluation contents of all the nodes under the evaluation items 205;
in practical applications, the decision tree is a tree structure of the decision tree by organizing the logic of each evaluated index, the internal node of each decision tree is a problem related to a scene proposed according to the evaluated index, and the leaf nodes are different diagnosis results. When the decision tree is constructed, a corresponding decision tree is obtained according to the evaluation index type of a scene, a final evaluation result of the scene is obtained by answering a question, and the decision tree meeting the condition is output based on the evaluation result.
206, extracting vehicle control parameters in the control data based on the evaluation items;
in the step, after the evaluation items in the decision tree are extracted, the control data are extracted and classified based on the evaluation items, and a vehicle control parameter structure chart matched with the decision tree is generated, wherein the structure chart is in a tree structure form.
207, taking the evaluation content of each node as an evaluation marking rule, and sequentially evaluating and marking the corresponding vehicle control parameters to obtain a marking result;
and 208, uploading the labeling result to the algorithm simulation platform for diagnosis and evaluation, and generating a corresponding simulation test report.
Understandably, simulation test reports are output independently based on different problems or scenes, for example, scene unreal problems are fed back to smart agent teams to develop more real interaction between the obstacles and the main vehicle; the evaluation index algorithm problem is fed back to the accuracy of the optimization algorithm of an evaluation algorithm engineer; feeding the scene incomparable problem back to the simulation group to optimize the version comparison simulation mode; and feeding back the problem of the planning algorithm to the engineer of the planning algorithm for planning algorithm iteration.
Specifically, the step is realized by determining the corresponding algorithm fault problem type based on the marking result;
carrying out simulation diagnosis by using a scene model corresponding to the type of the algorithm fault problem through the algorithm simulation platform to obtain a diagnosis result;
and generating a corresponding simulation test report according to the diagnosis result and a report format corresponding to the type of the algorithm fault problem.
In practical application, the performance evaluation of the automatic driving algorithm is mainly realized on the basis of a system formed by a simulation test platform, a labeling platform and a database, as shown in fig. 3, firstly, an algorithm engineer submits a test task on a cloud simulation computing platform, and after the task is completed, a scene with a changed evaluation index is sent to the labeling platform for diagnosis; the labeling platform displays corresponding decision trees according to different evaluation index categories; and (4) marking according to decision tree logic by a marker, finally writing the diagnosis result of the marking platform into the simulation result database, generating a test report and sending the test report to an algorithm engineer.
And the decision tree of the evaluation algorithm diagnoses the simulation comparison result by using the tree structure according to the evaluation algorithm type, and classifies the internal nodes by using the accident problem type. The results of each diagnosis are fed back to the corresponding algorithm team to drive the algorithm development. The feedback flow chart of the diagnosis result is shown in a figure 4, the unreal scene problem is fed back to a smart agent team to develop and enable the obstacle to interact with the main vehicle more truly; the evaluation index algorithm problem is fed back to the accuracy of the optimization algorithm of an evaluation algorithm engineer; feeding the scene incomparable problem back to the simulation group to optimize the version comparison simulation mode; and feeding back the problem of the planning algorithm to the engineer of the planning algorithm for iteration of the planning algorithm.
In practical application, the development of a decision tree for evaluating indexes is classified by accident problem classification, the authenticity of scenes on two sides is determined by diagnosing problems, the quality of the behavior of the main vehicle is judged on the real basis, and by taking a security reduction decision tree as an example, an internal node in the decision tree is judged by using the following problems:
1. judging the error type of the original accident main vehicle driving behavior according to the camera and the audio frequency of the security officer;
2. whether the collision object is sensed and detected too late or not, the host vehicle is not in time to brake, and collision is caused;
3. if the collided object knocks the main vehicle, whether the main vehicle has a brake smaller than-2 m/s/s before the collision;
4. in the road test, whether the behavior of the obstacle with safety risk to the main vehicle in the simulation is reckless or not;
5. whether the interaction of the obstacle with the host vehicle in the two software versions is real;
6. in version b, whether the host vehicle is at risk of colliding with an object whose assessment algorithm is exposed to danger.
In the embodiment of the invention, the abnormal indexes in the simulation test data are evaluated in a decision tree mode to obtain the evaluation result so as to output the test report, the marking and evaluation contents of the indexes are converted into the decision tree in such a mode, and the indexes can be automatically evaluated by executing the decision tree, so that the situation that whether a diagnostician knows the man-vehicle algorithm and the evaluation algorithm is realized in a version comparison diagnosis task, and the training period is longer. By converting the diagnosis logic into the decision tree, the data annotator can perform diagnosis work of version comparison, and the personnel training time and the personnel cost are greatly shortened.
Meanwhile, judgment and comparison of version comparison diagnosis are subjective, and the number of misdiagnoses can be effectively reduced by standardizing diagnosis logic into a tree-type logic structure.
Furthermore, the version comparison diagnosis has high requirement on effectiveness, and previously, because the limited number of test engineers cannot quickly finish all diagnosis requirements, a marker can more quickly finish a diagnosis task, thereby effectively accelerating the iteration speed of a planning algorithm.
Referring to fig. 5, the method for evaluating the performance of the autopilot algorithm in the embodiment of the present invention is described above, and the apparatus for evaluating the performance of the autopilot algorithm in the embodiment of the present invention is described below, in which one embodiment of the apparatus for evaluating the performance of the autopilot algorithm in the embodiment of the present invention includes:
the system comprises an acquisition module 501, a processing module and a display module, wherein the acquisition module 501 is used for acquiring simulation test data of respective automatic driving algorithms in an automatic driving system and analyzing the simulation test data to obtain abnormal indexes and corresponding scene data;
a matching module 502, configured to determine a category of the abnormal index, and match, based on the category, a corresponding decision tree from a pre-constructed index diagnosis logic table, where the decision tree includes at least one node, and each node corresponds to different evaluation contents of the index;
and the evaluation module 503 is configured to evaluate the corresponding scene data according to the evaluation content corresponding to each node in the decision tree, and generate a simulation test report of the automatic driving algorithm based on the evaluation result.
In the embodiment of the invention, the evaluation indexes in the algorithm are used for constructing the evaluated decision tree, and the index data in the acquired scene data for simulating and executing the algorithm is evaluated based on the decision tree so as to realize automatic evaluation of the indexes. Meanwhile, the decision tree mode avoids the problem that misdiagnosis is easy to occur due to subjective judgment.
Referring to fig. 6, another embodiment of the performance evaluation apparatus for an autopilot algorithm according to an embodiment of the present invention includes:
the acquisition module 501 is configured to acquire simulation test data of respective automatic driving algorithms in an automatic driving system, and analyze the simulation test data to obtain abnormal indexes and corresponding scene data;
a matching module 502, configured to determine a category of the abnormal index, and match, based on the category, a corresponding decision tree from a pre-constructed index diagnosis logic table, where the decision tree includes at least one node, and each node corresponds to different evaluation contents of the index;
and the evaluation module 503 is configured to evaluate the corresponding scene data according to the evaluation content corresponding to each node in the decision tree, and generate a simulation test report of the automatic driving algorithm based on the evaluation result.
Optionally, the acquisition module 501 includes:
a determination unit 5011 for determining an automatic driving algorithm to be evaluated;
the acquisition unit 5012 is configured to acquire corresponding real driving data and simulation test data from an automatic driving system and an algorithm simulation platform respectively based on the automatic driving algorithm;
an extracting unit 5013, configured to compare the real driving data with the simulation test data, and extract corresponding scene data from the simulation test data based on a comparison result;
the first analyzing unit 5014 is configured to analyze the index of the scene data to obtain an abnormal index.
Optionally, the first parsing unit 5014 is specifically configured to:
identifying a main vehicle running track and control data in the running process in the scene data;
matching the main vehicle running track and the control data with preset scene definition information, and determining a scene type based on a matching result;
according to the scene type, performance parameters related to the scene are extracted from the control data, and an abnormality index is determined based on the performance parameters.
Optionally, the matching module 502 includes:
a type querying unit 5021, configured to determine a category of the abnormal indicator in a scene of a scene type according to a correspondence between the scene type and each indicator;
a labeling unit 5022, configured to determine a corresponding scene-related question according to the scene type, and determine corresponding labeling information based on the scene-related question, where the scene-related question is an evaluation item for evaluating control performance of the automatic driving algorithm in a corresponding scene;
a matching unit 5023, configured to match a corresponding decision tree from a pre-constructed index diagnosis logic table based on the tagging information, where the index diagnosis logic table includes decision trees for evaluating different indexes.
Optionally, the evaluation module 503 includes:
a second parsing unit 5031, configured to parse evaluation items of each trunk in the decision tree and evaluation contents of each node under the evaluation items;
a parameter extraction unit 5032 configured to extract a vehicle control parameter in the control data based on the evaluation item;
an evaluation unit 5033, configured to use the evaluation content of each node as an evaluation labeling rule, sequentially evaluate and label corresponding vehicle control parameters to obtain a labeling result;
a report generating unit 5034, configured to upload the annotation result to the algorithm simulation platform for diagnosis and evaluation, and generate a corresponding simulation test report.
Optionally, the report generating unit 5034 is specifically configured to:
determining the type of the corresponding algorithm fault problem based on the labeling result;
carrying out simulation diagnosis by using a scene model corresponding to the type of the algorithm fault problem through the algorithm simulation platform to obtain a diagnosis result;
and generating a corresponding simulation test report according to the diagnosis result and a report format corresponding to the type of the algorithm fault problem.
Optionally, the performance evaluation apparatus for an automatic driving algorithm further includes a decision tree construction module 504, which is specifically configured to:
extracting an automatic driving algorithm in an automatic driving system, and determining a scene corresponding to the algorithm;
determining test content according to the scene, and constructing an accident problem based on the test content;
analyzing scene-related problems corresponding to the accident problems, and configuring evaluation contents;
and constructing nodes and leaf nodes by using a tree structure according to the evaluation content corresponding to each accident problem to generate a corresponding decision tree, wherein leaf nodes in the decision tree are connected with the nodes, and all the nodes are connected to form the decision tree.
In the embodiment of the invention, the abnormal indexes in the simulation test data are evaluated in a decision tree mode to obtain the evaluation result so as to output the test report, the marking and evaluation contents of the indexes are converted into the decision tree in such a mode, and the indexes can be automatically evaluated by executing the decision tree, so that the unmanned vehicle algorithm and the evaluation indexes are deeply known, a test engineer spends a great deal of time in the task, and the cost is higher. Under the condition that the number of test engineers is limited, if a large number of version comparison and diagnosis requirements cannot be completed in time, the iteration cycle of the algorithm is delayed, and the development efficiency is influenced. Meanwhile, the decision tree mode avoids the problem that misdiagnosis is easy to occur due to subjective judgment.
Fig. 5 and 6 describe the performance evaluation device of the automatic driving algorithm in the embodiment of the present invention in detail from the perspective of the modular functional entity, and the computer device in the embodiment of the present invention is described in detail from the perspective of the hardware processing.
Fig. 7 is a schematic structural diagram of a computer device 700 according to an embodiment of the present invention, where the computer device 700 may have a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 710 (e.g., one or more processors) and a memory 720, one or more storage media 730 (e.g., one or more mass storage devices) for storing applications 733 or data 732. Memory 720 and storage medium 730 may be, among other things, transient storage or persistent storage. The program stored on the storage medium 730 may include one or more modules (not shown), each of which may include a series of computer program operations for the computer device 700. Still further, the processor 710 may be arranged to communicate with the storage medium 530 to execute a series of computer program operations in the storage medium 730 on the computer device 700.
The computer device 700 may also include one or more power supplies 740, one or more wired or wireless network interfaces 550, one or more input-output interfaces 760, and/or one or more operating systems 731, such as Windows Server, Mac OS X, Unix, Linux, FreeBSD, and the like. Those skilled in the art will appreciate that the computer device configuration illustrated in FIG. 7 does not constitute a limitation of computer devices, and may include more or fewer components than those illustrated, or some components may be combined, or a different arrangement of components.
The present invention also provides a computer device comprising a memory and a processor, wherein the memory stores a computer-readable computer program, and when the computer-readable computer program is executed by the processor, the processor executes the steps of the performance evaluation method of the automatic driving algorithm in the embodiments.
The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, and which may also be a volatile computer-readable storage medium, having stored therein a computer program, which, when run on a computer, causes the computer to perform the steps of the method for evaluating the performance of an autopilot algorithm.
It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.
The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes several computer programs to enable a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.
The above embodiments are only used to illustrate the technical solution of the present invention, and not to limit the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.
Claims (10)
1. A performance evaluation method of an automated driving algorithm, characterized by comprising:
acquiring simulation test data of each automatic driving algorithm in an automatic driving system, and analyzing the simulation test data to obtain abnormal indexes and corresponding scene data;
determining the category of the abnormal index, and matching a corresponding decision tree from a pre-constructed index diagnosis logic table based on the category, wherein the decision tree comprises at least one node, and each node corresponds to different evaluation contents of the index;
and evaluating corresponding scene data according to the evaluation content corresponding to each node in the decision tree, and generating a simulation test report of the automatic driving algorithm based on the evaluation result.
2. The method for evaluating performance of an autopilot algorithm according to claim 1, wherein the step of obtaining simulation test data of respective autopilot algorithms in an autopilot system and analyzing the simulation test data to obtain an abnormal index and corresponding scene data comprises:
determining an automatic driving algorithm to be evaluated;
respectively acquiring corresponding real driving data and simulation test data from an automatic driving system and an algorithm simulation platform based on the automatic driving algorithm;
comparing the real driving data with the simulation test data, and extracting corresponding scene data from the simulation test data based on the comparison result;
and analyzing the indexes of the scene data to obtain abnormal indexes.
3. The performance evaluation method of the automatic driving algorithm according to claim 2, wherein the analyzing the scene data to obtain an abnormal index comprises:
identifying a main vehicle running track and control data in the running process in the scene data;
matching the main vehicle running track and the control data with preset scene definition information, and determining a scene type based on a matching result;
according to the scene type, performance parameters related to the scene are extracted from the control data, and an abnormality index is determined based on the performance parameters.
4. The method of evaluating performance of an autonomous driving algorithm according to claim 3, wherein said determining a category of said abnormal index, based on which a corresponding decision tree is matched from a pre-constructed index diagnosis logic table, comprises:
determining the category of the abnormal index in the scene under the scene type according to the corresponding relation between the scene type and each index;
determining corresponding scene-related problems according to the scene types, and determining corresponding labeling information based on the scene-related problems, wherein the scene-related problems are evaluation items used for evaluating the control performance of the automatic driving algorithm in the corresponding scene;
and matching a corresponding decision tree from a pre-constructed index diagnosis logic table based on the labeling information, wherein the index diagnosis logic table comprises decision trees for evaluating different indexes.
5. The method for evaluating performance of an automatic driving algorithm according to claim 3, wherein the evaluating corresponding scene data according to the evaluation content corresponding to each node in the decision tree and generating the simulation test report of the automatic driving algorithm based on the evaluation result comprises:
analyzing the evaluation items of all the trunks in the decision tree and the evaluation contents of all the nodes under the evaluation items;
extracting vehicle control parameters in the control data based on the evaluation item;
taking the evaluation content of each node as an evaluation marking rule, and sequentially evaluating and marking the corresponding vehicle control parameters to obtain marking results;
and uploading the labeling result to the algorithm simulation platform for diagnosis and evaluation, and generating a corresponding simulation test report.
6. The method for evaluating the performance of the automatic driving algorithm according to claim 5, wherein the uploading the labeling result to the algorithm simulation platform for diagnosis evaluation and generating a corresponding simulation test report comprises:
determining the type of the corresponding algorithm fault problem based on the labeling result;
carrying out simulation diagnosis by using a scene model corresponding to the type of the algorithm fault problem through the algorithm simulation platform to obtain a diagnosis result;
and generating a corresponding simulation test report according to the diagnosis result and a report format corresponding to the type of the algorithm fault problem.
7. The method for evaluating performance of an autopilot algorithm according to one of claims 1-6, characterized in that before the obtaining of simulation test data of respective autopilot algorithms in an autopilot system and the analysis of the simulation test data to obtain anomaly indicators and corresponding scene data, it further comprises:
extracting an automatic driving algorithm in an automatic driving system, and determining a scene corresponding to the algorithm;
determining test content according to the scene, and constructing an accident problem based on the test content;
analyzing scene-related problems corresponding to the accident problems, and configuring evaluation contents;
and constructing nodes and leaf nodes by using a tree structure according to the evaluation content corresponding to each accident problem to generate a corresponding decision tree, wherein leaf nodes in the decision tree are connected with the nodes, and all the nodes are connected to form the decision tree.
8. A performance evaluation device of an automated driving algorithm, characterized by comprising:
the system comprises an acquisition module, a data processing module and a data processing module, wherein the acquisition module is used for acquiring simulation test data of respective automatic driving algorithms in an automatic driving system and analyzing the simulation test data to obtain abnormal indexes and corresponding scene data;
the matching module is used for determining the category of the abnormal index and matching a corresponding decision tree from a pre-constructed index diagnosis logic table based on the category, wherein the decision tree comprises at least one node, and each node corresponds to different evaluation contents of the index;
and the evaluation module is used for evaluating the corresponding scene data according to the evaluation content corresponding to each node in the decision tree and generating a simulation test report of the automatic driving algorithm based on the evaluation result.
9. A computer device, characterized in that the computer device comprises: a memory and at least one processor, the memory having stored therein a computer program;
the at least one processor invokes the computer program in the memory to cause the computer device to perform the steps of the performance evaluation method of the autopilot algorithm of any of claims 1-7.
10. A computer-readable storage medium, on which a computer program is stored which, when being executed by a processor, carries out the steps of the method for performance evaluation of an autopilot algorithm according to one of the claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210319751.4A CN114935918A (en) | 2022-03-29 | 2022-03-29 | Performance evaluation method, device and equipment of automatic driving algorithm and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210319751.4A CN114935918A (en) | 2022-03-29 | 2022-03-29 | Performance evaluation method, device and equipment of automatic driving algorithm and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN114935918A true CN114935918A (en) | 2022-08-23 |
Family
ID=82861800
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210319751.4A Pending CN114935918A (en) | 2022-03-29 | 2022-03-29 | Performance evaluation method, device and equipment of automatic driving algorithm and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114935918A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117077043A (en) * | 2023-10-17 | 2023-11-17 | 深圳翱翔锐影科技有限公司 | Evaluation method of CdZnTe photon counting detector based on leakage current response |
-
2022
- 2022-03-29 CN CN202210319751.4A patent/CN114935918A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117077043A (en) * | 2023-10-17 | 2023-11-17 | 深圳翱翔锐影科技有限公司 | Evaluation method of CdZnTe photon counting detector based on leakage current response |
| CN117077043B (en) * | 2023-10-17 | 2024-01-30 | 深圳翱翔锐影科技有限公司 | Evaluation method of CdZnTe photon counting detector based on leakage current response |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103392187B (en) | Utilize from the statistics of object trajectory data acquisition and the scene activity analysis of semantic feature | |
| CN116450399B (en) | Fault diagnosis and root cause positioning method for micro service system | |
| CN107147639A (en) | A kind of actual time safety method for early warning based on Complex event processing | |
| CN109204389B (en) | Fault diagnosis and self-healing method and system for subway equipment | |
| CN112949715A (en) | SVM (support vector machine) -based rail transit fault diagnosis method | |
| CN106776339A (en) | Automated testing method and device | |
| CN113890821B (en) | Log association method and device and electronic equipment | |
| CN115830399B (en) | Classification model training method, device, equipment, storage medium and program product | |
| CN112507623A (en) | Algorithm middling station construction method and system | |
| CN109871002A (en) | The identification of concurrent abnormality and positioning system based on the study of tensor label | |
| CN114935918A (en) | Performance evaluation method, device and equipment of automatic driving algorithm and storage medium | |
| CN116523284A (en) | Automatic evaluation method and system for business operation flow based on machine learning | |
| CN113672506B (en) | Dynamic proportion test case sorting and selecting method and system based on machine learning | |
| CN114880312A (en) | Flexibly-set application system service data auditing method | |
| CN117114420B (en) | Image recognition-based industrial and trade safety accident risk management and control system and method | |
| KR102411291B1 (en) | Method of evaluating quality of smart factory data | |
| CN115034278A (en) | Performance index abnormality detection method and device, electronic equipment and storage medium | |
| CN113138906A (en) | Call chain data acquisition method, device, equipment and storage medium | |
| CN117057142A (en) | Digital twinning-based vehicle test data processing method and system | |
| CN112115173A (en) | Automatic driving data processing and tracking method and system and vehicle | |
| Agostinelli | Automated segmentation of user interface logs using trace alignment techniques | |
| Schuh et al. | Event log-based weaknesses detection in business processes | |
| CN113204246A (en) | Unmanned aerial vehicle running state detection method | |
| CN109685453B (en) | Method for intelligently identifying effective paths of workflow | |
| CN109754159B (en) | Method and system for extracting information of power grid operation log |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |