CN112541708A - Index determination method and device and electronic equipment - Google Patents

Index determination method and device and electronic equipment Download PDF

Info

Publication number
CN112541708A
CN112541708A CN202011559016.8A CN202011559016A CN112541708A CN 112541708 A CN112541708 A CN 112541708A CN 202011559016 A CN202011559016 A CN 202011559016A CN 112541708 A CN112541708 A CN 112541708A
Authority
CN
China
Prior art keywords
index
values
indexes
indicators
determining
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202011559016.8A
Other languages
Chinese (zh)
Other versions
CN112541708B (en
Inventor
李建平
李丹
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baidu Netcom Science and Technology Co Ltd
Original Assignee
Beijing Baidu Netcom Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baidu Netcom Science and Technology Co Ltd filed Critical Beijing Baidu Netcom Science and Technology Co Ltd
Priority to CN202011559016.8A priority Critical patent/CN112541708B/en
Priority claimed from CN202011559016.8A external-priority patent/CN112541708B/en
Publication of CN112541708A publication Critical patent/CN112541708A/en
Application granted granted Critical
Publication of CN112541708B publication Critical patent/CN112541708B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G06Q10/063—Operations research, analysis or management
    • G06Q10/0639—Performance analysis of employees; Performance analysis of enterprise or organisation operations
    • G06Q10/06393—Score-carding, benchmarking or key performance indicator [KPI] analysis
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N20/00—Machine learning

Landscapes

  • Business, Economics & Management (AREA)
  • Engineering & Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • Theoretical Computer Science (AREA)
  • Development Economics (AREA)
  • Educational Administration (AREA)
  • Economics (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Software Systems (AREA)
  • Strategic Management (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • Tourism & Hospitality (AREA)
  • Data Mining & Analysis (AREA)
  • General Business, Economics & Management (AREA)
  • Operations Research (AREA)
  • Marketing (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Quality & Reliability (AREA)
  • Evolutionary Computation (AREA)
  • Medical Informatics (AREA)
  • Game Theory and Decision Science (AREA)
  • Computing Systems (AREA)
  • General Engineering & Computer Science (AREA)
  • Mathematical Physics (AREA)
  • Traffic Control Systems (AREA)

Abstract

The application discloses an index determination method, an index determination device and electronic equipment, and relates to the technical field of artificial intelligence such as automatic driving in computer technology. The specific implementation scheme is as follows: determining values of a plurality of first indicators for each of a plurality of segments of autonomous driving data; sorting the values of each first index respectively, and dividing the sorted values of each first index into a plurality of index value sets respectively; determining a correlation coefficient between the plurality of first indexes and the second index by using the index value sets of the plurality of first indexes and the value of the second index of the data segment sets of the index value sets of the plurality of first indexes; a target index is determined from the plurality of first indexes based on the correlation coefficient. Through the process, the incidence relation between the first indexes and the second indexes can be determined, the target index is determined according to the incidence relation, and the accuracy of the determined target index can be improved.

Description

Index determination method and device and electronic equipment
Technical Field
The present application relates to the field of artificial intelligence technologies such as automatic driving in computer technologies, and in particular, to an index determination method and apparatus, and an electronic device.
Background
With the continuous development of automatic driving technology, more and more automatic driving vehicles are provided, and the automatic driving vehicles are more and more intelligent. Because the scenes to be processed by the automatic driving vehicle are very complex and various, the automatic driving algorithm becomes an ultra-complex system along with continuous iteration of the automatic driving algorithm, and the system is generally divided into a plurality of modules of perception, prediction, decision, planning and control. In the actual development process, each algorithm module iterates independently, and the actual effect of the final vehicle end is concerned from the perspective of users or products, rather than the individual quality of a certain module. Usually, each module can be subjected to independent iteration to obtain a series of evaluation indexes for measuring the algorithm effect of the module, the indexes can evaluate the algorithm effect from multiple dimensions, and the index items are more.
At present, in order to analyze the relationship between a single module index and a final vehicle end effect index and select a key single module iteration index, a commonly adopted method is to obtain the characteristics of the single module index by manually analyzing the actual case of sudden braking of an automatic driving vehicle and then select the key index.
Disclosure of Invention
The application provides an index determination method and device and electronic equipment.
In a first aspect, an embodiment of the present application provides an index determining method, including:
determining values of a plurality of first indicators for each of a plurality of segments of autonomous driving data;
sorting the values of each first index respectively, and dividing the sorted values of each first index into a plurality of index value sets respectively;
determining a correlation coefficient between the plurality of first indexes and a second index by using values of the second index of a data segment set of the index value sets of the plurality of first indexes, wherein the data segment set of a target index value set is a set of data segments corresponding to index values in the target index value set, the target index value set is any one of the index value sets of the plurality of first indexes, and the plurality of automatic driving data segments comprise the data segment set of the target index value set;
determining a target index from the plurality of first indexes based on the correlation coefficient.
In the index determination method according to the embodiment of the application, instead of manually analyzing the automatic driving data to determine the target index, the values of the first indexes of each of the automatic driving data segments are determined, the values of the first indexes are sorted respectively, the sorted values of the first indexes are divided into the index value sets respectively, then the values of the second indexes of the data segment sets of the index value sets of the first indexes and the index value sets of the first indexes are utilized to determine the correlation coefficients between the first indexes and the second indexes, and the target index is determined according to the correlation coefficients. That is, in the present embodiment, the association relationship between the plurality of first indexes and the second index may be determined through the above-described procedure, and the target index is determined according to the association relationship, so that the accuracy of the determined target index may be improved.
In a second aspect, an embodiment of the present application provides an index determining apparatus, including:
a first determination module to determine values of a plurality of first indicators for each of a plurality of segments of autopilot data;
the processing module is used for sorting the values of each first index respectively and dividing the sorted values of each first index into a plurality of index value sets respectively;
a second determining module, configured to determine a correlation coefficient between the plurality of first indicators and a second indicator of a data segment set of the indicator value sets of the plurality of first indicators by using values of the plurality of first indicators and a second indicator of the data segment set of the indicator value sets of the plurality of first indicators, where a data segment set of a target indicator value set is a set of data segments corresponding to indicator values in the target indicator value set, the target indicator value set is any one of the indicator value sets of the plurality of first indicators, and the plurality of automatic driving data segments include the data segment set of the target indicator value set;
a third determining module for determining a target index from the plurality of first indexes based on the correlation coefficient.
In a third aspect, an embodiment of the present application further provides an electronic device, including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the index determination methods provided by the embodiments of the present application.
In a fourth aspect, an embodiment of the present application further provides a non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the index determining method provided by the embodiments of the present application.
In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program that, when being executed by a processor, implements the index determination method provided in the embodiments of the present application.
Drawings
The drawings are included to provide a better understanding of the present solution and are not intended to limit the present application. Wherein:
FIG. 1 is a schematic flow chart diagram illustrating an index determination method according to an embodiment of the present disclosure;
FIG. 2 is a schematic diagram of 100 sorting bins for the ith first indicator of one embodiment provided herein;
FIG. 3 is a graph of a correlation coefficient between an ith first indicator and an emergency brake indicator according to an embodiment provided herein;
FIG. 4 is a block diagram of an index determination device according to an embodiment provided herein;
fig. 5 is a block diagram of an electronic device for implementing the index determination method according to the embodiment of the present application.
Detailed Description
The following description of the exemplary embodiments of the present application, taken in conjunction with the accompanying drawings, includes various details of the embodiments of the application for the understanding of the same, which are to be considered exemplary only. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the present application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
As shown in fig. 1, according to an embodiment of the present application, there is provided an index determining method applicable to an electronic device, the method including:
step S101: values of a plurality of first indicators for each of a plurality of segments of autonomous driving data are determined.
The plurality of automatic driving data segments are data recorded in the driving process of the automatic driving vehicle, the automatic driving vehicle can encounter various driving events in the whole driving process, for example, the automatic driving vehicle is switched at a certain intersection and can encounter various obstacles, for example, the automatic driving vehicle encounters obstacles at a certain intersection and the like, the data can be recorded in the automatic driving data in the driving process, and the automatic driving data can be divided into a plurality of automatic driving data segments.
In the present embodiment, a plurality of pieces of automatic driving data are first acquired, and then values of the plurality of first indexes for each piece of data may be determined based on the plurality of pieces of automatic driving data. The plurality of first indexes may be determined in advance, and after the plurality of segments of automatic driving data are obtained, the values of the plurality of first indexes for each segment of data may be determined. For example, if the plurality of first indicators includes 8 first indicators and the plurality of segments of automatic driving data includes 60 ten thousand segments of data, then for each of the 60 ten thousand segments of data, there are 8 values of the first indicators, respectively, and thus, the value of each of the first indicators includes 60 ten thousand values.
As an example, the plurality of first indicators may include a location prediction error indicator and a speed prediction error indicator, and in addition, the plurality of first indicators may further include a route prediction accuracy indicator, a predicted route recall rate, and the like. Alternatively, the plurality of first indexes may be indexes based on an obstacle of the autonomous vehicle during traveling. Alternatively, the position prediction error indicator may be a position prediction error indicator within a predetermined time period in the future, and the speed prediction error indicator may be a speed prediction error indicator within a predetermined time period, for example, the predetermined time period may include 1 second in the future, 3 seconds in the future, 5 seconds in the future, and the like, and the position prediction error indicator includes a position prediction error indicator of 1 second in the future, a position prediction error indicator of 3 seconds in the future, and a position prediction error indicator of 5 seconds in the future, and the speed prediction error indicator includes a speed prediction error indicator of 1 second in the future, a speed prediction error indicator of 3 seconds in the future, and a speed prediction error indicator of 5 seconds in the future. It should be noted that the future is relative to the current time, for example, if the current time is eight points, then 1 second in the future refers to eight points and one second.
For example, when the autonomous vehicle encounters an obstacle a at time t0 during traveling, the speed of the obstacle a is V0 and the position thereof is P0, after 1 second has elapsed, the speed of the obstacle is V1 and the position thereof is P1, these actual data are recorded in the autonomous driving data, during this time period, the speed V11 and the position P11 of the obstacle after 1 second can be predicted after t0 encounters the obstacle a, the predicted speed and position are also recorded in the autonomous driving data, and then the predicted speed and position can be compared with the actual speed and position according to the predicted speed and position, and the value of the position prediction error index, the value of the speed prediction error index, and the like can be determined.
Step S102: and sorting the values of each first index respectively, and dividing the sorted values of each first index into a plurality of index value sets respectively.
Since each data segment is respectively corresponding to a plurality of values of the first index, a certain first index includes a plurality of values, and the number of the values of the first index is the same as the number of the data segments, and the values of each first index can be respectively sorted. For example, if there are 60 ten thousand data segments, for index a, there are corresponding values under each data segment, and index a has 60 ten thousand values, and for index B, there are corresponding values under each data segment, and index a has 60 ten thousand values, so that the values of index a and the values of index B can be sorted, and the sorted values of each index can be obtained. Then, the sorted values of each first index are divided into a plurality of index value sets, that is, for each first index, a plurality of index value sets are respectively corresponding to each first index. For example, there are a plurality of index value sets corresponding to index a, and a plurality of index value sets corresponding to index B. For example, each first index has 60 ten thousand values, and the values of each first index can be equally divided into 100 index value sets, so that each index value set includes 6000 index values. For example, the index value set of the index a includes 6000 index a values, and the index value set of the index B includes 6000 index B values.
As an example, the values of each first index may be sorted in descending order, that is, the values are arranged in the higher order, and thus, the sorted values of each first index may be divided into a plurality of index value sets in descending order, and optionally, the sorted values of each first index may be divided into a plurality of index value sets in descending order, or the sorted values of each first index may be divided into a plurality of index value sets in ascending order, that is, each index value set includes a predetermined number of index values (that is, in a case where the number of the plurality of data segments is divisible by the predetermined number, each index value set includes a predetermined number of index values, and if the number of the plurality of index values is not divisible, the number of index values in one index value set is a remainder of dividing the number of the plurality of data segments by the predetermined number, the number of the index values of the other index value sets is the same, and is a result rounding value obtained by dividing the number of the plurality of data fragments by the preset number), or the number of the plurality of index value sets of any first index is the preset set number and the number of the index values in the index value set is the same (in the case that the number of the plurality of data fragments can be evenly divided by the preset set number, if the number of the index values in one index value set cannot be evenly divided, the number of the index values in one index value set is the sum of the remainder of dividing the number of the plurality of data fragments by the preset set number and the result rounding value obtained by dividing the number of the plurality of data fragments by the preset set number, and the number of the index values in the other index value sets is the same and is the result rounding value obtained by dividing the number of the plurality of data fragments by the preset set number). It will be appreciated that the closer values are in one set.
As an example, for 60 ten thousand values of the index a, the preset number is 6000, after sorting the values in the descending order, every 6000 values form an index value set, the 6000 values arranged in front are used as an index value set, the values sorted from 6001 to 12000 are used as an index value set, and so on, until the 6000 values arranged at the end are used as an index value set, that is, the segmentation of 60 ten thousand values of the index a is completed, and 100 index value sets are obtained.
Step S103: the method includes determining a correlation coefficient between a plurality of first indexes and a second index using values of the second index of a data segment set of a plurality of index value sets of the first indexes and the index value sets of the first indexes.
The data fragment set of the target index value set is a set of data fragments corresponding to the index values in the target index value set, the target index value set is any one set of the index value sets of the first indexes, and the automatic driving data fragments comprise the data fragment set of the target index value set. Namely, the data segment set corresponding to any index value set is in a plurality of automatic driving data segments. The correlation coefficient may characterize an association between the first index and the second index. In addition, before determining the correlation coefficient between the plurality of first indexes and the second index by using the values of the second indexes in the data segment sets of the index value sets of the plurality of first indexes, the value of the second index in the data segment sets of the index value sets of the plurality of first indexes may be determined.
It should be noted that, since each first index corresponds to a plurality of index value sets, the index value sets of the plurality of first indexes may be understood as a plurality of index value sets including each first index, for example, the plurality of first indexes includes an index a and an index B, the index a corresponds to the plurality of index value sets, the index B corresponds to the plurality of index value sets, and the index value sets of the plurality of first indexes include the plurality of index value sets corresponding to the index a and the plurality of index value sets corresponding to the index B. The data segment set of the index value sets of the first indexes is a data segment set comprising a plurality of index value sets of each first index, each index value set corresponds to one data segment set, and one value of the second index can be determined on each data segment set. For example, if one index set of index a includes one value X and another value Y of index a, the X value of index a is obtained on data segment P1, and the Y value of index a is obtained on data segment P2, the data segment set corresponding to the index set includes data segment P1 and data segment P2. As described above, the data segment sets of the index value sets of the first indexes include data segment sets corresponding to the index value sets corresponding to a and data segment sets corresponding to the index value sets corresponding to B.
As an example, the second indicator may include, but is not limited to, a sudden braking ratio, and the value of the second indicator of the data segment set is a ratio of the number of data segments in the data segment set where sudden braking occurs to the number of data segments in the data segment set, for example, if one data segment set includes 6000 data segments and there are 600 data segments where sudden braking occurs, the value of the second indicator of the data segment set is 300/6000, that is, 0.05.
Step S104: a target index is determined from the plurality of first indexes based on the correlation coefficient.
And determining correlation coefficients between the plurality of first indexes and the second indexes by using the values of the second indexes of the data segment sets of the index value sets of the plurality of first indexes, namely obtaining a plurality of correlation coefficients, wherein the number of the correlation coefficients is the same as that of the plurality of first indexes, and then determining a target index from the plurality of first indexes according to the correlation coefficients. It should be noted that the index determining method of the present embodiment may be applied to an algorithm iteration scenario of an automatic driving project.
In the index determination method according to the embodiment of the application, instead of manually analyzing the automatic driving data to determine the target index, the values of the first indexes of each of the automatic driving data segments are determined, the values of the first indexes are sorted respectively, the sorted values of the first indexes are divided into the index value sets respectively, then the values of the second indexes of the data segment sets of the index value sets of the first indexes and the index value sets of the first indexes are utilized to determine the correlation coefficients between the first indexes and the second indexes, and the target index is determined according to the correlation coefficients. That is, in the present embodiment, the association relationship between the plurality of first indexes and the second index may be determined through the above-described procedure, and the target index is determined according to the association relationship, so that the accuracy of the determined target index may be improved. Meanwhile, the labor cost can be reduced.
As an example, the target index is determined from the plurality of first indexes based on the correlation coefficient, that is, the determination of the key index is completed, the target index may be understood as an index having a larger influence on the second index from the plurality of first indexes, and the target index has a larger association with the second index, and the change of the target index can cause an obvious change of the second index. In this way, after the target index is determined from the plurality of first indexes based on the correlation coefficient, the target algorithm corresponding to the target index can be determined from the automatic driving algorithm, and the target algorithm can be iterated, so that the effect of optimizing the second index of the automatic driving vehicle can be improved. The method can better guide the iteration of the developer on the algorithm and the like through the target index, and improves the optimization effect of the target algorithm, thereby improving the second index optimization effect of the vehicle end.
In one embodiment, determining a correlation coefficient between a plurality of first indexes and a second index using values of the second index of a data segment set of a plurality of index value sets of the first indexes comprises:
determining a correlation coefficient between the ith first index and the second index using a first mean of each set of index values of the ith first index, a second mean of a plurality of sets of index values of the ith first index, and a value of the second index of the set of data fragments of each set of index values of the ith first index;
the number of the first indexes is N, N is an integer larger than 1, the first indexes comprise the ith first index, i is larger than or equal to 1 and is smaller than or equal to N, and the second average value is the average value of the first average values.
In the above process for determining the correlation coefficient between the ith first indicator and the second indicator, since the ith first indicator belongs to the plurality of first indicators, i is greater than or equal to 1 and less than or equal to N, that is, any first indicator in the plurality of first indicators can determine the correlation coefficient between the first indicator and the second indicator through the above process, so that the correlation coefficient between the plurality of first indicators and the second indicator can be determined.
In the process of determining the correlation coefficient between the first index and the second index in this embodiment, the first mean value of each index value set of the first index, the second mean values of the index value sets of the first index, and the value of the second index of the data segment set of each index value set of the first index are considered, so that the accuracy of the obtained correlation coefficient can be improved.
In one embodiment, the correlation coefficient between the ith first index and the second index is determined by the following formula:
Figure BDA0002859703640000081
wherein M isiNumber of sets of a plurality of index value sets, r, for the ith first indexiIs the correlation coefficient between the ith first index and the second index, YijIs a first mean of a jth set of index values from the plurality of sets of index values of the ith first index,
Figure BDA0002859703640000091
is the second mean value of the ith first index, XijIs a value of a second index of the data fragment set of the jth set of index values of the plurality of sets of index values of the ith first index,
Figure BDA0002859703640000092
is the mean of the values of the second index of the plurality of sets of index values of the ith first index. Therefore, the correlation coefficient between the first index and the second index is calculated through the formula, and the accuracy of the correlation coefficient can be improved.
In one embodiment, determining a target metric from a plurality of first metrics based on the correlation coefficients comprises:
sorting correlation coefficients between the plurality of first indexes and the second indexes;
determining at least one of the sorted correlation coefficients as a target index;
at least one correlation coefficient is larger than the rest correlation coefficients, and the rest correlation coefficients are coefficients except for at least one correlation coefficient in the correlation coefficients between the first indexes and the second indexes.
Since the larger the value of the correlation coefficient between the first index and the second index is, the stronger the correlation between the first index and the second index is, the larger the influence of the first index on the second index is, in this embodiment, at least one correlation coefficient with a larger correlation coefficient is selected from the plurality of first indexes as the target index, that is, the index with the stronger correlation with the second index is selected as the target index, so that the accuracy of the target index is improved, and the strength of correlation between the target index and the second index is ensured.
In one embodiment, determining the value of the first indicator for each of the plurality of autodrive data segments further comprises: obtaining autodrive data for an autodrive vehicle; and segmenting the automatic driving data by using the preset time length to obtain a plurality of automatic driving data segments.
That is, the automatic driving data is averagely segmented according to the preset time length, if the total time length of the automatic driving data can be divided by the preset time length, the time length of each automatic driving data segment obtained by segmentation is the same, if the total time length of the automatic driving data cannot be divided by the preset time length, the time length of one automatic driving data segment is smaller than the preset time length, the remainder of the total time length of the automatic driving data divided by the preset time length is the same, the time lengths of the rest automatic driving data segments are the same, and the integral value of the result of the total time length of the automatic driving data divided by the preset time length is downwards. For example, the preset time length may be 6 seconds, the automatic driving data is 1000 hours, and the automatic driving data of 1000 hours is divided according to 6 seconds, so that 60 ten thousand automatic driving data segments are obtained, and the time length of each data segment is 6 seconds.
In this embodiment, according to a preset duration, segmenting the automatic driving data to obtain a plurality of automatic driving data segments, determining values of a plurality of first indexes of each data segment in the plurality of automatic driving data segments, sorting the values of each first index, dividing the sorted values of each first index into a plurality of index value sets, determining correlation coefficients between the plurality of first indexes and a second index of the data segment set of the index value sets of the plurality of first indexes by using the index value sets of the plurality of first indexes and the values of the second index of the data segment set of the index value sets of the plurality of first indexes, determining the target index according to the correlation coefficients, and improving accuracy of the determined target index.
In one embodiment, determining the value of the first indicator for each of the plurality of autodrive data segments further comprises:
determining a plurality of reference indicators;
determining a reference index in each reference dimension based on a plurality of reference dimensions of the obstacle;
wherein the plurality of first indicators comprise reference indicators in a plurality of reference dimensions.
For example, there are m reference dimensions, and the number of the reference indexes is h, the reference indexes in each reference dimension can be determined, and then the reference indexes in the plurality of reference dimensions include h reference indexes in each reference dimension, that is, m is multiplied by h reference indexes, and then the plurality of first indexes include m is multiplied by h reference indexes.
In this embodiment, a plurality of reference dimensions of the obstacle are taken into consideration, and the reference indexes in the plurality of reference dimensions are taken as the plurality of first indexes, so that the accuracy of the value of the first index of the statistics can be improved.
As an example, the plurality of reference indicators may include a position prediction error indicator and a velocity prediction error indicator, such that the plurality of first indicators includes a position prediction error indicator and a velocity prediction error indicator in a plurality of reference dimensions, e.g., the reference dimensions include a reference dimension W1 and a reference dimension W2, and the plurality of first indicators may include a position prediction error indicator and a velocity prediction error indicator in a reference dimension W1 and a position prediction error indicator and a velocity prediction error indicator in a reference dimension W2, i.e., the plurality of first indicators includes four indicators. In addition, the plurality of reference indicators may further include a route prediction accuracy indicator and a predicted route recall rate, and the plurality of first indicators may further include a route prediction accuracy indicator and a predicted route recall rate in a plurality of reference dimensions.
The procedure of the above-described index determination method is specifically described below with an embodiment.
Triggered from the perspective of a user or a product, the driving effect of the whole automatic driving vehicle is concerned, generally, evaluation is performed according to factors such as body feeling, safety, driving efficiency, intelligence and the like, and each factor is measured by some determined indexes, for example, body feeling can be represented by a sudden brake ratio in each kilometer, safety can be represented by a collision ratio in each kilometer, and the like. All modules of the whole automatic driving system can influence the final vehicle driving effect, key indexes of single modules in iteration are extracted, and the relevance between the indexes of the single modules and the final vehicle driving effect indexes is essentially analyzed. Therefore, the embodiment of the application provides an index determining method for index relevance evaluation through large-scale data analysis, the relevance between a single module index and a final vehicle driving effect index is measured through a correlation coefficient, and a final key index, namely a target index, is extracted through the sequencing of the correlation coefficient. The procedure of the above-mentioned index determination method is illustrated below by taking as an example the analysis of the correlation between the first index of the module itself and the sudden braking index of the final vehicle (which may be the sudden braking ratio, etc.).
Take the example that the reference index includes the position prediction error index and the speed prediction error index of the future 1s, 3s and 5s of the obstacle. Firstly, each reference index is classified in a plurality of dimensions in a fine-grained manner, the specific dimensions and the categories are shown in the following table 1, the categories of each dimension are mutually crossed, a plurality of reference dimensions are formed after the crossing, and the reference indexes of the plurality of reference dimensions form a plurality of first indexes. The purpose of classifying the indexes in a finer granularity is to accurately and finely know which obstacle attributes have a greater influence on the sudden braking index of the vehicle. For example, in table 1, a reference dimension formed by combining the automobile, the key obstacle, the full life cycle of the obstacle, and the intersection within 60 meters around the vehicle is a first index, and the position prediction error index that can be obtained in the reference dimension in the future of the obstacle by 1s is a position prediction error index in the reference dimension formed by combining the automobile, the key obstacle, the full life cycle of the obstacle, and the intersection within 60 meters around the vehicle.
TABLE 1
Dimension (d) of Categories
Class of obstacle Automobiles, non-motor vehicles, pedestrians, all
Interaction intensity with the host vehicle Key obstacles, obstacles with interaction, all
Dimension of time Full life cycle of obstacle and T seconds before emergency brake of main vehicle
Range of obstacles Within and all Z meters around the main vehicle
Area of obstacle Crossing, non-crossing, all
Wherein T may be 8, Z may be 60, etc.
And then, segmenting the large-scale automatic driving data into a plurality of independent short scene data segments. A 6s cut of, for example, 1000h of autopilot data may yield 60 ten thousand segments of autopilot data.
A statistical algorithm is run to determine the value of the first indicator on each data segment and the results are saved.
The values of each first index are sorted and equally divided into 100 classification slots, i.e., index value sets, as shown in fig. 2. Then, the classification slots segmented according to the value of the first index can classify a plurality of automatic driving data segments, namely, each classification slot has a corresponding data segment set, and the value of the emergency brake index of the data segment set of each classification slot can be determined.
For the ith first index, the mean value of the ith first index in the jth classification slot can be counted as YijWith the sudden braking index (e.g., sudden braking ratio) in the jth classification slot as XijThe correlation between the ith first index and the sudden braking index can be characterized by using a Pearson correlation coefficient, and the formula is as follows:
Figure BDA0002859703640000121
Mithe number of sets of the plurality of index value sets, r, of the ith first index is 100iIs the correlation coefficient between the ith first index and the emergency brake index, YijIs the first mean of the jth set of index values in the ith first index 100 set of index values,
Figure BDA0002859703640000122
is the ith first indexTwo mean values, XijFor the value of the second index of the jth index value set data segment set in the 100 index value set of the ith first index,
Figure BDA0002859703640000123
is the mean of the values of the hard brake indicator of the plurality of sets of indicator values of the ith first indicator. Y isi1
Taking the first index as the position prediction error index of the obstacle in the future 3s of a reference dimension formed by combining the automobile, the key obstacle, the full life cycle of the obstacle, the area within 60 meters around the automobile and the intersection as an example, the first mean value and the emergency brake index of the first index are distributed as shown in fig. 3, the abscissa is the first mean value of the first index, the ordinate is the value of the emergency brake index, the emergency brake index and the first index are approximately in a positive correlation in trend, but the correlation is not strong, and the correlation coefficient r is 0.52.
For each first index, a corresponding correlation coefficient is obtained, and according to the sorting of the multiple correlation coefficients, the target index most relevant to the emergency brake index can be obtained from the sorted correlation coefficients.
In summary, in the method implemented by the present application, a key single module index, i.e., a target index, is extracted by analyzing the correlation (expressed by a mutual coefficient) between the index of the module and the vehicle effect index. And classifying the plurality of automatic driving data segments according to the value interval division of the single module indexes, and further establishing the relevance between the single module indexes and the vehicle effect indexes in each class. According to the method, on one hand, the problem that key indexes are invalid due to complexity of the system and the scene to be processed through an algorithm principle is avoided, the target indexes are guaranteed not to be influenced by complexity of the system and the scene, and accuracy of the determined target indexes is improved. On the other hand, the problems of cost and poor accuracy caused by manual data analysis are solved, partial poor cases are not preferred, and relevance is considered on the whole. Thus, the accuracy of the determined target index can be improved.
As shown in fig. 4, according to an embodiment of the present application, the present application further provides an index determining apparatus 400, including:
a first determination module 401 for determining values of a plurality of first indicators for each of a plurality of segments of autopilot data;
a processing module 402, configured to sort the values of each first index, and divide the sorted values of each first index into a plurality of index value sets;
a second determining module 403, configured to determine a correlation coefficient between the plurality of first indicators and the second indicator by using a value of a second indicator in a data segment set of the index value sets of the plurality of first indicators, where a data segment set of the target index value set is a set of data segments corresponding to an index value in the target index value set, the target index value set is any one of the index value sets of the plurality of first indicators, and the plurality of automatic driving data segments include the data segment set of the target index value set;
a third determining module 404 for determining a target indicator from the plurality of first indicators based on the correlation coefficient.
In one embodiment, determining a correlation coefficient between a plurality of first indexes and a second index using values of the second index of a data segment set of a plurality of index value sets of the first indexes comprises:
determining a correlation coefficient between the ith first index and the second index using a first mean of each set of index values of the ith first index, a second mean of a plurality of sets of index values of the ith first index, and a value of the second index of the set of data fragments of each set of index values of the ith first index;
the number of the first indexes is N, N is an integer larger than 1, the first indexes comprise the ith first index, i is larger than or equal to 1 and is smaller than or equal to N, and the second average value is the average value of the first average values.
In one embodiment, the correlation coefficient between the ith first index and the second index is determined by the following formula:
Figure BDA0002859703640000141
wherein M isiNumber of sets of a plurality of index value sets, r, for the ith first indexiIs the correlation coefficient between the ith first index and the second index, YijIs a first mean of a jth set of index values from the plurality of sets of index values of the ith first index,
Figure BDA0002859703640000142
is the second mean value of the ith first index, XijFor a value of a second index of the jth index set data segment set of the plurality of index value sets of the ith first index,
Figure BDA0002859703640000143
is the mean of the values of the second index of the plurality of sets of index values of the ith first index.
In one embodiment, the third determining module includes:
the sorting module is used for sorting the correlation coefficients between the first indexes and the second indexes;
the determining submodule is used for determining at least one of the sorted correlation coefficients as a target index;
at least one correlation coefficient is larger than the rest correlation coefficients, and the rest correlation coefficients are coefficients except for at least one correlation coefficient in the correlation coefficients between the first indexes and the second indexes.
In one embodiment, the above apparatus further comprises:
a data acquisition module for acquiring autopilot data of an autopilot vehicle;
and the segmentation module is used for segmenting the automatic driving data by using the preset time length to obtain a plurality of automatic driving data segments.
In one embodiment, the above apparatus further comprises:
a reference indicator determination module for determining a plurality of reference indicators;
a first index determination module for determining a first index under each reference index based on a plurality of reference dimensions of the obstacle;
the plurality of first indexes include first indexes under a plurality of reference indexes.
The index determining device of each embodiment is a device for implementing the index determining method of each embodiment, and the technical features correspond to the technical effects, and are not described herein again.
There is also provided, in accordance with an embodiment of the present application, an electronic device, a readable storage medium, and a computer program product.
A non-transitory computer-readable storage medium of an embodiment of the present application stores computer instructions for causing a computer to execute the index determination method provided herein.
The computer program product of the embodiments of the present application includes a computer program, and the computer program is used for causing a computer to execute the index determining method provided by the embodiments of the present application.
FIG. 5 illustrates a schematic block diagram of an example electronic device 500 that can be used to implement embodiments of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be examples only, and are not meant to limit implementations of the present application that are described and/or claimed herein.
As shown in fig. 5, the electronic device 500 includes a computing unit 501, which can perform various appropriate actions and processes according to a computer program stored in a Read Only Memory (ROM)502 or a computer program loaded from a storage unit 508 into a Random Access Memory (RAM) 503. In the RAM503, various programs and data required for the operation of the electronic apparatus 500 can also be stored. The calculation unit 501, the ROM 502, and the RAM503 are connected to each other by a bus 504. An input/output (I/O) interface 505 is also connected to bus 504.
A number of components in the electronic device 500 are connected to the I/O interface 505, including: an input unit 506 such as a keyboard, a mouse, or the like; an output unit 507 such as various types of displays, speakers, and the like; a storage unit 508, such as a magnetic disk, optical disk, or the like; and a communication unit 509 such as a network card, modem, wireless communication transceiver, etc. The communication unit 509 allows the electronic device 500 to exchange information/data with other devices through a computer network such as the internet and/or various telecommunication networks.
The computing unit 501 may be a variety of general-purpose and/or special-purpose processing components having processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, and so forth. The calculation unit 501 executes the respective methods and processes described above, such as the index determination method. For example, in some embodiments, the metric determination method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 508. In some embodiments, part or all of the computer program may be loaded and/or installed onto the device 600 via the ROM 502 and/or the communication unit 509. When the computer program is loaded into the RAM503 and executed by the computing unit 501, one or more steps of the index determination method described above may be performed. Alternatively, in other embodiments, the computing unit 501 may be configured to perform the metric determination method by any other suitable means (e.g., by means of firmware). Various implementations of the systems and techniques described here above may be implemented in digital electronic circuitry, integrated circuitry, Field Programmable Gate Arrays (FPGAs), Application Specific Integrated Circuits (ASICs), Application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, receiving data and instructions from, and transmitting data and instructions to, a storage system, at least one input device, and at least one output device.
Program code for implementing the methods of the present application may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the program codes, when executed by the processor or controller, cause the functions/operations specified in the flowchart and/or block diagram to be performed. The program code may execute entirely on the machine, partly on the machine, as a stand-alone software package partly on the machine and partly on a remote machine or entirely on the remote machine or server.
In the context of this application, a machine-readable medium may be a tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. The machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of a machine-readable storage medium would include an electrical connection based on one or more wires, a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to a user; and a keyboard and a pointing device (e.g., a mouse or a trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic, speech, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a back-end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back-end, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), Wide Area Networks (WANs), the internet, and blockchain networks.
The computer system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The Server can be a cloud Server, also called a cloud computing Server or a cloud host, and is a host product in a cloud computing service system, so as to solve the defects of high management difficulty and weak service expansibility in the traditional physical host and VPS service ("Virtual Private Server", or simply "VPS"). The server may also be a server of a distributed system, or a server incorporating a blockchain.
It should be understood that various forms of the flows shown above may be used, with steps reordered, added, or deleted. For example, the steps described in the present application may be executed in parallel, sequentially, or in different orders, and the present invention is not limited thereto as long as the desired results of the technical solutions disclosed in the present application can be achieved.
The above-described embodiments should not be construed as limiting the scope of the present application. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims (13)

1. An index determination method, the method comprising:
determining values of a plurality of first indicators for each of a plurality of segments of autonomous driving data;
sorting the values of each first index respectively, and dividing the sorted values of each first index into a plurality of index value sets respectively;
determining a correlation coefficient between the plurality of first indexes and a second index by using values of the second index of a data segment set of the index value sets of the plurality of first indexes, wherein the data segment set of a target index value set is a set of data segments corresponding to index values in the target index value set, the target index value set is any one of the index value sets of the plurality of first indexes, and the plurality of automatic driving data segments comprise the data segment set of the target index value set;
determining a target index from the plurality of first indexes based on the correlation coefficient.
2. The method of claim 1, wherein the determining a correlation coefficient between the plurality of first indicators and a second indicator of a set of data segments of the set of indicator values of the plurality of first indicators using values of the plurality of first indicators comprises:
determining a correlation coefficient between an ith first index and a second index using a first mean of each set of index values of the ith first index, a second mean of a plurality of sets of index values of the ith first index, and a value of the second index of a set of data segments of each set of index values of the ith first index;
the number of the first indexes is N, N is an integer larger than 1, the first indexes comprise the ith first index, i is larger than or equal to 1 and is smaller than or equal to N, and the second mean value is the mean value of the first mean values.
3. The method of claim 1, wherein said determining a target metric from the plurality of first metrics based on the correlation coefficient comprises:
ranking correlation coefficients between the plurality of first indicators and the second indicator;
determining at least one of the sorted correlation coefficients as the target index;
wherein the at least one correlation coefficient is larger than the rest of correlation coefficients, and the rest of correlation coefficients are coefficients other than the at least one correlation coefficient among the correlation coefficients between the plurality of first indexes and the second index.
4. The method of claim 1, wherein prior to determining the value of the first indicator for each of the plurality of segments of autonomous driving data, further comprising:
obtaining autodrive data for an autodrive vehicle;
and segmenting the automatic driving data by using preset time length to obtain a plurality of automatic driving data segments.
5. The method of claim 1, wherein prior to determining the value of the first indicator for each of the plurality of segments of autonomous driving data, further comprising:
determining a plurality of reference indicators;
determining a first index under each reference index based on a plurality of reference dimensions of the obstacle;
wherein the plurality of first indicators includes first indicators under the plurality of reference indicators.
6. An index determination apparatus, the apparatus comprising:
a first determination module to determine values of a plurality of first indicators for each of a plurality of segments of autopilot data;
the processing module is used for sorting the values of each first index respectively and dividing the sorted values of each first index into a plurality of index value sets respectively;
a second determining module, configured to determine a correlation coefficient between the plurality of first indicators and a second indicator of a data segment set of the indicator value sets of the plurality of first indicators by using values of the plurality of first indicators and a second indicator of the data segment set of the indicator value sets of the plurality of first indicators, where a data segment set of a target indicator value set is a set of data segments corresponding to indicator values in the target indicator value set, the target indicator value set is any one of the indicator value sets of the plurality of first indicators, and the plurality of automatic driving data segments include the data segment set of the target indicator value set;
a third determining module for determining a target index from the plurality of first indexes based on the correlation coefficient.
7. The apparatus of claim 6, wherein the determining a correlation coefficient between the plurality of first indicators and a second indicator of a set of data segments of the set of indicator values of the plurality of first indicators using values of the plurality of first indicators comprises:
determining a correlation coefficient between an ith first index and a second index using a first mean of each set of index values of the ith first index, a second mean of a plurality of sets of index values of the ith first index, and a value of the second index of a set of data segments of each set of index values of the ith first index;
the number of the first indexes is N, N is an integer larger than 1, the first indexes comprise the ith first index, i is larger than or equal to 1 and is smaller than or equal to N, and the second mean value is the mean value of the first mean values.
8. The apparatus of claim 6, wherein the third determining means comprises:
a sorting module, configured to sort correlation coefficients between the plurality of first indicators and the second indicator;
the determining submodule is used for determining at least one of the sorted correlation coefficients as the target index;
wherein the at least one correlation coefficient is larger than the rest of correlation coefficients, and the rest of correlation coefficients are coefficients other than the at least one correlation coefficient among the correlation coefficients between the plurality of first indexes and the second index.
9. The apparatus of claim 6, further comprising:
a data acquisition module for acquiring autopilot data of an autopilot vehicle;
and the segmentation module is used for segmenting the automatic driving data by using preset time length to obtain a plurality of automatic driving data segments.
10. The apparatus of claim 6, further comprising:
a reference indicator determination module for determining a plurality of reference indicators;
a first index determination module for determining a first index under each reference index based on a plurality of reference dimensions of the obstacle;
wherein the plurality of first indicators includes first indicators under the plurality of reference indicators.
11. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein the content of the first and second substances,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the index determination method of any of claims 1-5.
12. A non-transitory computer-readable storage medium storing computer instructions for causing a computer to execute the index determination method of any one of claims 1 to 5.
13. A computer program product comprising a computer program which, when executed by a processor, implements an index determination method according to any one of claims 1-5.
CN202011559016.8A 2020-12-25 Index determination method and device and electronic equipment Active CN112541708B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202011559016.8A CN112541708B (en) 2020-12-25 Index determination method and device and electronic equipment

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011559016.8A CN112541708B (en) 2020-12-25 Index determination method and device and electronic equipment

Publications (2)

Publication Number Publication Date
CN112541708A true CN112541708A (en) 2021-03-23
CN112541708B CN112541708B (en) 2024-05-24

Family

ID=

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108362957A (en) * 2017-12-19 2018-08-03 东软集团股份有限公司 Equipment fault diagnosis method, device, storage medium and electronic equipment
CN110723152A (en) * 2019-10-23 2020-01-24 成都信息工程大学 Artificial intelligence detection method, device, equipment or storage medium for guaranteeing automatic driving safety
CN111009153A (en) * 2019-12-04 2020-04-14 珠海深圳清华大学研究院创新中心 Training method, device and equipment of trajectory prediction model
US20200180647A1 (en) * 2018-12-10 2020-06-11 Perceptive Automata, Inc. Neural network based modeling and simulation of non-stationary traffic objects for testing and development of autonomous vehicle systems

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108362957A (en) * 2017-12-19 2018-08-03 东软集团股份有限公司 Equipment fault diagnosis method, device, storage medium and electronic equipment
US20200180647A1 (en) * 2018-12-10 2020-06-11 Perceptive Automata, Inc. Neural network based modeling and simulation of non-stationary traffic objects for testing and development of autonomous vehicle systems
CN110723152A (en) * 2019-10-23 2020-01-24 成都信息工程大学 Artificial intelligence detection method, device, equipment or storage medium for guaranteeing automatic driving safety
CN111009153A (en) * 2019-12-04 2020-04-14 珠海深圳清华大学研究院创新中心 Training method, device and equipment of trajectory prediction model

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
李野;鲁淑;袁翔;袁林;: "基于相关性分析的宏观经济指标预测算法", 指挥信息系统与技术, no. 01, 28 February 2020 (2020-02-28) *

Similar Documents

Publication Publication Date Title
EP4020315A1 (en) Method, apparatus and system for determining label
CN112559371B (en) Automatic driving test method and device and electronic equipment
CN112258093A (en) Risk level data processing method and device, storage medium and electronic equipment
CN113837596B (en) Fault determination method and device, electronic equipment and storage medium
CN111681417B (en) Traffic intersection canalization adjusting method and device
CN115794578A (en) Data management method, device, equipment and medium for power system
CN116028730A (en) Search resource abnormality identification method and device and electronic equipment
CN114677653A (en) Model training method, vehicle key point detection method and corresponding devices
CN111341096B (en) Bus running state evaluation method based on GPS data
CN112541708B (en) Index determination method and device and electronic equipment
CN112541708A (en) Index determination method and device and electronic equipment
US20110015967A1 (en) Methodology to identify emerging issues based on fused severity and sensitivity of temporal trends
CN112559272A (en) Quality information determination method and device of vehicle-mounted equipment, equipment and storage medium
EP4109431A1 (en) Method, apparatus and storage medium of determining state of intersection
CN114863715A (en) Parking data determination method and device, electronic equipment and storage medium
CN115936522A (en) Vehicle stop station evaluation method, device, equipment and storage medium
CN115330067A (en) Traffic congestion prediction method and device, electronic equipment and storage medium
CN115691117A (en) Method and device for evaluating influence of traffic incident on road traffic and electronic equipment
CN115062687A (en) Enterprise credit monitoring method, device, equipment and storage medium
CN114884813A (en) Network architecture determination method and device, electronic equipment and storage medium
CN112926135A (en) Scene information determination method, apparatus, device, storage medium, and program product
CN113326885A (en) Method and device for training classification model and data classification
CN114281808A (en) Traffic big data cleaning method, device, equipment and readable storage medium
CN114067565B (en) Method and device for determining congestion identification accuracy
CN114419876B (en) Road saturation evaluation method and device, electronic equipment and storage medium

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant