EP3866044B1 - Road test method and apparatus for autonomous driving vehicle, device and storage medium - Google Patents

Road test method and apparatus for autonomous driving vehicle, device and storage medium Download PDF

Info

Publication number
EP3866044B1
EP3866044B1 EP21178545.6A EP21178545A EP3866044B1 EP 3866044 B1 EP3866044 B1 EP 3866044B1 EP 21178545 A EP21178545 A EP 21178545A EP 3866044 B1 EP3866044 B1 EP 3866044B1
Authority
EP
European Patent Office
Prior art keywords
autonomous driving
vehicle
test
evaluation
driving scenarios
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP21178545.6A
Other languages
German (de)
English (en)
French (fr)
Other versions
EP3866044A3 (en
EP3866044A2 (en
Inventor
Jun Zhao
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Beijing Baidu Netcom Science and Technology Co Ltd
Original Assignee
Beijing Baidu Netcom Science and Technology Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Beijing Baidu Netcom Science and Technology Co Ltd filed Critical Beijing Baidu Netcom Science and Technology Co Ltd
Publication of EP3866044A2 publication Critical patent/EP3866044A2/en
Publication of EP3866044A3 publication Critical patent/EP3866044A3/en
Application granted granted Critical
Publication of EP3866044B1 publication Critical patent/EP3866044B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01M—TESTING STATIC OR DYNAMIC BALANCE OF MACHINES OR STRUCTURES; TESTING OF STRUCTURES OR APPARATUS, NOT OTHERWISE PROVIDED FOR
    • G01M17/00—Testing of vehicles
    • G01M17/007—Wheeled or endless-tracked vehicles
    • G01M17/06—Steering behaviour; Rolling behaviour
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01M—TESTING STATIC OR DYNAMIC BALANCE OF MACHINES OR STRUCTURES; TESTING OF STRUCTURES OR APPARATUS, NOT OTHERWISE PROVIDED FOR
    • G01M17/00—Testing of vehicles
    • G01M17/007—Wheeled or endless-tracked vehicles
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W40/00—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models
    • B60W40/02—Estimation or calculation of non-directly measurable driving parameters for road vehicle drive control systems not related to the control of a particular sub unit, e.g. by using mathematical models related to ambient conditions
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/04—Monitoring the functioning of the control system
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W50/04—Monitoring the functioning of the control system
    • B60W50/045—Monitoring control system parameters
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W60/00—Drive control systems specially adapted for autonomous road vehicles
    • B60W60/001—Planning or execution of driving tasks
    • B60W60/0011—Planning or execution of driving tasks involving control alternatives for a single driving scenario, e.g. planning several paths to avoid obstacles
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/01—Detecting movement of traffic to be counted or controlled
    • G08G1/0104—Measuring and analyzing of parameters relative to traffic conditions
    • G08G1/0137—Measuring and analyzing of parameters relative to traffic conditions for specific applications
    • G08G1/0145—Measuring and analyzing of parameters relative to traffic conditions for specific applications for active traffic flow control
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/07—Controlling traffic signals
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/09—Arrangements for giving variable traffic instructions
    • G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • G08G1/0967—Systems involving transmission of highway information, e.g. weather, speed limits
    • G08G1/096708—Systems involving transmission of highway information, e.g. weather, speed limits where the received information might be used to generate an automatic action on the vehicle control
    • G08G1/096716—Systems involving transmission of highway information, e.g. weather, speed limits where the received information might be used to generate an automatic action on the vehicle control where the received information does not generate an automatic action on the vehicle control
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/09—Arrangements for giving variable traffic instructions
    • G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • G08G1/0967—Systems involving transmission of highway information, e.g. weather, speed limits
    • G08G1/096733—Systems involving transmission of highway information, e.g. weather, speed limits where a selection of the information might take place
    • G08G1/096741—Systems involving transmission of highway information, e.g. weather, speed limits where a selection of the information might take place where the source of the transmitted information selects which information to transmit to each vehicle
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/09—Arrangements for giving variable traffic instructions
    • G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • G08G1/0967—Systems involving transmission of highway information, e.g. weather, speed limits
    • G08G1/096766—Systems involving transmission of highway information, e.g. weather, speed limits where the system is characterised by the origin of the information transmission
    • G08G1/096775—Systems involving transmission of highway information, e.g. weather, speed limits where the system is characterised by the origin of the information transmission where the origin of the information is a central station
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/09—Arrangements for giving variable traffic instructions
    • G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • G08G1/0967—Systems involving transmission of highway information, e.g. weather, speed limits
    • G08G1/096766—Systems involving transmission of highway information, e.g. weather, speed limits where the system is characterised by the origin of the information transmission
    • G08G1/096783—Systems involving transmission of highway information, e.g. weather, speed limits where the system is characterised by the origin of the information transmission where the origin of the information is a roadside individual element
    • G—PHYSICS
    • G08—SIGNALLING
    • G08G—TRAFFIC CONTROL SYSTEMS
    • G08G1/00—Traffic control systems for road vehicles
    • G08G1/09—Arrangements for giving variable traffic instructions
    • G08G1/0962—Arrangements for giving variable traffic instructions having an indicator mounted inside the vehicle, e.g. giving voice messages
    • G08G1/0967—Systems involving transmission of highway information, e.g. weather, speed limits
    • G08G1/096766—Systems involving transmission of highway information, e.g. weather, speed limits where the system is characterised by the origin of the information transmission
    • G08G1/096791—Systems involving transmission of highway information, e.g. weather, speed limits where the system is characterised by the origin of the information transmission where the origin of the information is another vehicle
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W50/00—Details of control systems for road vehicle drive control not related to the control of a particular sub-unit, e.g. process diagnostic or vehicle driver interfaces
    • B60W2050/0001—Details of the control system
    • B60W2050/0002—Automatic control, details of type of controller or control system architecture
    • B60W2050/0004—In digital systems, e.g. discrete-time systems involving sampling
    • B60W2050/0005—Processor details or data handling, e.g. memory registers or chip architecture
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2552/00—Input parameters relating to infrastructure
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2554/00—Input parameters relating to objects
    • B60W2554/40—Dynamic objects, e.g. animals, windblown objects
    • B60W2554/404—Characteristics
    • B60W2554/4049—Relationship among other objects, e.g. converging dynamic objects
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2554/00—Input parameters relating to objects
    • B60W2554/40—Dynamic objects, e.g. animals, windblown objects
    • B60W2554/406—Traffic density
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2554/00—Input parameters relating to objects
    • B60W2554/80—Spatial relation or speed relative to objects
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B60—VEHICLES IN GENERAL
    • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
    • B60W2556/00—Input parameters relating to data
    • B60W2556/10—Historical data
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04W—WIRELESS COMMUNICATION NETWORKS
    • H04W4/00—Services specially adapted for wireless communication networks; Facilities therefor
    • H04W4/30—Services specially adapted for particular environments, situations or purposes
    • H04W4/40—Services specially adapted for particular environments, situations or purposes for vehicles, e.g. vehicle-to-pedestrians [V2P]
    • H04W4/44—Services specially adapted for particular environments, situations or purposes for vehicles, e.g. vehicle-to-pedestrians [V2P] for communication between vehicles and infrastructures, e.g. vehicle-to-cloud [V2C] or vehicle-to-home [V2H]
    • Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
    • Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
    • Y02T—CLIMATE CHANGE MITIGATION TECHNOLOGIES RELATED TO TRANSPORTATION
    • Y02T10/00—Road transport of goods or passengers
    • Y02T10/10—Internal combustion engine [ICE] based vehicles
    • Y02T10/40—Engine management systems

Definitions

  • the present application relates to the field of artificial intelligence, in particular, to autonomous driving and intelligent transportation technologies, and in particular to a road test method and apparatus for an autonomous driving vehicle, a device and a storage medium.
  • a road test is a test stage where an autonomous driving vehicle behaves most close as in a scenario with practical requirements, where interactions with actual pedestrians and vehicles are conducted in an open road network to test a travelling experience on a real road with respect to the autonomous driving vehicle, and the road test is a comprehensive test on safety, comfort, intelligence and travelling efficiency of the autonomous driving vehicle, where scenarios as many as possible can be covered by covering different traffic throughputs and different traffic flows.
  • CN 107782564 A discloses that, before an onsite driving test, a test route is indicated to a vehicle; during the onsite driving test, the vehicle drives on the test route and feeds back driving test result data; and a driving test result report is determined based on test route data indicating the test route and the test result data fed back by the vehicle.
  • US 2019/145860 A1 discloses a method for autonomous system performance metric generation and benchmarking, according to which a control data indicative of a driver performance is received, and the driver performance is graded based on the control data, where the control data may be generated in response to a human driver performance or an autonomous driving control system performance.
  • CN 111765 903 A discloses a solution to determine a mileage required for a road test according to a corresponding relationship between a mileage and a ratio between a current number of accumulated problems monitored in a testing process and a total number of problems monitored in the testing process.
  • the present invention is set out in the appended set of claims.
  • the present application provides a road test method and apparatus for an autonomous driving vehicle, a device and a storage medium.
  • the present application provides a road test method for an autonomous driving vehicle, including:
  • the present application provides a road test apparatus for an autonomous driving vehicle, including:
  • the present application provides an electronic device, including:
  • the present application provides a non-transitory computer readable storage medium, storing thereon computer instructions which are configured to cause a computer to implement the method described in the above.
  • the present application provides a computer program product, including a computer program which, when being executed by a processor, implements the method described in the above.
  • the present application provides a computer program, the computer program is stored in a computer-readable storage medium, when being executed by a processor, implements the method described in the above.
  • the technology according to the present application improves an accuracy of a road test result of an autonomous driving vehicle.
  • the present application provides a road test method and apparatus for an autonomous driving vehicle, a device and a storage medium, which are applied to the field of artificial intelligence and, in particular, to autonomous driving and intelligent transportation technologies, to achieve a technical effect of improving an accuracy of a road test result of an autonomous driving vehicle.
  • the road test method for an autonomous driving vehicle may be applied to a road test scenario for an autonomous driving vehicle shown in FIG. 1 .
  • the vehicle shown in FIG. 1 is a to-be-test autonomous driving vehicle
  • the electronic device is a device to execute the road test method for an autonomous driving vehicle.
  • a to-be-test vehicle loaded with an evaluation personnel acquires, from another vehicle, a road side device or a server, road test travelling data and vehicle surroundings information of the vehicle in a process while travelling along a test route.
  • the evaluation personnel submits experience evaluation data based on an actual travelling experience in the process during which the vehicle is travelling along the test route.
  • the electronic device evaluates an autonomous driving ability of the vehicle to obtain a first test result, according to objective test data such as the road test travelling data of the vehicle and the vehicle surroundings information; evaluates an autonomous driving ability of the vehicle to obtain a second test result, according to subjective evaluation data such as the experience evaluation data based on subjective feelings of the evaluation personnel; and obtains a road test result of the vehicle by combining the first test result and the second test result.
  • objective test data such as the road test travelling data of the vehicle and the vehicle surroundings information
  • subjective evaluation data such as the experience evaluation data based on subjective feelings of the evaluation personnel
  • the road test result of the vehicle includes an objective test result based on objective test data such that rigorousness and preciseness of the road test of the vehicle are guaranteed, besides, the road test result of the vehicle also includes a quantitative test result based on subjective experience evaluation data, which reflects an evaluation result on travelling safety, comfort and intelligence and travelling efficiency of a vehicle from a subjective perspective of a passenger, thereby a more comprehensive road test result is obtained, and an overall performance of an autonomous driving vehicle in the road test is evaluated more accurately, such that the accuracy of a road test result of an autonomous driving vehicle is improved.
  • FIG. 2 is a flowchart of a road test method for an autonomous driving vehicle according to a first embodiment of the present application. As shown in FIG. 2 , the specific steps of the method are as the following: S101: acquiring autonomous driving scenarios, information of test parameters corresponding to the autonomous driving scenarios and experience evaluation data in a process during which a vehicle is travelling along a test route, where different autonomous driving scenarios correspond to different test parameters.
  • the autonomous driving scenario in the process during which the vehicle is travelling along the test route includes but not limited to: travelling to follow, travelling straight, switching a lane, travelling through an intersection, cutting in or cutting out, identifying and coping with a close obstacle, turning around, merging into traffic, intelligent nudge, travelling through a main/side road, pulling over, starting up.
  • test parameters corresponding to each autonomous driving scenario are predetermined, the test parameters are indicative of evaluating whether an autonomous driving behavior of a vehicle in an autonomous driving scenario meets a requirement.
  • the parameters corresponding to different autonomous driving scenarios are different.
  • the information of the test parameters corresponding to the autonomous driving scenario includes values of the test parameters corresponding to the autonomous driving scenario.
  • the experience evaluation data is: evaluation data on coping abilities corresponding to the autonomous driving scenarios in a travelling process of the vehicle, where the evaluation data is submitted by an evaluation personnel who rides in the vehicle based on an actual travelling experience in the travelling process of the vehicle in the process during which the vehicle is travelling along the test route.
  • S102 analyzing the information of the test parameters corresponding to the autonomous driving scenarios according to evaluation baselines corresponding to the autonomous driving scenarios, to determine a first test result of the vehicle.
  • evaluation baselines corresponding to the autonomous driving scenarios are used for determining whether the test parameters corresponding to the autonomous driving scenarios meet a requirement.
  • the information of the test parameters corresponding to the autonomous driving scenarios is analyzed according to the evaluation baselines corresponding to the autonomous driving scenarios to determine the first test result of the vehicle.
  • the first test result is an objective, rigorous and precise test result obtained based on objective test data.
  • the experience evaluation data corresponding to the autonomous driving scenarios is subjected to a quantitation process to obtain the second test result.
  • the second test result is a test result on a travelling experience on the vehicle, which reflects a subjective evaluation result of a passenger on travelling safety, comfort and intelligence and travelling efficiency of the vehicle.
  • S104 determining a road test result of the vehicle according to the first test result and the second test result of the vehicle.
  • the road test result of the vehicle is determined by combining the first test result and the second test result.
  • weights of the first test result based on objective data and the second test result based on subjective data may be assigned, such that a weighted sum of the first test result and the second test result is calculated as the road test result of the vehicle.
  • a sum of the first test result based on objective data and the second test result based on subjective data may serve as the road test result of the vehicle; or an eventual road test result may be determined according to the first test result and the second test result through other methods, which are not specifically limited herein.
  • autonomous driving scenarios and information of test parameters corresponding to the autonomous driving scenarios in a process during which a vehicle is travelling along a test route are acquired, the information of the test parameters corresponding to the autonomous driving scenario is analyzed according to evaluation baselines corresponding to the autonomous driving scenarios, to determine a first test result of the vehicle, that is, the first test result is obtained by evaluating an autonomous driving ability of the vehicle according to objective test data such as the information of the test parameters corresponding to the autonomous driving scenarios; a second test result is obtained by evaluating an autonomous driving ability of the vehicle from a travelling experience perspective of an evaluation personnel according to subjective evaluation data such as experience evaluation data; and a road test result of the vehicle is obtained by combining the first test result and the second test result.
  • the road test result of the vehicle includes an objective test result based on objective test data such that objectivity, rigorousness and preciseness of the road test of the vehicle are guaranteed, besides, the road test result of the vehicle also includes a quantitative test result based on subjective experience evaluation data, which reflects an evaluation result on travelling safety, comfort and intelligence and travelling efficiency of a vehicle from a subjective travelling experience perspective of a passenger, thereby a more comprehensive road test result is obtained, and an overall performance of an autonomous driving vehicle in the road test is evaluated more accurately, such that the accuracy of a road test result of an autonomous driving vehicle is improved.
  • FIG. 3 is a flowchart of a road test method for an autonomous driving vehicle according to a second embodiment of the present application.
  • the road test method for an autonomous driving vehicle includes a test section based on subjective evaluation data, and a test section based on objective road test travelling data and vehicle surroundings information, in the embodiment, the road test method for the autonomous driving vehicle is explained in detail in conjunction with FIG. 3 .
  • steps S201-S204 are the test section based on objective road test travelling data and vehicle surroundings information
  • steps S205-S208 are the test section based on subjective evaluation data.
  • Step S201 acquiring road test travelling data and vehicle surroundings information of a vehicle in a process during which the vehicle is travelling along a test route.
  • the first is to fix an evaluation personnel, regarding road tests to different versions of an autonomous driving system, a fixed evaluation personnel is selected for testing, and a baseline for subjective evaluation is established through an one-time experience or evaluation.
  • the second is to fix a road test period or to follow up the vehicle for a long time for testing, since scenario complexities will be different due to different road test traffic flows in different periods of every week or different periods of a day, and an accuracy can be easily affected for autonomous driving vehicle tests under two scenarios with a huge difficulty gap.
  • the third is to fix a test route.
  • An autonomous driving ability of an autonomous driving vehicle required to carry out left/right turning, necessary lane switching, turning around is basically guaranteed to be the same under the same test route, while different test routes require different autonomous driving abilities, which may result in an accuracy problem.
  • a test route is planned in advance, finally, one or more test routes are determined. Where in the process during which an autonomous driving vehicle is travelling along the test route, all autonomous driving scenarios that need to be tested should be covered.
  • the road test travelling data includes: vehicle travelling data in the process during which the autonomous driving vehicle is travelling along the test route, which includes vehicle controlling information and vehicle travelling behavior information or the like.
  • vehicle controlling information refers to instruction information for controlling the autonomous driving vehicle to travel, for example a starting up instruction, a pulling over instruction, a lane switching instruction, a turning around instruction, acceleration/deceleration instruction sent by a controlling system or the like.
  • vehicle travelling behavior information includes data relates to a vehicle behavior, such as a travelling speed, an acceleration, an orientation angle or a travelling trajectory of the autonomous driving vehicle in the travelling process.
  • the vehicle surroundings information is data of a surrounding environment of the vehicle in the process during which the autonomous driving vehicle is travelling along the test route.
  • the vehicle surroundings information includes one or more of the following: a distance between the vehicle and a vehicle in front, information of a lane on which the vehicle is travelling, intersection information of an intersection to which the vehicle is travelling, information of traffic around the vehicle.
  • Step S202 analyzing the road test travelling data, or the road test travelling data and the vehicle surroundings information to identify autonomous driving scenarios which occur in the process during which the vehicle is travelling along the test route.
  • the road test travelling data and the vehicle surroundings information of the vehicle, which are recorded in the processing during which the vehicle is travelling along the test route, are subjective data that can affect or inflect autonomous driving behaviors of the autonomous driving vehicle. Based on the subjective data, autonomous driving scenarios which occur in the process during which the vehicle is travelling along the test route can be identified.
  • the following manner may be employed to identify the autonomous driving scenarios which occur in the process during which the vehicle is travelling along the test route: analyzing vehicle controlling information in the road test travelling data; determining that one of the autonomous driving scenarios occurs in the process during which the vehicle is travelling along the test route, if the vehicle controlling information includes controlling information corresponding to the one of the autonomous driving scenarios.
  • the road test travelling data may include a controlling log of the vehicle where controlling information of the vehicle is recorded.
  • the controlling information of the vehicle can be acquired by analyzing the controlling log of the vehicle.
  • autonomous driving scenarios In a practical application, with respect to different autonomous driving scenarios, driving behaviors of a vehicle corresponding to the autonomous driving scenarios are carried out under the control of one or more controlling instructions sent out by a controlling system. Controlling instructions corresponding to different autonomous driving scenarios are not entirely the same, some autonomous driving scenarios even have controlling instructions that are specific (unique) to these autonomous driving scenarios. Therefore, an autonomous driving scenario can be identified according to a controlling instruction. Autonomous driving scenarios which occur in the process during which the vehicle is travelling along the test route can be accurately identified through controlling information obtained from analysis.
  • a controlling log generated by an autonomous driving vehicle in a travelling process is analyzed, it is found that a lane switching instruction is sent out, then it is determined that a lane switching scenario occurs at the time when the lane switching instruction is sent out by the controlling system.
  • the following manner may be employed to identify the autonomous driving scenarios which occur in the process during which the vehicle is travelling along the test route: analyzing the road test travelling data and the vehicle surroundings information to acquire a behavior characteristic of the vehicle; according to the behavior characteristic of the vehicle, if the vehicle has a behavior characteristic of one of the autonomous driving scenario, then determining that the one of the autonomous driving scenarios occurs in the process during which the vehicle is travelling along the test route.
  • driving behaviors of a vehicle corresponding to the autonomous driving scenarios have different characteristics, some autonomous driving scenarios even have driving behaviors that are specific to the autonomous driving scenarios.
  • Behavior characteristics of a vehicle can be identified by analyzing vehicle travelling behavior information in the road test travelling data, and the vehicle surroundings information. If the vehicle has behavior characteristics of a certain autonomous driving scenario, then it is determined that the vehicle is met with the autonomous driving scenario. The time when the vehicle represents these behavior characteristics, is the time when the vehicle is met with the corresponding autonomous driving scenario.
  • the autonomous driving scenarios occurring in the process during which the vehicle is travelling along the test route can be identified accurately by analyzing behavior characteristics of the vehicle.
  • Step S203 acquiring information of test parameters corresponding to the autonomous driving scenarios.
  • test parameters corresponding to each autonomous driving scenario are predetermined, the test parameters are indicative of evaluating whether an autonomous driving behavior of a vehicle in an autonomous driving scenario meets a requirement.
  • the parameters corresponding to different autonomous driving scenarios are different.
  • the test parameter corresponding to each of the autonomous driving scenarios After identifying the autonomous driving scenarios which occur in the process during which the vehicle is travelling along the test route, acquiring the information of the test parameter corresponding to each of the autonomous driving scenarios.
  • the information of the test parameters corresponding to the autonomous driving scenario includes values of the test parameters corresponding to the autonomous driving scenario.
  • the following manner may be employed to acquire the information of the test parameter(s) corresponding to any one of the autonomous driving scenarios: determining an occurrence time of the autonomous driving scenario; acquiring, from the road test travelling data and the vehicle surroundings information, an actual value(s) of the test parameter(s) corresponding to the autonomous driving scenario at the occurrence time.
  • the information of the test parameter(s) corresponding to the autonomous driving scenario is acquired accurately based on the occurrence time of the autonomous driving scenario.
  • test parameters corresponding to a scenario of switching a lane may include: a time to collision (TTC) at the rear when switching a lane, a TTC at the front when switching a lane, a vehicle speed when switching a lane.
  • TTC time to collision
  • a TTC at the rear, a TTC at the front and a speed of the vehicle at the occurrence time of the scenario of switching a lane are acquired from the road test travelling data, so as to determine the information of the test parameters corresponding to the scenario of switching a lane.
  • test parameters corresponding to the scenario of switching a lane provided in the example are merely for illustrative purposes, the test parameters corresponding to the scenario of switching a lane may further include a highest speed, an average speed or the like during a process of switching a lane, or further include other parameters that are not specified herein, which are not specifically limited herein.
  • test parameters corresponding to the autonomous driving scenarios may be configured according to a practical application scenario, for road tests of multiple versions of an autonomous driving vehicle, the test parameters corresponding to the autonomous driving scenarios are not specifically limited herein.
  • the autonomous driving scenarios the information of the test parameters corresponding to the autonomous driving scenarios in the process during which the vehicle is travelling along the test route are acquired to provide subjective data for a road test of the vehicle, based on which the road test of the vehicle may be carried out subjectively, thereby improving objectivity and accuracy of a road test result of the autonomous driving vehicle.
  • Step S204 analyzing the information of the test parameters corresponding to the autonomous driving scenarios according to evaluation baselines corresponding to the autonomous driving scenarios, to determine a first test result of the vehicle.
  • the evaluation baselines corresponding to the autonomous driving scenarios include baseline ranges of the test parameters corresponding to the autonomous driving scenarios.
  • the evaluation baselines corresponding to the autonomous driving scenarios are used for determining whether the test parameters corresponding to the autonomous driving scenarios meet a requirement. Regarding a certain test parameter corresponding to a certain autonomous driving scenario, if an actual value of the test parameter is within a corresponding baseline range, then it is indicated that the test parameter meets a requirement, if the actual value of the test parameter is not within the corresponding baseline range, then it is indicated that the test parameter does not meet the requirement.
  • the evaluation baselines corresponding to the autonomous driving scenarios are obtained according to analysis and statistics of driving data of a standard human driver.
  • the step may be specifically implemented in the following manner: determining test parameters whose actual values are within corresponding baseline ranges according to the actual values of the test parameters corresponding to the autonomous driving scenarios and the baseline ranges of the test parameters corresponding to the autonomous driving scenarios; and determining a sum of evaluation values corresponding to the test parameters whose actual values are within corresponding baseline ranges as the first test result of the vehicle; where each of the test parameters corresponding to each of the autonomous driving scenarios corresponds to an evaluation value.
  • all of information of test parameters corresponding to all of the autonomous driving scenarios are combined for evaluating an autonomous driving ability of an autonomous driving vehicle effectively and objectively.
  • a score deducting method may be employed: setting a total evaluation value of a road test, an evaluation value of each autonomous driving scenario, and an evaluation value of each test parameter corresponding to an autonomous driving scenario.
  • a sum of evaluation values of all of the autonomous driving scenarios equals to the total evaluation value of a road test
  • a sum of evaluation values of test parameters corresponding to an autonomous driving scenario equals to the evaluation value of the autonomous driving scenario.
  • an evaluation value of a test parameter is deducted if the test parameter does not meet a requirement, when evaluation values of all of test parameters that do not meet a requirement are deducted, the left evaluation value of a road test is the first test result.
  • evaluation values corresponding to different autonomous driving scenarios may be different, evaluation values of different test parameters corresponding to the same autonomous driving scenario may also be different.
  • a score deducting method may be employed: assuming that a total score of a road test of a vehicle is 100 points, 10 autonomous driving scenarios are tested, and a full score for each of the autonomous driving scenarios is 10 points, there are 5 parameters corresponding to a scenario of switching a lane where each of the parameters takes 2 points. If an actual value of one of the parameters is not within a corresponding baseline range, then 2 points are deducted from the total score of a road test. When evaluation values of all of test parameters that do not meet a requirement are deducted, the left evaluation value of a road test is the first test result.
  • the above steps S202-S204 achieves the effects that an objective test result of a vehicle in a road test which is based on objective test data is obtained, and objectivity, rigorousness and preciseness of the road test of the vehicle are guaranteed by: acquiring road test travelling data in the process during which the vehicle is travelling along the test route; analyzing the road test travelling data, or the road test travelling data and the vehicle surroundings information to identify autonomous driving scenarios which occur in the process during which the vehicle is travelling along the test route; and acquiring the information of the test parameters corresponding to the autonomous driving scenarios; determining test parameters whose actual values are within corresponding baseline ranges according to the actual values of the test parameters corresponding to the autonomous driving scenarios and the baseline ranges of the test parameters corresponding to the autonomous driving scenarios; and determining a sum of evaluation values corresponding to the test parameters whose actual values are within corresponding baseline ranges as the first test result of the vehicle.
  • Step S205 acquiring experience evaluation data corresponding to the autonomous driving scenarios in the process during which the vehicle is travelling along the test route.
  • the experience evaluation data is: evaluation data on coping abilities corresponding to the autonomous driving scenarios in a travelling process of the vehicle, where the evaluation data is submitted by an evaluation personnel who rides in the vehicle based on an actual travelling experience in the travelling process of the vehicle in the process during which the vehicle is travelling along the test route.
  • a membership theory of fuzzy mathematics can be employed to transform a qualitative problem into a quantitative problem. Based on an intuitive experience in the process during which a vehicle is travelling along a test route, and a test result of a previous version, an evaluation personnel evaluates an autonomous ability under each of the autonomous driving scenarios according to evaluation factors corresponding to the respective autonomous driving scenario, and submits the experience evaluation data.
  • experience evaluation data corresponding to the one of the autonomous driving scenarios includes evaluation levels corresponding to the one of the autonomous driving scenarios, where different evaluation levels correspond to different quantitative information.
  • the number of the evaluation levels and quantitative information corresponding to each of the evaluation levels may be set according to a practical application scenario, which are not specifically limited herein.
  • each of the autonomous driving scenarios corresponds to the following 5 evaluation levels: good, quite good, average, bad, quite bad, quantitative information corresponding to the 5 evaluation levels are: 1.0, 0.8, 0.6, 0.4 and 0.2.
  • the step may be implemented in the following manner: sending an evaluation factor corresponding to a respective autonomous driving scenario to multiple experience evaluation terminals; receiving experience evaluation data corresponding to the respective autonomous driving scenario, the experience evaluation data being collected based on the evaluation factor in the process during which the vehicle is travelling along the test route and submitted by the multiple experience evaluation terminals.
  • a subject evaluation is carried out by an evaluation personnel with reference to unified evaluation factors, such that evaluation data usability is achieved; additionally, the experience evaluation data is submitted directly by the experience evaluation terminals, compared with a manner of submitting a paper questionnaire, the efficiency is improved.
  • the evaluation factors corresponding to the autonomous driving scenarios may be shown in the following table: Table 1 No. Autonomous driving scenario Evaluation factor 1 Travelling to follow Whether a following distance is reasonable, whether it will overtake a low-speed obstacle vehicle 2 Travelling straight Whether braking for no reason, braking unreasonably, driving abruptly 3 Switching a lane Whether a switching decision is hesitant, whether failing to switch when it is permitted, switching for no reason 4 Travelling through an intersection Whether there is any hesitation, indecision, unreasonable braking and collision risk 5 Coping with cutting in or cutting out Whether a deceleration is too fierce, whether there is any collision risk 6 Identifying and coping with a close obstacle Whether a yielding distance is too long or short, whether slowing down unreasonably 7 Turning around Whether failing to turn around at some intersections, resulting in detours, whether a lane selection is reasonable 8 Merging into traffic Whether failing to merge often, whether there is a collision risk at a merging spot 9 Intelligent nudge
  • the 11 autonomous driving scenarios shown in Table 1 are autonomous driving scenarios corresponding to key autonomous driving abilities of L4 level, which are selected from an autonomous driving scenario library combining with road test methods and practice, to evaluate typical autonomous driving abilities which are frequently practiced while driving on a road and influence autonomous driving noticeably.
  • Step S206 calculating quantitative experience information corresponding to the autonomous driving scenarios according to the experience evaluation data corresponding to the autonomous driving scenarios in the process during which the vehicle is travelling along the test route.
  • experience evaluation data corresponding to each of the autonomous driving scenarios is counted and analyzed to determine the quantitative experience information corresponding to the respective autonomous driving scenario.
  • the step may be specifically implemented in the following manner: regarding any one of the autonomous driving scenarios, experience evaluation data corresponding to the one of the autonomous driving scenarios includes evaluation levels corresponding to the one of the autonomous driving scenarios, where different evaluation levels correspond to different quantitative information; determining a total evaluation count for which an experience evaluation is carried out with respect to the one of the autonomous driving scenarios and an evaluation count of a respective evaluation level corresponding to the one of the autonomous driving scenarios, according to the evaluation levels corresponding to the one of the autonomous driving scenarios; determining a ratio of the evaluation count of the respective evaluation level corresponding to the one of the autonomous driving scenarios to the total evaluation count as an evaluation weight of the respective evaluation level corresponding to the one of the autonomous driving scenarios; and calculating a weighted sum of quantitative information of the evaluation levels based on the evaluation weights of the evaluation levels corresponding to the one of the autonomous driving scenarios, to obtain quantitative experience information corresponding to the one of the autonomous driving scenarios.
  • subjective evaluation contents are accurately quantitated.
  • the total evaluation count for which the experience evaluation is carried out with respect to the one of the autonomous driving scenarios equals to a sum of evaluation counts of all of the evaluation levels corresponding to the one of the autonomous driving scenarios. That is, the total evaluation count for which the experience evaluation is carried out with respect to the one of the autonomous driving scenarios, is the number of evaluation personnels who correctly submit experience evaluation data.
  • each of the autonomous driving scenarios corresponds to the following 5 evaluation levels: good, quite good, average, bad, quite bad, quantitative information corresponding to the 5 evaluation levels are: 1.0, 0.8, 0.6, 0.4 and 0.2.
  • Table 2 a counting result of the evaluation counts of the evaluation levels corresponding to the autonomous driving scenarios is shown in the following Table 2: Table 2 No. Autonomous driving scenario Good Quite good Average Bad Quite bad 1 Travelling to follow R1 R2 R3 R4 R5 2 Travelling straight S1 S2 S3 S4 S5
  • R1, R2, R3, R4 and R5 are respective evaluation counts of the 5 evaluation levels given to an autonomous driving vehicle in a scenario of travelling to follow.
  • the total evaluation count for which the experience evaluation is carried out with respect to the scenario of travelling to follow equals to a sum of R1, R2, R3, R4 and R5, which will be denoted as R;
  • the weights of the 5 evaluation levels corresponding to the scenario of travelling to follow are: R1/R, R2/R, R3/R, R4/R and R5/R;
  • S1, S2, S3, S4 and S5 are respective evaluation counts of the 5 evaluation levels given to an autonomous driving vehicle in a scenario of travelling straight.
  • the total evaluation count for which the experience evaluation is carried out with respect to the scenario of travelling straight equals to a sum of S1, S2, S3, S4 and S5, which will be denoted as S;
  • Counting results for other test scenarios are similar, which will not be exemplified one by one herein.
  • the quantitative experience information corresponding to the respective autonomous driving scenario may be multiply by a coefficient, or represented as a percentage.
  • Step S207 acquiring weights corresponding to the autonomous driving scenarios.
  • the weights corresponding to the autonomous driving scenarios are acquired before determining a second test result according to the quantitative experience information corresponding to the autonomous driving scenarios and the weights corresponding to the autonomous driving scenarios.
  • the step may be specifically implemented in the following manner: acquiring multiple evaluation coefficients corresponding to a respective autonomous driving scenario, where the evaluation coefficients are used for evaluating a level at which the respective autonomous driving scenario influences a driving experience of the vehicle; calculating, according to the evaluation coefficients corresponding to the autonomous driving scenarios, an evaluation coefficient totality corresponding to all of the autonomous driving scenarios, and a sum of evaluation coefficients corresponding to the respective autonomous driving scenario; and determining a ratio of the sum of the evaluation coefficients corresponding to the respective autonomous driving scenario to the evaluation coefficient totality, as the weight corresponding to the respective autonomous driving scenario.
  • a weight corresponding to an autonomous driving scenario which is determined through an improved analytic hierarchy process, accurately reflects a level at which the autonomous driving scenario influences an overall autonomous driving effect.
  • the evaluation coefficients may be classified into multiple ranks, different ranks correspond to different eligible values, an evaluation coefficient may be one of the multiple eligible values.
  • the evaluation coefficients corresponding to the autonomous driving scenarios are constrained in a range to avoid a large deviation among evaluation coefficients corresponding to different autonomous driving scenarios. For example, the evaluation coefficients are classified into 5 ranks, and the corresponding eligible values are 9, 7, 5, 3, 1.
  • an expert evaluation method can be employed to evaluate a level at which an autonomous driving scenario influences an overall autonomous driving effect, where coefficients corresponding to each of the autonomous driving scenarios submitted by multiple experts may be acquired.
  • evaluation coefficients corresponding to the autonomous driving scenarios submitted by multiple experts are shown in the following table : Table 3 Travelling to follow Travelling straight Intelligent nudge Expert 1 U11 U21 U31 Expert 2 U12 U22 U32 ... Expert n U1n U2n U3n Weight T1 T2 T3
  • U11, U21 and U31 are respective evaluation coefficients corresponding to 3 autonomous driving scenarios and given by Expert 1
  • U1n, U2n and U3n are respective evaluation coefficients corresponding to 3 autonomous driving scenarios and given by Expert n.
  • a weight of a scenario of travelling to follow T1 (U11+U12+...U1n)/U
  • a weight of a scenario of intelligent nudge T3 (U31+U32+...U3n)/U.
  • the method for calculating weights of other autonomous driving scenarios is similar, which will not be exemplified one by one herein.
  • Step S208 determining the second test result according to the quantitative experience information corresponding to the autonomous driving scenarios and the weights corresponding to the autonomous driving scenarios.
  • a weighted sum of the quantitative experience information corresponding to the autonomous driving scenarios may be calculated to obtain the second test result.
  • a subjective evaluation result of autonomous driving is quantitated through an improved analytic hierarchy process and fuzzy evaluation method, a quantitative test result is obtained based on subjective experience evaluation data, which reflects an evaluation result on travelling safety, comfort and intelligence and travelling efficiency of a vehicle from a subjective travelling experience perspective of a passenger.
  • Step S209 determining a road test result of the vehicle according to the first test result and the second test result of the vehicle.
  • the road test result of the vehicle is determined by combining the first test result and the second test result.
  • weights of the first test result based on objective data and the second test result based on subjective data may be assigned, such that a weighted sum of the first test result and the second test result is calculated as the road test result of the vehicle.
  • a sum of the first test result based on objective data and the second test result based on subjective data may serve as the road test result of the vehicle; or an eventual road test result may be determined according to the first test result and the second test result through other methods, which are not specifically limited herein.
  • Step S210 acquiring, according to a current version of an autonomous driving system of the vehicle, a road test result of the vehicle when a previous version of the autonomous driving system is employed.
  • the road test result of the vehicle when the previous version of the autonomous driving system is employed may be obtained through the above steps S201-S209, the specific details will not be elaborated herein.
  • Step S211 comparing road test results of the vehicle when the previous version and the current version of the autonomous driving system are employed, and displaying a comparing result.
  • the process in the road test method for an autonomous driving vehicle to determine the second test result based on the experience evaluation data, and to determine the first test result based on the objective road test travelling data and the vehicle surroundings information is explained in detail, and the road test result of the vehicle is obtained by combining the first test result and the second test result.
  • the road test result of the vehicle includes an objective test result based on objective test data such that rigorousness and preciseness of the road test of the vehicle are guaranteed, besides, the road test result of the vehicle also includes a quantitative test result based on subjective experience evaluation data, which reflects an evaluation result on travelling safety, comfort and intelligence and travelling efficiency of a vehicle from a subjective perspective of a passenger, thereby a more comprehensive road test result is obtained, and an overall performance of an autonomous driving vehicle in the road test is evaluated more accurately, such that the accuracy of a road test result of an autonomous driving vehicle is improved.
  • FIG. 4 is a schematic diagram of a road test apparatus for an autonomous driving vehicle according to a third embodiment of the present application.
  • the road test apparatus for an autonomous driving vehicle provided in the embodiment of the present application can carry out the processing procedure provided in the embodiments of the road test method for an autonomous driving vehicle.
  • a road test apparatus 30 for an autonomous driving vehicle includes: a road test data processing module 301, an objective testing module 302, a subjective testing module 303 and a comprehensive testing module 304.
  • the road test data processing module 301 is configured to acquire autonomous driving scenarios, information of test parameters corresponding to the autonomous driving scenarios and experience evaluation data in a process during which a vehicle is travelling along a test route, where different autonomous driving scenarios correspond to different test parameters.
  • the objective testing module 302 is configured to analyze the information of the test parameters corresponding to the autonomous driving scenarios according to evaluation baselines corresponding to the autonomous driving scenarios, to determine a first test result of the vehicle.
  • the subjective testing module 303 is configured to generate a second test result according to the experience evaluation data.
  • the comprehensive testing module 304 is configured to determine a road test result of the vehicle according to the first test result and the second test result of the vehicle.
  • the apparatus provided in the embodiment of the present application may be specifically configured to carry out the method embodiment provided in the forgoing first embodiment, whose functions will not be elaborated herein.
  • autonomous driving scenarios and information of test parameters corresponding to the autonomous driving scenarios in a process during which a vehicle is travelling along a test route are obtained, the information of the test parameters corresponding to the autonomous driving scenario is analyzed according to evaluation baselines corresponding to the autonomous driving scenarios, to determine a first test result of the vehicle, that is, the first test result is obtained by evaluating an autonomous driving ability of the vehicle according to objective test data such as the information of the test parameters corresponding to the autonomous driving scenarios; a second test result is obtained by evaluating an autonomous driving ability of the vehicle from a travelling experience perspective of an evaluation personnel according to subjective evaluation data such as experience evaluation data; and a road test result of the vehicle is obtained by combining the first test result and the second test result.
  • the road test result of the vehicle includes an objective test result based on objective test data such that objectivity, rigorousness and preciseness of the road test of the vehicle are guaranteed, besides, the road test result of the vehicle also includes a quantitative test result based on subjective experience evaluation data, which reflects an evaluation result on travelling safety, comfort and intelligence and travelling efficiency of a vehicle from a subjective travelling experience perspective of a passenger, thereby a more comprehensive road test result is obtained, and an overall performance of an autonomous driving vehicle in the road test is evaluated more accurately, such that the accuracy of a road test result of an autonomous driving vehicle is improved.
  • the road test data processing module 301 is further configured to: acquire road test travelling data in the process during which the vehicle is travelling along the test route; analyze the road test travelling data, or the road test travelling data and the vehicle surroundings information to identify autonomous driving scenarios which occur in the process during which the vehicle is travelling along the test route; and acquire the information of the test parameters corresponding to the autonomous driving scenarios; where the vehicle surroundings information includes one or more of the following: a distance between the vehicle and a vehicle in front, information of a lane on which the vehicle is travelling, intersection information of an intersection to which the vehicle is travelling, information of traffic around the vehicle.
  • the road test data processing module is further configured to: analyze vehicle controlling information in the road test travelling data; and determine that one of the autonomous driving scenarios occurs in the process during which the vehicle is travelling along the test route, if the vehicle controlling information includes controlling information corresponding to the one of the autonomous driving scenarios.
  • the road test data processing module is further configured to: analyze the road test travelling data and the vehicle surroundings information to acquire a behavior characteristic of the vehicle; and determine, according to the behavior characteristic of the vehicle, that one of the autonomous driving scenarios occurs in the process during which the vehicle is travelling along the test route if the vehicle has a behavior characteristic of the one of the autonomous driving scenario.
  • the road test data processing module is further configured to: determine occurrence times of the autonomous driving scenarios; and acquire, from the road test travelling data and the vehicle surroundings information, actual values of the test parameters corresponding to the autonomous driving scenarios at the occurrence times.
  • the objective testing module is further configured to: the information of the test parameters corresponding to the autonomous driving scenarios includes actual values of the test parameters corresponding to the autonomous driving scenarios, the evaluation baselines corresponding to the autonomous driving scenarios include baseline ranges of the test parameters corresponding to the autonomous driving scenarios; determine test parameters whose actual values are within corresponding baseline ranges according to the actual values of the test parameters corresponding to the autonomous driving scenarios and the baseline ranges of the test parameters corresponding to the autonomous driving scenarios; and determine a sum of evaluation values corresponding to the test parameters whose actual values are within corresponding baseline ranges as the first test result of the vehicle; where each of the test parameters corresponding to each of the autonomous driving scenarios corresponds to an evaluation value.
  • the subjective testing module is further configured to: calculate quantitative experience information corresponding to the autonomous driving scenarios according to experience evaluation data corresponding to the autonomous driving scenarios in the process during which the vehicle is travelling along the test route; and determine the second test result according to the quantitative experience information corresponding to the autonomous driving scenarios and weights corresponding to the autonomous driving scenarios.
  • the subjective testing module is further configured to: regarding any one of the autonomous driving scenarios, experience evaluation data corresponding to the one of the autonomous driving scenarios includes evaluation levels corresponding to the one of the autonomous driving scenarios, where different evaluation levels correspond to different quantitative information; determine a total evaluation count for which an experience evaluation is carried out with respect to the one of the autonomous driving scenarios and an evaluation count of a respective evaluation level corresponding to the one of the autonomous driving scenarios, according to the evaluation levels corresponding to the one of the autonomous driving scenarios; determine a ratio of the evaluation count of the respective evaluation level corresponding to the one of the autonomous driving scenarios to the total evaluation count as an evaluation weight of the respective evaluation level corresponding to the one of the autonomous driving scenarios; and calculate a weighted sum of quantitative information of the evaluation levels based on the evaluation weights of the evaluation levels corresponding to the one of the autonomous driving scenarios, to obtain quantitative experience information corresponding to the one of the autonomous driving scenarios.
  • the subjective testing module is further configured to: acquire multiple evaluation coefficients corresponding to a respective autonomous driving scenario, where the evaluation coefficients are used for evaluating a level at which the respective autonomous driving scenario influences a driving experience of the vehicle; calculate, according to the evaluation coefficients corresponding to the autonomous driving scenarios, an evaluation coefficient totality corresponding to all of the autonomous driving scenarios, and a sum of evaluation coefficients corresponding to the respective autonomous driving scenario; and determine a ratio of the sum of the evaluation coefficients corresponding to the respective autonomous driving scenario to the evaluation coefficient totality, as the weight corresponding to the respective autonomous driving scenario.
  • the road test data processing module is further configured to: send an evaluation factor corresponding to a respective autonomous driving scenario to multiple experience evaluation terminals; and receive experience evaluation data corresponding to the respective autonomous driving scenario, the experience evaluation data being collected based on the evaluation factor in the process during which the vehicle is travelling along the test route and submitted by the multiple experience evaluation terminals.
  • the comprehensive testing module is further configured to: after determining the road test result of the vehicle according to the first test result and the second test result of the vehicle, acquire, according to a current version of an autonomous driving system of the vehicle, a road test result of the vehicle when a previous version of the autonomous driving system is employed; and compare road test results of the vehicle when the previous version and the current version of the autonomous driving system are employed, and display a comparing result.
  • the apparatus provided in the embodiment of the present application may be specifically configured to carry out the method embodiment provided in the foregoing second embodiment, whose functions will not be elaborated herein.
  • the process in the road test method for an autonomous driving vehicle to determine the second test result based on the experience evaluation data, and to determine the first test result based on the objective road test travelling data and the vehicle surroundings information is explained in detail, and the road test result of the vehicle is obtained by combining the first test result and the second test result.
  • the road test result of the vehicle includes an objective test result based on objective test data such that rigorousness and preciseness of the road test of the vehicle are guaranteed, besides, the road test result of the vehicle also includes a quantitative test result based on subjective experience evaluation data, which reflects an evaluation result on travelling safety, comfort and intelligence and travelling efficiency of a vehicle from a subjective perspective of a passenger, thereby a more comprehensive road test result is obtained, and an overall performance of an autonomous driving vehicle in the road test is evaluated more accurately, such that the accuracy of a road test result of an autonomous driving vehicle is improved.
  • the present application provides a computer program, the computer program is stored in a computer-readable storage medium, at least one processor of an electronic device can read the computer program from the computer-readable storage medium, and the at least one processor executes the computer program to enable the electronic device to execute the method provided by any of the above embodiments.
  • the present application further provides an electronic device and a readable storage medium.
  • FIG. 5 is a schematic diagram of an electronic device to implement the road test method for an autonomous driving vehicle of the present application.
  • the electronic device is intended to represent a digital computer of various forms, such as a laptop, a desktop, a workstation, a personal digital assistant, a server, a blade server, a mainframe computer, or other suitable computers.
  • the electronic device may also represent a mobile device of various forms, such as a personal digital assistant, a cellular phone, a smart phone, a wearable device or other similar computing devices.
  • the components shown herein, the connections and relationships thereof, and functions thereof, are only examples and are not intended to limit the implementation of the application described and/or claimed herein.
  • the electronic device 500 includes: one or more processors 501, a memory 502 and interfaces interconnecting the respective components, including a high-speed interface and a low-speed interface.
  • the various components are connected to each other with different buses which may be installed on a common motherboard or otherwise as needed.
  • the processor 501 may process instructions executed by the electronic device 500, including instructions stored in or on the memory to display graphical information of a GUI on an external input/output apparatus (such as a display device coupled to an interface).
  • an external input/output apparatus such as a display device coupled to an interface.
  • multiple processors and/or multiple buses can be combined with multiple memories.
  • multiple electronic devices 500 can be interconnected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system).
  • FIG. 5 shows an example of one processor 501.
  • the memory 502 is the non-transitory computer-readable storage medium provided in the present application.
  • the memory stores instructions that can be executed by at least one processor to cause the at least one processor to carry out the road test method for an autonomous driving vehicle provided in the present application.
  • the non-transitory computer-readable storage medium of the present application stores thereon computer instructions, and the computer instructions are intended to cause a computer to carry out the road test method for an autonomous driving vehicle provided in the present application.
  • the memory 502 can be used to store non-transitory software programs, non-transitory computer-executable programs and modules, such as the program instructions/modules corresponding to the road test method for an autonomous driving vehicle in the embodiments of the present application, for example, the road test data processing module 301, the objective testing module 302, the subjective testing module 303, and the comprehensive testing module 304 shown in FIG. 4 .
  • the processor 501 executes various functional applications and data processing of a server by running non-transitory software programs, instructions, and modules stored in the memory 502, that is, implements the road test method for an autonomous driving vehicle in the foregoing method embodiments.
  • the memory 502 may include a program storage area and a data storage area, where the program storage area may store an operating system and an application program required by at least one function; the data storage area may store data created by the use of the electronic device 500 when implementing the road test method for an autonomous driving vehicle, or the like.
  • the memory 502 may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid-state storage devices.
  • the memory 502 may optionally include memories remotely provided with respect to the processor 501, and these remote memories may be connected via a network to the electronic device 500 implementing the road test method for an autonomous driving vehicle. Examples of the above network includes but not limited to, the Internet, a corporate intranet, a local area network, a mobile communication network, or a combination thereof.
  • the electronic device 500 implementing the road test method for an autonomous driving vehicle may further include: an input apparatus 503 and an output apparatus 504.
  • the processor 501, the memory 502, the input apparatus 503, and the output apparatus 504 may be interconnected with a bus or otherwise.
  • FIG. 5 shows an example of connection by a bus.
  • the input apparatus 503 may receive input number or character information, and generate key signal input related to the user settings and function controlling of the electronic device 500 implementing the road test method for an autonomous driving vehicle, which may be an input apparatus such as a touch screen, a keypad, a mouse, a track board, a touchpad, a pointing stick, one or more mouse buttons, a trackball or a joystick.
  • the output apparatus 504 may include a display device, an auxiliary lighting device (for example, LED), a tactile feedback device (for example, a vibration motor) and the like.
  • the display device may include but not limited to, a liquid crystal display (LCD), a light emitting diode (LED) display, a monitor or a plasma display. In some embodiments, the display device may be a touch screen.
  • Various implementations of the systems and technologies described herein may be implemented in a digital electronic circuit system, an integrated circuit system, an application-specific ASIC (application-specific integrated circuits), a computer hardware/firmware/software, and/or a combination thereof.
  • These various embodiments may be implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted in a programmable system including at least one programmable processor, the programmable processor may be a special-purpose or a general-purpose programmable processor that can receive data and instructions from a storage system, at least one input apparatus, or at least one output apparatus, and transmit data and instructions to the storage system, the at least one input apparatus, or the at least one output apparatus.
  • machine-readable medium refers to any computer program product, device, and/or apparatus used to provide machine instructions and/or data to a programmable processor, for example, a magnetic disk, an optical disk, a memory, a programmable logic devices (PLD), which includes a machine-readable medium that receives machine instructions as machine-readable signals.
  • machine-readable signal refers to any signal used to provide machine instructions and/or data to a programmable processor.
  • a display device for displaying information to the user (for example, a CRT (cathode ray tube) or LCD (liquid crystal display) monitor), a keyboard and a pointing device (for example, a mouse or a trackball) through which input can be provided to the computer by a user.
  • a feedback provided to a user may be sensory feedback of any form, for example, a visual feedback, an auditory feedback, or a tactile feedback, and an input may be received from a user in any form (including an acoustic input, a voice input, or a tactile input).
  • the systems and technologies described here may be implemented in a computing system that includes a back-end component, for example, as a data server; or in a computing system that includes a middleware component, for example, an application server, or in a computing system that includes a front-end component, for example, a user computer with a graphical user interface or a web browser through which a user can interact with the implementations of the system and technology described herein, or in a computing system that includes any combination of such background component, intermediate component or the front-end component.
  • the components of the system may be interconnected through any form or medium of digital data communication, for example, a communication network. Examples of the communication network include: a local area network (LAN), a wide area network (WAN), and the Internet.
  • a computer system may include a client and a server.
  • the client and server are generally remote from each other and usually interact through a communication network.
  • the relationship of a client and a server is established by computer programs that run on the corresponding computers and have a client-server relationship with each other.
  • autonomous driving scenarios and information of test parameters corresponding to the autonomous driving scenarios in a process during which a vehicle is travelling along a test route are obtained, the information of the test parameters corresponding to the autonomous driving scenario is analyzed according to evaluation baselines corresponding to the autonomous driving scenarios, to determine a first test result of the vehicle, that is, the first test result is obtained by evaluating an autonomous driving ability of the vehicle according to objective test data such as the information of the test parameters corresponding to the autonomous driving scenarios; a second test result is obtained by evaluating an autonomous driving ability of the vehicle from a travelling experience perspective of an evaluation personnel according to subjective evaluation data such as experience evaluation data; and a road test result of the vehicle is obtained by combining the first test result and the second test result.
  • the road test result of the vehicle includes an objective test result based on objective test data such that objectivity, rigorousness and preciseness of the road test of the vehicle are guaranteed, besides, the road test result of the vehicle also includes a quantitative test result based on subjective experience evaluation data, which reflects an evaluation result on travelling safety, comfort and intelligence and travelling efficiency of a vehicle from a subjective travelling experience perspective of a passenger, thereby a more comprehensive road test result is obtained, and an overall performance of an autonomous driving vehicle in the road test is evaluated more accurately, such that the accuracy of a road test result of an autonomous driving vehicle is improved.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Automation & Control Theory (AREA)
  • Transportation (AREA)
  • Mechanical Engineering (AREA)
  • Human Computer Interaction (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Atmospheric Sciences (AREA)
  • Mathematical Physics (AREA)
  • Chemical & Material Sciences (AREA)
  • Analytical Chemistry (AREA)
  • Traffic Control Systems (AREA)
  • Control Of Driving Devices And Active Controlling Of Vehicle (AREA)
EP21178545.6A 2020-12-10 2021-06-09 Road test method and apparatus for autonomous driving vehicle, device and storage medium Active EP3866044B1 (en)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202011436678.6A CN112525551B (zh) 2020-12-10 2020-12-10 自动驾驶车辆的路测方法、装置、设备及存储介质

Publications (3)

Publication Number Publication Date
EP3866044A2 EP3866044A2 (en) 2021-08-18
EP3866044A3 EP3866044A3 (en) 2021-12-29
EP3866044B1 true EP3866044B1 (en) 2023-05-24

Family

ID=74999256

Family Applications (1)

Application Number Title Priority Date Filing Date
EP21178545.6A Active EP3866044B1 (en) 2020-12-10 2021-06-09 Road test method and apparatus for autonomous driving vehicle, device and storage medium

Country Status (5)

Country Link
US (1) US20210316753A1 (ja)
EP (1) EP3866044B1 (ja)
JP (1) JP7297846B2 (ja)
KR (1) KR102582657B1 (ja)
CN (1) CN112525551B (ja)

Families Citing this family (22)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113468238A (zh) * 2021-06-21 2021-10-01 上汽通用五菱汽车股份有限公司 自适应巡航系统的评测方法、装置及计算机可读存储介质
CN113504056A (zh) * 2021-06-30 2021-10-15 中国科学技术大学先进技术研究院 一种自动驾驶汽车测试评价系统及方法
CN113535816A (zh) * 2021-07-13 2021-10-22 江苏智多行网联科技有限公司 一种智能网联云控车辆的行车性能评价方法及系统
CN113609016B (zh) * 2021-08-05 2024-03-15 北京赛目科技股份有限公司 车辆自动驾驶测试场景的构建方法、装置、设备及介质
EP4141406A1 (en) * 2021-08-31 2023-03-01 Volvo Autonomous Solutions AB Remote perception station as maintenance trigger for autonomous vehicles deployed in autonomous transport solutions
CN113515463B (zh) * 2021-09-14 2022-04-15 深圳佑驾创新科技有限公司 自动化测试方法、装置、计算机设备和存储介质
CN113790903A (zh) * 2021-09-15 2021-12-14 江苏智行未来汽车研究院有限公司 低速智能网联汽车封闭场景静态避障测试数据评估方法
CN113887037B (zh) * 2021-09-28 2024-04-26 中国科学技术大学先进技术研究院 一种不同渗透率的网联环境下自动驾驶系统测评方法
CN113741700A (zh) * 2021-09-29 2021-12-03 奇瑞新能源汽车股份有限公司 车辆的自动驾驶评价系统及方法
CN114194204A (zh) * 2021-11-30 2022-03-18 际络科技(上海)有限公司 自动驾驶车辆体感数据处理方法及系统
CN114065549B (zh) * 2021-11-30 2023-04-07 广州文远知行科技有限公司 自动驾驶水平评估方法、装置、设备及存储介质
CN114415628A (zh) * 2021-12-28 2022-04-29 阿波罗智联(北京)科技有限公司 自动驾驶测试方法及装置、电子设备和存储介质
CN114526929A (zh) * 2022-02-25 2022-05-24 智己汽车科技有限公司 一种车道变换测试方法、装置及车辆
CN114581873B (zh) * 2022-03-11 2023-08-18 阿波罗智联(北京)科技有限公司 筛选路侧设备的方法、装置、设备、介质及产品
CN114572233B (zh) * 2022-03-25 2022-11-29 阿波罗智能技术(北京)有限公司 基于模型集合的预测方法、电子设备及自动驾驶车辆
CN115077936B (zh) * 2022-06-16 2023-04-07 中国汽车工程研究院股份有限公司 一种车辆自适应巡航系统驾乘性能测评方法
CN114782926B (zh) * 2022-06-17 2022-08-26 清华大学 驾驶场景识别方法、装置、设备、存储介质和程序产品
DE102022126555A1 (de) 2022-10-12 2024-04-18 Dr. Ing. H.C. F. Porsche Aktiengesellschaft Verfahren, System und Computerprogrammprodukt zur Prädiktion von gruppenspezifischen Bewertungen eines ADAS/ADS-Systems
CN115655752B (zh) * 2022-12-09 2023-03-17 成都鲁易科技有限公司 新能源车辆自动测试方法及装置、电子设备和存储介质
CN115689259B (zh) * 2023-01-04 2023-06-30 交通运输部公路科学研究所 路段车路协同场景优先级确定方法、设备及介质
CN115774680B (zh) * 2023-01-16 2023-04-11 小米汽车科技有限公司 自动驾驶软件的版本测试方法、装置、设备及存储介质
CN115979679B (zh) * 2023-03-22 2023-06-23 中国汽车技术研究中心有限公司 自动驾驶系统实际道路测试方法、设备和存储介质

Family Cites Families (23)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103234763B (zh) * 2013-04-09 2015-04-15 北京理工大学 一种无人驾驶车辆定量评价系统及评价方法
AT514754B1 (de) * 2013-09-05 2018-06-15 Avl List Gmbh Verfahren und Vorrichtung zur Optimierung von Fahrassistenzsystemen
DE102016220913A1 (de) * 2015-11-06 2017-05-11 Ford Global Technologies, Llc Verfahren und Vorrichtung zur Generierung von Testfällen für autonome Fahrzeuge
US10606276B2 (en) * 2016-09-30 2020-03-31 Faraday & Future Inc. User data-based autonomous vehicle system
KR101938064B1 (ko) * 2017-05-17 2019-01-11 현대오토에버 주식회사 자율주행 테스트 시스템
CN107782564B (zh) * 2017-10-30 2019-12-13 青岛慧拓智能机器有限公司 一种自动驾驶车辆测评系统及方法
CN107843440B (zh) * 2017-10-30 2019-12-13 青岛慧拓智能机器有限公司 一种自动驾驶车辆性能测试系统及方法
US10345811B2 (en) * 2017-11-14 2019-07-09 GM Global Technology Operations LLC Method and apparatus for scenario generation and parametric sweeps for the development and evaluation of autonomous driving systems
US10677686B2 (en) * 2017-11-14 2020-06-09 GM Global Technology Operations LLC Method and apparatus for autonomous system performance and grading
CN108458880A (zh) * 2018-01-29 2018-08-28 上海测迅汽车科技有限公司 无人驾驶整车受控场景测试方法
CN108645628B (zh) * 2018-05-14 2020-08-21 公安部交通管理科学研究所 基于道路驾驶技能的自动驾驶汽车测试系统
CN109242251B (zh) * 2018-08-03 2020-03-06 百度在线网络技术（北京）有限公司 行车行为安全性检测方法、装置、设备及存储介质
CN109187048A (zh) * 2018-09-14 2019-01-11 盯盯拍（深圳）云技术有限公司 自动驾驶性能测试方法以及自动驾驶性能测试装置
CN109406166A (zh) * 2018-10-30 2019-03-01 百度在线网络技术（北京）有限公司 无人车的分级方法、装置、设备、存储介质和车辆
CN109520744B (zh) * 2018-11-12 2020-04-21 百度在线网络技术（北京）有限公司 自动驾驶车辆的驾驶性能测试方法和装置
CN111413958B (zh) * 2018-12-18 2021-09-24 北京航迹科技有限公司 用于确定自动驾驶中的驾驶路径的系统和方法
CN110196994A (zh) * 2019-04-23 2019-09-03 同济大学 一种自动驾驶汽车交通协调性评测模型及方法
CN110779730A (zh) * 2019-08-29 2020-02-11 浙江零跑科技有限公司 基于虚拟驾驶场景车辆在环的l3级自动驾驶系统测试方法
CN110606093A (zh) * 2019-09-26 2019-12-24 中国第一汽车股份有限公司 车辆性能评测方法、装置、设备和存储介质
CN110825093B (zh) * 2019-11-28 2021-04-16 安徽江淮汽车集团股份有限公司 自动驾驶策略生成方法、装置、设备及存储介质
CN111487070A (zh) * 2020-04-28 2020-08-04 北京百度网讯科技有限公司 自动驾驶车辆的驾驶性能测试方法、装置、设备及介质
CN111735639B (zh) * 2020-05-26 2022-03-22 清华大学苏州汽车研究院（相城） 一种面向智能网联汽车示范区的自动驾驶场景最小集生成方法
CN111765903B (zh) * 2020-06-29 2022-08-09 阿波罗智能技术(北京)有限公司 自动驾驶车辆的测试方法、装置、电子设备和介质

Also Published As

Publication number Publication date
JP7297846B2 (ja) 2023-06-26
JP2022017446A (ja) 2022-01-25
US20210316753A1 (en) 2021-10-14
EP3866044A3 (en) 2021-12-29
CN112525551A (zh) 2021-03-19
KR20210088457A (ko) 2021-07-14
KR102582657B1 (ko) 2023-09-25
EP3866044A2 (en) 2021-08-18
CN112525551B (zh) 2023-08-29

Similar Documents

Publication Publication Date Title
EP3866044B1 (en) Road test method and apparatus for autonomous driving vehicle, device and storage medium
US11527155B2 (en) Time-segmented signal timing method and apparatus for traffic light, electronic device and storage medium
US11379296B2 (en) Intelligent responding to error screen associated errors
US20210233405A1 (en) Parking lot free parking space predicting method, apparatus, electronic device and storage medium
CN112527748B (zh) 用于分析用户操作行为的方法、装置、设备以及存储介质
US20210232986A1 (en) Parking lot free parking space predicting method, apparatus, electronic device and storage medium
US10509649B2 (en) Value stream graphs across heterogeneous software development platforms
CN111552646A (zh) 用于回归测试的方法和装置
CN112035587A (zh) 基于轨迹数据校验交规的方法、装置、设备和介质
CN112634631A (zh) 交通信号灯的控制方法、装置、系统以及电子设备
CN112700112B (zh) Rpa流程调整方法、装置、电子设备及存储介质
CN110555486A (zh) 模型结构的延时预测方法、装置以及电子设备
CN111724598B (zh) 用于自动驾驶规划路径的方法、装置、设备以及存储介质
CN112270532B (zh) 一种数据处理方法、装置、电子设备以及存储介质
CN112528846A (zh) 障碍物检测的测评方法、装置、设备以及存储介质
CN110825951A (zh) 网页处理方法、装置和电子设备
CN112540904B (zh) 机器操作行为识别方法、装置、电子设备以及计算机介质
CN110887671B (zh) 自动驾驶中定位车辆故障的根本原因的方法和装置
US20160253061A1 (en) Evaluating user interface efficiency
CN112270414A (zh) 算子监控方法、装置、电子设备及存储介质
CN111782794A (zh) 问答响应方法和装置
CN111599174A (zh) 一种交通指标的异常检测方法和电子设备
CN111753955A (zh) 一种模型参数调整方法、装置、电子设备和存储介质
CN113177077B (zh) 用于自动驾驶的异常事件确定方法、装置及电子设备
CN111538656A (zh) 梯度检查的监控方法、装置、设备及存储介质

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20210609

AK Designated contracting states

Kind code of ref document: A2

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

PUAL Search report despatched

Free format text: ORIGINAL CODE: 0009013

AK Designated contracting states

Kind code of ref document: A3

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

RIC1 Information provided on ipc code assigned before grant

Ipc: G01M 17/007 20060101ALI20211119BHEP

Ipc: G06F 30/15 20200101AFI20211119BHEP

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20220826

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20230215

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602021002439

Country of ref document: DE

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1569987

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230615

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG9D

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20230524

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 1569987

Country of ref document: AT

Kind code of ref document: T

Effective date: 20230524

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230925

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230824

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

P01 Opt-out of the competence of the unified patent court (upc) registered

Effective date: 20231025

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230924

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230825

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: FR

Payment date: 20230707

Year of fee payment: 3

Ref country code: DE

Payment date: 20230703

Year of fee payment: 3

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20230630

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602021002439

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230609

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230609

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230609

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230609

26N No opposition filed

Effective date: 20240227

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20230524

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20230630