EP4673830A1

EP4673830A1 - Determination of safety criticality of test scenarios for autonomous vehicles

Info

Publication number: EP4673830A1
Application number: EP23716892.7A
Authority: EP
Inventors: Vijaya Sarathi Indla; Shehzaman Salim KHATIB; Saikat Mukherjee; Sagar PATHRUDKAR
Original assignee: Siemens AG; Siemens Corp
Current assignee: Siemens AG; Siemens Corp
Priority date: 2023-04-05
Filing date: 2023-04-05
Publication date: 2026-01-07
Also published as: WO2024208416A1

Abstract

A test scenario discretization system (102), a computer implemented method (300) and a computer program product for discretizing a test scenario set (Ts) including test scenarios (S1-Sn) and scenario parameters (P1-Pm) affecting an autonomous vehicle (AV) when deployed in real-world, are provided. The test scenario discretization system (102) determines, by employing a trained safety criticality estimation model (103), an impact on safety associated with the AV corresponding to a value assumable by each of the scenario parameters (P1-Pm) which is varied between a lower range value and a higher range value thereof, clusters the scenario parameters (P1-Pm) based on the impact and a predetermined impact variation threshold, and generates, based on the clustered scenario parameters (P1-Pm), a focused test scenario set (Tsf) having focused test scenarios (Sf1-Sfn) and focused scenario parameters (Pf1-Pfm) transferrable to an autonomous system (101) for testing of the AV.

Description

DETERMINATION OF SAFETY CRITICALITY OF TEST SCENARIOS FOR AUTONOMOUS VEHICLES

The present disclosure relates to scenarios involved in testing of autonomous vehicle systems. More particularly, the present disclosure relates to a system and a method for discretizing a test scenario set employable in testing of an autonomous vehicle.

With rapid progress being made in the field of autonomous driving technology, gaining trust and confidence of drivers and passengers of the autonomous vehicles and acceptance from the environment in which these vehicles are expected to be driven, are necessary factors to foster the development of autonomous driving technologies. To achieve such trust and acceptance, it is imperative that the advanced driver assist systems (ADAS) and/or the automated driving system (ADS), associated with the autonomous vehicles as well as semi- autonomous vehicles, are tested thoroughly with a wide range of test scenarios.

There are many variables that contribute to the criticality or unsafe-ness of a test scenario for the ADAS and ADS systems. For example, environmental factors such as weather which affects visibility and road friction, time of day which affects illumination and consequently the visibility, map geometry at the geolocation where the vehicle is to be driven, curvature of the road, lane width, junction configuration including approach angles, junction radius, etc., behavior of traffic in that geography and/or demography, driver behavior such as conservative, aggressive, reckless, etc.

Each of these different scenario variables can take a large number of possible values thus exploding the test space that contains test scenarios to be used for testing the ADAS and ADS systems, commonly referred to as the test-space explosion problem. Typically, not each value of a scenario variable and/or combinations of these values hold equal importance when it comes to the safety criticality of a scenario. For example, some of the values and/or value combinations may be more safety critical than others. Accordingly, it is an object of the present disclosure to provide a system and a method that prioritizes the safety critical scenarios while testing the autonomous vehicles, thereby leading to a reduction in the test scenario -space and correspondingly faster deployment of the ADAS and ADS systems.

The present disclosure achieves the aforementioned object by providing a test discretization system, hereinafter referred to as TSDS, and a computer implemented method for discretizing a test scenario set Ts comprising one or more test scenarios Sl-Sn.

The TSDS comprises a non-transitory computer readable storage medium storing computer program instructions defined by module(s) of the TSDS, and at least one processor communicatively coupled to the non-transitory computer readable storage medium, wherein the at least one processor is configured to execute the computer program instructions, thereby performing the computer implemented method for discretizing a test scenario set Ts comprising one or more test scenarios Sl-Sn.

Each test scenario Sl-Sn including multiple scenario parameters Pl-Pm affecting an autonomous vehicle (AV) when deployed in a real world environment.

According to an embodiment, the computer implemented mehtod employs the TSDS for discretizing a test scenario set Ts comprising one or more test scenarios Sl-Sn.

As used herein, the test scenario set Ts is associated with historical real-world scenarios and/or historical simulated scenarios. Advantageously, the test scenario set Ts includes historical safety reports associated with the historical real-world scenarios and/or historical safety reports associated with the historical simulated scenarios.

The safety reports, for example, are in a .json file format and include different safety metrics associated with the historical real-world and/or simulated scenarios. The safety metrics include, for example, a likelihood of occurrence of a safety critical event, an average severity associated with the safety critical event, an aggregate safety score, a coverage score, etc. One such example of the safety report is included below:

{...

"safety _scores":

{

"hkelihood_safety_critical_events" ■ 0.82,

" avg_severity_safety_critical_events" ■ 0.53 , "aggregate_safety_score": 0.16, "no_variations" ■ 243, "coverage_score": 0.9,

} ...}

According to an embodiment, the TSDS obtains the test scenario set Ts from an autonomous system such as an advanced driver assist systems (ADAS) and/or the automated driving system (ADS), employable in testing of the AV.

According to another embodiment, the TSDS obtains the test scenario set Ts stored in a database such as a test space database of the TSDS, a database on a central server with which the TSDS can communicate, a database in a cloud with which the TSDS can communicate, etc.

The TSDS determines, by employing a safety criticality estimation model, an impact on safety associated with the AV corresponding to a value of each of the scenario parameters Pl-Pm, by varying the value of each of the scenario parameters Pl-Pm between a lower range value and a higher range value assumable by each of the scenario parameters Pl-Pm. Advantageously, the values of the scenario parameters Pl-Pm are considered in combination of two or more scenario parameters Pl-Pm, while determining the impact. As used herein, the term “impact” refers to and includes a safety score associated with the AV corresponding to the value of each of the scenario parameters Pl-Pm. Advantageously, the safety score refers to a combination of values of two or more scenario parameters Pl-Pm of a test scenario Sl-Sn as scenario parameters Pl- Pm typically occur in combination to cause a safety critical event. According to an embodiment, the values of the scenario parameters Pl-Pm are varied at a predefined interval. Advantageously, the predefined interval is selected based on an overall range of the scenario parameter Pl-Pm associated therewith and/or a likelihood of the scenario parameter Pl-Pm affecting safety associated with the AV. For example, if illumination and traffic density are two of the scenario parameters Pl-Pm being considered for a foggy environmental test scenario Si such that illumination has a wide overall range between its lower range value and its higher range value compared to that of traffic density and wherein illumination is known to have a greater likelihood of impacting safety of the AV especially for a foggy environmental test scenario Si when compared to traffic density, then the interval at which the values of illumination as a scenario parameter is varied will be lower than the interval at which traffic density as a scenario parameters is varied. Advantageously, such interval selection ensures that the safety critical values of each scenario parameter Pl-Pm are not missed out on during the variation.

According to another embodiment, the values of the scenario parameters Pl-Pm are varied so as to generate value groups of each scenario parameter Pl-Pm based on the predetermined interval. According to this embodiment, the TSDS determines the impact per value group of the scenario parameters Pl-Pm by employing the safety criticality estimation model.

The safety criticality estimation model is trained to estimate criticality of safety associated with each value of each of the scenario parameters Pl-Pm. The TSDS trains the safety criticality estimation model. The TSDS, for training the safety criticality estimation model, generates a semantic graph corresponding to each of the test scenarios Sl-Sn of the test scenario set Ts, feeds the semantic graph to the safety criticality estimation model for generating predictions for the impact on safety criticality of the AV under each of the test scenarios Sl-Sn, and tunes the predictions for the impact by comparing the predictions with safety reports associated with the historical real-world scenarios and/or the historical simulated scenarios from the test scenario set Ts.

The semantic graph comprises multiple nodes connected therebetween via arrows. The node represents an AV, an event associated with the AV, or a scenario parameter(s) (Pl-Pm). The arrow represents a relationship between the scenario parameters Pl-Pm, a relationship between the AV and the scenario parameter(s) Pl-Pm, or a relationship between the event associated with the AV and one or more of the scenario parameters Pl-Pm.

The TSDS clusters the scenario parameters Pl-Pm based on the impact and a predetermined impact variation threshold. Advantageously, the predetermined impact variation threshold is defined based on a minimum variation in the impact corresponding to each of the scenario parameters Pl-Pm. For example, by defining a boundary condition of +/- 5% variation as an absolute change in the end outcome of the impact, that is, the safety scores associated with the scenario parameters Pl-Pm. Alternatively, based on a relative change between the impacts corresponding to two or more of the scenario parameters Pl-Pm and by selecting a least count out of these relative changes as the predetermined impact variation threshold.

The TSDS, for clustering the scenario parameters Pl-Pm based on the impact and the predetermined impact variation threshold, determines a variation in the impact by employing sensitivity analysis and/or causality analysis across the scenario parameters Pl-Pm, and aggregates the values of the scenario parameters Pl-Pm having minimal variation therebetween based on the predetermined impact variation threshold.

Advantageously, the sensitivity analysis and/or the causality analysis help in determining a level of impact of each individual scenario parameter Pl-Pm and in identifying the value range of the scenario parameter Pl-Pm that results in same level of significance of impact on safety criticality.

Advantageously, aggregating the values of the scenario parameters reduces the value range of the scenario parameter Pl-Pm, that is, discretizes the range to a coarser bin which ultimately results in reduction of test space associated with testing the AV.

The TSDS generates, based on the clustered scenario parameters Pl-Pm, a focused test scenario set Tsf comprising focused test scenarios Sfl-Sfn. Each focused test scenario Sfl-Sfn comprises focused scenario parameters Pfl-Pfm. The TSDS, for generating the focused test scenario set Tsf, selects sample values from the aggregated values of the scenario parameters Pl-Pm to form the focused scenario parameters Pfl-Pfm of the focused test scenarios Sfl-Sfn of the focused test scenario set Tsf. According to an embodiment, the TSDS selects the sample values based on a simple mean of the aggregated values. According to another embodiment, the TSDS selects the sample values by employing statistical methods to choose a value most suited to maximize coverage or a value most representational of the aggregated values, such as, by employing probabilistic function to determine a value that is most likely to occur from the aggregated values based on historical and/or simulated scenarios.

The focused test scenario set Tsf is transferrable, by the TSDS, to the autonomous system such as the ADAS and/or the ADS for testing of the AV.

According to another aspect of the present disclosure disclosed herein is the computer implemented method for discretizing a test scenario set (Ts) employable in testing of an AV. The computer implemented method obtains, from an autonomous system, employable in testing of the AV, the test scenario set Ts comprising test scenario(s) Sl-Sn, wherein each test scenario Sl-Sn comprises a plurality of scenario parameters Pl-Pm affecting the AV when deployed in a real world environment, determines, by employing a safety criticality estimation model, an impact on safety associated with the AV corresponding to a value of each of the scenario parameters Pl-Pm, by varying the value of each of the scenario parameters Pl-Pm between a lower range value and a higher range value assumable by each of the scenario parameters Pl-Pm, wherein the safety criticality estimation model is trained to estimate criticality of safety associated with each value of each of the scenario parameters Pl-Pm, clusters the scenario parameters Pl-Pm based on the impact and a predetermined impact variation threshold, and generates, based on the clustered scenario parameters Pl-Pm, a focused test scenario set Tsf comprising focused test scenarios Sfl-Sfn, wherein each focused test scenario Sfl-Sfn comprises focused scenario parameters Pfl- Pfm, and wherein the focused test scenario set Tsf is transferrable to the autonomous system 101 for testing of the AV. The computer implemented method trains the safety criticality estimation model by generating a semantic graph corresponding to each of the test scenarios Sl-Sn of the test scenario set Ts, wherein the semantic graph comprises a plurality of nodes connected therebetween via arrows, feeding the semantic graph to the safety criticality estimation model for generating predictions for the impact on safety criticality of the AV under each of the test scenarios Sl-Sn, and tuning the predictions for the impact by comparing the predictions with safety reports associated with one or more of historical real- world scenarios and historical simulated scenarios from the test scenario set Ts.

The computer implemented method clusters the scenario parameters Pl-Pm based on the impact and the predetermined impact variation threshold by determining a variation in the impact by employing sensitivity analysis and/or causality analysis across the scenario parameters Pl-Pm and aggregating the values of the scenario parameters Pl-Pm having minimal variation therebetween based on the predetermined impact variation threshold.

The computer implemented method generates a focused test scenario set Tsf by selecting sample values from the aggregated values of the scenario parameters Pl-Pm to form the focused scenario parameters Pfl-Pfm of the focused test scenarios Sfl-Sfn of the focused test scenario set Tsf.

According to yet another aspect of the present disclosure, disclosed herein is a computer program product having machine-readable instructions stored therein, which when executed by one or more processors, cause the processors to perform the computer implemented method for discretizing a test scenario set (Ts) employable in testing of an AV.

The computer program product disclosed herein comprises a first computer program code for obtaining, from an autonomous system, employable in testing of the AV, the test scenario set Ts comprising test scenario(s) Sl-Sn, wherein each test scenario Sl-Sn comprises a plurality of scenario parameters Pl-Pm affecting the AV when deployed in a real world environment; a second computer program code for determining, by employing a safety criticality estimation model, an impact on safety associated with the AV corresponding to a value of each of the scenario parameters Pl-Pm, by varying the value of each of the scenario parameters Pl-Pm between a lower range value and a higher range value assumable by each of the scenario parameters Pl-Pm, wherein the safety criticality estimation model is trained to estimate criticality of safety associated with each value of each of the scenario parameters Pl-Pm; a third computer program code for clustering the scenario parameters Pl-Pm based on the impact and a predetermined impact variation threshold; and a fourth computer program code for generating, based on the clustered scenario parameters Pl-Pm, a focused test scenario set Tsf comprising focused test scenarios Sfl-Sfn, wherein each focused test scenario Sfl-Sfn comprises focused scenario parameters Pfl-Pfm, and wherein the focused test scenario set Tsf is transferrable to the autonomous system 101 for testing of the AV.

The computer program product comprises a fifth computer program code for training the safety criticality estimation model. The fifth computer program code comprises a sixth computer program code for generating a semantic graph corresponding to each of the test scenarios Sl-Sn of the test scenario set Ts, wherein the semantic graph comprises a plurality of nodes connected therebetween via arrows; a seventh computer program code for feeding the semantic graph to the safety criticality estimation model for generating predictions for the impact on safety criticality of the AV under each of the test scenarios Sl-Sn; and an eight computer program code for tuning the predictions for the impact by comparing the predictions with safety reports associated with one or more of historical real- world scenarios and historical simulated scenarios from the test scenario set Ts.

The third computer program code comprises a ninth computer program code for determining a variation in the impact by employing one or more of sensitivity analysis and causality analysis across the scenario parameters Pl-Pm; and a tenth computer program code for aggregating the values of the scenario parameters Pl-Pm having minimal variation therebetween based on the predetermined impact variation threshold.

The fourth computer program code comprises an eleventh computer program code for selecting sample values from the aggregated values of the scenario parameters Pl-Pm to form the focused scenario parameters Pfl-Pfm of the focused test scenarios Sfl-Sfn of the focused test scenario set Tsf.

The TSDS and the computer implemented method disclosed herein enable in identifying the factors more likely to cause safety concerns with respect to various leading and lagging safety metrics by modelling and analysing the relationships between scenario parameters such as driving behaviors and traffic events. This is achieved by constructing a model, that is, the safety criticality estimation model by mapping combinations of scenario parameters to the safety criticality of the test scenarios thereby assessing the safety of test scenarios without the corresponding test drive output data either from simulation or real world tests, and at least partially, eliminating the need to test large number of scenarios. The TSDS and the computer implemented method disclosed herein quantify an avoidability of safety critical test scenarios using such a model, that is, by quantifying whether the safety criticality of a test scenario would reduce or increase when the scenario conditions were to be different.

The above mentioned and other features of the invention will now be addressed with reference to the accompanying drawings of the present invention. The illustrated embodiments are intended to illustrate, but not limit the invention.

The present invention is further described hereinafter with reference to illustrated embodiments shown in the accompanying drawings, in which:

FIG 1 illustrates a system having a test scenario discretization system for discretizing a test scenario set (Ts) employable in testing of an autonomous vehicle (AV), according to an embodiment of the present disclosure.

FIG 2 is a block diagram illustrating an architecture of a computer system employed by the test scenario discretization system shown in FIG 1, for discretizing a test scenario set (Ts) employable in testing of an autonomous vehicle (AV), according to an embodiment of the present disclosure. FIGS 3A-3B illustrate process flowcharts of a computer implemented method for discretizing a test scenario set (Ts) employable in testing of an autonomous vehicle (AV), according to an embodiment of the present disclosure.

FIGS 4A-4B illustrate exemplary semantic graphs corresponding to a test scenario associated with one or more autonomous vehicles (AVs), according to an embodiment of the present disclosure.

FIGS 5A-5B illustrate graphical representations of clustering and generating a focused test scenario having focused scenario parameters associated with one or more autonomous vehicles (AVs), according to an embodiment of the present disclosure.

Various embodiments are described with reference to the drawings, wherein like reference numerals are used to refer like elements throughout. In the following description, for the purpose of explanation, numerous specific details are set forth in order to provide thorough understanding of one or more embodiments. It may be evident that such embodiments may be practiced without these specific details.

FIG 1 illustrates a system 100 having a test scenario discretization system 102 for discretizing a test scenario set (Ts) employable in testing of an autonomous vehicle (AV), according to an embodiment of the present disclosure. The system 100 also includes an autonomous system 101 employable in testing of the AV. The autonomous system 101 refers to a driving automation system spanning across various levels of driving automation and includes, for example, an automated driving system (ADS) and/or an advanced drives assistance system (ADAS). The test scenario discretization system 102 hereinafter referred to as TSDS 102 is in communication with the autonomous system 101.

The TSDS 102 communicates with the autonomous system 101 via a wired and/or a wireless communication, for example, over a communication network (not shown). The communication network is, for example, a wired network, a wireless network, or a network formed from any combination thereof. The TSDS 102 disclosed herein is installable on and accessible by a user device, for example, a personal computing device, a workstation, a client device, a network enabled computing device, any other suitable computing equipment, and combinations of multiple pieces of computing equipment being used by a user (not shown). The TSDS 102 may also be installable on the autonomous system 101.

The TSDS 102 is configurable as a web-based platform, for example, a website hosted on a server or a network of servers, or, is implemented in the cloud computing environment as a cloud computing-based platform implemented as a service. A user of the TSDS 102 in this case accesses the TSDS 102 via the communication network. The TSDS 102 may have one or more users for example, an AV stack tester and operator.

The TSDS 102 disclosed herein comprises a non-transitory computer readable storage medium and at least one processor communicatively coupled to the non- transitory computer readable storage medium. As used herein, “non-transitory computer readable storage medium” refers to all computer readable media, for example, non-volatile media, volatile media, and transmission media except for a transitory, propagating signal. The non-transitory computer readable storage medium is configured to store computer program instructions defined by modules, for example, 104, etc., of the TSDS 102.

The processor is configured to execute the defined computer program instructions.

The TSDS 102 comprises a safety criticality estimation model 103, a data reception and processing module 104, a graphical user interface (GUI) 105, and a test space database 106. A user using the user device can access the TSDS 102 via the GUI 105. The GUI 105 is, for example, an online web interface, a web based downloadable application interface, etc.

The data reception and processing module 104 obtains from the autonomous system 101 employable in testing of the AV, the test scenario set Ts comprising test scenarios SbSn and scenario parameters Pl-Pm. The test scenario set is associated with historical real-world scenarios and/or historical simulated scenarios. The test scenario set comprises historical safety reports associated with the historical real-world scenarios and/or historical safety reports associated with the historical simulated scenarios. The safety reports may be in different file formats such as .json format. The safety reports may include information associated with the test scenarios such as a likelihood of safety critical event happening, an average severity of the safety critical event, an aggregate safety score, a number of variations associated with the test scenario, etc. For example, a test scenario Si from a test scenario set Ts for testing an AV refers to a real-world environmental scenario, that the AV has encountered in the past or may encounter once deployed in the real world, such as, a foggy weather test scenario Si. This foggy weather test scenario Si is defined with help of multiple scenario parameters Pl-Pm including, for example, humidity, temperature, pressure, illumination, etc. Thus, each test scenario Si includes a plurality of scenario parameters Pl-Pm. The data reception and processing module 104 stores the test scenario set Ts in the test space database 106.

The data reception and processing module 104 determines, by employing the safety criticality estimation model 103, an impact on safety associated with the AV corresponding to a value of each of the scenario parameters Pl-Pm in the test scenario set Ts. The impact on safety associated with the AV corresponding to the value of each of the scenario parameters Pl-Pm comprises a safety score. That is, an output of the safety criticality estimation model 103 is a safety score.

The safety criticality estimation model 103 is a data driven model trained to estimate criticality of safety associated with each value of each scenario parameter (Pl-Pm). The safety criticality estimation model 103 may reside inside the TSDS 102. The safety criticality estimation model 103 may reside outside the TSDS 102, for example, on a central server (not shown) accessible over a communication network by the data reception and processing module 104.

The data reception and processing module 104 trains the safety criticality estimation model 103 by generating a semantic graph corresponding to each of the test scenarios Sl-Sn of the test scenario set Ts. The semantic graph comprises multiple nodes connected therebetween via arrows. A node represents the AV, an event associated with the AV, or one of the scenario parameters Pl- Pm. The arrow represents a relationship between the scenario parameters Pl- Pm, a relationship between the AV and the scenario parameter(s) Pl-Pm, or a relationship between the event associated with the AV and the scenario parameter(s) Pl-Pm. For example, in a case where scenario parameters P1-P3 may occur together to cause a safety critical event, the nodes corresponding to these three scenario parameters P1-P3 are connected to a node representing a safety critical event.

The data reception and processing module 104 feeds the semantic graph to the safety criticality estimation model 103, that is an untrained safety criticality estimation model 103, for generating predictions for the impact on safety criticality of the AV under each of the test scenarios Sl-Sn.

The data reception and processing module 104 tunes the predictions for the impact by comparing the predictions with safety reports associated with the historical real-world scenarios and/or the historical simulated scenarios from the test scenario set Ts.

The data reception and processing module 104 employs a data driven machine learning technique, for example, in a supervised learning setting for training the safety criticality estimation model 103 by framing the problem of determination of impact on safety associated with the AV corresponding to the value of each of the scenario parameters Pl-Pm as a supervised learning problem. Given the historical real-world scenarios and/or the historical simulated scenarios stored in the test space database 106, the safety criticality estimation model 103 learns an estimation mapping between combinations of two or more scenario parameters Pl-Pm and safety criticality of test scenarios Sl-Sn generated based on the scenario parameters Pl-Pm.

The data reception and processing module 104 also employs a generative artificial intelligence technique such as a generative adversarial network (GAN) to learn underlying distributions of test scenarios Sl-Sn to generate realistic synthetic scenarios of similar safety criticality as observed in the historical real- world scenarios and/or the historical simulated scenarios stored in the test space database 106. For example, in a GAN setting, two sub-models including a generator safety criticality estimation sub -model trained to generate new examples and a discriminator safety criticality estimation sub -model trained to classify the generated examples as real or fake using various metrics such as closeness to statistics of real world dataset, values of safety criticality given the values of generated scenario parameters Pl-Pm, etc., are employed. As used herein, the term “examples” refers to the combinations of values of scenario parameters Pl-Pm associated with a particular test scenario Sl-Sn. The data reception and processing module 104 trains the aforementioned sub-models together until the discriminator safety criticality estimation sub-model fails to identify the fakeness of the generated examples, that is, when the generator safety criticality estimation sub-model starts generating plausible examples.

The data reception and processing module 104, while determining the impact using the trained safety criticality estimation model 103, varies the value of each of the scenario parameters Pl-Pm between a lower range value and a higher range value assumable by each of the scenario parameters Pl-Pm. For example, the range of humidity in air would vary from 0% to 100%.

The data reception and processing module 104 clusters the scenario parameters Pl-Pm based on the impact and a predetermined impact variation threshold. The data reception and processing module 104 determines a variation in the impact by employing sensitivity analysis and/or causality analysis across the scenario parameters Pl-Pm and aggregates the values of the scneario parameters Pl-Pm having minimal variation therebetween based on the predetermined impact variation threshold.

The sensitivity analysis and/or the causality analysis help in determining a level of impact of each individual scenario parameter Pl-Pm and in identifying the value range of the scenario parameter Pl-Pm that results in same level of significance of impact on safety criticality. Aggregating the values of the scenario parameters reduces the value range of the scenario parameter Pl-Pm, that is, discretizes the range to a coarser bin which ultimately results in reduction of test space associated with testing the AV.

The predetermined impact variation threshold is defined based on a minimum variation in the impact corresponding to each of the scenario parameters Pl-Pm. For example, by defining a boundary condition of +/- 5% variation as an absolute change in the end outcome. Alternatively, based on a relative change between the impacts corresponding to two or more of the scenario parameters Pl-Pm and by selecting least count of these relative changes as the predetermined impact variation threshold.

For example, the safety score that is generated as an output by the trained safety criticality estimation model 103 corresponding to a test scenario Sl-Sn is broken down into individual safety scores pertaining to each of the scenario parameters Pl-Pm of the test scenario Sl-Sn to assess their individual impact on the net safety score pertaining to a test scenario Sl-Sn. For example, humidity as a scenario parameter may hold the highest impact when compared to other scenario parameters such as illumination and traffic density.

The data reception and processing module 104 generates based on the clustered scenario parameters Pl-Pm, a focused test scenario set Tsf. The focused test scenario set Tsf comprises focused test scenarios Sfl-Sfn. Each focused test scenario Sfl-Sfn comprises focused scenario parameters Pfl-Pfm. The data reception and processing module 104 generates the focused test scenario set Tsf by selecting sample values from the aggregated values of the scenario parameters Pl-Pm to form the focused scenario parameters Pfl-Pfm of the focused test scenarios Sfl-Sfn of the focused test scenario set Tsf. The data reception and processing module 104 performs this selection based on, for example, a simple average from the aggregated values of the scenario parameters Pl-Pm, a value that is most likely to occur given the test scenario and/or the AV, etc.

The data reception and processing module 104 transfers the focused test scenario set Tsf to the autonomous system 101 for testing of the AV. FIG 2 is a block diagram illustrating an architecture of a computer system 200 employed by the test scenario discretization system 102, hereinafter referred to as TSDS 102, shown in FIG 1, for discretizing a test scenario set Ts employable in testing of an autonomous vehicle (AV), according to an embodiment of the present disclosure.

The TSDS 102 employs the architecture of the computer system 200. The computer system 200 is programmable using a high-level computer programming language. The computer system 200 may be implemented using programmed and purposeful hardware. The computer system 200 comprises a processor 201, a non-transitory computer readable storage medium such as a memory unit 202 for storing programs and data, an input/output (I/O) controller 203, a network interface 204, a data bus 205, a display unit 206, input devices 207, a fixed media drive 208 such as a hard drive, a removable media drive 209 for receiving removable media, output devices 210, etc.

The processor 201 refers to any one of microprocessors, central processing unit (CPU) devices, finite state machines, microcontrollers, digital signal processors, an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), etc., or any combination thereof, capable of executing computer programs or a series of commands, instructions, or state transitions. The processor 201 may also be implemented as a processor set comprising, for example, a general-purpose microprocessor and a math or graphics co-processor. The TSDS 102 disclosed herein is not limited to a computer system 200 employing a processor 201. The computer system 200 may also employ a controller or a microcontroller. The processor 201 executes the modules, for example, 104, etc., of the TSDS 102.

The memory unit 202 is used for storing programs, applications, and data. For example, the modules 104, 105, etc., of the TSDS 102 are stored in the memory unit 202 of the computer system 200. The memory unit 202 is, for example, a random-access memory (RAM) or another type of dynamic storage device that stores information and instructions for execution by the processor 201. The memory unit 202 also stores temporary variables and other intermediate information used during execution of the instructions by the processor 201. The computer system 200 further comprises a read only memory (ROM) or another type of static storage device that stores static information and instructions for the processor 201. The I/O controller 203 controls input actions and output actions performed by the TSDS 102.

The network interface 204 enables connection of the computer system 200 to the communication network 102. For example, the TSDS 102 connects to the communication network 102 via the network interface 204. In an embodiment, the network interface 204 is provided as an interface card also referred to as a line card. The network interface 204 comprises, for example, interfaces using serial protocols, interfaces using parallel protocols, and Ethernet communication interfaces, interfaces based on wireless communications technology such as satellite technology, radio frequency (RF) technology, near field communication, etc. The data bus 205 permits communications between the modules, for example, 104, 105, etc., of TSDS 102.

The display unit 206, via the graphical user interface (GUI) 105, displays information such as the test scenario set Ts, the focused test scenario set Tsf, etc. The display unit 206, via the GUI 105, also displays information such as user interface elements including text fields, buttons, windows, etc., for allowing a user to provide his/her inputs, if any. The display unit 206 comprises, for example, a liquid crystal display, a plasma display, an organic light emitting diode (OLED) based display, etc. The input devices 207 are used for inputting data into the computer system 200. The input devices 207 are, for example, a keyboard such as an alphanumeric keyboard, a touch sensitive display device, and/or any device capable of sensing a tactile input.

Computer applications and programs are used for operating the computer system 200. The programs are loaded onto the fixed media drive 208 and into the memory unit 202 of the computer system 200 via the removable media drive 209. In an embodiment, the computer applications and programs may be loaded directly via the communication network 102. Computer applications and programs are executed by double clicking a related icon displayed on the display unit 206 using one of the input devices 207. The output devices 210 output the results of operations performed by the TSDS 102. For example, the TSDS 102 provides graphical representation of clustered scenario parameters Pl-Pm based on their impact on safety criticality.

The processor 201 executes an operating system. The computer system 200 employs the operating system for performing multiple tasks. The operating system is responsible for management and coordination of activities and sharing of resources of the computer system 200. The operating system further manages security of the computer system 200, peripheral devices connected to the computer system 200, and network connections. The operating system employed on the computer system 200 recognizes, for example, inputs provided by the users using one of the input devices 207, the output display, files, and directories stored locally on the fixed media drive 208. The operating system on the computer system 200 executes different programs using the processor 201. The processor 201 and the operating system together define a computer platform for which application programs in high level programming languages are written.

The processor 201 of the computer system 200 employed by the TSDS 102 retrieves instructions defined by the modules 104, etc., of the TSDS 102 for performing respective functions disclosed in the detailed description of FIG 1. The processor 201 retrieves instructions for executing the modules, for example, 104, etc., of the TSDS 102 from the memory unit 202. A program counter determines the location of the instructions in the memory unit 202. The program counter stores a number that identifies the current position in the program of each of the modules, for example, 104, etc., of the TSDS 102. The instructions fetched by the processor 201 from the memory unit 202 after being processed are decoded. The instructions are stored in an instruction register in the processor 201. After processing and decoding, the processor 201 executes the instructions, thereby performing one or more processes defined by those instructions.

At the time of execution, the instructions stored in the instruction register are examined to determine the operations to be performed. The processor 201 then performs the specified operations. The operations comprise arithmetic operations and logic operations. The operating system performs multiple routines for performing several tasks required to assign the input devices 207, the output devices 210, and memory for execution of the modules, for example, 104, etc., of the TSDS 102. The tasks performed by the operating system comprise, for example, assigning memory to the modules, for example, 103, 104, etc., of the TSDS 102, and to data used by the TSDS 102, moving data between the memory unit 202 and disk units, and handling input/output operations. The operating system performs the tasks on request by the operations and after performing the tasks, the operating system transfers the execution control back to the processor 201. The processor 201 continues the execution to obtain one or more outputs. The outputs of the execution of the modules, for example, 104, etc., of the TSDS 102 are displayed to the user on the GUI 105.

For purposes of illustration, the detailed description refers to the TSDS 102 being run locally on the computer system 200, however the scope of the present invention is not limited to the TSDS 102 being run locally on the computer system 200 via the operating system and the processor 201, but may be extended to run remotely over the communication network (not shown) by employing a web browser and a remote server, a mobile phone, or other electronic devices. One or more portions of the computer system 200 may be distributed across one or more computer systems (not shown) coupled to the communication network.

Disclosed herein is also a computer program product comprising a non-transitory computer readable storage medium that stores computer program codes comprising instructions executable by at least one processor 201 for discretizing a test scenario set Ts employable in testing of an AV, as disclosed in aforementioned description.

The computer program product comprises a first computer program code for obtaining, from an autonomous system 101 shown in FIG 1, employable in testing of the AV, the test scenario set Ts comprising one or more test scenarios Sl-Sn, wherein each test scenario Sl-Sn comprises a plurality of scenario parameters Pl-Pm affecting the AV when deployed in a real world environment; a second computer program code for determining, by employing a safety criticality estimation model 103, an impact on safety associated with the AV corresponding to a value of each of the scenario parameters Pl-Pm, by varying the value of each of the scenario parameters Pl-Pm between a lower range value and a higher range value assumable by each of the scenario parameters Pl-Pm, wherein the safety criticality estimation model 103 is trained to estimate criticality of safety associated with each value of each of the scenario parameters Pl -Pm; a third computer program code for clustering the scenario parameters Pl- Pm based on the impact and a predetermined impact variation threshold; and a fourth computer program code for generating, based on the clustered scenario parameters Pl-Pm, a focused test scenario set Tsf comprising focused test scenarios Sfl-Sfn, wherein each focused test scenario Sfl-Sfn comprises focused scenario parameters Pfl-Pfm, and wherein the focused test scenario set Tsf is transferrable to the autonomous system 101 for testing of the AV.

The computer program product comprises a fifth computer program code for training the safety criticality estimation model 103. The fifth computer program code comprises a sixth computer program code for generating a semantic graph corresponding to each of the test scenarios Sl-Sn of the test scenario set Ts, wherein the semantic graph comprises a plurality of nodes connected therebetween via arrows; a seventh computer program code for feeding the semantic graph to the safety criticality estimation model 103 for generating predictions for the impact on safety criticality of the AV under each of the test scenarios Sl-Sn; and an eight computer program code for tuning the predictions for the impact by comparing the predictions with safety reports associated with one or more of historical real- world scenarios and historical simulated scenarios from the test scenario set Ts.

The fourth computer program code comprises an eleventh computer program code for selecting sample values from the aggregated values of the scenario parameters Pl-Pm to form the focused scenario parameters Pfl-Pfm of the focused test scenarios Sfl-Sfn of the focused test scenario set Tsf. In an embodiment, a single piece of computer program code comprising computer executable instructions, performs one or more steps of the computer implemented method according to the present disclosure, for discretizing a test scenario set Ts employable in testing of an AV. The computer program codes comprising computer executable instructions are embodied on the non-transitory computer readable storage medium. The processor 201 of the computer system 200 retrieves these computer executable instructions and executes them. When the computer executable instructions are executed by the processor 201, the computer executable instructions cause the processor 201 to perform the steps of the computer implemented method for discretizing a test scenario set Ts employable in testing of an AV.

FIGS 3A-3B illustrate process flowcharts of a computer implemented method 300 for discretizing a test scenario set Ts employable in testing of an autonomous vehicle (AV), according to an embodiment of the present disclosure. The computer implemented method 300 disclosed herein employs the test scenario discretization system 102, hereinafter referred to as TSDS 102, shown in FIG 1 for discretizing the test scenario set Ts employable in testing of the AV.

As shown in FIG 3A, at step 301, the computer implemented method 300 obtains from an autonomous system 101 shown in FIG 1, employable in testing of the AV, the test scenario set Ts comprising one or more test scenarios Sl-Sn, wherein each test scenario Sl-Sn comprises a plurality of scenario parameters Pl-Pm affecting the AV when deployed in a real world environment.

At step 301 A, the computer implemented method 300 establishes communication between the autonomous system 101 and the data reception and processing module 104 of the TSDS 102, for example, via a communication network (not shown).

Upon successfully establishing the communication therebetween, at step 30 IB, the computer implemented method 300 receives the test scenario set Ts which is associated with historical real-world scenarios and/or historical simulated scenarios and comprises historical safety reports associated with the historical

SUBSTITUTE SHEET (RULE 26) real-world scenarios and/or historical safety reports associated with the historical simulated scenarios.

Upon receiving the test scenario set Ts, the computer implemented method 300, at step 301C, stores the test scenario set Ts in a test space database 106 of the TSDS 102.

At step 302, the computer implemented method 300 determines, by employing a safety criticality estimation model 103 of the TSDS 102 shown in FIG 1, an impact on safety associated with the AV corresponding to a value of each of the scenario parameters Pl-Pm, by varying the value of each of the scenario parameters Pl-Pm between a lower range value and a higher range value assumable by each of the scenario parameters Pl-Pm. The impact on safety associated with the AV corresponding to the value of each of the scenario parameters Pl-Pm comprises a safety score.

At step 302A, the computer implemented method 300 determines data to be fed to the trained safety criticality estimation model 103, that is, per scenario parameter Pl-Pm, a value varied between the lower range of the scenario parameter Pl-Pm and the higher range of the scenario parameter Pl-Pm.

At step 302B, the computer implemented method 300 selects an interval at which the value is varied while feeding the data to the safety criticality estimation model 103. This interval selection maybe defined based on an overall range of the scenario parameter Pl-Pm and/or a likelihood of the scenario parameter Pl-Pm affecting safety associated with the AV. For example, if humidity and illumination are two of the scenario parameters Pl-Pm that do not have a wide overall range between their respective lower and higher range values and are known to have a greater likelihood of impacting safety of the AV especially for a foggy environmental test scenario Sl'Sn, then the interval with which their respective values are varied maybe smaller so as not to miss out on critical values.

At step 302C, the computer implemented method 300 feeds the values of each of the scenario parameters Pl-Pm selected at the respective intervals to the trained safety criticality estimation model 103. Alternatively, at step 302C the computer implemented method 300 feeds the values of a few selected scenario parameters Pl-Pm that are most hkely to have an impact on the safety associated with the AV. The selected scenarios maybe determined based on historical data from the safety reports from the test scenario set Ts. Alternatively, the selected scenarios maybe taken as a user input.

At step 302D, the computer implemented method 300 receives from the trained safety criticality estimation model 103 the impact on safety associated with the AV corresponding to a value of each of the scenario parameters Pl-Pm. For example, the safety score associated with each value of each scenario parameter Pl-Pm that is fed to the safety criticality estimation model 103.

At step 302E, the computer implemented method 300 stores the impact, that is, the safety scores in the test space database 106.

At step 303, the computer implemented method 300 clusters the scenario parameters Pl-Pm based on the impact and a predetermined impact variation threshold.

At step 303A, the computer implemented method 300 determines a variation in the impact by employing sensitivity analysis and/or causality analysis across the scenario parameters Pl-Pm. For example, by determining leading and lagging safety scores for each scenario parameter Pl-Pm by using analyses such as sensitivity and counterfactuabbased causality analyses to establish specific scenario parameters Pl-Pm and/or combinations of values of the scenario parameters Pl-Pm as safety-critical for the AV and rule out the correlation -is- not-causation, type of cases in the counterfactual causality analysis, that is, those cases that did not lead to a legitimate deduction of cause and effect relationship between values of the scenario parameter(s) Pl-Pm. The sensitivity and/or causality analysis is also useful for quantifying how avoidable some of the scenario parameters Pl-Pm are, for example, analogous to a stable, an unstable and a neutral equilibrium of potential energy typically referred to in such analyses. At step 303B, the computer implemented method 300 aggregates the values of the scenario parameters Pl-Pm having minimal variation therebetween based on the predetermined impact variation threshold. The predetermined impact variation threshold is defined based on a minimum variation in the impact corresponding to each of the scenario parameters Pl-Pm. For example, based on an absolute change in the safety scores for each scenario parameter Pl-Pm or based on relative changes in the safety scores for each scenario parameter Pl-Pm such that the least change is taken as the predetermined impact variation threshold from the relative changes.

At step 303C, the computer implemented method 300 stores the aggregated values of the scenario parameters Pl-Pm in the test space database 106.

At step 304, the computer implemented method 300 generates, based on the clustered scenario parameters Pl-Pm, a focused test scenario set Tsf comprising focused test scenarios Sfl-Sfn, wherein each focused test scenario Sfl-Sfn comprises focused scenario parameters Pfl-Pfm, and wherein the focused test scenario set Tsf is transferrable to the autonomous system 101 for testing of the AV.

At step 304A, the computer implemented method 300 selects sample values from the aggregated values of the scenario parameters Pl-Pm to form the focused scenario parameters Pfl-Pfm of the focused test scenarios Sfl-Sfn of the focused test scenario set Tsf. The sample values maybe selected by simple mean from the aggregated values. The sample values maybe selected by employing statistics to choose the value most suited to maximize coverage or the value most representational of the aggregated values, such as, most likely to occur value from the aggregated values based on historical and/or simulated scenarios.

At step 304B, the computer implemented method 300 stores the focused scenario set Tsf in the test space database 106.

At step 304C, the computer implemented method 300 transfers the focused scenario set Tsf to the autonomous system 101 thus, having reduced the test space. The computer implemented method 300, at step 302’ trains the safety criticality estimation model 103, that is, an untrained safety criticality estimation model 103 to estimate criticahty of safety associated with each value of each of the scenario parameters Pl-Pm.

As shown in FIG 3B, at step 302’A, the computer implemented method 300 generates a semantic graph corresponding to each of the test scenarios Sl-Sn of the test scenario set Ts. The semantic graph comprises a plurality of nodes connected therebetween via arrows. A node represents the AV, an event associated with the AV, or one of the scenario parameters Pl-Pm. An arrow represents a relationship between the scenario parameters Pl-Pm , a relationship between the AV and the scenario parameter(s) Pl-Pm, or a relationship between the event associated with the AV and the scenario parameter(s) Pl-Pm.

At step 302’B, the computer implemented method 300 feeds the semantic graph to the safety criticality estimation model 103 for generating predictions for the impact on safety criticality of the AV under each of the test scenarios Sl-Sn.

At step 302’C, the computer implemented method 300 tunes the predictions for the impact by comparing the predictions with safety reports associated with the historical real-world scenarios and/or the historical simulated scenarios from the test scenario set Ts.

The computer implemented method 300 employs data driven techniques such as generative adversarial networks (GAN), variational autoencoders, and/or graph data science algorithms to analyse various scenarios Sl-Sn from the historical real-world scenarios and/or the historical simulated scenarios, with respect to the scenario parameters Pl-Pm, for example, driving behaviour, etc., to train the safety criticality estimation model 103 to predict a safety score corresponding to each value of each scenario parameter Pl-Pm by generating joint distributions of relationships between scenario parameters Pl-Pm and/or between a safety event associated with the AV and the scenario parameters Pl-Pm. FIGS 4A-4B illustrate exemplary semantic graphs 400A, 400B corresponding to a test scenario Ts associated with one or more autonomous vehicles (AVs), according to an embodiment of the present disclosure. The computer implemented method 300 as shown in FIG 3B, generates a semantic graph 400A, 400B corresponding to each of the test scenarios Sl-Sn of the test scenario set Ts. The computer implemented method 300, generates the semantic graphs 400A, 400B by capturing relationships between scenario parameters Pl-Pm of each test scenario Sl-Sn and safety criticality associated therewith using historical real- world scenarios and/or the historical simulated scenarios.

The semantic graphs 400A, 400B describe the test scenario Sl-Sn semantically as shown in FIG 4A and in FIG 4B wherein different types of nodes 401 represent the AV shown as vehicle’, the safety event associated with the AV shown as ‘crash event’ or ‘near crash event’, or a scenario parameter Pl-Pm shown as ‘junction’, ‘road’, ‘lane’, ‘scene environment’, etc. similarly, the arrows 402 as shown in FIG 4A describe the relationships between the different nodes 401. For example, a vehicle ‘turns at’ a junction, a vehicle ‘travels in’ a lane, etc. Thus, a vehicle’s journey and trajectory through a geolocation is represented by modelling the vehicle as a node 401 and mapping variables associated therewith as other nodes 401 and the driving behavior including, for example, an average speed, a turning angle, a trajectory curvature, etc., therebetween as arrows 402. The nodes 401 are not orthogonal as they do affect each other, for example, road friction decreases in a foggy, rainy or snowy weather due to precipitation.

As shown in FIG 4B, the semantic graph 400B represents safety -critic al events such as near-crashes and crashes as nodes 401 and connects them to the nodes 401 representing scenario parameters Pl-Pm such as scene environment based on combinations of values of the scenario parameters Pl-Pm in which they occurred, for example, ‘junction’, ‘road’ and ‘scene environment’ have occurred in combination with a ‘crash event’ and a ‘near crash event’ thus, they are connected to one another. Thus, a vehicle’s journey through a geolocation under various scenario conditions and various events happening in the geolocation under those scenario conditions is represented by the semantic graph 400B.

These semantic graphs 400A, 400B are stored in the test space database 106. FIGS 5A-5B illustrate graphical representations of clustering and generating a focused test scenario Sfl having focused scenario parameters Pfl-Pf2, from a test scenario Si having scenario parameters Pl and P2, associated with one or more autonomous vehicles (AVs), according to an embodiment of the present disclosure.

The safety criticality estimation model 103 shown in FIG 1 generates an impact, that is, a safety score when values of these scenario parameters Pl and P2 are fed thereto. Based on which, the data reception and processing module 104 clusters the scenario parameters Pl and P2, that is, determines a variation in the impact by employing sensitivity analysis and/or causality analysis across the scenario parameters Pl and P2 and aggregates the values of the scneario parameters Pl and P2 such that there exists minimal variation therebetween based on the predetermined impact variation threshold.

As shown in FIG 5A, the test scenario Si defines a test space 500A created by the two scenario parameters Pl and P2 when considered in combination and when each of which are varied between their respective lower values and higher values. The bins 501-509 of the test space 500 A represent the clustered scenario parameters Pl and P2 when considered in combination.

The data reception and processing module 104 selects sample values from aggregated values, that is, the bins 501-509 of the scenario parameters Pl and P2 to generate the focused scenario parameters Pfl and Pf2 and thereby the focused test scenario Sfl, and therefore a focused test scenario set Tsf.

As shown in FIG 5B, the focused test scenario Sfl defines a discretized test space 500B comprising focused scenario parameters Pfl and Pf2 when considered in combination. The discretized test space 500B reduced as a result of selecting sample values from the bins 501-509. For example, as shown in FIG 5A, the bin 501 comprises about 16 values whereas in the FIG 5B, the bin 501 comprises a single value. Thus, the test space 500A defined by the test scenario Si is discretized to the test space 500B defined by the focused test scenario Sfl. This is achieved by varying the values of scenario parameters Pl and P2 separately while keeping the other scenario parameters P3-Pm constant and observing the effect of variation on the safety criticality, as shown by different bins 501-509. Thus, the test space 500A is effectively reduced to only 9 significant safety critical values as compared to 144 different combinations of the scenario parameters Pl and P2, that is, there exist only 9 scenario variations that are significantly different from each other in terms of scenario safety criticality in the discretized test space 500B. A test space thus, can be reduced exponentially as the number of scenario parameters Pl-Pm increases.

Where databases are described such as the test space database 106, it will be understood by one of ordinary skill in the art that (i) alternative database structures to those described may be readily employed, and (ii) other memory structures besides databases may be readily employed. Any illustrations or descriptions of any sample databases disclosed herein are illustrative arrangements for stored representations of information. Any number of other arrangements may be employed besides those suggested by tables illustrated in the drawings or elsewhere. Similarly, any illustrated entries of the databases represent exemplary information only! one of ordinary skill in the art will understand that the number and content of the entries can be different from those disclosed herein. Further, despite any depiction of the databases as tables, other formats including relational databases, object-based models, and/or distributed databases may be used to store and manipulate the data types disclosed herein. Likewise, object methods or behaviors of a database can be used to implement various processes such as those disclosed herein. In addition, the databases may, in a known manner, be stored locally or remotely from a device that accesses data in such a database. In embodiments where there are multiple databases in the system, the databases may be integrated to communicate with each other for enabling simultaneous updates of data linked across the databases, when there are any updates to the data in one of the databases.

The present disclosure can be configured to work in a network environment comprising one or more computers that are in communication with one or more devices via a network. The computers may communicate with the devices directly or indirectly, via a wired medium or a wireless medium such as the Internet, a local area network (LAN), a wide area network (WAN) or the Ethernet, a token ring, or via any appropriate communications mediums or combination of communications mediums. Each of the devices comprises processors, some examples of which are disclosed above, that are adapted to communicate with the computers. In an embodiment, each of the computers is equipped with a network communication device, for example, a network interface card, a modem, or other network connection device suitable for connecting to a network. Each of the computers and the devices executes an operating system, some examples of which are disclosed above. While the operating system may differ depending on the type of computer, the operating system will continue to provide the appropriate communications protocols to establish communication links with the network. Any number and type of machines may be in communication with the computers.

The present disclosure is not limited to a particular computer system platform, processor, operating system, or network. One or more aspects of the present disclosure may be distributed among one or more computer systems, for example, servers configured to provide one or more services to one or more client computers, or to perform a complete task in a distributed system. For example, one or more aspects of the present disclosure may be performed on a client-server system that comprises components distributed among one or more server systems that perform multiple functions according to various embodiments. These components comprise, for example, executable, intermediate, or interpreted code, which communicate over a network using a communication protocol. The present disclosure is not limited to be executable on any particular system or group of systems, and is not limited to any particular distributed architecture, network, or communication protocol.

While the present invention has been described in detail with reference to certain embodiments, it should be appreciated that the present invention is not limited to those embodiments. In view of the present disclosure, many modifications and variations would be present themselves, to those skilled in the art without departing from the scope of the various embodiments of the present invention, as described herein. The scope of the present invention is, therefore, indicated by the following claims rather than by the foregoing description. All changes, modifications, and variations coming within the meaning and range of equivalency of the claims are to be considered within their scope.

List of Reference Numerals

100 system

101 autonomous system

102 test scenario discretization system (TSDS)

103 safety criticality estimation model

104 data reception and processing module

105 graphical user interface (GUI)

106 test space database

200 computer system

201 processor

202 memory unit

203 input/output (I/O) controller

204 network interface

205 data bus

206 display unit

207 input devices

208 fixed media drive

209 removable media drive

210 output devices

400A, 400B semantic graph

401 node

402 arrow

500A test space

500B discretized test space

501-509 bins

Ts test scenario set

Sl-Sn test scenarios

Pl-Pm scenario parameters

Tsf focused test scenario set

Sfl-Sfn focused test scenarios

Pfl-Pfm focused scenario parameters

Claims

1. A computer implemented method (300) for discretizing a test scenario set (Ts) comprising one or more test scenarios (Sl'Sn), wherein each test scenario (Sl'Sn) comprises a plurality of scenario parameters (Pl-Pm) affecting an autonomous vehicle (AV) when deployed in a real world environment, the computer implemented method characterized by: determining, by employing a safety criticality estimation model (103), an impact on safety associated with the autonomous vehicle (AV) corresponding to a value of each of the scenario parameters (Pl-Pm), by varying the value of each of the scenario parameters (Pl-Pm) between a lower range value and a higher range value assumable by each of the scenario parameters (Pl-Pm), wherein the safety criticality estimation model is trained to estimate criticality of safety associated with each value of each of the scenario parameters (Pl-Pm); clustering the scenario parameters (Pl-Pm) based on the impact and a predetermined impact variation threshold; and generating, based on the clustered scenario parameters (Pl-Pm), a focused test scenario set (Tsf) comprising focused test scenarios (Sfl-Sfn), wherein each focused test scenario (Sfl-Sfn) comprises focused scenario parameters (Pfl-Pfm), and wherein the focused test scenario set (Tsf) is transferrable to the autonomous system (ADAS/ADS) for testing of the autonomous vehicle (AV).

2. The computer implemented method (300) according to claim 1, wherein the the test scenario set (Ts) is obtained from an autonomous system (101) employable in testing of the autonomous vehicle (AV).

3. The computer implemented method (300) according to any one of the claims 1 and 2, wherein the test scenario set (Ts) is associated with one or more of historical real-world scenarios and historical simulated scenarios, and wherein the test scenario set (Ts) comprises one or more of historical safety reports associated with the historical real-world scenarios and historical safety reports associated with the historical simulated scenarios.

4. The computer implemented method (300) according to claim 1, further comprising training the safety criticality estimation model (103) by: ■ generating a semantic graph (400A, 400B) corresponding to each of the test scenarios (Sl-Sn) of the test scenario set (Ts), wherein the semantic graph (400A, 400B) comprises a plurality of nodes (401) connected therebetween via arrows (402);

■ feeding the semantic graph (400A, 400B) to the safety criticality estimation model (103) for generating predictions for the impact on safety criticality of the autonomous vehicle (AV) under each of the test scenarios (Sl-Sn); and

■ tuning the predictions for the impact by comparing the predictions with safety reports associated with one or more of historical real-world scenarios and historical simulated scenarios from the test scenario set (Ts).

5. The computer implemented method (300) according to claim 4, wherein a node (401) represents one of the autonomous vehicle (AV), an event associated with the autonomous vehicle (AV), and one of the scenario parameters (Pl-Pm).

6. The computer implemented method (300) according to claim 4, wherein an arrow (402) represents one of: a relationship between the scenario parameters (Pl-Pm); a relationship between the autonomous vehicle (AV) and one or more of the scenario parameters (Pl-Pm); and a relationship between the event associated with the autonomous vehicle (AV) and one or more of the scenario parameters (Pl-Pm).

7. The computer implemented method (300) according to claim 1, wherein the impact on safety associated with the autonomous vehicle (AV) corresponding to the value of each of the scenario parameters (Pl-Pm) comprises a safety score.

8. The computer implemented method (300) according to claim 1, wherein clustering the scenario parameters (Pl-Pm) based on the impact and the predetermined impact variation threshold comprises: determining a variation in the impact by employing one or more of sensitivity analysis and causality analysis across the scenario parameters (Pl- Pm); and aggregating the values of the scenario parameters (Pl-Pm) having minimal variation therebetween based on the predetermined impact variation threshold.

9. The computer implemented method (300) according to any one of the claims 1 and 8, wherein the predetermined impact variation threshold is defined based on a minimum variation in the impact corresponding to each of the scenario parameters (Pl-Pm).

10. The computer implemented method (300) according to claim 1, wherein generating the focused test scenario set (Tsf) comprises selecting sample values from aggregated values of the scenario parameters (Pl-Pm) to form the focused scenario parameters (Pfl-Pfm) of the focused test scenarios (Sfl-Sfn) of the focused test scenario set (Tsf).

11. A test scenario discretization system (102) for discretizing a test scenario set (Ts) employable in testing of an autonomous vehicle (AV), characterized by: a non-transitory computer readable storage medium storing computer program instructions defined by one or more modules (103, 104, 105) of the test scenario discretization system! at least one processor (201) communicatively coupled to the non-transitory computer readable storage medium, wherein the at least one processor (201) is configured to execute the computer program instructions, thereby performing the method according to the claims 1 to 10.

12. A computer-program product having machine-readable instructions stored therein, which when executed by one or more processors (201), cause the processors (201) to perform the method according to the claims 1 to 10.