WO2021063486A1

WO2021063486A1 - Automatic root cause analysis of failures in autonomous vehicle

Info

Publication number: WO2021063486A1
Application number: PCT/EP2019/076528
Authority: WO
Inventors: Roman Talyansky; Pavel Kisilev; Dorin DANIAL; Edmond Chalom; Michael Katz; Xiaoli SHE; Shuyi JIN; Jiawei YU
Original assignee: Huawei Technologies Co., Ltd.
Priority date: 2019-10-01
Filing date: 2019-10-01
Publication date: 2021-04-08
Also published as: CN114008551A

Abstract

Automatically detecting failure root cause in an autonomous vehicle, by receiving sensor data captured during a period preceding the failure by sensor(s) deployed to sense an environment of the autonomous vehicle, analyzing the sensor data to identify object(s) in the environment, creating a failure scenario defining a time-lined motion pattern of each object, computing a feature vector comprising features extracted from an output generated by sub-systems of the autonomous vehicle during the failure scenario, applying to the feature vector machine learning classification model(s) trained with a plurality of labeled feature vectors computed for a plurality of failure scenarios and their corresponding success scenarios, identifying key features significantly contributing to an outcome of the trained machine learning classification model(s) by applying an interpretation model to the machine learning classification model(s), the feature vector(s) and/or the outcome and estimating root cause failure sub-system(s) according to its association with the key feature(s).

Description

AUTOMATIC ROOT CAUSE ANALYSIS OF FAILURES IN AUTONOMOUS VEHICLE

TECHNICAL FIELD

The present invention, in some embodiments thereof, relates to identifying a root cause of failures in autonomous vehicles and, more specifically, but not exclusively, to identifying a root cause of failures in automated sub-systems of autonomous vehicles using machine learning models trained with failure scenarios and corresponding success scenarios.

BACKGROUND

Autonomous vehicles technology, either ground, aerial and/or naval vehicles, is advancing rapidly. As partially autonomous vehicles are already common, the ultimate goal of fully autonomous vehicles seems to be just around the comer.

While autonomous vehicles will have numerous advantages, entrusting full control to the vehicle involves safety concerns which require a responsible, cautious and calculated advancement in this field.

As the autonomous vehicles involve ever increasing numbers of sub-systems for monitoring, controlling and/or planning operation of the autonomous vehicle, there is a likelihood of failures occurring in the sub-systems or in interactions between them.

Therefore, there is a need for techniques that allow identifying such failures and tracing them back to the sub-system(s) which caused them.

SUMMARY

It is an objective of this disclosure to provide a solution which mitigates or solves the drawbacks and problems of conventional solutions.

The above and further objectives are solved by the subject matter of the independent claims. Further advantageous embodiments can be found in the dependent claims.

The disclosure aims at providing a solution for identifying a root cause of failures in autonomous vehicle, in particular identifying root cause sub-systems leading to failures in the autonomous vehicle.

According to a first aspect of the present invention there is provided a system for automatically detecting a root cause of a failure in an autonomous vehicle, comprising one or more processors configured for:

Receiving sensor data captured during a predefined period preceding a failure of an autonomous vehicle by one or more sensors deployed to sense an environment of the autonomous vehicle. A hidden time period immediately preceding the failure is not included in the predefined period.

Analyzing the sensor data to identify one or more objects in the environment.

Creating a failure scenario defining a time-lined motion pattern of one or more of the identified object(s) with respect to the autonomous vehicle.

Computing a feature vector comprising a plurality of features extracted from an output generated by one or more of a plurality of sub-systems of the autonomous vehicle during the failure scenario.

Applying to the feature vector one or more machine learning classification models trained with a plurality of labeled feature vectors computed for a plurality of failure scenarios and their corresponding success scenarios.

Identifying one or more key features having a major contribution to an outcome of one or more of the trained machine learning classification models by applying an interpretation model to the machine learning classification model(s), the feature vector and/or the outcome.

Estimating one or more root cause failure sub-systems of the plurality of sub-systems according to their association with one or more of the key features.

According to a second aspect of the present invention there is provided a method of automatically detecting a root cause of a failure in an autonomous vehicle, comprising:

Analyzing the sensor data to identify one or more objects in the environment.

Automatically identifying the failure root cause may be significantly simpler and easier compared to manual RCA. The automated RCA may require very little time for tracing the failure to the root cause sub-system(s) thus significantly reducing the analysis time. This advantage is dramatically emphasized by the fact that autonomous vehicles may be deployed with an extremely large number of sub-systems interacting with each other and may log extremely large volumes of data during extensive trips while experiencing multiple failures. Manual analysis of such high volume and complex data may be unfeasible. The automated RCA however may support comprehensive and time efficient tracing of the failures to their root cause sub-system(s) irrespective of the large number of such sub-systems, their complex interaction, the extensive trips and/or the number of encountered failures. Using the ML classification model(s) may enable constant evolution by training ML classification model(s) using additional and newly encountered training failure scenarios. Using the success scenarios for training may significantly enhance the accuracy of the RCA as the ML classification model(s) may be highly trained to identify valid and invalid feature combinations. Adjusting the scenarios to remove the hidden time period may prevent the ML classification model(s) from adjusting to identify root cause features during the period immediately preceding the failures which may hide actual root cause features.

An optional implementation form of the first and/or second aspects comprises conducting the RCA based on analyzing segments of the failure scenario, by:

Splitting the failure scenario into a plurality of time intervals based on a velocity of the one or more objects relative to the autonomous vehicle.

Computing a plurality of feature vectors each for a respective one of the plurality of time intervals.

Applying the one or more machine learning classification models to the plurality of feature vectors, the one or more machine learning classification models are trained with a plurality of labeled feature vectors computed for a plurality of time intervals of the failure scenarios and the corresponding success scenarios.

Identifying the one or more key features by applying the interpretation model to the plurality of feature vectors, to the one or more trained machine learning classification models and to the outcome. Estimating one or more root cause failure sub-systems of the plurality of sub-systems according to its association with the one or more key features.

Splitting the failure scenario for the RCA session and splitting the plurality of failure scenarios and their corresponding success scenarios during the training session(s) according into a plurality of time intervals and using per-interval feature vectors which may be significantly limited in size may allow using shallow ML classification models, thus reducing complexity and possible cost of the RCA system.

In a further implementation form of the first and/or second aspects, the failure scenario is split into the plurality of time intervals based on variations in the velocity of the one or more objects such that a new time interval is created upon each velocity change exceeding a predefined threshold. Velocity of the object with respect to the autonomous vehicle may be one of the most critical behavioral attributes of the objects which may lead to the failure, in particular, a collision. Therefore, splitting the scenario according to the velocity of the object may enable improved tracing of the failure to a certain time according to the changes in the velocity.

In a further implementation form of the first and/or second aspects, the failure scenario is split into the plurality of time intervals based on one or more extremum values of the velocity of the one or more objects such that a new time interval is created after reaching the one or more extremum values. The one or more extremum values may include a maximum value and a minimum value. Therefore, splitting the scenario according to the velocity of the object may enable improved tracing of the failure to a certain time according to the changes in the velocity.

In a further implementation form of the first and/or second aspects, the failure scenario is split into the plurality of time intervals based on a polynomial approximation of the velocity of the one or more objects such that a new time interval is created upon deviation of the polynomial approximation outcome from a predefined threshold. Therefore, splitting the scenario according to the velocity of the object may enable improved tracing of the failure to a certain time according to the changes in the velocity.

In a further implementation form of the first and/or second aspects, the time-lined motion pattern includes one or more members of a group consisting of: a distance of the one or more objects from the autonomous vehicle, a direction of the one or more objects with respect to the autonomous vehicle, a velocity of the one or more objects with respect to the autonomous vehicle and an acceleration of the one or more objects with respect to the autonomous vehicle. Accurately mapping the motion pattern of the object with respect to the autonomous vehicle may be essential for tacking the object in correlation to the outputs of the sub-systems in attempt to trace the failure to the root cause sub-system(s). Accurate mapping of the object in time and space may relay on the spatiotemporal attributes of the other object.

In a further implementation form of the first and/or second aspects, each of the plurality of features relates to one or more of the autonomous vehicle and the one or more objects, each of the plurality of features is a member of a group consisting of: a relative distance between the one or more objects and the autonomous vehicle, a relative distance between the one or more objects and one or more another objects detected in the environment, a relative velocity of the one or more objects with respect to the autonomous vehicle, a relative velocity of the one or more objects with respect to the one or more another objects, a duration of detection of the one or more objects, a location of the autonomous vehicle on a road, a maneuvering operation of the autonomous vehicle, a distribution of the one or more objects in spatial regions defined in the environment and an identifier of the one or more sensor whose sensor data is analyzed to identify the one or more objects. Such features may be highly indicative of failures of the autonomous vehicle, for example, an accident and/or a near accident.

In a further implementation form of the first and/or second aspects, similarity between each of the plurality of failure scenarios and its corresponding success scenario is identified by applying one or more Dynamic Time Warping (DTW) algorithms configured to estimate similarity of one or more candidate subsequences identified in the time-lined motion pattern of the one or more objects in each failure scenario and the time-lined motion pattern of the one or more objects in a respective corresponding success scenario. The DTW algorithm may be highly efficient to identify similarity between sequences.

In an optional implementation form of the first and/or second aspects, one or more multi-dimensional DTW algorithms are applied to estimate the similarity between each failure scenario and its corresponding success in case multiple objects are detected in the respective failure scenario such that similarity is estimated between the time-lined motion pattern of each of the multitude of objects in each failure scenario and in the respective corresponding success scenario. Multi-dimensional DTW algorithms may be highly efficient to identify similarity between sequences having multiple dimensions, which in this case may result from the plurality of objects detected in the environment of the autonomous vehicle.

In an optional implementation form of the first and/or second aspects, one or more success scenarios are discarded in case a relative distance of the one or more objects from the autonomous vehicle in a certain failure scenario exceeds a certain threshold compared to the relative distance of the one or more objects from the autonomous vehicle in the respective corresponding success scenario. The threshold may be predefined to set the similarity level required between the candidate success scenario and its respective failure scenario. Exceeding the threshold may indicate that the success scenario is not sufficiently similar.

In an optional implementation form of the first and/or second aspects, one or more success scenarios are discarded based on a trajectory analysis which predicts that an object other than the one or more objects is estimated to collide with the autonomous vehicle. While a success scenario candidate defining the motion pattern of a certain object may seem to be successful (no failure) within a certain time sequence (window), it is possible that following the certain time sequence, the object collides with the autonomous vehicle. Ensuring that the object does not collide with the autonomous vehicle ensures the candidate scenario is indeed successful.

In an optional implementation form of the first and/or second aspects, one or more success scenarios which end with a failure are discarded. This is related to the insight that candidate scenarios which end with a failure, even a failure different than the failure of the corresponding failure scenario, may not serve as a valuable success scenario.

In a further implementation form of the first and/or second aspects, each of the plurality of labeled feature vectors is labeled with a label “FAILURE” or a label “SUCCESS” according to its respective failure scenario or success scenario. Labeling the feature vectors is applied to support supervised and/or semi-supervised training of the ML classification model(s).

In a further implementation form of the first and/or second aspects, the interpretation model is adapted to determine the contribution of at least some of the plurality of features to the outcome by manipulating one or more input feature vectors and identifying a perturbation in the outcome. Manipulating the (input) feature vector(s) injected to the ML classification model(s) and applying the interpretation model to analyze perturbation in the outcome with respect to the manipulation may enable efficient and/or accurate identification of the key feature(S) and hence of the root cause sub-system(s) associated with the key feature(s).

Other systems, methods, features, and advantages of the present disclosure will be or become apparent to one with skill in the art upon examination of the following drawings and detailed description. It is intended that all such additional systems, methods, features, and advantages be included within this description, be within the scope of the present disclosure, and be protected by the accompanying claims.

Unless otherwise defined, all technical and/or scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which the invention pertains. Although methods and materials similar or equivalent to those described herein can be used in the practice or testing of embodiments of the invention, exemplary methods and/or materials are described below. In case of conflict, the patent specification, including definitions, will control. In addition, the materials, methods, and examples are illustrative only and are not intended to be necessarily limiting.

Implementation of the method and/or system of embodiments of the invention can involve performing or completing selected tasks manually, automatically, or a combination thereof. Moreover, according to actual instrumentation and equipment of embodiments of the method and/or system of the invention, several selected tasks could be implemented by hardware, by software or by firmware or by a combination thereof using an operating system.

For example, hardware for performing selected tasks according to embodiments of the invention could be implemented as a chip or a circuit. As software, selected tasks according to embodiments of the invention could be implemented as a plurality of software instructions being executed by a computer using any suitable operating system. In an exemplary embodiment of the invention, one or more tasks according to exemplary embodiments of method and/or system as described herein are performed by a data processor, such as a computing platform for executing a plurality of instructions. Optionally, the data processor includes a volatile memory for storing instructions and/or data and/or a non-volatile storage, for example, a magnetic hard disk and/or removable media, for storing instructions and/or data. Optionally, a network connection is provided as well. A display and/or a user input device such as a keyboard or mouse are optionally provided as well.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

Some embodiments of the invention are herein described, by way of example only, with reference to the accompanying drawings. With specific reference now to the drawings in detail, it is stressed that the particulars shown are by way of example and for purposes of illustrative discussion of embodiments of the invention. In this regard, the description taken with the drawings makes apparent to those skilled in the art how embodiments of the invention may be practiced.

In the drawings:

FIG. 1 is a flowchart of an exemplary process of estimating a root cause of a failure in an autonomous vehicle using trained machine learning classification models, according to some embodiments of the present invention;

FIG. 2 is a schematic illustration of an exemplary system for estimating a root cause of a failure in an autonomous vehicle, according to some embodiments of the present invention; FIG. 3 is a schematic illustration of an exemplary system for training machine learning classification models for estimating a root cause of a failure in an autonomous vehicle, according to some embodiments of the present invention;

FIG. 4 is a flowchart of an exemplary process of training machine learning classification models for estimating a root cause of a failure in an autonomous vehicle, according to some embodiments of the present invention;

FIG. 5 is a schematic illustration of an exemplary time-lined sequence in which candidate scenarios are searched for a query scenario, according to some embodiments of the present invention;

FIG. 6 presents charts of segments of exemplary time-lined motion patterns smoothed using filters, according to some embodiments of the present invention;

FIG. 7 is a schematic illustration of an exemplary autonomous vehicle planning and control system receiving output of one or more sub-systems of the autonomous vehicle;

FIG. 8 is a chart of an exemplary time-lined motion pattern of a scenario split into a plurality of time intervals, according to some embodiments of the present invention;

FIG. 9 is a schematic illustration of tracing an autonomous vehicle failure resulting from a planning and control system to sub-system(s) of the autonomous vehicle, according to some embodiments of the present invention; and

FIG. 10 presents an exemplary flow for training machine learning models to map autonomous vehicle failures to root cause failure sub-systems and an exemplary flow for detecting a root cause of the autonomous vehicle using the trained ML models, according to some embodiments of the present invention.

DETAILED DESCRIPTION

According to some embodiments of the present invention, there are provided methods, systems and computer program products for identifying a root cause of a failure, for example, an accident, a near accident, a malfunction and/or the like in an autonomous vehicle and/or an at least partially autonomous vehicle, for example, a ground vehicle (e.g. car, truck, train, etc.), an aerial vehicle (e.g. drone, place, helicopter, etc.), a naval vehicle (e.g. boat, ship, submarine) and/or the like. In particular, the root cause of the failure may be traced to one or more of a plurality of sub-systems deployed in the vehicle to monitor, plan and/or control operation of the vehicle.

A Root Cause Analysis (RCA) system may obtain sensor data captured by one or more sensors deployed in the vehicle to sense an environment of the vehicle, in particular, sensor data captured by the sensor(s) during a predefined time period prior to (preceding) a time of the failure. The sensors may include, for example, imaging sensors (e.g. camera, infrared sensor, thermal imaging sensor, night vision sensor, etc.), ranging sensors (e.g. Light Detection and Ranging (LiDAR), RAdio Detection and Ranging (RADAR), Sound Navigation and Ranging (SONAR), etc.), environmental condition sensors (e.g. temperature sensor, humidity sensor, illumination (light) sensor, etc.) and/or the like. As such the sensor data may include, for example, images, video streams comprising sequences of images, ranging maps, thermal maps and/or the like according to the sensing type and/or technology of the sensor(s).

The RCA system may analyze the sensor data to identify one or more objects in the environment of the vehicle during a trip, condition and/or state of the vehicle prior to the failure. The RCA system may use one or more methods, tools and/or algorithms, for example, image processing, computer vision, signal processing and/or the like according to the type of the sensor data.

Based on analysis of the sensor data, the RCA system may create a failure scenario expressing a time-lined motion pattern of one or more of the objects detected in the environment of the vehicle prior to the time of the failure. As such the time-lined motion pattern of each detected object defined by the failure scenario describes a spatiotemporal (space and time) relation between the respective object and the vehicle, in particular in terms of distance, velocity (speed), acceleration and/or the like.

The RCA system may further obtain output data generated during the trip, condition and/or state of the vehicle prior to the time of the failure by one or more of the sub-systems deployed in the vehicle. The RCA system may compute one or more feature vectors (feature sets) each comprising features extracted from the output of the sub-system(s). The features may include, for example, a relative distance between the respective object and the vehicle, a relative distance between the respective object and one or more other objects detected in the environment, a relative velocity of the respective object with respect to the vehicle, a relative velocity of the respective object with respect to the other detected object(s), a duration of detection of the respective object, a location of the vehicle on a road, a maneuvering operation of the vehicle, a distribution of the respective object in spatial regions defined in the environment, an identifier of one or more of the sensor(s) whose sensor data is analyzed to identify the respective object and/or the like.

The features, in particular features combinations may be highly indicative of various defects, malfunctions and/or errors exhibited by one or more of the sub-systems deployed in the vehicle. As the features are extracted from the output of the output of the sub-system(s), each of the features may be directly associated with one or more of the sub-systems deployed in the vehicle.

The RCA system may further split the time-lined motion pattern of one or more of the detected objects into a plurality of time intervals each spanning a limited time period. The RCA system may split the time-lined motion pattem(s) into the plurality of time intervals according to one or more motion attributes of the respective vehicle, for example, velocity and/or the like. As such the RCA system may create a new time interval upon one or more velocity conditions, for example, a significant variation in the velocity, an extremum velocity value (maximum, minimum) and/or the like. In such case, the RCA system may compute one or more feature vectors for each of the plurality of time intervals constituting the respective time-lined motion pattern of one or more of the detected objects.

The feature vector(s) computed for the failure scenario may thus include one or more per-interval feature vectors computed for one or more of the time intervals constituting the time-lined motion pattern of one or more of the detected objects as well as scenario-wide features extending over several time intervals and potentially over the entire failure scenario.

The RCA system may then apply one or more trained Machine Learning (ML) classification models to the feature vector(s) created for the failure scenario. The ML classification model(s) for example, a neural network, a Support Vector Machine (SVM) and/or the like may be trained in advance with a plurality of labeled feature vectors created for a plurality of failure scenarios and corresponding success scenarios identified for each of the failure scenarios.

While each failure scenario represents a scenario ending with a failure, the corresponding success scenario(s) does not end with the failure exhibited by the respective failure scenario. The success scenarios are created by analyzing the sensor data to identify one or more candidate scenarios for each failure scenario. The candidate scenario(s) exhibit similar behavior, i.e. time-line motion pattern of the object(s) detected in the respective failure scenario. The candidate scenarios may be searched and selected using one or more techniques, methods and/or algorithms as known in the art, for example, Dynamic Time Warping (DTW) algorithm and/or the like configured to identify similarity between candidate subsequences identified in the time-lined motion pattern of an object in a failure scenario and the time-lined motion pattern of the same object in respective corresponding success scenarios. Moreover, one or more multi dimensional DTW algorithms may be applied to identify similarity of the time-lined motion pattern of multiple objects between the success scenario(s) and the respective failure scenario.

In addition, one or more of the failure scenarios may be adjusted to remove a hidden time period immediately preceding the time of the failure such that a predefined time period immediately prior (preceding) to the failure is removed from the failure scenario.

One or more of the failure scenarios and optionally their respective success scenario(s) may be created based on analysis of a plurality of sensor datasets comprising real sensor data captured by sensors deployed in one or more at least partially autonomous vehicles during a plurality of trips, conditions and/or states of the vehicle(s) prior to failures. However, optionally, one or more of the failure scenarios and optionally their respective success scenario(s) may be created based on simulated sensor data generated by one or more simulation systems, applications, tools and/or the like configured to generate simulated sensor data for one or more simulated vehicles during one or more simulated trips, conditions and/or states of the vehicle(s) ending with failures. Moreover, one or more of the failure scenarios and/or success scenarios may be generated based on a combination of simulated and/or real sensor data and/or outputs of one or more of the sub-systems.

Further, a plurality of output datasets may be obtained where each of the output datasets corresponds to a certain one of the sensor datasets. As such each of the scenarios, both the failure scenarios and the success scenarios, is associated with a respective output dataset comprising data items generated by one or more of the sub-systems deployed and/or simulated in the respective vehicle during the respective trip, condition and/or state of the vehicle defined by the respective scenario.

A plurality of feature vectors may be computed for the plurality of output datasets such that each of the plurality of failure scenarios and their corresponding success scenarios are correlated with one or more feature vectors. Each of the feature vectors may be labeled with a label, for example, “FAILURE” or “SUCCESS” according to its correlated scenario.

Using the plurality of feature vectors relating to the failure scenarios and their corresponding success scenarios, the ML classification model(s) may be trained, validated and tested as known in the art.

After the RCA system applies the ML classification model(s) to the feature vector(s) created for the failure scenario, the RCA system may further apply one or more interpretation models, for example, a LIME model and/or the like as known in the art to identify one or more key features having a major contribution to the outcome of the ML classification model(s). The interpretation model(s) applied to the ML classification model(s), to the outcome of the ML classification model(s) and/or to one or more input feature vectors may be used to explain the outcome of the ML classification model(s) in order to achieve meaningful and relevant interpretation of the outcome. The RCA system may manipulate one or more of the input feature vector(s) of the failure scenario injected to the ML classification model(s) such that, based on perturbations in the outcome of the ML classification model(s), the interpretation model(s) may identify the key features.

The RCA system may then estimate one or more of the sub-systems as a root cause of the failure according to the association of the key feature(s) with their respective sub-systems. The RCA system may further trace the failure to one or more time points along the failure scenario based on the time interval(s) of the respective feature vector(s) comprising the key feature(s).

The automated RCA based on machine learning utilizing the failure scenarios and corresponding success scenarios may present major advantages and benefits compared to the exiting RCA methods and systems.

First, automatically identifying the failure root cause may be significantly simpler and easier compared to manual RCA which may be time consuming and may typically require collaboration of multiple specialists to identify the root cause of the failure thus entailing extensive time and significant costs for RCA. The automated RCA on the other hand may require very little time for tracing the failure to the root cause sub-system(s) thus significantly reducing the analysis time. This may further benefit to reduce development and/or testing time and thus accelerate the development cycle. This aspect may be dramatically emphasized by the fact that autonomous vehicles may be deployed with tens and even hundreds of sub-systems operating in concert to control multiple operational aspects of the autonomous vehicle. Manually analyzing such complex systems to identify failures root cause sub-systems may naturally be an overwhelming and potentially impossible. Applying the automated RCA on the other hand may enable comprehensive and time efficient tracing of the failures to their root cause sub-system(s) irrespective of the number of such sub-systems and their complex interaction.

Moreover, identifying the success scenarios corresponding to the failure scenarios may significantly enhance the accuracy of the RCA as the ML classification model(s) may be highly trained to identify valid and invalid feature combinations to accurately trace the failures to their originating root cause sub-systems. This may also significantly improve safety of the autonomous vehicles. Moreover, automatically identifying and selecting the success scenarios may significantly reduce the effort as opposed to manual detection of the success scenarios as may be done by existing methods may significantly reduce time, effort and/or cost.

Furthermore, user (driver, passenger) experience of the vehicle may be significantly improved by adjusting and/or adapting the identified failure root cause sub-systems to employ user friendly operation and/or behavior, for example, reduce and potentially avoid superfluous breaking and/or acceleration of the vehicle.

In addition, the ML classification model(s) may constantly evolve by training them using additional training failure scenarios thus avoiding manual tuning and/or manual RCA to identify root cause of newly encountered failures.

Training the ML classification model(s) with the adjusted scenarios from which the hidden time period is removed may prevent the ML classification model(s) from adjusting to identify root cause features during the period immediately preceding the failures which may hide actual root cause features. This may significantly improve performance of the ML classification model(s) to accurately identify real root cause features and hence identify their associated sub-system(s) as the real root cause sub-system(s).

Also, using the per-interval feature vectors which may be significantly limited in size compared to the scenario-wide feature vectors may allow using shallow ML classification models. This may reduce complexity of the ML classification model(s), reduce classification processing time and/or require significantly reduced computing resources.

Before explaining at least one embodiment of the invention in detail, it is to be understood that the invention is not necessarily limited in its application to the details of construction and the arrangement of the components and/or methods set forth in the following description and/or illustrated in the drawings and/or the examples. The invention is capable of other embodiments or of being practiced or carried out in various ways.

The present invention may be a system, a method, and/or a computer program product. The computer program product may include a computer readable storage medium (or media) having computer readable program instructions thereon for causing a processor to carry out aspects of the present invention.

The computer readable storage medium can be a tangible device that can retain and store instructions for use by an instruction execution device. The computer readable storage medium may be, for example, but is not limited to, an electronic storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. A non-exhaustive list of more specific examples of the computer readable storage medium includes the following: a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), a static random access memory (SRAM), a portable compact disc read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanically encoded device such as punch-cards or raised structures in a groove having instructions recorded thereon, and any suitable combination of the foregoing. A computer readable storage medium, as used herein, is not to be construed as being transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide or other transmission media (e.g., light pulses passing through a fiber-optic cable), or electrical signals transmitted through a wire.

Computer readable program instructions described herein can be downloaded to respective computing/processing devices from a computer readable storage medium or to an external computer or external storage device via a network, for example, the Internet, a local area network, a wide area network and/or a wireless network. The network may comprise copper transmission cables, optical transmission fibers, wireless transmission, routers, firewalls, switches, gateway computers and/or edge servers. A network adapter card or network interface in each computing/processing device receives computer readable program instructions from the network and forwards the computer readable program instructions for storage in a computer readable storage medium within the respective computing/processing device.

Computer readable program instructions for carrying out operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, state setting data, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages.

The computer readable program instructions may execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic circuitry including, for example, programmable logic circuitry, field-programmable gate arrays (FPGA), or programmable logic arrays (PLA) may execute the computer readable program instructions by utilizing state information of the computer readable program instructions to personalize the electronic circuitry, in order to perform aspects of the present invention.

Aspects of the present invention are described herein with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer readable program instructions.

The flowchart and block diagrams in the Figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of instructions, which comprises one or more executable instructions for implementing the specified logical function(s). In some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and/or flowchart illustration, and combinations of blocks in the block diagrams and/or flowchart illustration, can be implemented by special purpose hardware-based systems that perform the specified functions or acts or carry out combinations of special purpose hardware and computer instructions.

Referring now to the drawings, FIG. 1 is a flowchart of an exemplary process of estimating a root cause of a failure in an autonomous vehicle using trained machine learning classification models, according to some embodiments of the present invention. An exemplary process 100 may be executed by a Root Cause Analysis (RCA) system to identify a root cause of a failure, for example, an accident, a near accident, a malfunction and/or the like in an autonomous vehicle and/or at least partially autonomous vehicle, for example, a ground vehicle, an aerial vehicle, a naval vehicle and/or the like. In particular, the process 100 may be executed to trace the failure root cause to one or more of a plurality of sub-systems deployed in the vehicle to monitor, plan and/or control operation of the vehicle. Reference is also made to FIG. 2, which is a schematic illustration of an exemplary system for estimating a root cause of a failure in an autonomous vehicle, according to some embodiments of the present invention. An exemplary RCA system 200, for example, a controller, a computer, a server, a computing node, a cluster of computing nodes and/or the like may execute a process such as the process 100 to identify a root cause of a failure in an at least partially autonomous vehicle 202, for example, a ground vehicle (e.g. car, truck, train, etc.), an aerial vehicle (e.g. drone, place, helicopter, etc.), a naval vehicle (e.g. boat, ship, submarine) and/or the like.

One or more sensors 204, for example, an imaging sensor, a ranging sensor, an environmental condition sensor and/or the like may be deployed in the vehicle 202 to sense an environment of the vehicle 202. The imaging sensor(s) may include, for example, a camera, an infrared sensor, a thermal imaging sensor camera, a night vision sensor and/or the like. The ranging sensor(s) may include, for example, LiDAR, RADAR, SONAR and/or the like. The environmental condition sensor(s) may include, for example, a temperature sensor, a humidity sensor, an illumination (light) sensor and/or the like.

The sensor(s) 204 may capture and/or generate sensor data according to the sensory technology applied by the sensor(s) 204, for example, images, video streams comprising sequences of images, ranging maps, thermal maps and/or the like.

The vehicle 202 may be deployed with a plurality of sub-systems 206 which are configured to monitor, plan and/or control operation, in particular, motion (movement) of the vehicle 202 which is, as stated herein before, at least partially autonomous. Each of the sub systems 206 may typically receive input data (input) and generate output data (output). For example, one or more of the sub-systems 206 may receive the sensor data from one or more of the sensor(s) 204. In another example, one or more of the sub-systems 206 may communicate with each other such that an output of one or more sub-systems 206 may be injected as input to one or more other sub-systems 206.

The sub-systems 206 may include, for example, a steering sub-system for controlling an angle of advancement of the vehicle 202. For example, assuming the vehicle 202 is a ground vehicle the steering sub-system may control an angle of a steering wheel of the vehicle 202 or an angle of one or more wheels of the vehicle 202 to apply a two-dimensional direction of advancement of the vehicle 202. In another example, assuming the vehicle 202 is an aerial vehicle, the steering sub-system may control an angle of one or more flaps of the vehicle 202 to apply a three-dimensional direction of advancement of the vehicle 202. In another example, assuming the vehicle 202 is a naval vehicle, the steering sub-system may control an angle of one or more rudders of the vehicle 202 to apply a 2D direction (boat, ship) and/or 3D direction (submarine) of advancement of the vehicle 202.

In another example, the sub-systems 206 may include a breaking sub-system for controlling break and/or deceleration of the vehicle 202.

In another example, the sub-systems 206 may include a speed and/or acceleration sub system for controlling speed (velocity) and/or acceleration of the vehicle 202.

In another example, the sub-systems 206 may include a path planning sub-system for planning a route of the vehicle 202.

In another example, the sub-systems 206 may include a lighting sub-system for controlling operation of one or more lights of the vehicle 202, for example, a headlight, a tail light and/or the like.

The RCA system 200 may be deployed in the vehicle 202. However, the RCA system 200 may optionally be a remote system separated from the vehicle 202. The RCA system 200 may include an Input/Output (I/O) interface 210, a processor(s) for executing the process 100 and storage 214 for storing code (program store) and/or data.

The I/O interface 210 may include one or more wired and/or wireless interconnection interfaces, for example, a Universal Serial Bus (USB) interface, a serial port, a Controller Area Network (CAN) bus interface and/or the like. The I/O interface 210 may further include one or more network interfaces for connecting to one or more wired and/or wireless networks, for example, a Local Area Network (LAN), a Wide Area Network (WAN), a Municipal Area Network (MAN), a cellular network, the internet and/or the like.

The I/O interface 210 may naturally be configured, adapted and/or employed according to the deployment of the RCA system 200. For example, in case the RCA system 200 is deployed in the vehicle 202, the I/O interface 210 may be used to communicate with the sensor(s) 204 and the sub-systems 206 to obtain the sensor data generated by the sensor(s) 204 and the output of the sub-systems 206 via one or more networks and/or interconnections deployed in the vehicle 202. In another example, in case the RCA system 200 a remote system, the I/O interface 210 may be used to communicate with the vehicle 202 via one or more of the networks to obtain the sensor data generated by the sensor(s) 204 and the output of the sub systems 206. In another example, in case the RCA system 200 a remote system, the I/O interface 210 include one or more ports and/or interfaces to host one or more attachable devices, for example, an attachable storage device storing media the sensor data generated by the sensor(s) 204 and the output of the sub-systems 206 logged at the vehicle 202. The storage 214 used for storing data and/or code (program store) may include one or more non-transitory memory devices, either persistent non-volatile devices, for example, a hard drive, a solid state drive (SSD), a magnetic disk, a Flash array and/or the like. The storage 214 may also include one or more volatile devices, for example, a Random Access Memory (RAM) device, a cache memory and/or the like.

The storage 214, in particular in case the RCA system 200 is remote and separate from the vehicle 202, may further comprise one or more network storage resources, for example, a storage server, a Network Attached Storage (NAS), a network drive, and/or the like accessible via one or more networks through the I/O interface 210.

The processor(s) 212 may include one or more processor(s), homogenous or heterogeneous, each comprising one or more processing nodes arranged for parallel processing, as clusters and/or as one or more multi-core processor(s). The processor(s) 212 may execute one or more software modules, for example, a process, an application, an agent, a utility, a tool, a script and/or the like each comprising a plurality of program instructions stored in a non- transitory medium such as the storage 214 and executed by one or more processors such as the processor(s) 212. The processor(s) 212 may further include one or more hardware components, for example, a circuit, an Integrated Circuit (IC), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Digital Signals Processor (DSP), a Graphic Processor Unit (GPU) and/or the like to support execution of the process 100.

The processor(s) 212 may therefore execute one or more functional modules which may be implemented by one or more of the software modules, one or more of the hardware components and/or a combination thereof. The functional modules may include, for example, an RCA analyzer 220 executed to perform the process 100.

Optionally, in case the RCA system 200 is remote and separate from the vehicle 202, the RCA system 200 and/or the RCA analyzer 220 are provided by one or more cloud computing services, for example, Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Software as a Service (SaaS) and/or the like provided by one or more cloud infrastructures and/or services such as, for example, Amazon Web Service (AWS), Google Cloud, Microsoft Azure and/or the like.

Reference is now made to FIG. 3, which is a schematic illustration of an exemplary system for training machine learning classification models for estimating a root cause of a failure in an autonomous vehicle, according to some embodiments of the present invention. An exemplary classification model training system 300, for example, a computer, a server, a computing node, a cluster of computing nodes and/or the like may be used to generate, create and/or train one or more ML classification models which may be used by one or more RCA systems such as the RCA system 200 to estimate a root cause of failures in at least partially autonomous vehicles such as the vehicle 202.

The classification model training system 300 may include an Input/Output (I/O) interface 310 such as the I/O interface 210, a processor(s) 312 such as the processor(s) 212 and storage 314 such as the storage 214.

Via the I/O interface 310, the classification model training system 300 may obtain a sensor data captured by the sensor(s) 204 and/or output of the sub-systems 206 deployed in a plurality of vehicles such as the vehicle 202. The classification model training system 300 may obtain the real sensor data and/or the real output of the sub-systems 206 by communicating directly with one or more of the vehicles 202 via the I/O interface 310 connecting one or more of the wired and/or wireless networks. Optionally, the classification model training system 300 may obtain the real sensor data and/or the real output of the sub-systems 206 by communicating via the I/O interface 310 with one or more storage resources, systems, platforms and/or services, for example, a storage server, a cloud storage service and/or the like which may store the real sensor data and/or the real output of the sub-systems 206 logged by the vehicles 202. In another example, the classification model training system 300 may obtain the real sensor data and/or the real output of the sub-systems 206, via the I/O interface 310, from one or more attachable storage devices storing the real sensor data and/or the real output of the sub-systems 206 logged by the vehicles 202.

The classification model training system 300 may optionally obtain simulated sensor data and/or simulated output of the sub-systems 206 generated by one or more simulation systems 304. The simulation system(s) 304 may be designed, configured and/or adapted to generate the simulated sensor data and/or the simulated output of the sub-systems 206 according to one or more simulated trips of the vehicle 202.

The processor(s) 312 may include one or more processor(s), homogenous or heterogeneous, each comprising one or more processing nodes arranged for parallel processing, as clusters and/or as one or more multi-core processor(s). The processor(s) 312 may execute one or more software modules, for example, a process, an application, an agent, a utility, a tool, a script and/or the like each comprising a plurality of program instructions stored in a non- transitory medium such as the storage 314 and executed by one or more processors such as the processor(s) 312. The processor(s) 312 may further include one or more hardware components, for example, a circuit, an Integrated Circuit (IC), an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Digital Signals Processor (DSP), a Graphic Processor Unit (GPU) and/or the like.

The processor(s) 312 may therefore execute one or more functional modules which may be implemented by one or more of the software modules, one or more of the hardware components and/or a combination thereof. The functional modules may include, for example, a trainer 320 executed to train one or more of the ML classification models.

Optionally, the classification model training system 300 and/or the trainer 320 are provided by one or more of the cloud computing services, for example, IaaS, PaaS, SaaS and/or the like provided by one or more of the cloud infrastructures and/or services such as, for example, Amazon Web Service (AWS), Google Cloud, Microsoft Azure and/or the like.

Reference is also made to FIG. 4, which is a flowchart of an exemplary process of training machine learning classification models for estimating a root cause of a failure in an autonomous vehicle, according to some embodiments of the present invention. An exemplary process 400 may be executed by the classification model training system 300 for creating, generating and/or training one or more ML classification models which may be used by the RCA system 200 to estimate a root cause of failures in at least partially autonomous vehicles such as the vehicle 202.

As shown at 402, the process 400 starts with the trainer 320 obtaining a plurality of sensor datasets. Each of the sensor datasets may include sensor data representing an environment of a vehicle such as the vehicle 202 during a respective one of a plurality of trips, situations and/or states of the respective vehicle 202.

The sensor data may include one or more data formats, typically according to the type and/or technology of the sensor(s) 204 which captured and/or generated the sensor data. The sensor data may therefore include, for example, images, video streams comprising sequences of images, ranging maps, thermal maps and/or the like.

Moreover, the sensor data included in each of the sensor datasets may be timestamped so that each sensor data item may be identified and allocated along a time line of the respective trip, situation and/or state of the vehicle 202.

One or more of the sensor datasets may include real sensor data captured during one or more trips, situations and/or states of one or more vehicles 202 by one or more sensors such as the sensor 204 deployed in the vehicle(s) 202. As described herein before, the trainer 320 may obtain the real sensor datasets logged by the vehicles 202 via one or more sources. For example, the trainer 320 may directly communicate, via the I/O interface 310, with the vehicles 202 to receive one or more of the real sensor datasets. In another example, the trainer 320 may obtain one or more of the real sensor datasets by communicating with one or more of the storage resources, systems, platforms and/or services storing the real sensor datasets logged by the vehicles 202. In another example, the trainer 320 may obtain one or more of the real sensor datasets, via the I/O interface 310, from one or more of the attachable devices storing the real sensor data logged by the vehicles 202.

However, one or more of the sensor datasets may include simulated sensor data generated by one or more simulation systems such as the simulation system 304, for example, a system, an application, a tool, a service and/or the like. The simulation system(s) 304 may be designed, configured and/or adapted to generate simulated sensor data as it may be captured by one or more of the sensors 204 during one or more simulated trips, conditions and/or states of a simulated vehicle simulating one or more vehicles 202.

As shown at 404, the trainer 320 obtains a plurality of output datasets generated by one or more of a plurality of sub-systems such as the sub-systems 206 deployed in the plurality of vehicles 202. Each of the output datasets may include one or more data items of a wide range of data types and formats, for example, status data, control data, instructions and/or the like generated by the sub-systems 206 during the respective trip, condition and/or state of the vehicle 202.

Each of the plurality of output datasets is associated with one of the plurality of sensor datasets such that each sensor dataset and its associated output dataset relate to the same trip, condition and/or state of the same vehicle 202. Moreover, the output data included in each of the output datasets may be timestamped so that each output data item may be identified and allocated along a time line of the respective trip, situation and/or state of the vehicle 202. As such each output data item may be correlated in time with one or more sensor data items included in the corresponding sensor dataset.

Similarly to the sensor data, the output datasets may include real output data and/or simulated output data. Since each output dataset corresponds to one of the sensor datasets, in case the sensor dataset comprises real sensor data captured by the sensor(s) 204 of a certain vehicle 202, the corresponding output dataset may be also real, i.e. generated by the sub system^) 206 of the certain vehicle 202. In case the sensor dataset comprises simulated sensor data generated by the simulation system 304 for a certain simulated vehicle 202, the corresponding output dataset may be also simulated real, i.e. generated by the simulation system 304 to simulate the output of the sub-system(s) 206 of the certain simulated vehicle 202.

However, one or more of the sensor datasets and their corresponding output datasets may be a combination of real and simulated datasets. For example, the simulation system 304 may use one or more real sensor datasets captured and/or generated by sensor(s) 304 of one or more of the vehicles 202 to generate one or more corresponding simulated output datasets simulating the output of one or more of the sub-systems 206 driven with the respective real sensor datasets. In another example, one or more simulated sensor datasets may be driven into one or more of the sub-systems 206 and the output of the sub-systems 206 may be captured and included in one or more corresponding real output datasets.

As shown at 406, the trainer 320 may analyze the plurality of sensor datasets in order to generate a plurality of respective failure scenarios. Specifically, the trainer 320 may analyze a plurality of sensor datasets captured, generated and/or simulated prior to a failure which the vehicle 202 experiences, exhibits and/or suffers for example, an accident, a near accident, an operational malfunction and/or the like.

Each failure scenario may relate to one or more other objects identified in the environment of the vehicle 202 during the respective trip, condition and/or state documented by the respective sensor dataset. These objects may include, for example, another vehicle, a pedestrian, a bicycle rider, a road infrastructure object (e.g. traffic light, traffic sign, road side fence, road safety fence, road and/or lane markings, etc.), a structure (e.g. a bridge, a tunnel, a building, etc.) and/or the like.

The trainer 320 may apply one or more analysis methods, algorithms and/or tools as known in the art to identify objects in the sensor datasets. For example, the trainer 320 may use one or more image processing and/or computer vision algorithms and/or tools to identify one or more objects in one or more imagery sensor datasets captured by one or more of the imaging sensor(s) 204. In another example, the trainer 320 may use one or more signal processing and/or image processing algorithms and/or tools to identify one or more objects in ranging sensor datasets (ranging maps) captured and/or generated by one or more of the LiDAR, SONAR and/or SONAR sensor(s) 204.

Based on the analysis of the sensor data contained in a sensor dataset which ends with a failure of the respective (real or simulated) vehicle 202, the trainer 320 may generate a respective failure scenario which defines a time-lined motion pattern of the identified object(s) with respect to the respective vehicle 202. As such, each failure scenario describes a spatio temporal relation between the respective vehicle 202 and the other object(s), in particular in terms of distance, velocity, acceleration and/or the like. The time-lined motion pattern defined by each of the failure scenarios may therefore include one or more motion attributes of the other object(s) with respect to the vehicle 202, for example, distance of the object(s) from the vehicle 202, a direction of the object(s) with respect to the vehicle 202, a velocity of the object(s) with respect to the vehicle 202, an acceleration of the object(s) with respect to the vehicle 202 and/or the like.

For example, in assuming a certain failure scenario defines the time-lined motion pattern of a certain object with respect to the vehicle 202 according to a relative distance between the certain object and the vehicle 202 and absolute velocities along x and y axes. The certain failure scenario defining a time-lined motion pattern b(Sf) of the certain object designated o with respect to the vehicle 202 designated av may be represented as a multi-dimensional time series as expressed in equation 1 below.

Equation 1:

for t = 1, ... , T

Where d_{x t} is the relative distance between the certain object (o) and the vehicle 202 (av) along the x axis, d_{y t} is the relative distance between the certain object (o) and the vehicle 202 (av) along the y axis, v_{x t} is the absolute velocity along x axis, v_{x t} is the absolute velocity along y axis and t is a timestamp.

Moreover, the trainer 320 may adjust one or more of the failure scenarios to remove a hidden time period immediately prior to the failure such that the failure scenario does not include a final time period before the failure of the vehicle 202. The hidden time period may be predefined, for example, 0.5 seconds, 1 second and/or the like. For example, the trainer 320 may set the hidden time period to span a time back from the time of the failure to a time in which the sub-system(s) 206 are still able to prevent the failure. The trainer 320 may identify the hidden time period since the failure scenarios are created offline after the failures occur.

As shown at 408, the trainer 320 may identify and generate a plurality of success scenarios corresponding to the plurality of failure scenarios by analyzing one or more of the sensor datasets where each success scenario is significantly similar to its corresponding failure scenario with the exception that the success scenarios does not end with a failure of the respective vehicle 202.

Similarly, to the failure scenarios, each of the success scenarios defines a time-lined motion pattern of one or more objects identified in the environment of the respective vehicle 202 by analyzing the sensor datasets with respect to the respective vehicle 202. Therefore, in order to identify and generate the success scenarios, the trainer 320 may analyze a plurality of sensor datasets captured, generated and/or simulated during trips, conditions and/or states of vehicle 202 which did not eventually end with a failure of the vehicle 202. Similarly, to the failure scenarios, each success scenario describes a spatiotemporal relation between the respective vehicle 202 and the other object(s), in terms of distance, velocity, acceleration and/or the like. The time-lined motion pattern defined by each of the failure scenarios may therefore include one or more motion attributes of the other object(s) with respect to the vehicle 202, for example, distance of the object(s) from the vehicle 202, a direction of the object(s) with respect to the vehicle 202, a velocity of the object(s) with respect to the vehicle 202, an acceleration of the object(s) with respect to the vehicle 202 and/or the like.

The trainer 320 may first analyze the plurality of sensor dataset(s) which did not eventually end with a failure of the vehicle 202 to generate one or more time-lined subsequences defining a time-lined motion pattern of one or more of the objects identified in the environment of the respective vehicle 202 associated with the sensor dataset(s).

In order to identify success scenarios that correspond to the failure scenarios, the trainer 320 may apply one or more similarity algorithms on the time-lined sequence(s) using each failure scenario and/or part thereof as a query (time-lined) sequence (series).

The similarity between the failure scenarios and their corresponding success scenarios may be defined using one or more similarity parameters applied to the time-lined motion pattern of an object compared to the motion pattern identified for the object in one or more of the time- lined subsequences. Such similarity parameters may include, for example, a type of the object(s) represented in the query failure scenario, a distance of the object(s) from the respective vehicle 202, a velocity of the object(s) with respect to the respective vehicle 202 and/or the like.

Moreover, one or more of the similarity parameters may be defined within a predefined range in which the trainer 320 may determine that a certain candidate sequence is sufficiently similar to the respective query failure scenario. For example, assuming the similarity is determined according to a distance between the other object(s) and the vehicle 202. Assuming that at any given time, the distance in the candidate sequence is +/-10 meters compared to the distance in the respective query failure scenario, the trainer 320 may determine that the candidate sequence is sufficiently similar and may be therefore designated a success scenario corresponding to the respective query failure scenario. In another example, assuming the similarity is determined according to a velocity of the other object(s) with respect to the vehicle 202. Assuming that at any given time, the velocity in the candidate sequence is +/-5 kilometer per hour compared to the velocity in the respective query failure scenario, the trainer 320 may determine that the candidate sequence is sufficiently similar and may be therefore designated a success scenario corresponding to the respective query failure scenario. For example, the trainer 320 may apply one or more DTW algorithms as known in the art to search and identify one or more candidate subsequences of the time-lined sequence generated based on the analysis of the sensor dataset(s) which did not end with a failure of the vehicle 202 which match one or more of the query failure scenarios. The DTW algorithms may compute a similarity score for each candidate subsequence based on the similarity between the time-lined motion pattern defined for an object in the respective candidate sequence and the time-lined motion pattern defined by the corresponding query failure scenario according to the similarity parameter(s) and the predefined range.

The trainer 320 may evaluate whether each such candidate subsequence qualifies as a candidate success scenario based on a threshold which may be predefined for one or more of the similarity parameters, for example, the relative distance between the object(s) and the vehicle 202 and/or the like. In case the similarity score computed for a certain candidate success scenario is below the predefined threshold, the trainer 320 may select the certain candidate success scenario as a valid success scenario corresponding to the query failure scenario. However, the trainer 320 may discard one or more candidate success scenario for which the similarity score computed by the DTW algorithm(s) is below the predefined threshold.

The similarity check applied by the trainer 320, in particular the DTW algorithm(s) to identify similarity between the time-lined motion pattern b (s^ ) of a certain candidate success scenario and the time-lined motion pattern

a corresponding failure scenario may be formulated by equation 2 below.

Equation 2:

The time-lined motion pattern b^s-^) of the candidate success scenario may be determined similar to the time-lined motion pattern b(sf) of the failure scenario in case the similarity score computed for the time-lined motion pattern b (s- with respect to the time-lined motion pattern

below a threshold TH predefined for the relative distance and the absolute velocity in the x- and y axes.

The trainer 320 may further check one or more other similarity parameters to determine whether each candidate success scenario is indeed a suitable scenario which corresponds to the respective failure scenario with significant and sufficient accuracy. For example, assuming the DTW algorithm(s) determined the similarity between a certain failure scenario and its corresponding candidate success scenario according to an x-axis distance between the object(s) and the vehicle 202. Assuming the similarity score computed by the DTW algorithm(s) based on the x-axis distance is below the predefined threshold, the trainer 320 may further use the DTW algorithm(s) to compute a similarity score based a y-axis distance between the object(s) and the vehicle 202. In case the score computed based on the y-axis distance is below the predefined threshold, the trainer 320 may select the candidate success scenario as a valid success scenario corresponding to the failure scenario. In another example, assuming the similarity score computed by the DTW algorithm(s) based on the x-axis distance is below the predefined threshold, the trainer 320 may further use the DTW algorithm(s) to compute a similarity score based a velocity of the object(s) and with respect to the vehicle 202. In case the score computed based on the velocity is below the predefined threshold, the trainer 320 may select the candidate success scenario as a valid success scenario corresponding to the failure scenario.

In order to ensure that the candidate success scenarios do not eventually end with a failure of the respective vehicle 202, the trainer 320 may further use one or more trajectory prediction algorithms to ensure that the object(s) defined by one or more of the candidate success scenarios do not collide, nearly collide and/or dangerously intersect with the respective vehicle 202. The trainer 320 may therefore discard one or more of the candidate success scenarios defining the time-lined motion pattern of an object that is predicted to collide with the vehicle 202.

Optionally, based on the trajectory prediction algorithm(s), the trainer 320 may select one or more candidate success scenarios in which the object(s) defined by the respective candidate success scenario(s) do not collide, nearly collide and/or dangerously intersect with the respective vehicle 202.

In addition, the trainer 320 may discard one or more candidate success scenarios which end with a failure of the vehicle 202, i.e. candidate success scenarios which may replicate a respective failure scenario that eventually ends with a failure of the vehicle 202.

The trainer 320 may further apply one or more multi-dimensional DTW algorithms as known in the art to search for one or more candidate sequences which match one or more the query failure scenarios in the time-lined sequence generated based on analysis of one or more of the sensor datasets, in particular for query failure scenarios defining the time-lined motion pattern of multiple objects. Using the multi-dimensional DTW algorithm(s), the trainer 320 may evaluate similarity of the time-lined motion pattern of each of the multitude of objects time-lined sequence with the time-lined motion pattern of the respective object defined by the query failure scenario(s). Reference is no made to FIG. 5, which is a schematic illustration of an exemplary time- lined sequence in which candidate scenarios are searched for a query scenario, according to some embodiments of the present invention. A graph line 502 of an exemplary query failure scenario Q defines a time-lined motion pattern of an object with respect to a vehicle such as the vehicle 202. A graph line 504 of a time-lined sequence D may be generated by a trainer such as the trainer 320 based on analysis of one or more of the sensor datasets, in particular sensor dataset(s) which did not end with a failure to the respective vehicle 202. The trainer 320, applying the DTW algorithm(s), may search the time-lined sequence D 504 to identify one or more candidate sequences C 506 (marked bold) which are similar to the query failure scenario Q 502. In particular, the trainer 320 may determine the similarity according to the predefined range (marked by the bold lines between the two graph lines) indicating a sufficient similarity between the query failure scenario Q 502 and the candidate sequence(s) C 506. The trainer 320 may designate one or more of the candidate sequences C 506 as a success scenario which may serve as success scenarios corresponding to the query failure scenario Q 502.

Optionally, the trainer 320 applies one or more filters to the time-lined motion pattern defined by one or more of the failure scenarios as well as to one or more of the time-lined sequence(s) generated based on the sensor data of the sensor datasets captured, generated and/or simulated for the trips, conditions and or states which do not end with a failure of the vehicle 202 and which are used to search for the success scenario(s). Applying the filters may serve to smooth the time-lined motion pattern and remove fluctuations, irregularities and/or outliers thus supporting the trainer 320 to more efficiently identify matching candidate success scenarios.

Reference is now made to FIG. 6, which presents graph charts of segments of exemplary time-lined motion patterns smoothed using filters, according to some embodiments of the present invention. A trainer such as the trainer 320 may apply one or more filters to the time- lined motion patterns defined by one or more of the failure scenarios as well as to one or more of the time-lined sequence(s) generated based on the sensor data of the sensor datasets captured, generated and/or simulated for the trips, conditions and or states which do not end with a failure of a vehicle such as the vehicle 202. For example, as seen at 602, the trainer 320 may apply a low-pass filter to smooth a time-lined motion pattern defining, for example, a velocity of an object identified in the environment of the vehicle 202. In another example, as seen at 604, the trainer 320 may apply a Gaussian process filter to smooth a time-lined motion pattern defining, for example, the velocity of the object identified in the environment of the vehicle 202.

Reference is made once again to FIG. 4. As shown at 410, the trainer 320 may compute a plurality of feature vectors each corresponding to and characterizing a respective one of the plurality of failure scenarios and success scenarios. As such each of the failure scenarios and success scenarios is associated (corresponding) with a respective feature vector.

Each feature vector comprises a plurality of features extracted by the trainer 320 from a respective one of the output datasets generated by the sub-system(s) 206 associated with a respective one of the vehicles 202, either a real vehicle 202 and/or a simulated vehicle 202.

As each output dataset includes a plurality of data items generated by the sub-system(s) 206, for example, status data, control data, instructions and/or the like, each of the features extracted by the trainer 320 from the respective output dataset is therefore associated with one or more of the sub-systems 206.

Each of the features extracted from each output dataset may relate to the vehicle 202 and/or to one or more of the objects whose time-lined motion pattern is defined by the respective failure or success scenario created from the respective sensor dataset associated with the respective output dataset.

The features may be highly indicative of one or more root causes for one or more of the failures experienced by the vehicle 202. Such root causes may include, for example, a fusion false negative, a fusion false positive, a Planning and Control (P&C) and/or the like. Fusion false negative relates to a collision between the vehicle 202 and an object that was in the environment of the vehicle 202 but was not detected by the sub-system(s) 206, for example, a fusion module or detected too late by the fusion module such that the vehicle 202 is incapable of braking in time to avoid the collision. Fusion false positive relates to a collision reported by the sub-system(s) 206 between the vehicle 202 and an object which was not actually present in the environment of the vehicle 202 but erroneously detected by the sub-system(s) 206. P&C relates to a collision between the vehicle 202 and an object detected in the environment of the vehicle 202 which results from an error in sub-system(s) 206, for example, a planning and control module. For example, assuming another vehicle object which is slightly ahead of the vehicle 202 accelerates, and then decelerates. Further assuming, the vehicle 202 responds efficiently to the acceleration of the other vehicle but fails to react to the deceleration in sufficient time to avoid a collision of the vehicle 202 with the other vehicle. Such failure may typically result from one or more P&C systems which control the operation of the vehicle 202. However, the failure resulting from the P&C system 206 may typically be the result of a failure to one or more of the sub-systems 206 driving their output to the P&C sub-system(s) 206. Reference is now made to FIG. 7, which is a schematic illustration of an exemplary autonomous vehicle planning and control system receiving output of one or more sub-systems of the autonomous vehicle. An exemplary P&C system deployed in an autonomous vehicle such as the autonomous vehicle 202 may receive as input the output of one or more sub-systems such as the sub-systems 206. The P&C system may plan, compute and/or generate control data for controlling the autonomous vehicle 202 according to the data received from the sub system^) 206. Therefore, one or more failed sub-systems 206 may generate erroneous data which is driven to the P&C system. The P&C system using the erroneous data received from the failed sub-system(s) 206 may generate error control data which may lead to the failure(s) of the autonomous vehicle 202.

The features extracted from each output datasets may include, for example, a relative distance between the respective object and the vehicle 202, a relative distance between the respective object and one or more other objects detected in the environment by analyzing the associated sensor dataset, a relative velocity of the respective object with respect to the vehicle 202, a relative velocity of the respective object with respect to the other object(s), a duration of detection of the respective object, a location of the vehicle 202 on a road, a maneuvering operation of the vehicle 202, a distribution of the respective object in spatial regions defined in the environment, an identifier of one or more of the sensor(s) 204 whose sensor data is analyzed to identify the respective object and/or the like.

Moreover, the trainer 320 may create a plurality of feature vectors for each output dataset where each of the feature vectors relates to a certain timestamp along the time line of the respective trip, situation and/or state of the vehicle 202. As such each of the failure scenarios and success scenarios is associated with a plurality of corresponding feature vectors each relating to certain one of the timestamps of the respective scenario. The trainer 320 may be therefore able to identify and allocate each of the feature vectors associated with the respective scenario along the time-lined motion pattern defined by the respective scenario.

Optionally, the trainer 320 splits the time-lined motion pattern defined by one or more of the failure scenarios and/or their corresponding success scenarios into a plurality of time intervals each spanning a limited time period. The trainer 320 may apply one or more criteria for splitting the time-lined motion pattern. For example, the trainer 320 may split the time-lined motion pattern into the plurality of time intervals according to one or more of the motion attributes describing the object(s) with respect to the vehicle 202.

For example, the trainer 320 may split the time-lined motion pattern into the plurality of time intervals according to a velocity of the object with respect to the vehicle 202. In such implementation, the trainer 320 may split the time-lined motion pattern into the plurality of time intervals, for example, based on variations in the velocity of the object such that a new time interval is created upon each velocity change exceeding a predefined threshold. In another example, the trainer 320 may split the time-lined motion pattern into the plurality of time intervals based on one or more extremum values of the velocity such that a new time interval is created upon after the velocity reaches one of the extremum values, for example, a maximum value, a minimum value and/or the like. In another example, the trainer 320 may split the time- lined motion pattern into the plurality of time intervals based on a polynomial approximation of the velocity of the object such that a new time interval is created upon deviation of the actual velocity of the object from the polynomial approximation outcome over a predefined threshold.

Reference is now made to FIG. 8, which is a graph chart of an exemplary time-lined motion pattern of a scenario split into a plurality of time intervals, according to some embodiments of the present invention. A trainer such as the trainer 320 may split a time-lined motion pattern of an object defined by a certain scenario into a plurality of time intervals according to one or more criteria, for example, the x-axis velocity of an object. A graph line 802 describes the x-axis velocity of a certain object with respect to a vehicle such as the vehicle as defined by a certain exemplary scenario while a graph line 804 describes the y-axis velocity of the certain object with respect to the vehicle as defined by the certain exemplary scenario. As seen, the trainer 320 splits the time-lined motion pattern defined by the certain scenario into a plurality of time intervals 810 based on the variations in the velocity of the certain object and based on extremum values of the velocity of the certain object. For example, the trainer 320 creates a new time interval 2 when the x-axis velocity of the object deviates and crosses a certain predefined threshold, for example, 12.5 meter per second. In another example, the trainer 320 creates a new time interval 3 when the x-axis velocity of the object reaches a maximum velocity value, which may be an absolute maximum value and/or a local maximum value. In another example, the trainer 320 creates a new time interval 4 when the x-axis velocity of the object reaches a minimum velocity value which may be an absolute minimum value and/or a local minimum value. In another example, the trainer 320 creates a new time interval 5 when the x- axis velocity of the object reaches the predefined threshold of 12.5 meter per second. In another example, the trainer 320 creates a new time interval 6 when the x-axis velocity of the object reaches again the predefined threshold of 12.5 meter per second. In another example, the trainer 320 creates a new time interval 7 when the x-axis velocity of the object reaches a maximum velocity, which as stated herein before may be the absolute maximum value and/or a local maximum value. After splitting the time-lined motion pattern of the object into the plurality of time intervals, the trainer 320 may compute a per-interval feature vector for each of the time intervals. The feature vector computed for each time interval therefore comprises features extracted from the associated output dataset which correspond in time to the same time interval. This may allow for further accuracy in identifying and/or allocating the features along the time- lined motion pattern.

The features included in the per-interval feature vectors may reflect possible failures in the associated sub-system(s) 206 which may extend and/or be apparent for a limited time, for example, unplanned out of lane, failure to break below time-to-collision, relative distance between objects, velocity of objects and/or the like.

The features included in the scenario -wide feature vectors may further include one or more features identified across multiple time intervals and optionally across the entire scenario, for example, duration of object observation, total duration of object observation and/or the like. Moreover, the scenario -wide feature vectors may further include features that exceed one or more boundaries of the time duration of the scenario, i.e. prior and/or following the scenario. For example, assuming a scenario is created for a certain object. Further assuming based on the analyses of the sensor data captured after (flowing) a collision of the certain object with the vehicle 202 the certain object is observed after the collision. This may indicate that the certain object is real.

Combination of such scenario -wide features may be indicative of one or more false positive failures as well as of one or more false-negative failures.

The trainer 320 may combine the scenario-wide feature vectors and all interval feature vectors into a single feature vector and label it. Each combined feature vector associated with a failure scenario is labeled with a label “FAILURE” while each combined feature vector associated with a success scenario is labeled with a label “SUCCESS”.

As shown at 412, the trainer 320 may use the labeled feature vectors computed to characterize the plurality of failure scenarios and their corresponding success scenarios for training one or more ML classification models, for example, a neural network, an SVM and/or the like to identify one or more root cause sub-systems 206 leading to failures in each of the plurality of failure scenarios.

The feature vectors used by the trainer 320 to train the ML classification model(s) may include the scenario-wide feature vectors computed for entire scenarios and/or per-interval feature vectors computed for the plurality of time interval constituting one or more of the scenarios. However, using the per-interval feature vectors which may be significantly limited in size compared to the scenario-wide feature vectors may allow employing shallow ML classification models. Employing shallow ML classification model(s) may significantly reduce complexity of the ML classification model(s), reduce processing time and/or require significantly reduced computing resources thus reducing cost and/or complexity of the classification model training system 300.

The trainer 320 apply one or more training methodologies, techniques and/or paradigms as known in the art to train the ML classification model(s) using the labeled feature vectors, for example, supervised training, semi-supervised training and/or the like.

The trainer 320 may further validate and test the trained ML classification model(s) as known in the art. Moreover, the trainer 320 may use separate subsets of the feature vectors for training the ML classification model(s), for validating the ML classification model(s) and/or for testing the ML classification model(s). Lor example, the trainer 320 may train the ML classification model(s) using feature vectors computed for scenarios constructed based on simulated sensor datasets and corresponding simulated output datasets. In another example, the trainer 320 may test the ML classification model(s) using feature vectors computed for scenarios constructed based on both real and simulated sensor datasets and corresponding simulated output datasets. The separate subsets of feature vectors used by the trainer 320 to train, validate and test the ML classification model(s) may be selected to be independent of each other and having no mutual correlation in order to improve performance of the ML classification model(s) and avoid one or more issues typical to machine learning models, for example, overfitting and/or the like.

Since the features, and to further extent, combinations of features may be highly indicative of root cause(s) of one or more of a plurality of failure scenarios, the trained ML classification model(s) may be able to identify the root cause(s) to such failures, in particular to estimate from which of the sub-system(s) 206 the root cause may originate.

Lor example, assuming a first feature includes a brief detection of another vehicle object for a very short time prior to a collision of the other vehicle with the vehicle 202. Lurther assuming, a second feature includes detection of the other vehicle for a significantly long period of time after the collision. Such combination of features may be interpreted by the trained ML classification model(s) as a fusion false negative failure since the other vehicle (object) may be correctly detected for short period(s) of time but at a certain point in time it may be detected too late which may lead to the collision.

In another example, assuming a first feature includes detection of another object (e.g. vehicle) for extended periods of time during a certain respective failure scenario. Lurther assuming, a second feature includes inconsistency between the velocity of the other vehicle and the distance that the other object is expected to travel. Such combination of features may be interpreted by the trained ML classification model(s) as a fusion false positive failure since the other vehicle (object) may be erroneously detected.

In another example, assuming a first feature includes detection of another object (e.g. vehicle) at a left most lane of a road on which the vehicle 202 is riding. Further assuming, a second feature includes detection, by a LiDAR sensor 204, of a plurality of additional objects to the left of the other object where the distance of the additional objects from the vehicle 202 is less than two meters in the y-axis. Assuming a third feature includes no detection indication form a fusion sub-system 206. Such combination of features may be interpreted by the trained ML classification model(s) as a fusion false negative failure since the combination may be indicative of a road fence (to the left of the left most lane) which is not detected.

As described in step 406 of the process 400, the trainer 320 may adjust one or more of the failure scenarios to remove the hidden time period immediately preceding the failure. As result the feature vectors computed by the trainer 320 for the failure scenarios do not include features relating to the hidden time period which may be trivial and/or may typically result from more acute previous root cause failures which may occur sooner in the scenario.

Training the ML classification model(s) with the adjusted scenarios from which the hidden time period is removed may therefore serve to prevent the ML classification model(s) from adjusting itself to identify root cause features during the period immediately preceding the failures which may hide actual root cause features. This may significantly improve performance of the ML classification model(s) to accurately identify real root cause features and hence identify their associated sub-system(s) 206 as the real root cause sub-system(s).

For example, assuming the feature vectors used to train the ML classification model(s) include features reflecting the distance and/or the relative velocity of a certain object during a time period of 0.5 seconds before the vehicle 202 collides with the certain object. In such case the ML classification model(s) may typically use these features for prediction instead of identifying the root cause(s) pointed by other features, for example, fusion false positive in which the object which was in the environment of the vehicle 202 was not detected by the sub system^) 206. As such, the root cause(s) for this fusion false positive failure may be hidden.

As shown at 414, the trainer 320 may output the trained ML classification model(s) which may be used for automatic detection of root cause to failures in vehicles such as the vehicles 202, in particular for tracing the root causes to one or more of the sub-systems 206 from which each failure is estimated to originate from. The trainer 320 may transmit the trained ML classification model(s) to one or more systems and/or services, store the trained ML classification model(s) in one or more attachable media devices for delivery to other systems and/or the like.

Reference is made once again to FIG. 1.

The RCA analyzer 220 executed by the RCA system 200 may conduct the process 100 using the ML classification model(s) created and/or trained by the trainer 320 to identify a root cause of a failure, for example, an accident, a near accident, a malfunction and/or the like in one or more at least partially autonomous vehicles 202.

As shown at 102, the process 100 starts with the RCA analyzer 220 receiving sensor data captured by one or more of the sensors 204 deployed in the vehicle 202 to sense the environment of the vehicle 202 during a certain trip, situation and/or state of the vehicle 202 which ends with a failure, for example, an accident, a near accident, a malfunction and/or the like.

As described in step 402 of the process 400, the sensor data may include one or more data formats according to the type and/or technology of the sensor(s) 204 deployed in the vehicle 202, for example, images, video streams, ranging maps, thermal maps and/or the like. The sensor data may be timestamped so that each sensor data item may be identified and allocated along a time line of the trip, situation and/or state of the vehicle 202.

As shown at 104, the RCA analyzer 220 may analyze the sensor data to identify one or more objects in the environment of the vehicle 202, for example, another vehicle, a pedestrian, a bicycle rider, a road infrastructure object, a structure and/or the like. The RCA analyzer 220 may apply one or more analysis methods, algorithms and/or tools to identify the object(s) in the sensor data as described in step 406 of the process 400.

As shown at 106, the RCA analyzer 220 may generate a failure scenario for the trip, situation and/or state of the vehicle 202 which ended with the failure describing a spatio temporal relation between each of the object(s) detected in the analysis of the sensor data with respect to the vehicle 202, in terms of distance, velocity, acceleration and/or the like. As described in step 406, the failure scenario therefore defines a time-lined motion pattern of each object with respect to the vehicle 202.

The RCA analyzer 220 may further split the time-lined motion pattern of one or more of the detected object(s) into a plurality of time intervals each spanning a limited time period. The RCA analyzer 220 may apply one or more criteria for splitting the time-lined motion pattern as described in the process 400, for example, according to one or more of the motion attributes describing the object(s) with respect to the vehicle 202. The failure scenario generated by the RCA analyzer 220 may be formulated by equation 1 as described in step 406 of the process 400.

As shown at 108, the RCA analyzer 220 may compute one or more feature vectors for the failure scenario based on the output generated by one or more of the plurality of sub-systems 206 deployed in the vehicle 202. The RCA analyzer 220 may compute scenario-wide feature vectors and/or per-interval feature vectors as describe in step 410 of the process 400. As describe in 410, since the output includes a plurality of data items generated by the sub system^) 206 of the vehicle 202, for example, status data, control data, instructions and/or the like, each of the features extracted by the RCA analyzer 220 from the output is therefore associated with one or more of the sub-systems 206. Each of the features extracted from the output may relate to the vehicle 202 and/or to one or more of the objects whose time-lined motion pattern is defined by the failure scenario created from the sensor data.

As shown at 110, the RCA analyzer 220 may apply to the feature vector(s) one or more of the trained ML classification model(s) created and/or trained by the trainer 320 executing the process 400. The trained ML classification model(s) analyzing the feature vector(s) may produce an outcome according to the internal architecture and/or path propagation adjusted during the training sessions according to the training feature vectors samples.

As shown at 112, the RCA analyzer 220 may apply one or more interpretation models as known in the art, for example, LIME and/or the like to the ML classification model(s), the outcome and/or one or more of the input feature vector(s) in order to identify one or more key features having a major contribution to the outcome of the ML classification model(s).

The RCA analyzer 220 applies the interpretation model(s) to explain the outcome of the ML classification model(s) since the predictions and/or estimations produced by the ML classification model(s) may often require further understanding of the internal workings of the ML classification model(s), the context and/or the significance of output elements in the final estimation.

The interpretation model(s), in particular, LIME, may generate an explanation of the outcome based on approximation of the underlying ML classification model by an interpretable one, for example, a linear model with only a few non-zero coefficients and/or the like which is trained on perturbations of the original input, i.e. the feature vector(s) injected to the ML classification model(s). The key intuition behind the interpretation model(s), specifically LIME is that it is may be significantly easier to approximate a black-box ML classification model(s) by replacing it with a local simple model in the neighborhood of the explained prediction, as opposed to trying to approximate a model globally. Locally approximating the ML classification model(s) to identify the key features having significantly higher contribution to the outcome of the ML classification model(s) compared to other features may be done by weighting the perturbed input features according to weather classification result of the ML classification model changes, as compared to the classification result of the ML classification model, when it is applied to the feature vector prior to the perturbation.

In order to train the LIME interpretable classification model, the RCA analyzer 220 may manipulate one or more of the feature vectors provided as input to the ML classification model(s) which thus produce new perturbed labelled feature vectors. These perturbed labelled feature vectors may be used to train the LIME interpretable classification model which compute the weights for each of the features. As such, a higher weight may indicate higher contribution (importance) of the respective feature and vice versa, a lower weight may indicate lower contribution of the respective feature.

The RCA analyzer 220 may then apply the trained LIME interpretable classification model to the ML classification model(s), the outcome of the ML classification model(s) for the failure scenario and/or to one or more of the input feature vector(s) computed for the failure scenario. The output of the trained LIME interpretable classification model may indicate one or more of the features having the highest contribution to the outcome of the ML classification model(s).

The key features,

, ,f_m having major (highest) contribution to the outcome of the ML classification model(s) may be thus highly indicative of the failure experience by the vehicle 202.

As shown at 114, the RCA analyzer 220 may estimate one or more of the sub-systems 206 of the vehicle 202 as the root cause failure sub-systems according to association of the key feature(s) with their respective sub-system(s) 206.

As described herein before, each of the features included in the feature vector(s) is associated with one or more of the sub-systems 206. The RCA analyzer 220 may therefore map each key feature to respective sub-system(s) 206.

Moreover, since the feature vectors may be associated with one or more of the time- intervals and or time stamps along the time-lined motion pattern of the other object(s), the RCA analyzer 220 may further associate each key feature with a certain time in the trip, condition and/or state of the vehicle 202. This may be formulated by A(/_£) ® (t, s ) where A(/_£) is the association of key feature /_£ to a certain sub-system 206 s at a certain timestamp and/or time interval t. Reference is now made to FIG. 9, which is a schematic illustration of tracing an autonomous vehicle failure resulting from a planning and control system to sub-system(s) of the autonomous vehicle, according to some embodiments of the present invention. A failure of an autonomous vehicle such as the autonomous vehicle 202 may result from one or more of a plurality of P&C systems configured to control operation of the autonomous vehicle 202. However, the failure of the P&C system(s) may typically result from erroneous data received by the P&C system(s) from one or more sub-systems such as the sub-systems 206 deployed in the autonomous vehicle 202.

An RCA analyzer such as the RCA analyzer 220 may execute a process such as the process 100 to estimate which of the plurality of sub-systems 206 is the root cause of the failure in the autonomous vehicle 202. In particular, the RCA analyzer 220 may apply one or more of the ML models to features extracted from output datasets comprising the data outputted from the sub-systems 206 to the P&C system(s) and further apply one or more of the interpretation models to the ML model(s) and/or to the outcome of the ML models to estimate which of the sub-systems 206 may be the root cause of the failure.

Moreover, each of the output datasets may be associated with a respective timestamp (e.g. C , t₂ , t₃, t₄, t₅) such that the respective features extracted from each output dataset are also associated with the time stamp. This may support the RCA analyzer 220 to estimate the timing and progress of the failure as it propagates through the sub-systems 206 and the P&C system(s).

Reference is now made to FIG. 10, which presents an exemplary flow for training machine learning models to map autonomous vehicle failures to root cause failure sub-systems and an exemplary flow for detecting a root cause of the autonomous vehicle using the trained ML models, according to some embodiments of the present invention. An exemplary flow 1002 for data preparation and ML model(s) training may follow the steps of the process 400 as described herein before. An exemplary flow 1004 for detecting a root cause of a failure in an autonomous vehicle such as the autonomous vehicle 202 may follow the steps of the process 100 as described herein before.

It is expected that during the life of a patent maturing from this application many relevant systems, methods and computer programs will be developed and the scope of the terms sensor, sub-system and ML classification model are intended to include all such new technologies a priori.

As used herein the term “about” refers to ± 10 %. The terms "comprises", "comprising", "includes", "including", “having” and their conjugates mean "including but not limited to".

The term “consisting of’ means “including and limited to”.

As used herein, the singular form "a", "an" and "the" include plural references unless the context clearly dictates otherwise. For example, the term "a compound" or "at least one compound" may include a plurality of compounds, including mixtures thereof.

Throughout this application, various embodiments of this invention may be presented in a range format. It should be understood that the description in range format is merely for convenience and brevity and should not be construed as an inflexible limitation on the scope of the invention. Accordingly, the description of a range should be considered to have specifically disclosed all the possible subranges as well as individual numerical values within that range. For example, description of a range such as from 1 to 6 should be considered to have specifically disclosed subranges such as from 1 to 3, from 1 to 4, from 1 to 5, from 2 to 4, from 2 to 6, from 3 to 6 etc., as well as individual numbers within that range, for example, 1, 2, 3, 4, 5, and 6. This applies regardless of the breadth of the range.

Whenever a numerical range is indicated herein, it is meant to include any cited numeral (fractional or integral) within the indicated range. The phrases “ranging/ranges between” a first indicate number and a second indicate number and “ranging/ranges from” a first indicate number “to” a second indicate number are used herein interchangeably and are meant to include the first and second indicated numbers and all the fractional and integral numerals therebetween.

It is appreciated that certain features of the invention, which are, for clarity, described in the context of separate embodiments, may also be provided in combination in a single embodiment. Conversely, various features of the invention, which are, for brevity, described in the context of a single embodiment, may also be provided separately or in any suitable subcombination or as suitable in any other described embodiment of the invention. Certain features described in the context of various embodiments are not to be considered essential features of those embodiments, unless the embodiment is inoperative without those elements.

Although the invention has been described in conjunction with specific embodiments thereof, it is evident that many alternatives, modifications and variations will be apparent to those skilled in the art. Accordingly, it is intended to embrace all such alternatives, modifications and variations that fall within the spirit and broad scope of the appended claims.

All publications, patents and patent applications mentioned in this specification are herein incorporated in their entirety by reference into the specification, to the same extent as if each individual publication, patent or patent application was specifically and individually indicated to be incorporated herein by reference. In addition, citation or identification of any reference in this application shall not be construed as an admission that such reference is available as prior art to the present invention. To the extent that section headings are used, they should not be construed as necessarily limiting. In addition, any priority document(s) of this application is/are hereby incorporated herein by reference in its/their entirety.

Claims

WHAT IS CLAIMED IS:

1. A system for automatically detecting a root cause of a failure in an autonomous vehicle, comprising: one or more processors, wherein the one or more processors are configured for: receiving sensor data captured during a predefined period preceding a failure of an autonomous vehicle by at least one sensor deployed to sense an environment of the autonomous vehicle, wherein a hidden time period immediately preceding the failure is not included in the predefined period; analyzing the sensor data to identify at least one object in the environment; creating a failure scenario defining a time-lined motion pattern of the at least one object with respect to the autonomous vehicle; computing a feature vector comprising a plurality of features extracted from an output generated by at least one of a plurality of sub-systems of the autonomous vehicle during the failure scenario; applying to the feature vector at least one machine learning classification model trained with a plurality of labeled feature vectors computed for a plurality of failure scenarios and their corresponding success scenarios; identifying at least one key feature having a major contribution to an outcome of the at least one trained machine learning classification model by applying an interpretation model to the at least machine learning classification model, the feature vector and the outcome; and estimating at least one root cause failure sub-system of the plurality of sub systems according to its association with the at least one key feature.

2. The system of claim 1, wherein the one or more processors are further configured for: splitting the failure scenario into a plurality of time intervals based on a velocity of the at least one object relative to the autonomous vehicle, computing a plurality of feature vectors each for a respective one of the plurality of time intervals, applying the at least one machine learning classification model to the plurality of feature vectors, the at least one machine learning classification model is trained with a plurality of labeled feature vectors computed for a plurality of time intervals of the failure scenarios and the corresponding success scenarios, identifying the at least one key feature by applying the interpretation model to the plurality of feature vectors, to the at least one trained machine learning classification model and to the outcome, and estimating at least one root cause failure sub-system of the plurality of sub-systems according to its association with the at least one key feature.

3. The system of claim 2, wherein the failure scenario is split into the plurality of time intervals based on variations in the velocity of the at least one object such that a new time interval is created upon each velocity change exceeding a predefined threshold.

4. The system of claim 2, wherein the failure scenario is split into the plurality of time intervals based on at least one extremum value of the velocity of the at least one object such that a new time interval is created after reaching the at least one extremum value, the at least one extremum value includes a maximum value and a minimum value.

5. The system of claim 2, wherein the failure scenario is split into the plurality of time intervals based on a polynomial approximation of the velocity of the at least one object such that a new time interval is created upon deviation of the polynomial approximation outcome from the actual velocity over a predefined threshold.

6. The system of claim 1, wherein the time-lined motion pattern includes at least one member of a group consisting of: a distance of the at least one object from the autonomous vehicle, a direction of the at least one object with respect to the autonomous vehicle, a velocity of the at least one object with respect to the autonomous vehicle and an acceleration of the at least one object with respect to the autonomous vehicle.

7. The system of claim 1, wherein each of the plurality of features relates to at least one of the autonomous vehicle and the at least one object, each of the plurality of features is a member of a group consisting of: a relative distance between the at least one object and the autonomous vehicle, a relative distance between the at least one object and at least one another object detected in the environment, a relative velocity of the at least one object with respect to the autonomous vehicle, a relative velocity of the at least one object with respect to the at least one another object, a duration of detection of the at least one object, a location of the autonomous vehicle on a road, a maneuvering operation of the autonomous vehicle, a distribution of the at least one object in spatial regions defined in the environment and an identifier of the at least one sensor whose sensor data is analyzed to identify the at least one object.

8. The system of claim 1 , wherein similarity between each of the plurality of failure scenarios and its corresponding success scenario is identified by applying at least one Dynamic Time Warping (DTW) algorithm configured to estimate similarity of at least one candidate subsequence identified in the time-lined motion pattern of the at least one object in each failure scenario and the time-lined motion pattern of the at least one object in a respective corresponding success scenario.

9. The system of claim 8, wherein the one or more processors are further configured for applying at least one multi-dimensional DTW to estimate the similarity between each failure scenario and its corresponding success in case multiple objects are detected in the respective failure scenario such that similarity is estimated between the time-lined motion pattern of each of the multitude of objects in each failure scenario and in the respective corresponding success scenario.

10. The system of claim 8, wherein the one or more processors are further configured for discarding at least one success scenario in which a relative distance of the at least one object from the autonomous vehicle in a certain failure scenario exceeds a certain threshold compared to the relative distance of the at least one object from the autonomous vehicle in the respective corresponding success scenario.

11. The system of claim 8, wherein the one or more processors are further configured for discarding at least one success scenario based on a trajectory analysis which predicts that an object other than the at least one object is estimated to collide with the autonomous vehicle.

12. The system of claim 8, wherein the one or more processors are further configured for discarding at least one success scenario which ends with a failure.

13. The system of claim 1, wherein each of the plurality of labeled feature vectors is labeled with a label “FAILURE” or a label “SUCCESS” according to its respective failure scenario or success scenario.

14. The system of claim 1, wherein the interpretation model is adapted to determine the contribution of at least some of the plurality of features to the outcome by manipulating at least one input feature vector and identifying a perturbation in the outcome.

15. A method of automatically detecting a root cause of a failure in an autonomous vehicle, comprising: receiving sensor data captured during a predefined period preceding a failure of an autonomous vehicle by at least one sensor deployed to sense an environment of the autonomous vehicle, a hidden time period immediately preceding the failure is not included in the predefined period; analyzing the sensor data to identify at least one object in the environment; creating a failure scenario defining a time-lined motion pattern of the at least one object with respect to the autonomous vehicle; computing a feature vector comprising a plurality of features extracted from an output generated by at least one of a plurality of sub-systems of the autonomous vehicle during the failure scenario; applying to the feature vector at least one machine learning classification model trained with a plurality of labeled feature vectors computed for a plurality of failure scenarios and their corresponding success scenarios; identifying at least one key feature having a major contribution to an outcome of the at least one trained machine learning classification model by applying an interpretation model to the at least machine learning classification model, the feature vector and the outcome; and estimating at least one root cause failure sub-system of the plurality of sub-systems according to its association with the at least one key feature.

16. A computer program which, when executed by a computer, causes the computer to perform the method of claim 15.