WO2024059354A1

WO2024059354A1 - Safety evaluation system for intelligent vehicle

Info

Publication number: WO2024059354A1
Application number: PCT/US2023/062678
Authority: WO
Inventors: Yixin SHEN; Jiangsheng Yu
Original assignee: Futurewei Technologies, Inc.
Priority date: 2023-02-15
Filing date: 2023-02-15
Publication date: 2024-03-21

Abstract

A method, system, and computer-readable storage medium are provided for testing a driving controller of a self-driving vehicle. The method includes applying one or more testing inputs to the driving controller to operate the driving controller in a. simulated driving setting, the one or more testing inputs including the intelligent vehicle interacting with a simulated stress vehicle; recording a behavior of the driving controller in response to the simulated, stress vehicle; and quantifying a collision risk of the behavior of the driving controller according to one or more of a direction of the simulated stress vehicle relative to the intelligent vehicle, a distance between the simulated stress vehicle and the intelligent vehicle, or a closure speed between the simulated, stress vehicle and the intelligent vehicle.

Description

SAFETY EVALUATION SYSTEM FOR INTELLIGENT VEHICLE TECHNICAL FIELD [0001] The present disclosure is related to intelligent vehicles, and more particularly, to systems and methods for evaluating the operation of intelligent vehicles. BACKGROUND [0002] Intelligent vehicles, such as self-driving vehicles or vehicles that monitor their environment to provide feedback to drivers, are becoming more commonplace. Makers of intelligent vehicles (e.g., self-driving cars, trucks, buses, etc.) need to evaluate the safety of their vehicles, such as by performing testing of vehicles to ensure safe operation in scenarios where a collision can be avoided or mitigated. An intelligent vehicle may include a driving controller or driving module that makes device-based decisions for operation of the vehicle. Such driving controller needs to be rigorously tested to ensure the intelligent vehicle performs in a way to achieve optimum driving safety over a wide range of scenarios. SUMMARY [0003] Various examples are now described to introduce a selection of concepts in a simplified form that are further described below in the detailed description. The Summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. [0004] The safety of driving modules can be evaluated by repeated independent computer simulations in an interaction scenario of interaction environment. Identifying the decisions of the driving modules and the behavior of the intelligent vehicles that may result in a collision is important in determining the risk of operating intelligent vehicles and may be useful in adjusting operation of the driving modules of the vehicles. [0005] According to one aspect of the present disclosure, there is provided a computer-implemented method for testing a driving controller of an intelligent vehicle. The method includes applying one or more testing inputs to the driving controller to operate the driving controller in a simulated driving setting; the one or more testing inputs including the intelligent vehicle interacting with a simulated stress vehicle; recording a behavior of the driving controller in response to the stress vehicle; and quantifying a collision risk of the behavior of the driving controller according to one or more of a direction of the simulated stress vehicle relative to the intelligent vehicle, a distance between the simulated stress vehicle and the intelligent vehicle, or a closure speed between the simulated stress vehicle and the intelligent vehicle. [0006] In an embodiment, the driving controller determining the direction of the simulated stress vehicle relative to the intelligent vehicle, determining the direction of the simulated stress vehicle relative to the intelligent vehicle, and determining the closure speed between the simulated stress vehicle and the intelligent vehicle. [0007] In an embodiment, the recording the behavior of the driving controller includes recording one or both of a test behavior or a response behavior of the driving controller. [0008] In an embodiment, the quantifying the collision risk includes computing a collision probability function according to one or more of the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, or the closure speed between the simulated stress vehicle and the intelligent vehicle. [0009] In an embodiment, the quantifying the collision risk includes computing a collision probability function according to the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, and the closure speed between the simulated stress vehicle and the intelligent vehicle. [0010] In an embodiment, the quantifying the collision risk includes computing the collision probability function for multiple directions of the simulated stress vehicle in relation to the intelligent vehicle. [0011] In an embodiment, the quantifying the collision risk includes computing the collision probability function according to a closure speed and a distance for a front direction, a left side direction, a right side direction, and a rear direction of the intelligent vehicle. [0012] In an embodiment, the quantifying the collision risk includes determining a threshold distance relative to the intelligent vehicle, determining whether the simulated stress vehicle is within the threshold distance, and quantifying the collision risk of the behavior of the driving controller according to the each direction. [0013] In an embodiment, the quantifying the collision risk includes determining a threshold distance for each direction of multiple directions relative to the intelligent vehicle, determining whether the simulated stress vehicle is within the threshold distance corresponding to the direction, and quantifying the collision risk of the behavior of the driving controller according to the each direction. [0014] In an embodiment, the method further includes setting the threshold distance according to the closure speed between the simulated stress vehicle and the intelligent vehicle. [0015] In an embodiment, the method further includes setting the threshold distance according to the direction of the simulated stress vehicle relative to the intelligent vehicle and the closure speed between the simulated stress vehicle and the intelligent vehicle. [0016] In an embodiment, the method further includes determining the behavior of the driving controller by inputting the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, and the closure speed between the simulated stress vehicle and the intelligent vehicle into a neural network implemented by processing circuitry of the driving controller, and changing the weighting of decision nodes of the neural network to reduce the quantified collision risk. [0017] In an embodiment, the applying the one or more testing inputs to the driving controller further includes setting the direction of the simulated stress vehicle relative to the intelligent vehicle, setting the closure speed between the simulated stress vehicle and the intelligent vehicle, and setting an initial distance between the simulated stress vehicle and the intelligent vehicle. [0018] In an embodiment, the method further includes detecting, by the driving controller during testing, a current distance between the simulated stress vehicle and the intelligent vehicle, and detecting, by the driving controller during testing, a current direction of the simulated stress vehicle relative to the intelligent vehicle. [0019] In an embodiment, the method further includes detecting, by the driving controller, a decreasing distance between the simulated stress vehicle and the intelligent vehicle. [0020] According to another aspect of the present disclosure there is provided a testing system for testing a driving controller of an intelligent vehicle. The system includes a memory storing computer-readable instructions and storing driving data recorded by the intelligent vehicle, and at least one processor in communication with the memory. The at least one processor is configured, upon execution of the instructions, to perform the steps of applying one or more testing inputs to the driving controller to operate the driving controller in a simulated driving setting, the one or more testing inputs including the intelligent vehicle interacting with a simulated stress vehicle; recording a behavior of the driving controller in response to the simulated stress vehicle; and quantifying a collision risk of the behavior of the driving controller according to one or more of a direction of the simulated stress vehicle relative to the intelligent vehicle, a distance between the simulated stress vehicle and the intelligent vehicle, or a closure speed between the simulated stress vehicle and the intelligent vehicle. [0021] Optionally in the preceding aspect, the at least one processor further executes the instructions to perform the steps of determining the direction of the simulated stress vehicle relative to the intelligent vehicle, determining the direction of the simulated stress vehicle relative to the intelligent vehicle, and determining the closure speed between the simulated stress vehicle and the intelligent vehicle. [0022] Optionally in any of the preceding aspects, the recording the behavior of the driving controller includes recording one or both of a test behavior or a response behavior of the driving controller. [0023] Optionally in any of the preceding aspects, the quantifying the collision risk includes computing a collision probability function according to one or more of the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, or the closure speed between the simulated stress vehicle and the intelligent vehicle. [0024] Optionally in any of the preceding aspects, the quantifying the collision risk includes computing a collision probability function according to the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, and the closure speed between the simulated stress vehicle and the intelligent vehicle. [0025] Optionally in any of the preceding aspects, the quantifying the collision risk including computing the collision probability function for multiple directions of the simulated stress vehicle in relation to the intelligent vehicle. [0026] Optionally in any of the preceding aspects, the quantifying the collision risk including computing the collision probability function according to a closure speed and a distance for a front direction, a left side direction, a right side direction, and a rear direction of the intelligent vehicle. [0027] Optionally in any of the preceding aspects, the quantifying the collision risk includes determining a threshold distance relative to the intelligent vehicle, determining whether the simulated stress vehicle is within the threshold distance, and quantifying the collision risk of the behavior of the driving controller according to the each direction. [0028] Optionally in any of the preceding aspects, the quantifying the collision risk includes determining a threshold distance for each direction of multiple directions relative to the intelligent vehicle, determining whether the simulated stress vehicle is within the threshold distance corresponding to the direction, and quantifying the collision risk of the behavior of the driving controller according to the each direction. [0029] Optionally in any of the preceding aspects, the at least one processor further executes the instructions to perform the steps of setting the threshold distance according to the closure speed between the simulated stress vehicle and the intelligent vehicle. [0030] Optionally in any of the preceding aspects, the at least one processor further executes the instructions to perform the steps of setting the threshold distance according to the direction of the simulated stress vehicle relative to the intelligent vehicle and the closure speed between the simulated stress vehicle and the intelligent vehicle. [0031] Optionally in any of the preceding aspects, the at least one processor further executes the instructions to perform the steps of determining the behavior of the driving controller by inputting the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, and the closure speed between the simulated stress vehicle and the intelligent vehicle into a neural network implemented by processing circuitry of the driving controller, and changing the weighting of decision nodes of the neural network to reduce the quantified collision risk. [0032] Optionally in any of the preceding aspects, the applying the one or more testing inputs to the driving controller further includes setting the direction of the simulated stress vehicle relative to the intelligent vehicle, setting the closure speed between the simulated stress vehicle and the intelligent vehicle, and setting an initial distance between the simulated stress vehicle and the intelligent vehicle. [0033] Optionally in any of the preceding aspects, the at least one processor further executes the instructions to perform the steps of detecting, by the driving controller during testing, a current distance between the simulated stress vehicle and the intelligent vehicle, and detecting, by the driving controller during testing, a current direction of the simulated stress vehicle relative to the intelligent vehicle. [0034] Optionally in any of the preceding aspects, the at least one processor further executes the instructions to perform the steps of detecting, by the driving controller, a decreasing distance between the simulated stress vehicle and the intelligent vehicle. [0035] According to another aspect of the present disclosure there is provided a computer-readable storage medium storing computer instructions that configure at least one processor, upon execution of the instructions, to perform the steps of applying one or more testing inputs to the driving controller to operate the driving controller in a simulated driving setting, the one or more testing inputs including the intelligent vehicle interacting with a simulated stress vehicle; recording a behavior of the driving controller in response to the simulated stress vehicle; and quantifying a collision risk of the behavior of the driving controller according to one or more of a direction of the simulated stress vehicle relative to the intelligent vehicle, a distance between the simulated stress vehicle and the intelligent vehicle, or a closure speed between the simulated stress vehicle and the intelligent vehicle. [0036] Optionally in the preceding aspect, the at least one processor executes the instructions to further perform the steps of determining the direction of the simulated stress vehicle relative to the intelligent vehicle, determining the direction of the simulated stress vehicle relative to the intelligent vehicle, and determining the closure speed between the simulated stress vehicle and the intelligent vehicle. [0037] Optionally in any of the preceding aspects, an implementation of the aspect provides the recording the behavior of the driving controller includes recording one or both of a test behavior or a response behavior of the driving controller. [0038] Optionally in any of the preceding aspects, an implementation of the aspect provides the quantifying the collision risk including computing a collision probability function according to one or more of the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, or the closure speed between the simulated stress vehicle and the intelligent vehicle. [0039] Optionally in any of the preceding aspects, an implementation of the aspect provides the quantifying the collision risk including computing a collision probability function according to the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, and the closure speed between the simulated stress vehicle and the intelligent vehicle. [0040] Optionally in any of the preceding aspects, an implementation of the aspect provides the quantifying the collision risk includes computing the collision probability function for multiple directions of the simulated stress vehicle in relation to the intelligent vehicle. [0041] Optionally in any of the preceding aspects, an implementation of the aspect provides the quantifying the collision risk includes computing the collision probability function according to a closure speed and a distance for a front direction, a left side direction, a right side direction, and a rear direction of the intelligent vehicle. [0042] Optionally in any of the preceding aspects, an implementation of the aspect provides the quantifying the collision risk includes determining a threshold distance relative to the intelligent vehicle, determining whether the simulated stress vehicle is within the threshold distance, and quantifying the collision risk of the behavior of the driving controller according to the each direction. [0043] Optionally in any of the preceding aspects, the quantifying the collision risk includes determining a threshold distance for each direction of multiple directions relative to the intelligent vehicle, determining whether the simulated stress vehicle is within the threshold distance corresponding to the direction, and quantifying the collision risk of the behavior of the driving controller according to the each direction. [0044] Optionally in any of the preceding aspects, the execution of the instructions further performs the step of setting the threshold distance according to the closure speed between the simulated stress vehicle and the intelligent vehicle. [0045] Optionally in any of the preceding aspects, the execution of the instructions further performs the step of setting the threshold distance according to the direction of the simulated stress vehicle relative to the intelligent vehicle and the closure speed between the simulated stress vehicle and the intelligent vehicle. [0046] Optionally in any of the preceding aspects, the execution of the instructions further performs the steps of determining the behavior of the driving controller by inputting the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, and the closure speed between the simulated stress vehicle and the simulated stress vehicle into a neural network implemented by processing circuitry of the driving controller, and changing the weighting of decision nodes of the neural network to reduce the quantified collision risk. [0047] Optionally in any of the preceding aspects, an implementation of the aspect provides the applying the one or more testing inputs to the driving controller further includes setting the direction of the simulated stress vehicle relative to the intelligent vehicle, setting the closure speed between the simulated stress vehicle and the intelligent vehicle, and setting an initial distance between the simulated stress vehicle and the intelligent vehicle. [0048] Optionally in any of the preceding aspects, the execution of the instructions further performs the steps of detecting, by the driving controller during testing, a current distance between the simulated stress vehicle and the intelligent vehicle, and detecting, by the driving controller during testing, a current direction of the simulated stress vehicle relative to the intelligent vehicle. [0049] Optionally in any of the preceding aspects, the execution of the instructions further performs the step of detecting, by the driving controller, a decreasing distance between the simulated stress vehicle and the intelligent vehicle. BRIEF DESCRIPTION OF THE DRAWINGS [0050] Some figures illustrating example embodiments are included with the text in the detailed description. [0051] FIG. 1 is an illustration of intelligent vehicles obtaining real- world driving data in an example embodiment. [0052] FIGS. 2A-2D illustrate examples of potential risk directions for an intelligent vehicle. [0053] FIG. 3 is a flowchart of a computer-implemented method of testing a driving controller of an intelligent vehicle in an embodiment. [0054] FIGS. 4A and 4B illustrate examples of behavior of a driving controller of an intelligent vehicle in response to a stress vehicle from different directions. [0055] FIG. 5 is a diagram of a computer system in an embodiment. [0056] FIG. 6 is a diagram of a computer system in an embodiment. [0057] FIG. 7 is an illustration of a testing input to a driving simulation system. [0058] FIG. 8 is a graph showing an example of a boundary condition for a threshold distance between safety and collision as a function of relative distance and relative speed of a stress vehicle with respect to an intelligent vehicle. [0059] FIG. 9 is a diagram of a computer system according to an embodiment. DETAILED DESCRIPTION [0060] In the following description, reference is made to the accompanying drawings that form a part hereof and, in which are shown, by way of illustration, specific embodiments that may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention, and it is to be understood that other embodiments may be utilized, and that structural, logical, and electrical changes may be made without departing from the scope of the present invention. The following description of example embodiments is, therefore, not to be taken in a limited sense, and the scope of the present invention is defined by the appended claims. [0061] The functions or algorithms described herein may be implemented in software in one embodiment. The software may consist of computer-executable instructions stored on computer-readable media or computer-readable storage device such as one or more non-transitory memories or other type of hardware-based storage devices, either local or networked. Further, such functions correspond to modules which may be software, hardware, firmware, or any combination thereof. Multiple functions may be performed in one or more modules as desired, and the embodiments described herein are merely examples. The software may be executed on a digital signal processor, application-specific integrated circuit (ASIC), microprocessor, or another type of processor operating on a computer system, such as a personal computer, server, or another computer system, turning such computer system into a specifically programmed machine. [0062] The functionality can be configured to perform an operation using, for instance, software, hardware, firmware, or the like. For example, the phrase “configured to” can refer to a logic circuit structure of a hardware element that is employed to implement the associated functionality. The phrase “configured to” can also refer to a logic circuit structure of a hardware element that is employed to implement the coding design of associated functionality of firmware or software. The term “module” refers to a structural element that can be implemented using any suitable hardware (e.g., a processor, among others), software (e.g., an application, among others), firmware, or any combination of hardware, software, and firmware. The term “logic” encompasses any functionality for performing a task. For instance, each operation illustrated in the flowcharts corresponds to logic for performing that operation. An operation can be performed using software, hardware, firmware, or the like. The terms “component,” “system,” and the like may refer to computer-related entities, hardware, and software in execution, firmware, or combination thereof. A component may be a process running on a processor, an object, an execution, a program, a function, a subroutine, a computer, or a combination of software and hardware. The terms “processor” and “processing circuitry” may refer to a hardware component, such as a processing unit of a computer system. [0063] Furthermore, the claimed subject matter may be implemented as a method, apparatus, or article of manufacture using standard programming and engineering techniques to produce software, firmware, hardware, or any combination thereof to control a computing device to implement the disclosed subject matter. The term “article of manufacture,” as used herein, is intended to encompass a computer program accessible from any computer-readable storage device or media. Computer-readable storage media can include, but are not limited to, magnetic storage devices, e.g., hard disk, floppy disk, magnetic strips, optical disk, compact disk (CD), digital versatile disk (DVD), smart cards, flash memory devices, among others. In contrast, computer-readable media (i.e., not limited to storage media) may additionally include communication media such as transmission media for wireless signals and the like. [0064] As noted previously herein, testing the safety of operation of intelligent vehicles is important. It is desirable for the testing to consider real- world data and take different driving scenarios into account. It is desirable to test the operation of intelligent vehicles using simulated environments to safely test the interactions of the intelligent vehicle with other vehicles and mitigate any risks of collisions from operating the intelligent vehicle. [0065] FIG. 1 is an illustration of intelligent vehicles 105 that are self- driving vehicles in an example embodiment. Each intelligent vehicle 105 includes a driving controller 115 that controls the movement of the intelligent vehicle 105 by steering, signaling, braking, accelerating, etc. The driving controller 115 is an onboard computing device that includes one or more hardware processors that execute computer-readable instructions (e.g., one or both of software and firmware) to perform the functions described. Each intelligent vehicle 105 includes sensors 110 (e.g., position sensors, time-of-flight sensors, one or more video cameras, etc.) to collect driving data. The sensors 110 can include, in general categories, position sensors, time-of-flight sensors, and directional sensors, for example. The sensors 110 can include radar, lidar, still cameras, video cameras, acoustic/ultrasonic detectors, IR sensors, magnetic sensors, optical sensors, infrared sensors, or any other suitable sensors. [0066] The driving controller 115 can use the collected driving data to collect environmental data. Driving data and environmental data as used herein includes roadway data, roadway condition data, weather data, and/or traffic data, among other things. Additional data is contemplated and is within the scope of the present description and claims. The driving and or environmental data can further include locations and movements of other vehicles, locations and movements of pedestrians, and/or locations and movements of obstacles or other objects that could pose a threat to the intelligent vehicle 105. The driving controller 115 obtains and processes the environmental data sensed using the sensors 110 of the intelligent vehicle 105 to make decisions about movement of the intelligent vehicle 105, such as to move forward, speed up, slow down, change lanes, turn left or right, stop, back-up, etc., and actuates steering mechanisms, brakes, transmission, etc., of the intelligent vehicle 105 to execute the movement of the intelligent vehicle 105. [0067] There can be a collision risk associated with operating an intelligent vehicle 105. The collision risk of the intelligent vehicle 105 colliding with another vehicle can vary based on the direction, closure speed, and/or operation in relation to the other vehicle. The testing scenarios can be classified according to the direction of the other vehicle relative to the intelligent vehicle 105 in some embodiments. The directions can be tested separately, so that the evaluations become more targeted. [0068] FIGS. 2A-2D illustrate examples of potential risk directions for an intelligent vehicle 105. In FIG. 2A, a second vehicle 207 is in front of the intelligent vehicle 105. There is a collision risk if the second vehicle 297 suddenly slows down or stops, wherein a collision may occur if the intelligent vehicle 105 does not perform an evasive maneuver, does not perform an optimal evasive maneuver, or does not timely perform an evasive maneuver. In this situation, the driving controller 115 of the intelligent vehicle 105 may need to quickly brake or take other evasive action to avoid a collision. The collision risk from the front direction of the intelligent vehicle 105 varies with the distance between the vehicles and the relative speed of the two vehicles. [0069] In FIG. 2B, the second vehicle 207 is behind the intelligent vehicle 105. There is a collision risk if the second vehicle 297 does not slow down when the intelligent vehicle slows down or comes to a stop (e.g., stopped waiting for a traffic light). The options for the intelligent vehicle 105 to avoid collision are more limited in this scenario, and the collision risk for the second vehicle approaching from behind can be higher than the scenario of FIG. 2A. [0070] In FIGS. 2C and 2D, the second vehicle 207 is on one side of the intelligent vehicle 105. In FIG. 2C, the second vehicle 207 is on the left side of the intelligent vehicle 105. In FIG. 2D, the second vehicle 207 is on the right side. The collision risk between the vehicles in the examples of FIGS. 2A-2D varies with the distance between the vehicles. A collision can occur if the second vehicle 207 moves over at least partially into the lane occupied by the intelligent vehicle 105, or could displace the intelligent vehicle 105 and cause the intelligent vehicle 105 to collide with a third vehicle. [0071] The behavior of the driving controller 115 of an intelligent vehicle 105 can be evaluated using computer simulation assessing the collision risk from autonomous operation of the intelligent vehicle in the real world, but without posing any risk of safety. The driving controller 115 generates the simulation by executing the software or firmware of the driving controller 115. Testing inputs can be applied to driving controller 115, with the testing inputs simulating interaction with a simulated stress vehicle i.e., the second vehicle 207. The interaction can be simulated by generating simulating sensor signals that are provided to the driving controller 115. The driving controller 115 makes driving decisions based on the simulated sensor signals. The interaction further comprises the driving controller 115 generating and providing driving commands (or responses) in reaction to the simulated sensor signals, i.e., the driving controller 115 generating response signals in attempts to avoid a collision or near-collision. [0072] The response signals generated by the driving controller 115 can be analyzed to assess whether the intelligent vehicle 105 has performed an optimal response. The response signals can be analyzed to assess whether the test controller 115 would avoid a collision with the simulated test vehicle (or with other nearby vehicles or objects). The response signals can be analyzed to assess whether the test controller 115 has maintained sufficient clearance distance from the simulated test vehicle. The response signals can be analyzed to assess whether the test controller 115 has reacted sufficiently quickly in response to the simulated test vehicle. [0073] The computer simulation can be used to test the response of the driving controller to different collision scenarios. The collision testing scenarios can be grouped into potential risk directions, and risk can be determined for both dynamic and static (i.e., moving and stationary) statuses of the intelligent vehicle 105. This can result in determining unsafe distance thresholds for relative speed and distance for each of the potential risk directions. [0074] FIG. 3 is a flow diagram of a computer-implemented method 300 of testing a driving controller 115 of an intelligent vehicle 105 in an embodiment. At block 305, the driving controller 115 is operated in a simulated driving setting. The simulated driving setting may include real-word driving data collected using one or more intelligent vehicles 105. The simulated driving setting can include data related to ordinary driving scenarios, including normal interactions between vehicles, interactions in parking lots, on streets, on highways, and/or on freeways. The simulated driving setting can include data regarding less ordinary (and more potentially hazardous) driving scenarios, including actions of drivers who are unaware of other vehicle traffic, road construction or repair, road hazards, pedestrians in the roadway, natural disasters, inclement weather, and other unusual or relatively unpredictable events. [0075] The data for the simulated driving setting may be stored in a local memory of the computer system implementing the method 300 or may be stored in a cloud or cloud memory. The data for the simulated driving setting may include road information recorded by the intelligent vehicle 105 whose driving controller 115 is being tested, data about other vehicles encountered by the intelligent vehicle 105, and information of non-vehicle objects such as cross walks, speed bumps, etc., encountered by the intelligent vehicle 105. Alternatively, the data may be obtained from previous testing sessions and/or from previously tested vehicles. Further, the data can comprise artificially- generated data, such as data created for implementing a test scenario. [0076] One or more hardware processors of the computer system implementing the method 300 execute the instructions of the driving controller 115 to operate the driving controller 115 in the simulated driving setting (see FIGS. 5-6 and the accompanying discussion). The instructions of the driving controller 115 may be the software or firmware of the driving controller 115. In some aspects, the driving controller 115 may be a “black box” module providing outputs in response to the inputs applied to the driving controller module. The one more one or more hardware processors apply the data of the simulated driving setting to the driving controller software or firmware to emulate the driving controller 115 operating in the simulated driving setting. [0077] At block 310, one or more testing inputs are applied to the driving controller 115. The driving controller 115 implements the simulated driving setting and implements a simulated test (or stress) vehicle in the setting, using the simulated stress vehicle to test the reactions of the driving controller 115. The testing can test the reactions of the driving controller 115 to the simulated stress vehicle, but without danger to the intelligent vehicle 105 or occupants. [0078] The simulated stress vehicle can present a danger in many ways. The simulated stress vehicle can simulate a poor driver. The simulated stress vehicle can simulate an inattentive driver. The simulated stress vehicle can simulate a dangerous change in road conditions. The simulated stress vehicle can simulate a driver lacking driving skills for a current driving scenario. The danger can comprise a danger caused by the simulated test vehicle driving too fast or too slow, driving too closely, performing sudden maneuvers, performing sudden maneuvers without sufficient warning, performing sudden maneuvers without sufficient clearance to other vehicles, pedestrians, or obstacles. [0079] As a result, the responses of the intelligent vehicle 105 can be tested, recorded, and evaluated. It can be determined whether the driving controller 115 performed a proper response action. It can be determined whether the driving controller 115 performed a response within a required response time. It can be determined whether the driving controller 115 performed a response that did not endanger other vehicles or nearby pedestrians, structures, et cetera. [0080] The testing inputs include introducing one or more stress vehicles that interact with the intelligent vehicle 105. The testing inputs can be simulated or stored sensor signals that emulate the outputs of the sensors 110 of the intelligent vehicle 105 in response to a simulated stress vehicle being simulated within the vicinity of the intelligent vehicle 105. The testing inputs can include sensor signals that emulate a stress vehicle approaching the intelligent vehicle 105, for example. [0081] The testing inputs can be changed, such as changing one or more of the direction of travel of the stress vehicle , the closure speed/speed difference between the vehicles, and/or an acceleration of the stress vehicle. Other testing inputs may include simulated non-vehicle objects (including moving and unmoving objects, such as simulated obstacles or simulated pedestrians, for example) interacting with the intelligent vehicle 105, erratic and/or unusual behavior by other vehicles, and other testable driving scenarios. [0082] At block 315, in an example, the sensor signals cause the driving controller 115 to detect a decreasing distance between the intelligent vehicle 105 and the stress vehicle (i.e., the second vehicle 207) and to determine the direction of the stress vehicle relative to the intelligent vehicle 105. [0083] At block 320, the driving controller determines a direction of the simulated test vehicle relative to the intelligent vehicle. The direction can be determined in any manner. The direction can be determined by the driving controller obtaining position coordinates of the simulated stress vehicle and determining the direction from the intelligent vehicle to the simulated stress vehicle. This obtaining can include further obtaining the position coordinates of the intelligent vehicle as well. The position coordinates can comprise position coordinates from the global positioning system (GPS), the global navigation satellite system (GLONASS), or any other suitable positioning system. The direction can be obtained by receiving radar, lidar, optical, acoustic, or other electromagnetic signal reflections from the simulated stress vehicle. [0084] At block 323, the driving controller determines a distance between the intelligent vehicle and the simulated stress vehicle. The distance can be determined from obtained position coordinates, as discussed above. Alternatively, the distance can be obtained from a distance measurement, such as from a sensor capable of measuring distance, as previously discussed. [0085] At block 326, the driving controller determines a closure speed between the simulated stress vehicle and the intelligent vehicle. The closure speed can be determined in any suitable way, such as by using a change in distance between the simulated test vehicle and the intelligent vehicle over time. Determining the closure speed can further include determining a rate of acceleration or deceleration, in addition to determining the closure speed. [0086] At block 330, a computing system records the behavior of the driving controller 115 in response to interaction with the stress vehicle. The behavior comprises one or both of a test behavior or a response behavior. The behavior in some embodiments includes the one or testing inputs supplied to the driving controller 115, with the behavior therefore capturing a test behavior. The behavior in some embodiments includes responses generated by the driving controller 115 for controlling the intelligent vehicle, with the behavior therefor capturing a response behavior. The behavior shows the actual decision making of the driving controller 115 under the simulated testing conditions. The behavior in some embodiments includes responses that are generated using one simulated stress vehicle. The behavior in other embodiments includes responses that are generated using multiple simulated stress vehicles, where the intelligent vehicle must take into account and react to the multiple simulated stress vehicles. [0087] At block 350, the collision risk as a result of the test behavior of the driving controller 115 is quantified. The risk in some embodiments is quantified according to the direction of the stress vehicle, the closure speed between the intelligent vehicle 105 and the stress vehicle 207, and/or the distance between the intelligent vehicle 105 and the stress vehicle 207. Based on the simulated testing, the programming and the operation of the driving controller 115 may be modified to adopt safer decision-making strategies. [0088] FIGS. 4A and 4B illustrate examples of behavior of a driving controller 115 in response to a stress vehicle from different directions. In FIG. 4A, the stress vehicle 207 approaches the intelligent vehicle 105 from behind. In response, the driving controller 115 of the intelligent vehicle should change speed to maintain or increase the distance to the stress vehicle 207, or perform some other appropriate action or evasive maneuver. In FIG. 4B, in this example, the intelligent vehicle 105 approaches a slower moving stress vehicle 207 (i.e., overtakes it). The driving controller 115 should change lanes to avoid a collision or near-collision with the stress vehicle 207. [0089] Given a relative distance between the intelligent vehicle 105 and the stress vehicle 207, the crash rate for a stress vehicle in a given direction can be defined as the ratio of crash scenarios to total scenarios for stress vehicles approaching or being approached from that direction. The crash rate for different directions has different distributions, and the different directions are associated with levels of risk. For example, the driving controller 115 has a better chance of avoiding front-end collisions than avoiding rear-end collisions. Driving decisions of the driving controller 115 can lead to safer decision-making strategies for the driving controller 115 operating in the real world when the driving controller 115 considers the direction of the stress vehicle 207 from the intelligent vehicle in trying to reduce the collision risk. [0090] FIG. 5 is a diagram of a computer system 500 in an embodiment. The computer system 500 can implement the method 300 of FIG. 3 in an embodiment. The computer system 500 can implement a testing system configured to test a driving controller 115 of an intelligent vehicle 105. The computer system 500 includes at least one processor 502 and a memory 504 in communication with the processor 502. The memory 504 stores driving data 506 that can be used to simulate a driving setting and/or a testing scenario. [0091] The memory 504 also includes computer-readable driving controller instructions 508 for the driving controller 115 of the intelligent vehicle 105. The driving controller instructions 508 are executed by the processor 502. The memory 504 also includes computer-readable instructions 510 that cause the processor 502 to perform the method 300 of FIG. 3. The computer system 500 can be used to set up experimental environments to test the driving controller 115 of an intelligent vehicle under-test. The testing of the driving controller 115 can be interactive. [0092] FIG. 6 is a diagram of a computer system 600 in an embodiment. The computer system 600 can implement the method of FIG.3, for example. The computer system 600 can incorporate and include the computer system 500 of FIG. 5. The computer system 600 can implement a testing system configured to test the driving controller 115 of an intelligent vehicle 105. The computer system 600 includes a user interface that includes a display 604. The computer system 600 is shown presenting an experimental environment on the display 604. The intelligent vehicle 105 is the “vehicle-under-test” in the display and the testing inputs are the stress vehicle(s) 207 that are used to test operation and reactions of the vehicle-under-test. [0093] Returning to FIG. 5, the processor 502 operates the intelligent vehicle 105 in the simulated setting by executing the driving controller instructions 508 and applying the simulated driving setting data to the driving controller 115. The processor 502 may generate video driving data of the operation of the intelligent vehicle 105 in a driving scenario using the stored driving setting data. In some examples, the stored driving setting data includes stored video driving data, and the stored video driving data can be replayed for the user. Video of the operation of the intelligent vehicle 105 can be presented to the user of the computer system 600. [0094] Through the user interface, the processor 502 can receive positions and directions of the stress vehicle 207 or vehicles relative to the vehicle-under-test. The processor 502 simulates operating the stress vehicle(s) 207 to interact with the vehicle-under-test (i.e., the intelligent vehicle 105) beginning from initial positions and directions in the simulated driving setting. As shown in FIG. 6, the computer system 500 in some embodiments generates video data that includes the one or more stress vehicles 207 interacting with the vehicle-under-test 105 in the video data and presents the video of the vehicle interactions to the user on the display 604. [0095] In some aspects, the processor 502 quantizes the collision risk of the behavior of the driving controller 115 by computing a collision probability function for a direction of the stress vehicle. The probability may include the ratio of crashes for testing inputs using the specific vehicle direction to the total amount of testing inputs for the direction. The collision probability function may also take into account the closure speed between the intelligent vehicle 105 and the stress vehicle 207, and the distance between the intelligent vehicle 105 and the stress vehicle 207. A collision probability function can be computed for multiple directions, such as for the front direction, the left side and right side directions, and the rear direction. It should be understood that each of the front, left, right, or rear directions can comprise a single direction or can comprise multiple front, left, right, or rear directions, including ranges or subsets of front, left, right, or rear directions. [0096] According to some aspects, the processor 502 quantizes the collision risk by analysis of a combination of the potential risk directions. FIG. 7 is an illustration of a testing input to a driving simulation system. The testing input is used to simulate an interaction between a stress vehicle(s) and an intelligent vehicle. The testing input is employed for the stress vehicle over multiple angles or a combination of angles. In the figure, the stress vehicle 207 cuts in front of the intelligent vehicle 105. The potential risk direction of the stress vehicle 207 is a combination of the left side direction and the front end direction of the intelligent vehicle 105, wherein the stress vehicle could potentially collide with one or both of the front or the left side of the intelligent vehicle 105. The collision risk also varies with the distance “D” between vehicles and the relative speed of the vehicles (e.g., the collision risk increases for a greater closure speed of the intelligent vehicle 105 with respect to the stress vehicle 207), with the closure speed comprising a speed the two vehicles are moving together, irrespective of their individual speeds and/or individual directions. [0097] The processor 502 of the computer system 500 quantizes the collision risk using a threshold distance between the vehicles and the closure speed between the stress and the intelligent vehicle (i.e., the closure speed between vehicles). The threshold distance for collision detection may be a function of one or more of the relative direction, relative distance, and relative speeds and closure speed of the vehicles. [0098] FIG. 8 is a graph showing an example of the boundary condition for a threshold distance between safety and collision as a function of relative distance versus closure speed of the vehicles. The processor 502 determines the collision risk of the vehicles based on whether the distance between the vehicles is less than the threshold distance determined for the relative direction, relative distance, and closure speed of the vehicles. [0099] In some examples, a machine learning algorithm can be used to determine the driving decisions made by the driving controller 115. For example, the processing circuitry of the driving controller 115 may perform software or firmware instructions to implement a neural network. The neural network may have weighted decision nodes. Inputs to the neural network may include one or more of the direction of a stress vehicle relative to the intelligent vehicle, and the relative speed and distance of the vehicles as determined by the processing circuitry from the sensor data. The output of the neural network may comprise driving decisions or driving commands such as decrease speed, change lanes, brake quickly, etc. [00100] The processor 502 may implement a machine learning algorithm to train the neural network. The quantified collision risk determined during the driving simulation can be used as input to the machine learning algorithm to change the weights of the decision nodes of the neural network implemented by the driving controller 115. The machine learning algorithm changes the weighting of the decision nodes to minimize the quantized collision risk determined from one or more emulated driving sessions. [00101] The methods, systems, and devices described herein provide techniques for testing an intelligent vehicle. The collision risk by the intelligent vehicle changes with the direction of other approaching vehicles. This risk can be quantized by emulating the operation of the driving controller of the intelligent vehicle in a simulated driving setting. The collision risk can be quantized and used to assess the driving decisions made by the driving controller. [00102] FIG. 9 is a diagram of a computer system 900 according to an embodiment. The computer system 900 is configured to perform the method embodiments disclosed herein (e.g., the method of FIG. 3) and to execute the algorithms herein, according to example embodiments. All components need not be used in various embodiments. It should be understood that the computer system 900 may comprise one or both of the computer system 500 of FIG. 5 and the computer system 600 of FIG. 6. [00103] One example of the computing system 900 may include processing circuitry (e.g., a processing unit or processor 902), memory 904, removable storage 906, and non-removable storage 908. Although the example computing device is illustrated and described as computer system 900 (or computer 900), the computing device may be in different forms in different embodiments. For example, the computing device may be a server, a router, or a virtual router. [00104] Although the various data storage elements are illustrated as part of the computer 900, the storage may also include cloud-based storage accessible via a network, such as Internet or server-based storage. Memory 904 may include volatile memory 910 and non-volatile memory 912. Computer 900 may include, or have access to, a computing environment that includes a variety of computer-readable media, such as volatile memory 910, non-volatile memory 912, removable storage 906, and/or non-removable storage 908. Computer storage can include random-access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD ROM), Digital Versatile Disks (DVD) or other optical disk storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other medium capable of storing computer-readable instructions. [00105] Computer 900 may include or have access to a computing environment that includes an input interface 914 (or user interface), output interface 916, and a communication interface 918. Output interface 916 in some embodiments includes a display device, such as a touchscreen, that also may serve as an input device. The input interface 914 in some embodiments includes one or more of a touchscreen, touchpad, mouse, keyboard, camera, one or more device-specific buttons, one or more sensors integrated within or coupled via wired or wireless data connections to the computer 900, and other input devices. [00106] The computer 900 may operate in a networked environment using a communication connection to connect to one or more remote computers, such as database servers. The remote computer may include a personal computer (PC), server, router, network PC, a peer device or other common data flow network switch, or the like. The communication connection may include a Local Area Network (LAN), a Wide Area Network (WAN), cellular, Wi-Fi, Bluetooth, or other networks. According to one embodiment, the various components of computer 900 are connected by a system bus 920. [00107] Computer-readable instructions stored on a computer-readable medium are executable by the processing unit 902 of the computer 900, such as a program instructions 922. In some embodiments, program instructions 922 comprises software to implement one or more methods described herein. Alternatively, the program instructions 922 can be stored in memory in the processing unit 902. A hard drive, CD-ROM, and RAM are some examples of articles including a non-transitory computer-readable medium, such as a storage device. The terms “computer-readable medium” and “storage device” do not include carrier waves to the extent carrier waves are deemed too transitory. Storage can also include networked storage, such as a storage area network (SAN). Computer program instructions 922 may be used to cause processing unit 902 to perform one or more methods or algorithms described herein. In some examples, the computer 900 is a testing system for driver controllers of self-driving vehicles. [00108] In an example embodiment, a testing system to test a driving controller of an intelligent vehicle includes a simulation module simulating a driving setting. The simulated driving setting includes recorded environmental data (including data recorded by the intelligent vehicle), an operational module operating the driving controller in the simulated driving setting and using the recorded environmental data, an application module applying one or more testing inputs to the operating driving controller, the one or more testing inputs including the intelligent vehicle interacting with the one or more simulated stress vehicles using the recorded environmental data, and a test module recording a behavior of the driving controller, including the behavior in response to the one or more testing inputs. In some embodiments, the testing system may include other or additional modules for performing any one of or combination of steps described in the embodiments. Further, any of the additional or alternative embodiments or aspects of the method, as shown in any of the figures or recited in any of the claims, are also contemplated to include similar modules. [00109] Although embodiments have been described in detail above, other modifications are possible. For example, the logic flows depicted in the figures do not require the particular order shown, or sequential order, to achieve desirable results. Other steps may be provided, or steps may be eliminated, from the described flows, and other components may be added to, or removed from, the described systems. Other embodiments may be within the scope of the following claims.

Claims

CLAIMS What is claimed is: 1. A computer-implemented method for testing a driving controller of an intelligent vehicle, the method comprising: applying one or more testing inputs to the driving controller to operate the driving controller in a simulated driving setting, the one or more testing inputs including the intelligent vehicle interacting with a simulated stress vehicle; recording a behavior of the driving controller in response to the simulated stress vehicle; and quantifying a collision risk of the behavior of the driving controller according to one or more of a direction of the simulated stress vehicle relative to the intelligent vehicle, a distance between the simulated stress vehicle and the intelligent vehicle, or a closure speed between the simulated stress vehicle and the intelligent vehicle.

2. The method of claim 1, further comprising the driving controller: determining the direction of the simulated stress vehicle relative to the intelligent vehicle; determining the direction of the simulated stress vehicle relative to the intelligent vehicle; and determining the closure speed between the simulated stress vehicle and the intelligent vehicle.

3. The method of any of claims 1-2, the recording the behavior of the driving controller comprising recording one or both of a test behavior or a response behavior of the driving controller.

4. The method of any of claims 1-3, the quantifying the collision risk includes computing a collision probability function according to one or more of the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, or the closure speed between the simulated stress vehicle and the intelligent vehicle.

5. The method of any of claims 1-4, the quantifying the collision risk includes computing a collision probability function according to the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, and the closure speed between the simulated stress vehicle and the intelligent vehicle.

6. The method of any of claims 4-5, the quantifying the collision risk includes computing the collision probability function for multiple directions of the simulated stress vehicle in relation to the intelligent vehicle.

7. The method of any of claims 4-5, the quantifying the collision risk includes computing the collision probability function according to a closure speed and a distance for a front direction, a left side direction, a right side direction, and a rear direction of the intelligent vehicle.

8. The method of any of claims 1-7, the quantifying the collision risk comprising: determining a threshold distance relative to the intelligent vehicle; determining whether the simulated stress vehicle is within the threshold distance; and quantifying the collision risk of the behavior of the driving controller according to the each direction.

9. The method of any of claims 1-8, the quantifying the collision risk comprising: determining a threshold distance for each direction of multiple directions relative to the intelligent vehicle; determining whether the simulated stress vehicle is within the threshold distance corresponding to the direction; and quantifying the collision risk of the behavior of the driving controller according to the each direction.

10. The method of claim 8, further comprising setting the threshold distance according to the closure speed between the simulated stress vehicle and the intelligent vehicle.

11. The method of claim 8, further comprising setting the threshold distance according to the direction of the simulated stress vehicle relative to the intelligent vehicle and the closure speed between the simulated stress vehicle and the intelligent vehicle.

12. The method of any of claims 1-11, further comprising: determining the behavior of the driving controller by inputting the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, and the closure speed between the simulated stress vehicle and the intelligent vehicle into a neural network implemented by processing circuitry of the driving controller; and changing the weighting of decision nodes of the neural network to reduce the quantified collision risk.

13. The method of any of claims 1-12, the applying the one or more testing inputs to the driving controller further comprising: setting the direction of the simulated stress vehicle relative to the intelligent vehicle; setting the closure speed between the simulated stress vehicle and the intelligent vehicle; and setting an initial distance between the simulated stress vehicle and the intelligent vehicle.

14. The method of any of claims 1-13, further comprising: detecting, by the driving controller during testing, a current distance between the simulated stress vehicle and the intelligent vehicle; and detecting, by the driving controller during testing, a current direction of the simulated stress vehicle relative to the intelligent vehicle.

15. The method of any of claims 1-14, further comprising detecting, by the driving controller, a decreasing distance between the simulated stress vehicle and the intelligent vehicle.

16. A testing system for testing a driving controller of an intelligent vehicle, the system comprising: a memory storing computer-readable instructions and storing driving data recorded by the intelligent vehicle; and at least one processor in communication with the memory, the at least one processor configured, upon execution of the instructions, to perform the following steps: applying one or more testing inputs to the driving controller to operate the driving controller in a simulated driving setting, the one or more testing inputs including the intelligent vehicle interacting with a simulated stress vehicle; recording a behavior of the driving controller in response to the simulated stress vehicle; and quantifying a collision risk of the behavior of the driving controller according to one or more of a direction of the simulated stress vehicle relative to the intelligent vehicle, a distance between the simulated stress vehicle and the intelligent vehicle, or a closure speed between the simulated stress vehicle and the intelligent vehicle.

17. The system of claim 16, the at least one processor further executing the instructions to perform the steps of: determining the direction of the simulated stress vehicle relative to the intelligent vehicle; determining the direction of the simulated stress vehicle relative to the intelligent vehicle; and determining the closure speed between the simulated stress vehicle and the intelligent vehicle.

18. The system of any of claims 16-17, the recording the behavior of the driving controller comprising recording one or both of a test behavior or a response behavior of the driving controller.

19. The system of any of claims 16-18, the quantifying the collision risk includes computing a collision probability function according to one or more of the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, or the closure speed between the simulated stress vehicle and the intelligent vehicle.

20. The system of any of claims 16-18, the quantifying the collision risk includes computing a collision probability function according to the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, and the closure speed between the simulated stress vehicle and the intelligent vehicle.

21. The system of any of claims 19-20, the quantifying the collision risk including computing the collision probability function for multiple directions of the simulated stress vehicle in relation to the intelligent vehicle.

22. The system of any of claims 19-20, the quantifying the collision risk including computing the collision probability function according to a closure speed and a distance for a front direction, a left side direction, a right side direction, and a rear direction of the intelligent vehicle.

23. The system of any of claims 16-22, the quantifying the collision risk comprising: determining a threshold distance relative to the intelligent vehicle; determining whether the simulated stress vehicle is within the threshold distance; and quantifying the collision risk of the behavior of the driving controller according to the each direction.

24. The system of any of claims 16-23, the quantifying the collision risk comprising: determining a threshold distance for each direction of multiple directions relative to the intelligent vehicle; determining whether the simulated stress vehicle is within the threshold distance corresponding to the direction; and quantifying the collision risk of the behavior of the driving controller according to the each direction.

25. The system of claim 24, further comprising setting the threshold distance according to the closure speed between the simulated stress vehicle and the intelligent vehicle.

26. The system of claim 24, further comprising setting the threshold distance according to the direction of the simulated stress vehicle relative to the intelligent vehicle and the closure speed between the simulated stress vehicle and the intelligent vehicle.

27. The system of any of claims 16-26, further comprising: determining the behavior of the driving controller by inputting the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, and the closure speed between the simulated stress vehicle and the intelligent vehicle into a neural network implemented by processing circuitry of the driving controller; and changing the weighting of decision nodes of the neural network to reduce the quantified collision risk.

28. The system of any of claims 16-27, the applying the one or more testing inputs to the driving controller further comprising: setting the direction of the simulated stress vehicle relative to the intelligent vehicle; setting the closure speed between the simulated stress vehicle and the intelligent vehicle; and setting an initial distance between the simulated stress vehicle and the intelligent vehicle.

29. The system of any of claims 16-28, further comprising: detecting, by the driving controller during testing, a current distance between the simulated stress vehicle and the intelligent vehicle; and detecting, by the driving controller during testing, a current direction of the simulated stress vehicle relative to the intelligent vehicle.

30. The system of any of claims 16-29, further comprising detecting, by the driving controller, a decreasing distance between the simulated stress vehicle and the intelligent vehicle.

31. A computer-readable storage medium storing computer instructions that configure at least one processor, upon execution of the instructions, to perform the following steps comprising: applying one or more testing inputs to the driving controller to operate the driving controller in a simulated driving setting, the one or more testing inputs including the intelligent vehicle interacting with a simulated stress vehicle; recording a behavior of the driving controller in response to the simulated stress vehicle; and quantifying a collision risk of the behavior of the driving controller according to one or more of a direction of the simulated stress vehicle relative to the intelligent vehicle, a distance between the simulated stress vehicle and the intelligent vehicle, or a closure speed between the simulated stress vehicle and the intelligent vehicle.

32. The computer-readable storage medium of claim 31, wherein the execution of the instructions further performs the steps of: determining the direction of the simulated stress vehicle relative to the intelligent vehicle; determining the direction of the simulated stress vehicle relative to the intelligent vehicle; and determining the closure speed between the simulated stress vehicle and the intelligent vehicle.

33. The computer readable storage medium of any of claims 31-32, the recording the behavior of the driving controller comprising recording one or both of a test behavior or a response behavior of the driving controller.

34. The computer readable storage medium of any of claims 31-33, the quantifying the collision risk including computing a collision probability function according to one or more of the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, or the closure speed between the simulated stress vehicle and the intelligent vehicle.

35. The computer readable storage medium of any of claims 31-33, the quantifying the collision risk including computing a collision probability function according to the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, and the closure speed between the simulated stress vehicle and the intelligent vehicle.

36. The computer readable storage medium of any of claims 34-35, the quantifying the collision risk includes computing the collision probability function for multiple directions of the simulated stress vehicle in relation to the intelligent vehicle.

37. The computer readable storage medium of any of claims 34-35, the quantifying the collision risk includes computing the collision probability function according to a closure speed and a distance for a front direction, a left side direction, a right side direction, and a rear direction of the intelligent vehicle.

38. The computer readable storage medium of any of claims 31-37, the quantifying the collision risk comprising: determining a threshold distance relative to the intelligent vehicle; determining whether the simulated stress vehicle is within the threshold distance; and quantifying the collision risk of the behavior of the driving controller according to the each direction.

39. The computer readable storage medium of any of claims 31-38, the quantifying the collision risk comprising: determining a threshold distance for each direction of multiple directions relative to the intelligent vehicle; determining whether the simulated stress vehicle is within the threshold distance corresponding to the direction; and quantifying the collision risk of the behavior of the driving controller according to the each direction.

40. The computer readable storage medium of claim 39, wherein the execution of the instructions further performs the step of setting the threshold distance according to the closure speed between the simulated stress vehicle and the intelligent vehicle.

41. The computer readable storage medium of claim 39, wherein the execution of the instructions further performs the step of setting the threshold distance according to the direction of the simulated stress vehicle relative to the intelligent vehicle and the closure speed between the simulated stress vehicle and the intelligent vehicle.

42. The computer readable storage medium of any of claims 31-41, wherein the execution of the instructions further performs the steps of: determining the behavior of the driving controller by inputting the direction of the simulated stress vehicle relative to the intelligent vehicle, the distance between the simulated stress vehicle and the intelligent vehicle, and the closure speed between the simulated stress vehicle and the simulated stress vehicle into a neural network implemented by processing circuitry of the driving controller; and changing the weighting of decision nodes of the neural network to reduce the quantified collision risk.

43. The computer readable storage medium of any of claims 31-42, the applying the one or more testing inputs to the driving controller further comprising: setting the direction of the simulated stress vehicle relative to the intelligent vehicle; setting the closure speed between the simulated stress vehicle and the intelligent vehicle; and setting an initial distance between the simulated stress vehicle and the intelligent vehicle.

44. The computer readable storage medium of any of claims 31-43, wherein the execution of the instructions further performs the steps of: detecting, by the driving controller during testing, a current distance between the simulated stress vehicle and the intelligent vehicle; and detecting, by the driving controller during testing, a current direction of the simulated stress vehicle relative to the intelligent vehicle.

45. The computer readable storage medium of any of claims 31-44, wherein the execution of the instructions further performs the step of detecting, by the driving controller, a decreasing distance between the simulated stress vehicle and the intelligent vehicle.