JP2024509086A - Agent transformation in driving simulation - Google Patents

Abstract

Described herein are techniques for performing log-based driving simulations to evaluate the performance and functionality of vehicle control systems. The simulation system may perform a log-based driving simulation that includes a playback agent that operates based on log data captured by vehicles operating within the environment. The simulation system may determine interactions associated with the playback agent and may transform the playback agent into a smart agent during a driving simulation. During a driving simulation, a playback agent transformed into a smart agent may interact with additional playback agents and cause a cascading effect of additional transformations. Converting the playback agent to a smart agent may include initiating a planning component for controlling the smart agent, which planning component is based on determining the destination and/or driving attributes based on the playback agent. obtain.

Description

The present disclosure relates to agent transformation in driving simulation.

This PCT International Application is related to U.S. Patent Application No. 17/184128, filed on February 24, 2021, and entitled “AGENT CONVERSIONS IN DRIVING SIMULATIONS”; 17/184,169, entitled ``BASED ON DRIVING LOG DATA,'' the entire contents of which are hereby incorporated by reference in their entirety for all purposes. be incorporated into.

Simulations are used to test and validate system features and functionality, including those that may be prohibited from testing in a real environment due to, for example, safety concerns, time limitations, reproducibility, etc. can be done. For example, autonomous vehicles may be used with driving simulation to test and improve the performance of vehicle control systems with respect to passenger safety, vehicle decision making, sensor data analysis, route optimization, etc. However, driving simulations that accurately reflect real-world scenarios are difficult to create because the data used to create such simulations can be noisy, inconsistent, or incomplete. Can be difficult and costly to create and implement. Additionally, running a driving simulation can involve running multiple different interacting systems and components, such as the vehicle control system being evaluated, agents and other objects within the simulation environment, and can be resource and computationally expensive. It can take a while.

The detailed description will be explained with reference to the accompanying figures. In the figures, the left-most digit of a code identifies the figure in which the code first appears. The use of the same symbols in different figures indicates similar or identical components or features.

3 illustrates an example process for performing a log-based driving simulation and converting a playback agent into a smart agent during the simulation in accordance with one or more embodiments of the present disclosure. 4 illustrates an example technique for determining interactions between agents during a log-based driving simulation in accordance with one or more embodiments of the present disclosure. 4 illustrates an example approach for determining interactions between agents during a log-based driving simulation in accordance with one or more embodiments of the present disclosure. 4 illustrates an example technique for determining interactions between agents during a log-based driving simulation in accordance with one or more embodiments of the present disclosure. 3 depicts scenes at four different times during a log-based driving simulation in accordance with one or more embodiments of the present disclosure. 4 depicts scenes at four different times during a log-based driving simulation in accordance with one or more embodiments of the present disclosure. 4 depicts scenes at four different times during a log-based driving simulation in accordance with one or more embodiments of the present disclosure. 4 depicts scenes at four different times during a log-based driving simulation in accordance with one or more embodiments of the present disclosure. FIG. 3 illustrates scenes at four different times during another log-based driving simulation in accordance with one or more embodiments of the present disclosure. FIG. 4 illustrates scenes at four different times during another log-based driving simulation in accordance with one or more embodiments of the present disclosure. FIG. 3 illustrates scenes at four different times during another log-based driving simulation in accordance with one or more embodiments of the present disclosure. FIG. 4 illustrates scenes at four different times during another log-based driving simulation in accordance with one or more embodiments of the present disclosure. 1 is a block diagram of a computing environment that includes a vehicle and a simulation system in accordance with one or more embodiments of the present disclosure. FIG. FIG. 2 is a flow diagram illustrating an example process for converting a playback agent into a smart agent during a driving simulation in accordance with one or more embodiments of the present disclosure. 2 is a flowchart illustrating an example process for cascading playback agents to smart agents during driving simulation in accordance with one or more embodiments of the present disclosure.

Discussed herein are techniques for performing log-based driving simulations to evaluate the functionality and performance of vehicle control systems. Some techniques described herein perform log-based driving simulations using a simulation system configured to run a vehicle control system and evaluate the response of the vehicle control system to a dynamic simulation environment. including doing. In a log-based driving simulation, the simulation environment may include a playback agent that operates based on log data associated with a corresponding agent observed in a real (eg, non-simulated or physical) environment. For example, a real (e.g., non-simulated) vehicle traversing an environment may store log data collected by the vehicle's sensors and/or perception systems, including those of other vehicles, bicyclists, pedestrians, etc. Log data representing agents and other objects observed by vehicles such as persons may be included. A simulation system may generate and run a log-based simulation based on the log data, in which playback agents are generated and provided to correspond to agents observed in a real environment. Ru. Each playback agent may be based on a corresponding agent represented in the log data, and may operate in a similar or identical manner in the simulation to a corresponding agent observed in a real environment.

During execution of a log-based driving simulation, the simulation system, instead of or in addition to the log data, converts one or more playback agents into "smart agents" that can be controlled by the planning component. There is. Unlike a playback agent, a smart agent in a simulation may react to the simulation environment and make behavioral decisions or actions that have deviations from the behavior of the corresponding playback agent in the log data. For example, a smart agent may detect potential interactions (eg, collisions) with other agents in the simulation and change speed or trajectory to avoid the interaction. Playback agents are limited to logged agent actions and may not have independent decision-making capabilities.

In some examples, the simulation system may monitor log-based simulation execution to determine interactions that have occurred or will occur between the playback agent and other agents during the simulation. be. In response to detecting an interaction involving a playback agent, the simulation system may convert the playback agent to a smart agent, and the smart agent may perform actions as necessary within the simulation to prevent the interaction. It allows you to control the route. After a playback agent is converted into a smart agent, it may have deviations from the behavior of the corresponding playback agent in the log data. As the driving simulation continues, such deviations result in the smart agent interacting with additional playback agents, which causes the simulation system to assign additional playback agents to the smart agent based on the interaction. Convert. As a result, there may be a cascading effect in which playback agents are transformed into smart agents during the execution of a driving simulation.

Additionally, in some implementations, the simulation system may analyze the log data to determine behaviors, attributes, and/or destinations associated with the playback agents within the log data. Once the playback agent is converted to a smart agent, the simulation system may execute a planning component to control the operation and/or operation of the smart agent. In some cases, execution of the planning component for the smart agent may include initiating and/or controlling the planning component based on the determined destination and/or driving behavior/attributes of the associated playback agent. For example, the simulation system analyzes the route taken by the playback agent and the driving behaviors and actions performed by the playback agent during the route to improve one or more driving styles of the playback agent and/or the driver's driving style. Personality types (eg, aggression metrics, driving skill metrics, reaction time metrics, legal compliance metrics, etc.) may be determined. The simulation system includes a planning component to control the smart agent based on the destination of the playback agent and/or based on parameters corresponding to the driving style and/or personality of the driver indicated by the playback agent. can be configured.

The techniques described herein can be implemented in many ways. An example implementation is provided below with reference to the figures. Although particular examples are described in the context of autonomous vehicles and log-based driving simulations, in other examples the methods, apparatus, systems, computer-readable media, and/or other implementations described herein may be used. , use in semi-autonomous or non-autonomous vehicles, and/or use within an aviation or navigational context.

FIG. 1 illustrates an example process 100 for performing a log-based driving simulation that includes converting a playback agent into a smart agent at one or more times during the simulation. In various implementations, some or all of the operations of process 100 may be performed by a driving simulation system, such as the simulation system described in more detail below.

At operation 102, the simulation system may receive log data associated with a vehicle traversing an environment. In various examples, log data may include data stored by a real or simulated vehicle traversing a real or simulated environment. As shown in box 104, vehicle 106 may traverse an environment that includes multiple agents 108 and/or other additional objects. Log data may include characteristics of the environment, agents 108 detected within proximity of vehicle 106, attributes or characteristics of the environment, agents, and/or other objects (e.g., classification, size, shape, position, yaw, velocity, trajectory). etc.) may include data observed and/or perceived by the vehicle 106. In some cases, log data includes information about the actions and/or behavior of agents 108 and other objects, as well as accidents or near-accidents, traffic violations, pedestrians, bicyclists, or animals crossing or ignoring traffic lights, weather abnormalities, construction zones, etc. It may include events observed by the vehicle 106, such as detours, school zones, etc. These specific events and actions may be designated from a list of events/actions that are desirable to be used as a driving simulation.

The log data received in operation 102 includes data based on raw sensor data and/or sensor data associated with vehicle 106, such as bounding boxes, speed, classification, object dimensions, predictions, object tracking information, control signals, etc. obtain. Examples of data generated by vehicle 106 that may be included in log data include, for example, U.S. patent application Ser. No. 15/693,700 entitled “Occupant Protection System Including Expandable Curtain and/or Expandable Bladder,” filed September 1, 2017; US patent application Ser.

In some implementations, the simulation system may generate one or more log-based driving simulations based on the log data received in operation 102. To generate a log-based driving simulation, the simulation generator may use the log data received in operation 102 to generate simulation instructions that, when executed by the simulator, generate a simulated environment. . Additional log data that may be used to generate log-based driving simulations includes perceptual data, predictive data, and/or state data indicative of diagnostic information, trajectory information, and other information generated by vehicle 106. be able to. In some cases, the simulation generator generates simulation scenarios by identifying static and/or dynamic objects in the environment (e.g., agents 108) and various attributes and behaviors of the objects in the log data. It is possible. In some examples, when generating a log-based simulation, the simulation system may exclude objects represented in the logs that do not meet or exceed a threshold level of interaction with the vehicle 106. By excluding objects that do not meet or exceed a threshold level of interaction with the vehicle 106, the simulation system can reduce the amount of computational resources required to generate and run the log-based driving simulation. can.

In operation 110, the simulation system executes a log-based driving simulation that includes a playback agent based on the agent 108 detected in the log data. In various instances, log-based driving simulations are used to test and validate the response of a simulated vehicle to various simulated scenarios that are similar or identical to those that may be encountered in the real world. can be done. For example, a log-based driving simulation may use various simulated environments, objects, and/or agents to model normal or abnormal driving and/or vehicle conditions. Log-based driving simulations can model different traffic conditions, environmental conditions, road obstacles, accidents, etc. to test and verify passenger safety, vehicle routing, decision making, efficiency, etc. Certain driving simulations may also test the simulated vehicle's response to defective and/or malfunctioning sensors on the vehicle. Certain driving simulations may test individual components or systems of the simulated vehicle (e.g., sensor data perception components, decision-making or planning components, etc.); other driving simulations may test various components or systems of the vehicle. The entire simulated vehicle, including interactions between components or systems, may be tested.

During execution of a log-based driving simulation, the simulation system may execute a set of simulation instructions to simulate an environment similar or identical to the environment represented by the log data. As shown in box 112, during a log-based simulation, the simulated vehicle 114 may be controlled by one or more vehicle control systems (or vehicle controllers), which The behavior of the simulated vehicle 114 may be controlled in response to the simulated environment, objects, and agents. In some cases, the simulator component of the simulation system executes the simulation scenario and represents the motion and behavior of the simulated vehicle 114 during the simulation to the process executing the vehicle controller to the process executing the simulation scenario. Data may be provided representing a simulated environment and objects to which the user may respond by providing data.

At the beginning of a log-based driving simulation, each of the simulated agents may initially be a playback agent 116. As discussed above, the playback agent may be configured to operate based on previously collected log data within a non-simulation environment, and thus the playback agent 116 within the simulation may may operate in a similar or identical manner to that of its corresponding agent. Because the previously collected log data is fixed and unchanging, neither the simulated environment nor any objects or agents within the simulation react or respond to the behavior of the simulated vehicle 114 or other objects within the simulation. It may not be configured as such. Accordingly, each agent depicted in box 112 may be a playback agent 116 at the beginning of a log-based simulation and may operate in a fixed, predetermined manner.

In contrast, a simulated vehicle 114 in a log-based driving simulation may behave differently than an agent 108 that collected log data associated with a non-simulated environment. In some examples, agent 108 and simulated vehicle 114 may be different vehicle types (and/or different agent types). Even if agent 108 and simulated vehicle 114 are both implemented as autonomous vehicles, simulated vehicle 114 may include a different vehicle controller than that of agent 108. For example, simulated vehicle 114 may include a variety of different and/or updated software systems (e.g., perception components, planning components, etc.) with respect to the software and/or hardware systems used by agent 108. , or may include different sets of vehicle capabilities, different planning/routing algorithms, or different sensor systems and/or sensor configurations. Therefore, the simulated vehicle 114 may respond differently within a log-based simulation than the agent 108 that captured the log data. Such differences between the vehicle controller and/or other vehicle software or hardware systems of the agent 108 and the simulated vehicle 114 may cause the behavior of the simulated vehicle 114 in the log-based simulation to differ from the behavior of the agent 108. There may be deviations. In this example, a simulated vehicle 114 is drawn within a box 112 at one location within the simulation scene, and a vehicle outline 118 is drawn at another location corresponding to the location of the agent 108 within the log data at the same relative time. It shows the location.

Simulated vehicle 114 can be, for example, an autonomous vehicle that is tested in a simulation by determining the response of simulated vehicle 114 to a simulated agent (such as agent 108). In the simulation, various aspects of the simulated vehicle 114 may be changed (e.g., with respect to perception, prediction, and/or planning components), for example, to test new versions of components. . Log data can be used in simulations to recreate real-world driving conditions and agent interactions. However, if the simulated vehicle 114 behaves differently than that recorded in the log data, the agent created from the log data may also interact differently. Maintaining the integrity of the simulation means preserving the real-world behavior and interactions of the agent (e.g., playback agent) while accounting for differences in the interaction of the simulated vehicle 114 with the environment. (e.g., via smart agents).

At act 120, the simulation system may determine whether any of the playback agents 116 within the driving simulation interact with non-playback agents that have a deviation. As used herein, a deviation agent may refer to an agent, such as a simulated vehicle 114 or a smart agent, whose state in the driving simulation differs from the state of the corresponding agent in the log data. . For example, the position of the simulated vehicle 114 is different from the corresponding position of the agent 108 in the log data (indicated by the vehicle outline 118), so in this example, if the simulated vehicle 114 has a deviation, may be identified. The determination of whether an agent has a deviation may be based on a comparison of position, as in this example, and/or yaw between the simulated agent and the corresponding agent in the log data. It may also be based on comparisons of other vehicle attributes or conditions such as speed, trajectory, etc. Additionally, as described below, the simulation system uses deviation distance thresholds, other types of threshold comparisons (e.g., yaw deviation thresholds), and/or a combination of multiple deviation criteria to determine whether the agent has a deviation. may determine whether it is classified as doing so.

The simulation system may periodically and/or continuously perform the determination during execution of the log-based simulation in operation 120, and the interaction between the playback agent 116 and the different deviation agents may occur during the simulation. may be determined at any time. To determine interactions between agents, the simulation system may use bounding boxes and/or other techniques (eg, path polygons, corridors, etc.) to represent the positions of the agents at various times during the simulation. In some examples, an agent's bounding box may be determined for any point in time during the simulation based on the agent's size, position, yaw, and/or velocity at that time. The simulation system may determine interactions between two agents based on the overlap between the bounding boxes of the two agents at a particular point in time during the simulation. Thus, an interaction may represent a collision, or a potential collision or near miss, between two agents at the point where the bounding boxes of the agents overlap.

If the simulation system determines that the playback agent 116 interacted (or will interact) with the deviation agent during the simulation (120: YES), in operation 122 the simulation system avoids the interaction. Playback agent 116 may be converted to a smart agent for this purpose. For example, as shown in box 124, the simulation system has converted the playback agent directly behind the simulated vehicle 114 into a smart agent 126. This transformation indicates that the associated playback agent 116 has interacted (e.g., collided) with a simulated vehicle 114 that has a deviation from the expected position based on log data (e.g., vehicle profile 118). may be performed based on a determination that the When converting an interacting playback agent 116 into a smart agent 126, the simulation system may execute or initiate a planning component for the smart agent 126, using the planning component to create a simulated environment. A path for smart agent 126 through the simulated environment may be analyzed and determined. In various examples, the planning component used by smart agent 126 may be the same as the planning component used by simulated vehicle 114, and may be configured to avoid interactions with minimal trajectory adjustments. It may be a simplified version of the configured component. For example, if smart agent 126 is moving faster than the agent immediately preceding it, at a predetermined distance threshold, the planning component may reduce the speed of smart agent 126 to match the speed of the agent immediately preceding it. It's okay. In other cases, the component may cause smart agent 126 to brake, steer, pause, accelerate, or perform other navigational maneuvers in the simulation to avoid interactions with other agents.

As mentioned above, at the start of a log-based driving simulation, some or all of the agents in the simulation may be playback agents 116 that act based on log data and/or do not include a planning component. It's okay. However, the simulated vehicle 114 may include a different vehicle controller and/or configuration that causes deviations from the behavior of the corresponding agent 108 in the log data. Deviations in the simulated vehicle 114 may cause the simulation system to convert one or more of the playback agents 116 to smart agents. After the playback agent 116 is converted to a smart agent 126, it may begin using a planning component to navigate the simulated environment, rather than relying solely on log data. Therefore, as the driving simulation continues, the smart agent 126 may also have deviations from the corresponding agent 108 in the log data. During this period of driving simulation, both the simulated vehicle 114 and the smart agent 126 may be deviation agents, and both may interact with additional playback agents 116 and be converted to smart agents. . In some instances, this may result in a cascading effect of the conversion of playback agents to smart agents, which may last for a period of time until it lapses locally within the simulation. It may continue until the end.

If there are no playback agents to be converted to smart agents at the current time during the simulation (120: NO), then in operation 128 the simulation system may determine whether a termination condition for the simulation is met. Simulation termination conditions may include, for example, a successful termination of the simulation, a component timeout, a conflict or error occurring within the simulation, or manual termination of the simulation by a user. If the simulation termination condition has not occurred (128: NO), the process 100 returns to operation 110 and continues the simulation. However, if a simulation termination condition occurs (128: YES), the simulation system may terminate the simulation and output or store the simulation results in operation 130. In some examples, the simulation results may identify the behavior and/or performance of the simulated vehicle 114 during the simulation, which corresponds to the behavior and/or performance of the vehicle controller being evaluated. It's okay. If the driving simulation is one of a batch or series of simulations, the process 100 returns to operation 102 to perform the next batch or series of simulations using the same log data and the same vehicle controller for the simulated vehicle 114. A simulation may also be performed.

As illustrated in these and other examples, the techniques described herein provide many improvements and technical advantages for generating and performing driving simulations. These benefits include providing log-based simulations that are more robust and durable with respect to vehicle control changes being tested, and providing simulations that more accurately represent real-world driving scenarios. included. For example, in some simulation systems, if a log-based simulation results in a collision between a simulated vehicle and a playback agent, the simulation may be invalidated and discarded from the simulation test battery. As a result, log-based tests in such systems are often short-lived and quickly become obsolete, as changes to the vehicle controller of the simulated vehicle can invalidate the simulation. There is a possibility that it will be stored away. In contrast, the techniques described herein involve selectively converting playback agents into smart agents, resulting in fewer failures, less need for intervention and manual analysis, and longer lasting This provides a more robust and durable simulation that can operate over a long period of time, thereby increasing the number of available simulation tests and improving the efficiency and quality of the simulation system.

Additionally, some conventional systems run simulations that include multiple (or exclusively) smart agents rather than playback agents. In such systems, smart agents can successfully avoid collisions during simulation, but in some cases may provide a more realistic and less valuable driving simulation. For example, because the playback agent is based on real-world log data, it provides a more authentic scenario that simulates an environment with real drivers, who may drive in unexpected and unpredictable ways. playback agents may be more advantageous than smart agents in simulations. Furthermore, if playback agents are converted to smart agents during log-based simulation, the simulation system may have limited sensor and sensory data to control the smart agents. Additionally, both the conversion of playback agents into smart agents and the planning component of the smart agent's behavior during the simulation may require additional computing resources, reducing the speed and efficiency of the simulation system. The number of simulations that can be run is limited. Therefore, the approach described herein is limited and selective, converting playback agents into smart agents only when needed during simulation, allowing more time and more time during simulation. A number of playback agents may be maintained.

2A-2C illustrate an approach to detecting and analyzing agent interactions within a log-based driving simulation. As described above, in some implementations, the simulation system performs a process during a driving simulation based on determining that an interaction has occurred between a playback agent and a non-playback agent that has a deviation within the simulation. The playback agent may be changed to a smart agent. In this example, FIGS. 2A-2B illustrate an example of detecting interactions between two agents in a driving simulation using trajectories and bounding boxes associated with the two agents. FIG. 2C shows an image associated with a non-playback agent (e.g., a simulated vehicle or another smart agent) with a deviation corresponding to the distance between the agent in the driving simulation and the corresponding agent in the log data. Determine the deviation distance.

2A and 2B each illustrate driving simulations 200 and 218, which may be different simulations or part of the same simulation. In these examples, the simulation system is configured to determine interactions representing potential conflicts between a simulated vehicle traveling along a trajectory and a playback agent traveling along a different trajectory. can be done. Although only two vehicles are discussed in these examples, in other examples the simulation system may similarly analyze trajectory, path, bounding box overlap, and/or Other techniques described herein for determining interactions between two agents may be implemented. Such interactions may include interactions between a playback agent and a simulated vehicle or between a playback agent and a smart agent.

As shown in this example, to determine interactions within the driving simulation, the simulation system may determine the location, trajectory, size, and/or spatial region associated with the vehicle within the driving simulation. , size and/or spatial area, along with vehicle trajectories, may be used to determine the likelihood or likelihood of interaction between vehicles. As shown in FIG. 2A, simulation 200 represents an interaction analysis between a simulated vehicle 202 traveling along a trajectory 204 and a playback agent 206 traveling along a trajectory 208. In some examples, the simulation system may determine a number of bounding boxes 210 and 212 associated with simulated vehicle 202 and additional bounding boxes 214 and 216 associated with playback agent 206. . The simulation system determines each bounding box 210-216 based on the dimensions (e.g., length, width, and height) and shape of each vehicle, as well as the trajectory of each vehicle and the resulting orientation of the vehicle. You may.

In some examples, bounding boxes 210-216 may also include a safety buffer that adds to the projected orientation area of the respective vehicle. The simulation system may determine the size and shape of the safety buffer to add to bounding box 210 based on the vehicle classification, speed, yaw, and/or other attributes of each vehicle. For example, the simulation system may determine the playback agent's 206 position based on the playback agent's 206's perceived edges (frontmost and last points, leftmost and rightmost points) along the playback agent's 206's perceived trajectory. Bounding boxes 214 and 216 may be dimensioned to include additional safety buffers around the perceived edges. In various examples, a larger safety buffer may be due to faster vehicles, more vulnerable vehicles/objects (e.g., bicycles or pedestrians), or because the simulation system has a larger safety buffer due to the size, shape, trajectory, or other state parameters of the playback agent 206. It can be used for scenarios where the perceptual data on the sensor is unreliable.

In FIG. 2A, the positions of simulated vehicle 202 and playback agent 206 may be depicted at a particular (e.g., current) time during a driving simulation, and bounding boxes 210-216 are depicted at the current time. Shown as projections at two subsequent time intervals. In this example, bounding boxes 210 and 212 are drawn for simulated vehicle 202 at a first time (t=1) and a second time (t=2) after the current time, respectively. Similarly, bounding boxes 214 and 216 are drawn for playback agent 206 at a first time (t=1) and a second time (t=2), respectively. As shown in this example, the simulation system also considers predicted vehicle maneuvers (e.g., turns) and vehicle orientation and Corresponding effects on position may also be considered.

In some examples, the simulation system compares the bounding box associated with the simulated vehicle 202 and the bounding box associated with the playback agent 206 at each time interval to determine whether the simulated vehicle It may be determined whether there is an overlap that may indicate an intersection between 202 and playback agent 206. In this example, the simulation system may compare the size, shape, and position of bounding boxes 210 and 214 and determine that there is no overlap at a first time (t=1); Bounding boxes 212 and 216 may be compared to determine that there is no overlap at a second time (t=2). Accordingly, in the example simulation 200, the simulation system may determine that no intersection exists between the simulated vehicle 202 and the playback agent 206.

In contrast, FIG. 2B shows an interaction analysis of a second simulation 218 between a simulated vehicle 220 traveling along a trajectory 222 and a playback agent 224 traveling along a trajectory 226. There is. Similar to the example above, at each time interval, the simulation system compares the bounding box associated with simulated vehicle 220 and the bounding box associated with playback agent 224 to determine if an overlap exists. , which may indicate an intersection between simulated vehicle 220 and playback agent 224 . In this example, the simulation system may generate a set of bounding boxes 228-230 for simulated vehicle 220 and another set of bounding boxes 232-234 for playback agent 224. The simulation system may then compare the size, shape, and position of bounding boxes 228 and 232 to determine whether an overlap exists at a first time (t=1), and similarly , bounding boxes 230 and 234 may be compared to determine whether an overlap exists at a second time (t=2). As shown in this example, simulated vehicle 220 of simulation 218 may be traveling faster than simulated vehicle 202 of simulation 200. However, the velocity and trajectory of playback agent 224 may remain the same as the velocity and trajectory of playback agent 206 in simulation 200. As a result, an overlap exists between bounding boxes 230 and 234, as shown in FIG. gender) is indicated.

Although the above example described determining interactions by projecting bounding boxes at discrete time intervals, the simulation system may implement various other techniques in other examples. For example, in some cases, the simulation system may determine projected path polygons or freeform corridors for each vehicle simulation 200 and 218 based on the respective trajectories of the vehicles, and the overlapping path polygons (or corridors). A spatio-temporal overlap analysis may be performed within the potential collision zone determined based on. See, for example, U.S. patent application Ser. As described in further detail in , the simulation system determines potential collision zones between the vehicles based on intersection points between the vehicles' trajectories and one or more offset distances associated with the vehicles. It's okay. The simulation system may determine the offset distance based on the length and/or width of the vehicle and may apply a safety buffer or other distance that represents a safe distance from which the vehicle will not crash. For example, the simulation system may calculate an offset distance that is used to define the dimensions of the potential collision area based on the overlap of each vehicle's predicted path of travel, and measurements are taken at the intersection of the vehicles' trajectories. is executed for the points before and after. In various examples, the size of the travel corridor may be measured from the center point of each vehicle and/or from the foremost and last points of each vehicle along the trajectory. The simulated vehicle may also consider vehicle maneuvers (eg, turns) and corresponding effects on the vehicle's position when calculating the offset distance of a potential collision area.

In some examples, after an interaction between a playback agent (e.g., playback agent 224) and a non-playback agent (e.g., simulated vehicle 220) in a driving simulation, the simulation system A deviation distance of a non-playback agent associated with a non-playback agent may be determined. An example of deviation distance determination is illustrated for simulation 218 in FIG. 2C. In this example, bounding boxes 230 and 234 are shown depicting the interaction between simulated vehicle 220 and playback agent 224 at a second time (t=2) in simulation 218. Further, FIG. 2C depicts the simulated vehicle position 236 and the corresponding agent vehicle position 238 as represented in the log data at a second time (t=2). As shown in this example, the simulated vehicle 220 deviates a distance 240 from its position in the log data because the function/behavior of the simulated vehicle 220 differs from the function/behavior of the corresponding vehicle in the log data. It's off by just that. As mentioned above, the simulated vehicle may have deviations from the log data based on different versions of the vehicle controller software and/or hardware system, and the smart agent may deviate from the driver's decisions in the log data. You can have deviations from the log data based on the planning component's decisions. However, in some instances, the playback agent's operations for navigating within the simulated environment may be solely based on logged data, and therefore, in such instances, the playback agent may There may be no deviation from the position in the data.

As shown in FIG. 2C, because the position 236 of the simulated vehicle 220 differs from the position 238 of the corresponding vehicle in the log data at the same relative time, the simulation system detects that the simulated vehicle 220 has a deviation. It was decided that it had. In some examples, to determine whether the agent has a deviation, the simulation system may compare the deviation distance 240 to a deviation distance threshold (e.g., 0.3 meters), and the deviation distance An agent that has a deviation greater than or equal to a threshold may be determined to be an agent that has a deviation, and an agent that has a deviation that is less than a repayment separation threshold may be determined to be an agent that does not have a deviation. . In some examples, a deviation distance threshold of zero may be used such that any deviation between the agent's location and the corresponding log data is classified by the simulation system as a deviation agent.

Additionally, although this example describes determining the deviation agent based on the amount of deviation distance 240, in other examples, determining the deviation agent may determine the agent's yaw deviation, and determine the yaw deviation of the agent. It may include comparing to a yaw deviation threshold. The deviation threshold for deviation distance, yaw, and/or other attributes of the agent (eg, velocity, trajectory, etc.) may optionally be a predetermined fixed threshold. In other cases, the simulation system may use different deviation thresholds for velocity-based agents and/or different types of agents. For example, pedestrian agents, bicycle agents, car agents, truck agents, etc. may each have different deviation distance and/or yaw deviation thresholds to determine which agents are considered deviation agents.

3A-3D depict four different scenes within a log-based driving simulation 300. Scene 302 (FIG. 3A) may represent a state of simulation 300 at a first time (t=0), and scene 304 (FIG. 3B) may represent a state of simulation 300 at a second time (t=1). In this case, scene 306 (FIG. 3C) may represent the state of simulation 300 at a third time (t=2), and scene 308 (FIG. 3D) may represent the state of simulation 300 at a fourth time (t=3). obtain. Simulation 300 includes a simulated vehicle 310 driving within a simulated environment that includes a number of additional agents and other objects. As discussed below with reference to individual scenes 302-308, simulated vehicle 310 in this example does not interact with any playback agents within simulation 300, and therefore simulation 300 Does not result in the transformation of the agent into a smart agent.

As shown in scene 302, a simulated vehicle 310 is stopped at a crosswalk before an intersection in a simulated environment that includes a number of additional agents and other objects. Playback agent 312 is a vehicle directly behind simulated vehicle 310 and is approaching an intersection while moving toward simulated vehicle 310 .

In scene 304, playback agent 312 is stopped directly behind simulated vehicle 310, and a second playback agent 314 approaches playback agent 312 from behind as it approaches an intersection.

In scene 306, simulated vehicle 310 and playback agent 312 remain stopped at the intersection, with additional playback agents 314 and 316 also stopped alongside playback agent 312 at the intersection. Agents 310-318 in scene 306 may be at an intersection, for example, at a stop sign, waiting for the light to change, waiting for the crosswalk to break for a pedestrian to cross, etc. There is. Additionally, a bicycle playback agent 318 and a pedestrian playback agent 320 are shown in scene 306, also waiting to cross the intersection in an upward direction.

In scene 308, simulated vehicle 310 begins moving and is making a left turn at an intersection. Playback agents 312, 314, and 316 are also advancing toward the intersection, and bicycle playback agent 318 and pedestrian playback agent 320 are advancing to cross the road in an upward direction.

During the simulation 300, the simulated vehicle 310 does not interact with any of the playback agents 312-320 and, as a result, does not interact with any of the playback agents 312-320 (or any other agent depicted in the simulation). (none of them) are converted to smart agents. In some cases, the simulation 300 indicates that the simulated vehicle 310 does not have a significant deviation (or no deviation at all) from the driving path/behavior of the corresponding vehicle in the log data on which the simulation 300 is based. ) may represent a simulation. In other cases, the simulated vehicle 310 may have a deviation from the corresponding vehicle in the log data, but not enough to cause an interaction with any of the playback agents 312-320 within the simulation 300. There is no deviation or there is a possibility that there is no deviation.

4A-4D illustrate four different scenes within another driving simulation 400 similar to the driving simulation 300 described above. For example, driving simulation 400 may be based on the same log data as driving simulation 300 and may include the same simulated environment, the same simulated objects, and the same initial set of playback agents. However, as discussed below with reference to individual scenes 402-408, simulated vehicle 410 in this example behaves differently than simulated vehicle 310 in simulation 300. Differences in the behavior of the simulated vehicle 410 occur as a result of the interaction with the playback agent, causing the simulation system to perform a cascading sequence of transformation of the playback agent into a smart agent.

As shown in scene 402, simulated vehicle 410 is stopped at the same crosswalk before the same intersection shown in simulation 300. However, simulated vehicle 410 in this example is stopped several feet in front of simulated vehicle 310 in the previous example. Unlike simulated vehicle 310, simulated vehicle 410 in this example comes to a complete stop before the crosswalk in front of the intersection. Playback agent 412 is a vehicle directly behind simulated vehicle 410 and is approaching an intersection while moving toward simulated vehicle 410 .

In scene 404, playback agent 412 followed the same path as playback agent 312 in the previous example. However, because the simulated vehicle 410 stopped several feet earlier than in the previous example, the simulation system determines that an interaction will occur between the playback agent 412 and the simulated vehicle 410 with a deviation. did. The interaction in this example is the possibility of a rear-end collision between the playback agent 412 and the simulated vehicle 410 that had a deviation by stopping earlier at the intersection. As a result, to prevent possible collisions, the simulation system transforms the agent 412 into a smart agent, such that the smart agent 412 uses the planning component to identified the possibility of a collision with vehicle 410, and applied the brakes early at the intersection to avoid the possibility of a collision.

In scene 406, simulated vehicle 410 and smart agent 412 remain stopped at the intersection. As a result of the smart agent 412 braking earlier at the intersection, the simulation system determines that another interaction (e.g., a potential rear-end collision) between the playback agent 414 and the smart agent 412 that currently has the deviation It was decided that this would occur. In response, the simulation system also converted agent 414 into a smart agent, and smart agent 414 braked early at the intersection to avoid a possible collision with smart agent 412. Next, after converting agent 414 into a smart agent, the simulation system determines that another interaction occurs between playback agent 416 and the now varying smart agent 414. Therefore, the simulation system converts agent 416 into a smart agent, and smart agent 416 brakes early at the intersection to avoid a possible collision with smart agent 414.

In scene 408, simulated vehicle 410 begins moving and is making a left turn at an intersection. Agents 412, 414, and 416, which have been converted to smart agents, are also moving forward toward the intersection. However, unlike simulation 300, in this example bicycle agent 418 and pedestrian agent 420 have also been converted to smart agents. For example, because the simulated vehicle 410 in scene 408 stops further back at the intersection and/or accelerates more slowly, the playback trajectory and velocity of bicycle playback agent 418 will change for the interaction immediately following scene 408. At the western crosswalk, slower speeds may cause interaction with simulated vehicles 410. As a result of the simulated vehicle 410 arriving at the crosswalk later than the corresponding vehicle in the log data, the simulation system may interact between the bicycle regeneration agent 418 and the simulated vehicle 410 with a deviation. (e.g., a possible collision between a bicycle and a vehicle). Accordingly, the simulation system also converts bicycle agent 418 into a smart bicycle agent, which exits at the crosswalk in scene 408 and allows simulated vehicle 410 to pass through the intersection before crossing. I did it like that. Next, after converting bicycle agent 418 to a smart bicycle agent, the simulation system determines that another interaction occurs between pedestrian agent 420 and smart bicycle agent 418, which now has a deviation. Therefore, the simulation system converts pedestrian agent 420 into a smart pedestrian agent, and smart pedestrian agent 420 does not begin walking in scene 408 because the smart bicycle agent remains directly in front of smart pedestrian agent 420.

As illustrated by simulation 400, and in contrast to similar simulation 300, when the simulation system converts a first playback agent into a smart agent, an agent that has a deviation from the first playback agent (e.g. By transformation, the smart agent uses a planning component within the simulation to prevent interactions and makes its path independent can be controlled by However, after the initial playback agent is converted to a smart agent, it may have deviations from the behavior of the corresponding agent in the log data, resulting in additional smart agents being added as the driving simulation continues. May interact with playback agents. This additional interaction with the currently disassociated smart agent causes the simulation system to convert the additional playback agent into a smart agent, such that during the running of the driving simulation, the playback agent becomes smart. There can be a cascading effect of being converted into an agent.

FIG. 5 illustrates an example computing environment 500 that may be used to implement the driving simulation systems and techniques described herein. The computing environment 500 includes a vehicle 502 and a simulation system 532 configured to generate and execute a log-based driving simulation. Vehicle 502 may include various software-based and/or hardware-based components of an autonomous vehicle to control an autonomous vehicle traversing a physical environment and/or a simulated vehicle operating within a log-based driving simulation. can be used to Vehicle 502 may be similar or identical to any or all of the real and/or simulated vehicles or vehicle controls described herein. In some examples, vehicle 502 may correspond to a vehicle traversing a physical environment and may capture and store log data that may be provided to simulation system 532 and used to generate a log-based simulation. Additionally or alternatively, vehicle 502 may operate as one or more separate vehicle control systems and interact with and be evaluated by simulation system 532 during a log-based driving simulation.

In at least one example, vehicle 502 may correspond to an autonomous or semi-autonomous vehicle configured to perform object perception and prediction functions, path planning and/or optimization. Exemplary vehicle 502 describes a vehicle that is capable of performing all safety-critical functions of the entire trip without the driver (or occupants) being expected to control the vehicle at any time, National Highway Traffic Safety Administration It may be a driverless vehicle, such as an autonomous vehicle configured to operate in accordance with the Level 5 classification issued by the United States. In such examples, the vehicle 502 may be configured to control all functions from initiation to completion of the trip, including all parking functions, such that the driver and/or steering wheel, accelerator pedal, etc. , and/or control devices for operating the vehicle 502, such as a brake pedal. This is just one example, and the systems and methods described herein are useful for any ground vehicle, including vehicles that always require manual control by a driver, to vehicles that are partially or fully autonomously controlled. It may be incorporated into a moving vehicle, an airborne vehicle, or a waterborne vehicle.

In this example, the vehicle 502 includes a vehicle computing device 504, one or more sensor systems 506, one or more emitters 508, one or more communication connections 510, at least one direct connection 512, and One or more drive systems 514 may be included.

Vehicle computing device 504 may include one or more processors 516 and memory 518 communicatively coupled to one or more processors 516 . In the illustrated example, vehicle 502 is an autonomous vehicle, but vehicle 502 may be other types of vehicles or robotic platforms. In the illustrated example, memory 518 of vehicle computing device 504 stores a localization component 520, a perception component 522, one or more system controllers 524, a prediction component 526, and a planning component 528. Although depicted in FIG. 5 as residing in memory 518 for illustrative purposes, one or more of stereotaxic component 520, perception component 522, system controller 524, prediction component 526, and planning component 528 may include: Additionally or alternatively, it may be accessible to vehicle 502 (eg, stored on memory remote from vehicle 502 or otherwise accessible).

In at least one example, the orientation component 520 is configured to determine the position and/or orientation (e.g., one or more of x-, y-, z-position, roll, pitch, or yaw) of the vehicle 502. can include the ability to receive data from sensor system 506. For example, localization component 520 can include and/or request/receive a map of the environment and can continually determine the position and/or orientation of the autonomous vehicle within the map. In some examples, the localization component 520 utilizes simultaneous localization and mapping (SLAM), calibration, localization and mapping, simultaneously (CLAMS), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc. It can receive lidar data, radar data, time-of-flight data, IMU data, GPS data, wheel encoder data, etc. to accurately determine the position of an autonomous vehicle. In some examples, the stereotaxic component 520 is configured to generate a trajectory and/or to determine that an object is proximate to one or more crosswalk areas; As discussed, data may be provided to various components of vehicle 502 to identify candidate reference lines and to determine an initial position of the autonomous vehicle.

In some examples, and generally, perception component 522 may include functionality to perform object detection, segmentation, and/or classification. In some examples, perceptual component 522 indicates the presence of an entity in proximity to vehicle 502 and/or indicates the type of entity (e.g., car, pedestrian, bicycle, animal, building, tree, road surface, curb, sidewalk, Processed sensor data may be provided that indicates a classification of the entity as a stop light, stop sign, unknown, etc.). In additional or alternative examples, the perception component 522 includes a processed component that is indicative of one or more characteristics associated with the detected entity (e.g., tracked object) and/or the environment in which the entity is located. sensor data can be provided. In some examples, characteristics associated with an entity include x position (global and/or local position), y position (global and/or local position), z position (global and/or local position), orientation (e.g. , roll, pitch, yaw), type of entity (e.g., classification), velocity of the entity, acceleration of the entity, range (size) of the entity, and the like. Characteristics related to the environment include, but are not limited to, the presence of other entities in the environment, the state of other entities in the environment, time of day, day of the week, season, weather, darkness/lightness indications, etc. Not done.

In some examples, memory 518 may include one or more maps that may be used by vehicle 502 to navigate within the environment. For purposes of this disclosure, a map can be defined in two, three, or N dimensions, which can provide information about the environment, such as topology (such as intersections), roads, mountain ranges, roads, terrain, and the environment in general. Any number of data structures may be modeled, including but not limited to. In some examples, the map includes texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV/HSL color information, etc.), intensity information (e.g., lidar information, radar information, etc.), spatial information (e.g., image data projected onto a mesh, individual "surfels" (e.g., polygons associated with individual colors and/or intensities)), reflectance information (e.g., specular reflectance information, retroreflectance information) , BRDF information, BSSRDF information, etc.), but is not limited to these. In one example, the map may include a three-dimensional mesh of the environment. In some examples, the map can be stored in tile format and loaded into working memory as needed, such that individual tiles of the map represent discrete portions of the environment.

In some examples, vehicle 502 can be controlled based at least in part on the map. That is, the map is used in conjunction with localization component 520, perception component 522, prediction component 526, and/or planning component 528 to determine the location of vehicle 502, identify objects in the environment, and/or Paths and/or trajectories can be generated for navigating within.

In at least one example, vehicle computing device 504 may include one or more system controllers 524 that provide steering, propulsion, braking, safety, emitter, communications, and Can be configured to control other systems. System controller 524 may communicate with and/or control corresponding systems of drive system 514 and/or other components of vehicle 502 .

Generally, prediction component 526 may include functionality to generate predictive information associated with objects in the environment. As an example, the prediction component 526 predicts that a pedestrian proximate to a crosswalk area (or otherwise an area or location associated with a pedestrian crossing the street) will cross or prepare to cross the crosswalk area. can be implemented to predict the location of pedestrians in the environment when As another example, the techniques discussed herein can be implemented to predict the position of other objects (e.g., vehicles, bicycles, pedestrians, etc.) as the vehicle 502 traverses the environment. . In some examples, prediction component 526 determines one or more predicted positions, predicted velocities, predicted trajectories, etc. of the target object based on attributes of the target object and/or other objects proximate to the target object. can be generated.

Generally, planning component 528 can determine a route that vehicle 502 should follow to traverse the environment. For example, planning component 528 may determine various routes and trajectories and various levels of detail. For example, the planning component 528 can determine a route to travel from a first location (eg, a current location) to a second location (eg, a target location). For purposes of this discussion, a route may be a sequence of waypoints for traveling between two locations. By way of non-limiting example, waypoints include streets, intersections, Global Positioning System (GPS) coordinates, and the like. Further, the planning component 528 can generate instructions to guide the autonomous vehicle along at least a portion of the route from the first location to the second location. In at least one example, the planning component 528 can determine how to guide the autonomous vehicle from a first waypoint in the sequence of waypoints to a second waypoint in the sequence of waypoints. In some examples, the command may be a trajectory, or a portion of a trajectory. In some examples, multiple trajectories can be generated substantially simultaneously (e.g., within technical tolerances) according to a receding horizon approach, and one of the multiple trajectories can be generated when vehicle 502 Selected to navigate.

In some examples, planning component 528 can generate one or more trajectories for vehicle 502 based at least in part on predicted positions associated with objects in the environment. In some examples, planning component 528 may evaluate one or more trajectories of vehicle 502 using time logic, such as linear time logic and/or signal time logic.

As will be appreciated, the components described herein (e.g., stereotactic component 520, perception component 522, one or more system controllers 524, prediction component 526, and planning component 528) have been separated for illustrative purposes. be described as something. However, the operations performed by the various components may be combined or performed in any other component. Additionally, any components described as being implemented in software can also be implemented in hardware, and vice versa. Additionally, any functionality implemented in vehicle 502 may also be implemented in one or more remote computing devices (e.g., simulated system 532) or another component (and vice versa). be.

In at least one example, sensor system 506 includes a time-of-flight sensor, a lidar sensor, a radar sensor, an ultrasound transducer, a sonar sensor, a position sensor (e.g., GPS, compass, etc.), an inertial sensor (e.g., an inertial measurement unit (IMU), etc.). , accelerometers, magnetometers, gyroscopes, etc.), cameras (RGB, IR, illuminance, depth, etc.), microphones, wheel encoders, environmental sensors (temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), etc. can. Sensor system 506 may include multiple instances of each of these or other types of sensors. For example, the time-of-flight sensors may include individual time-of-flight sensors located on the corners, front, back, sides, and/or top of vehicle 502. As another example, camera sensors may include multiple cameras located at various locations on the exterior and/or interior of vehicle 502. Sensor system 506 can provide input to vehicle computing device 504. Additionally or alternatively, the sensor system 506 may communicate with one or more external computing devices via the one or more networks 530 at a certain frequency, after a predetermined period of time, in near real time. Sensor data can be transmitted.

Vehicle 502 may also include one or more emitters 508 for emitting light and/or sound, as described above. Emitter 508 in this example includes internal audio and video emitters for communicating with passengers of vehicle 502. By way of example and not limitation, internal emitters include speakers, lights, signage, display screens, touch screens, tactile emitters (e.g., vibration and/or force feedback), mechanical actuators (e.g., seatbelt tensioners, seat positioners, headrest positioners). etc.). Emitter 508 in this example also includes an external emitter. By way of example and not limitation, external emitters in this example include lights (e.g., turn signals, signs, light arrays, etc.) to signal heading or other indicators of vehicle movement, and pedestrian or other It includes one or more audio emitters (eg, speakers, speaker arrays, horns, etc.) for audibly communicating with nearby vehicles, one or more of which include acoustic beam steering technology.

Vehicle 502 may also include one or more communication connections 510 that enable communication between vehicle 502 and one or more other local or remote computing devices. For example, communications connection 510 may facilitate communication with other local computing devices on vehicle 502 and/or drive system 514. Communication connections 510 may also enable the vehicle to communicate with other nearby computing devices (eg, other nearby vehicles, traffic lights, etc.). Communications connection 510 also allows vehicle 502 to communicate with remote remote operating computing devices or other remote services.

Communication connections 510 may include physical and/or logical interfaces for connecting vehicle computing device 504 to another computing device or a network, such as network 530. For example, the communications connection 510 may be configured to support Wi-Fi-based communications such as through frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth®, cellular communications (e.g., 2G, 3G, etc.). , 4G, 4G LTE, 5G, etc.), or any suitable wired or wireless communication protocol that allows each computing device to interface with other computing devices.

In at least one example, vehicle 502 may include one or more drive systems 514. In some examples, vehicle 502 may have a single drive system 514. In at least one example, when vehicle 502 has multiple drive systems 514, individual drive systems 514 can be located at opposite ends of vehicle 502 (eg, front and rear, etc.). In at least one example, drive system 514 can include one or more sensor systems to detect conditions around drive system 514 and/or vehicle 502. By way of example and not limitation, the sensor system may include one or more wheel encoders (e.g., rotary encoders) for sensing rotation of the wheels of the drive module, inertial sensors (e.g., for measuring orientation and acceleration of the drive module) , inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.), cameras or other image sensors, sonar sensors, lidar sensors, radar sensors, etc. for acoustically detecting objects around the drive system. can. Some sensors, such as wheel encoders, may be unique to drive system 514. In some cases, the sensor systems of drive system 514 can overlap or supplement corresponding systems of vehicle 502 (eg, sensor system 506).

Drive system 514 includes a high voltage battery, a motor to propel the vehicle, an inverter that converts direct current from the battery to alternating current for use in other vehicle systems, a steering motor, and a steering rack (which may be electric). ); braking systems including hydraulic or electric actuators; suspension systems including hydraulic and/or pneumatic components; stability control systems to distribute braking force to mitigate loss of traction and maintain control. , HVAC system, lighting (e.g., head/tail lights illuminating the exterior surroundings of the vehicle), and one or more other systems (cooling system, safety system, onboard charging system, DC/DC converter, etc.) electrical components, high voltage junctions, high voltage cables, charging systems, charging ports, etc.). Additionally, drive system 514 can include a drive system controller that can receive and preprocess data from sensor systems and control operation of various vehicle systems. In some examples, the drive system controller may include one or more processors and memory communicatively coupled to the one or more processors. Memory may store one or more components for performing various functionality of drive system 514. Additionally, drive systems 514 also include one or more communication connections that enable the respective drive system to communicate with one or more other local or remote computing devices.

In at least one example, direct connections 512 can provide a physical interface for coupling one or more drive systems 514 to the body of vehicle 502. For example, direct connection 512 can enable the transfer of energy, fluid, air, data, etc. between drive system 514 and a vehicle. In some cases, direct connection 512 can releasably secure drive system 514 to the body of vehicle 502.

In at least one example, the orientation component 520, the perception component 522, the one or more system controllers 524, the prediction component 526, and the planning component 528 can process sensor data and one or more of the Each output can be sent via multiple networks 530 to one or more external computing devices, such as a simulation system 532. In at least one example, the output of each of the components can be sent to the simulation system 532 at a certain frequency, after a predetermined period of time, and in near real time. Additionally or alternatively, vehicle 502 may transmit sensor data, including raw sensor data, processed sensor data, and/or representations of sensor data, to simulation system 532 via network 530. Such sensor data may be transmitted to simulation system 532 as one or more files of log data 550, such as at a specified frequency, after a predetermined period of time, in near real time, etc.

As discussed above, simulation system 532 performs steps 600 and 700 using various components and systems described herein to perform techniques similar or identical to those described above with reference to FIGS. 1-4D. A driving simulation, such as a log-based driving simulation, may be generated and executed for reference and execution below. Simulation system 532 may communicate/interact with one or more vehicles 502 to perform a driving simulation in which vehicles 502 correspond to the simulated vehicle. Although not shown in this example, simulation system 532 may also include a log data store and/or a simulation scenario that includes simulated environments and objects. In various examples, simulation system 732 generates and instantiates a driving simulation for the simulated vehicle, including monitoring and receiving simulation-based responses from vehicle 502 and/or other vehicle control systems; It can be executed.

Simulation system 532 may include one or more processors 534 and memory 536 communicatively coupled to one or more processors 534. In the illustrated example, memory 536 of simulation system 532 stores a log-based simulation generator 538 and a simulator 540 configured to perform log-based driving simulations. Simulator 540 includes an agent interaction monitor 542, an agent translator 544, a planning component 546, and a playback agent analyzer 548. In this example, agent interaction monitor 542 tracks agents and simulated vehicles during log-based simulations, generates bounding boxes, and uses the present invention to determine interactions between agents during simulations. may include the functionality described in . Agent converter 544 may include functionality described herein for converting playback agents into smart agents during log-based driving simulations. Planning component 546 may include functionality described herein for providing planning component functionality to a smart agent during a driving simulation. For example, a function, thread, or process may be initiated within the planning component 546 for each smart agent transformation during the simulation to analyze the simulated environment and determine the path of the smart agent through the simulated environment. In some examples, planning component 546 may be a simplified version of planning component 528 configured to avoid interactions with minimal trajectory adjustments by smart agents. Playback agent analyzer 548 analyzes the route taken by the playback agent, vehicle conditions, and driving actions performed by the playback agent to determine one or more driving styles and/or driving actions for the playback agent. may include the functions described herein for determining a person's personality type. Although depicted in FIG. 5 as residing in memory 536 for illustrative purposes, some or all of systems and components 538-548 may additionally or alternatively be stored remotely and communicated via network 530. It is contemplated that the simulation system 532 may be accessible to all users.

Log-based simulation generator 538 may generate log-based driving simulations using techniques similar or identical to those described herein. For example, log-based simulation generator 538 may receive log data previously collected by vehicle 502 and/or other vehicle control systems. Log data may correspond to data captured by vehicle 502 as it traverses the physical environment. To generate a log-based driving simulation, log-based simulation generator 538 uses the received log data to generate a simulated environment and simulated objects (e.g., , agent). During execution of a log-based simulation, simulator 540 may provide simulated environment and object data to a simulated vehicle (eg, vehicle 502) based on log data. For example, log-based simulation generator 538 may receive and analyze log data to identify specific objects in the environment and attribute data associated with those objects (e.g., size, location, trajectory, waypoints, etc.). may be detected, and simulator 540 may convert the object data into sensor data that may be sent to vehicle 502 during the performance of the driving simulation.

In some cases, log-based simulation generator 538 may determine that a particular object has a property and apply that property to the simulated object. For purposes of example only, the log data may indicate that the object is traveling approximately 10 miles below the speed limit and accelerating slowly. Log-based simulation generator 538 may determine that the object is a simulated vehicle and apply a conservative object model to the corresponding simulated object in the simulated environment. In some examples, the log-based simulation generator 538 determines whether an object is an aggressive object, a passive object, a neutral object, and/or has other types of behavior based on behavioral data in the log data. and apply behavioral instructions associated with the behavior (eg, passive behavior, cautious behavior, neutral behavior, and/or aggressive behavior) to the simulated object.

In some examples, simulator 540 may use filters to remove objects represented in the log data from the simulated scenario based on attributes associated with the objects. In some examples, simulator 540 may include object/classification type (e.g., car, pedestrian, motorcycle, bicycle, etc.), object size (e.g., length, width, height, and/or volume), confidence level, track Objects can be filtered based on the length of the log, the amount of interaction between the object and the vehicle that generated the log data, and/or the time period.

By way of example and not limitation, log data may include objects of various sizes, such as mailboxes and buildings. The log-based simulation generator 538 is represented in a simulated scenario in which objects are associated with a volume equal to or greater than a threshold volume of 3 cubic meters, such as a building, and objects with a volume less than 3 cubic meters, such as a mailbox. Volume-based filters can be used so that objects associated with the ``Device'' are not represented in the simulated scenario. In some examples, log-based simulation generator 538 determines whether an object with a track length (e.g., data related to physical distance or time period) that does not meet or exceed a track length threshold is simulated. A track length filter can be used that is filtered from a given scenario. As a result, it is possible to create a simulation in which objects related to poor detection during data acquisition are omitted. In some examples, log-based simulation generator 538 may use motion-based filters such that objects that are related to motion or trajectory according to the log data are represented in the simulation. In some examples, filters may be applied in combination or mutually exclusive.

In some examples, log-based simulation generator 538 may filter objects that do not meet or exceed a confidence threshold. By way of example and not limitation, the log data may indicate that the object is associated with a pedestrian classification attribute and that the confidence value associated with the classification is 5%. The log-based simulation generator 538 has a 75% confidence value threshold and can filter objects based on confidence values that are less than or above the confidence value threshold. In some examples, the user can perform one or more steps such that the log-based simulation generator 538 can filter objects that do not meet or exceed one or more attribute thresholds indicated by the user-generated filter. A user-generated filter can be provided that includes one or more attribute thresholds.

Simulator 540 can perform driving simulations as a set of simulation commands and generate simulation data. In some examples, simulator 540 can execute multiple simulated scenarios simultaneously and/or in parallel. This allows the user to edit simulations and run permutations of simulations with variations between each simulation. Further, simulator 540 can determine the results of the simulation. For example, simulator 540 can perform log-based driving simulations for use in testing and validation simulations. Simulator 540 generates simulation data indicating how vehicle 502 performed (e.g., responded), compares the simulation data to predetermined results, and/or determines whether predetermined rules/assertions were violated/ It is possible to determine whether it has been triggered.

In some examples, the predetermined rules/assertions may be based on simulation (e.g., a traffic rule regarding a crosswalk may be enabled based on a crosswalk scenario, or a traffic rule regarding crossing a lane marker may be enabled based on a crosswalk scenario). Rules can be disabled for stopped vehicle scenarios). In some cases, simulator 540 can dynamically enable and disable rules/assertions as the simulation progresses. For example, rules/assertions related to the school zone may be enabled when the simulated subject approaches the school zone, and disabled when the simulated subject moves away from the school zone. In some cases, the rules/assertions may include, for example, comfort metrics related to how fast an object can accelerate in a simulated scenario.

determining that the vehicle 502 exhibited performance consistent with a predetermined outcome (i.e., the autonomous controller did everything it was supposed to do) and/or that no rules were broken; or Based at least in part on determining that the assertion was not triggered, simulator 540 may determine that vehicle 502 was successful. determining that the performance of the vehicle 502 was inconsistent with a predetermined outcome (i.e., the autonomous controller did something it was not supposed to do) and/or that a rule was broken or an assertion was made. Based at least in part on determining the triggered twist, simulator 540 may determine that vehicle 502 has failed. Accordingly, based at least in part on the execution of the simulations, the simulation data may indicate how the vehicle 502 responds to each simulation, as described above, and based at least in part on the simulation data, the simulation data may indicate successful outcomes. Or a failure result can be determined.

Processor 516 of vehicle 502 and processor 534 of simulation system 532 may be any suitable processor capable of processing data and executing instructions to perform operations, as described herein. By way of example and not limitation, processors 516 and 534 may include one or more central processing units (CPUs), graphics processing units (GPUs), or processors that process electronic data and store the electronic data in registers and/or memory. It can be comprised of any other device or part of a device that converts the data obtained into other electronic data. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices may also be integrated into the processor, so long as they are configured to implement the encoded instructions. It can be considered as

Memories 518 and 536 are examples of non-transitory computer readable media. Memories 518 and 536 may store an operating system and one or more software applications, instructions, programs, and/or data for performing the functions ascribed to the methods and various systems described herein. can. In various implementations, the memory can be any type of memory, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile/flash memory, or any other type of memory that can store information. can be implemented using appropriate memory technology. The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, including those illustrated in the accompanying figures. Just an example relevant to the discussion.

Note that although FIG. 5 is illustrated as a distributed system, in alternative examples, components of vehicle 502 may be associated with simulation system 532 and/or components of simulation system 532 may be associated with vehicle 502. . That is, vehicle 502 may perform one or more of the functions associated with simulation system 532, and vice versa. Additionally, aspects of simulator 540 (and its subcomponents) can be implemented in any of the devices or systems discussed herein.

As described above, simulation system 532 analyzes log data 550 received from vehicle 502 to determine actions, attributes, and/or destinations associated with the vehicle and other objects represented in the log data. obtain. In some examples, the simulation system 532 uses the behaviors, attributes, and/or destinations of the objects in the log data to determine the corresponding behaviors, attributes, and/or destinations of the smart agent during the simulation. It is possible. For example, playback agent analyzer 548 may analyze log data 550 to identify any objects in the environment, including motorized vehicle objects, non-motorized vehicle objects (e.g., bicycles, skateboards, etc.), and/or pedestrian objects. One or more behaviors performed by a dynamic object may be identified. Based on the object classification and object behavior determined from the log data, playback agent analyzer 548 may determine attributes (eg, driving attributes, personality attributes, or other behavioral attributes) associated with the object. The planning component 546 then uses the object attributes determined by the playback agent analyzer 548 to control the smart agent within the simulation, thereby providing more realistic agent behavior and decision making, and the simulation can improve the overall quality and value of

The actions determined by playback agent analyzer 548 based on log data may include any observable actions performed (or not performed) by objects in the environment. Actions may be expressed as binary values and/or numbers that quantify actions performed by an object. In a particular example, different values of an attribute may be recorded for a particular agent. The value of an attribute may correspond to a particular agent's behavior. For example, a particular range of an attribute may correspond to a particular behavior, or values of an attribute may be aggregated and/or scaled to correspond to a particular behavior. In some examples, the playback agent analyzer 548 determines the behavior of the object, such as one set of behaviors for a car object, another set of behaviors for a bicycle, and another set of behaviors for a pedestrian. Different types of behavior may be determined for different classifications. Various examples of the types of behaviors and/or attributes that may be determined by playback agent analyzer 548 are described in the sections below for different object classifications. It can be understood from the context of this disclosure that the behaviors and attributes described herein are non-limiting examples, and that any additional types of object behaviors and attributes can be used in other examples.

For each vehicle (e.g., car, truck, bus, motorcycle, etc.) represented in log data 550, playback agent analyzer 548 determines the vehicle's position, trajectory, and speed as captured in log data 550 associated with the vehicle. A variety of different driving behaviors may be determined based on , attitude data, and/or other vehicle attributes. Examples of driving behavior determined from vehicle log data 550 may include the vehicle's speed relative to speed limits, the vehicle's stopping distance relative to stop lines (eg, stop signs, traffic lights, crosswalks, etc.). Additional examples of driving behavior include whether the vehicle travels in a bicycle lane, bus lane, or taxi lane, whether the vehicle uses lane splitting or lane sharing, and whether the vehicle travels on the shoulder or parking lane. Includes whether or not. Other examples of driving behavior may include any traffic violations committed by the vehicle. Other examples of driving behavior include vehicle average (or preferred) acceleration and braking, vehicle maximum acceleration and braking, vehicle average (or preferred) cruising speed, vehicle maximum and minimum cruising speed, vehicle average and/or braking. Or comfort parameters such as maximum allowable jerkiness measurements may be included. Additional driving behavior may include the observed reaction time of the vehicle in response to a potential interaction, or the estimated trajectory scan distance (e.g., how far forward the vehicle will scan due to the interaction). obtain. Another driving behavior may include a stop delay before the vehicle resumes movement while the vehicle is stopped at a stop sign or crosswalk. Another driving behavior may include a low speed tolerance parameter corresponding to the speed of an agent in front of the vehicle that causes the vehicle to change lanes or pass the agent. Additional driving behaviors include whether the vehicle uses the turn signal before turning at a fork, the distance before the turn that the vehicle uses the turn signal, and whether the vehicle uses the turn signal before changing lanes. Vehicle signaling parameters may be included, such as whether the vehicle uses turn signals and the distance before changing lanes.

For pedestrians and/or other non-vehicular objects (e.g., bicycles, skateboards, etc.) represented in log data 550, playback agent analyzer 548 analyzes the object's position, trajectory, velocity, attitude, and /Or other attributes may be used to determine any of the same actions as described above for automobiles. Further, playback agent analyzer 548 may alternatively determine different sets of actions for objects in the environment based on the object classification. Examples of behaviors determined from log data 550 for objects other than vehicles include the object's desired (or normal) speed and maximum speed (e.g., walking speed, bicycle speed, etc.), and the object's normal acceleration and maximum speed. Acceleration and deceleration may be included. Additional behavior for objects other than vehicles may include the object's normal speed and acceleration when crossing a road and/or running a red light. Other examples of behavior for non-vehicle objects include the object's normal and maximum turning speed, the distance an object maintains between itself and the road, wall, or other moving object (safety area, following distance). etc.).

For each driving behavior determined for a vehicle, or other behavior determined for an object other than a vehicle, playback agent analyzer 548 determines individual instances, averages, ranges, and/or distributions of the behavior. obtain. As an example, playback agent analyzer 548 may determine a vehicle stopping at a crosswalk as a behavior based on a single instance of a vehicle stopping at a crosswalk from log data 550. In other examples, the playback agent analyzer 548 may identify multiple instances of a behavior (e.g., driving speed measurements, following distance measurements, acceleration rate values, braking rate values, etc.), where the driving behavior is , may correspond to an instance mean, an instance range, or a probability distribution based on the instances. For example, with respect to driving behavior representing the speed of the vehicle relative to the speed limit, the following distance of the vehicle, etc., the driving behavior may be calculated as an average or distribution based on instances of the vehicle behavior.

In some examples, playback agent analyzer 548 may use the object's behavior determined based on log data 550 to determine behavioral attributes associated with the object. For electric vehicles in log data 550, behavioral attributes may be referred to as driving attributes. In some cases, driving attributes and/or behavioral attributes may correspond to a single behavior or a group of behaviors. For example, low-level driving attributes of a vehicle may include attributes such as vehicle reaction time, average speed relative to speed limit, average following distance, average turn signal distance, and the like.

Additionally or alternatively, particular driving and/or behavioral attributes may include higher level attributes based on a combination of behaviors. For example, playback agent analyzer 548 may use multiple driving behaviors such as acceleration/braking rate, turn rate, travel speed, following distance, etc. to determine aggressive driving attributes of the vehicle. Different combinations of driving behaviors (eg, lane use, traffic signal use, driving speed, stop/yield behavior, etc.) may be combined and analyzed to determine compliant driving attributes. Further combinations of driving behaviors (eg, following distance, acceleration/braking speed, reaction time, trajectory scan distance, etc.) may be combined and analyzed to determine driving skill attributes of the vehicle. For these examples and other high-level driving attributes for vehicles (or high-level behavioral attributes for other types of objects), playback agent analyzer 548 determines from log data 550 Scores or metrics may be assigned to each of the high-level attributes based on various combinations of behaviors. In some instances, a high-level driving attribute or combination of attributes may be referred to as a driving style or driving personality, and similar styles or personalities may be used to differentiate pedestrians and non-motorized vehicles based on their respective behaviors/attributes. may be determined for the object.

After playback agent analyzer 548 determines the behavior and/or attributes of objects in the environment based on log data 550, planning component 536 uses the behaviors and attributes to create one or more smarts in the simulation. Agents can be controlled. In some examples, a single smart agent within a driving simulation may be assigned the same behavior and/or attributes as a corresponding dynamic object within log data 550. To assign a set of behaviors/attributes to a smart agent, simulation system 532 may instantiate a particular version of planning component 536 based on the behaviors/attributes and/or assign the smart agent a set of desired behaviors/attributes. Certain parameters may be provided when instantiating the plan component 536 that corresponds to the set. As mentioned above, a smart agent may be instantiated at the beginning of a simulation or may be converted from a playback agent during a driving simulation.

As mentioned above, the planning component 536 may control the actions and behaviors of the smart agent during the simulation, including any driving decisions of objects within the simulation environment. As used herein, a simulated vehicle or other simulated object determination refers to any decision of the planning component 536 to control a simulated vehicle/object during a simulation. may be included. The driving decisions determined by the planning component 536 for the simulated vehicle may correspond to any driving behavior determined by the playback agent analyzer 548 for the corresponding vehicle in the log data 550.

In some examples, planning component 536 may control the simulated vehicle based on a single corresponding vehicle from log data 550. For example, the planning component 536 may plan to drive at the same speed relative to the speed limit, have the same following distance, or use the same turn signals for any of the driving behaviors described herein. may control the simulated vehicle. To determine specific driving decisions for the simulated vehicle, such as the speed at which it is traveling on the road, the rate of deceleration approaching the stop sign, and the location of the stop relative to the stop sign, the planning component 536 determines the corresponding driving decisions in the log data 550. The same behavior from the vehicle may be matched.

In other examples, planning component 536 may determine driving decisions for a simulated vehicle even without matching behavior from a corresponding vehicle in log data 550. For example, if the corresponding vehicle did not stop at a stop sign or signal during the period in which log data 550 was collected, planning component 536 controls the simulated vehicle in connection with the stop sign or signal. may not have the direct driving behavior to use when In such cases, planning component 536 may use one or more indirect techniques to determine driving decisions for the simulated vehicle. The planning component 536 may in some cases use predetermined and/or default values for the simulated vehicle driving behavior/decisions. Planning component 536 may also determine driving decisions for the simulated vehicle based on correlations between different driving behaviors and/or weight values associated with the correlations. For example, the planning component 536 determines that a correlation exists between certain driving behaviors (e.g., time delay at a stop sign) and other driving behaviors (e.g., average speed relative to speed limit, following distance, steering rate, etc.). can be determined. The planning component 536 may also determine weight values that indicate the strength of correlation between different driving behaviors (eg, a higher weight value may indicate a stronger correlation between driving behaviors). The planning component 536 then determines driving decisions for the simulated vehicle in the absence of a corresponding driving behavior in the log data 550 for the corresponding vehicle. values and associated weight values may be used.

Additionally or alternatively, the planning component 536 determines higher level driving attributes for the corresponding vehicle in the log data 550, such as aggressive driving attribute scores, evasive driving attribute scores, reaction time driving attribute scores, and/or or driving skill attribute scores, etc., may be used to determine driving decisions for the simulated vehicle. For example, the planning component 536 may make lower-level driving decisions (e.g., speed, acceleration, following distance, turn signal usage, etc.) for the simulated vehicle based on the corresponding vehicle's higher-level attributes in the log data 550. Machine learning models and/or heuristic-based algorithms may be implemented to determine the As an example, the planning component 536 determines the following distance of the first simulated vehicle based on the driver aggressiveness metric and the driver skill metric of the corresponding vehicle in the log data, while the second The following distance of a simulated vehicle may be determined based on aggressiveness metrics and skill metrics of different drivers of different corresponding vehicles in the log data.

Simulation system 532, in some examples, provides a one-to-one mapping in which a single simulated vehicle is controlled based on the driving behavior and attributes of a single corresponding vehicle in log data 550. Can be implemented. In other examples, the simulation system 532 may determine driving behaviors and attributes for multiple vehicles in the log data 550 such that the aggregation of behaviors/attributes from multiple vehicles controls a single simulated vehicle. A multiple-to-one mapping may be used for the purpose. One-to-multiple mapping may also be used by simulation system 532, in which the behavior and/or attributes of a single vehicle in log data 550 map to multiple simulated vehicles in one or more driving simulations. can be used to control.

In yet other examples, simulation system 532 may configure the simulation to include a group of simulated vehicles with a configuration of behaviors/attributes based on the group of vehicles in one or more vehicle logs. For example, simulation system 532 may analyze groups of vehicles within one or more environments and times associated with log data 550 and determine the number or percentage of vehicles within the environment that exhibit different driving behaviors or attributes. . An exemplary configuration of a vehicle from log data 550 is: 20% Highly Aggressive Driver, 85% Highly Compliant Driver, 45% Highly Skilled Driver, 70% Overtaking This may include drivers using bus lanes, etc. Simulation system 532 may configure different driving behaviors/attributes associated with different locations and/or different times of day. For example, a driving behavior/attribute configuration determined based on log data 550 collected during rush hour in one region may be different from a configuration of driving behavior/attributes determined based on log data 550 collected on a weekend afternoon in another region. /It may be different from the attribute configuration. Simulation system 532 may select a particular configuration of driving behaviors/attributes associated with time and domain, and may configure groups of simulated vehicles within the simulation to match the selected configuration, and This allows simulation system 532 to more realistically model groups of driver personality types in different regions and at different times.

Further, while the specific examples herein refer to vehicles and driving simulations, using similar or identical techniques to the objects described above, non-vehicles (e.g., pedestrians, bicycles, etc.) in the log data 550 can be and corresponding non-vehicle objects in the simulation.

FIG. 6 is a flow diagram of an example process 600 for converting a playback agent into a smart agent during a driving simulation in accordance with various systems and techniques described herein. In some examples, the operations of process 600 may be performed by one or more components of simulation system 532, alone or in conjunction with one or more vehicles 502. For example, a planning component for analyzing log data associated with playback agents, determining playback agent destinations and/or driving behavior/attributes, and controlling smart agent decisions (e.g., routes and behaviors). The techniques described herein for determining and applying parameters may be performed by the planning component 546 and/or the playback agent analyzer 548 described above.

At act 602, simulation system 532 may receive and analyze log data associated with playback agents within the simulation. For example, simulation system 532 may receive log data 550 from vehicle 502 that includes various data observed and/or perceived by vehicle 502 while traversing the environment. In particular, the log data received in operation 602 may include specific attributes or characteristics (e.g., agent classification, size, shape, location, yaw, speed, trajectory, etc.) detected within proximity of vehicle 502. agents (e.g., other vehicles, pedestrians, bicyclists, etc.). In some cases, log data may also include agent actions and/or incidents such as accidents or near-accidents, traffic violations, pedestrians, cyclists, or animals crossing or ignoring traffic lights, weather abnormalities, construction zones, detours, school zones, etc. or may include behavior. Log data 550 received in operation 602 may include data based on raw sensor data and/or sensor data captured on vehicle 502.

At act 604, simulation system 532 may determine the destination of the playback agent within the simulation based on the log data received at act 602. The playback agent's destination may be determined and/or inferred by the simulation system 532 based on the route/path traveled by the corresponding agent in the log data. In some cases, determining the destination for the playback agent requires additional log data for the corresponding agent captured at times before or after the period associated with the log-based driving simulation. may include analyzing.

At act 606, simulation system 532 may determine one or more driving and/or personality attributes associated with the playback agent based on the log data received at act 602. For example, playback agent analyzer 548 detects and analyzes one or more driving behaviors and/or actions performed by a corresponding agent in the log data, and detects and analyzes one or more driving styles and/or actions of the playback agent. /or may determine the driver's personality type. As an example, playback agent analyzer 548 analyzes log data associated with a particular agent to determine one or more aggressiveness metrics, driving skill metrics, reaction time metrics, legal compliance metrics, etc. for the agent. It is possible.

At act 608, simulation system 532 may determine whether the playback agent is converted to a smart agent at some point during the driving simulation. As mentioned above, in the course of a log-based driving simulation, an agent may initially act as a playback agent and act within the simulation based on the actions of the corresponding agent in the log data. Simulation system 532 is responsive to determining that an interaction between a playback agent and a non-playback agent (e.g., a smart agent or a simulated vehicle) having a deviation may occur within the simulation. can be used to convert playback agents into smart agents. Not all playback agents are converted to smart agents, and during various driving simulations, simulation system 532 may convert some, all, or all of the playback agents to smart agents. In this example, if the particular playback agent analyzed in acts 602-606 is not converted to a smart agent (608: NO), the process 600 monitors the operation of the playback agent as the simulation continues. One may return to operation 602 to continue.

However, if simulation system 532 determines during simulation that a particular playback agent is converted to a smart agent (608: YES), then in act 610 simulation system 532 determines that the playback agent determined in act 604 is a smart agent. One or more planning component parameters are determined based on the agent's destination and/or based on the playback agent's driving/personality attributes determined in operation 606. Such parameters are: a parameter that determines whether the smart agent uses Viking lanes when determining a route, a parameter that determines whether the smart agent uses lane splitting, a parameter that determines whether the smart agent uses lane sharing. a parameter that determines whether the desired velocity of the smart agent (based on the agent type), a parameter that represents the maximum possible velocity and acceleration of the smart agent (based on the agent type), a parameter that represents the desired velocity of the smart agent (based on the agent type), a parameter that determines whether the Parameters that represent the desired distance of the smart agent from the agent (based on agent type), parameters that represent how the smart agent reacts to traffic lights, stop signs, exit signs, stop lines, school zones, and construction zones , may include, but are not limited to, parameters representing when the smart agent uses turn signals, turns on/off lights, and/or performs other vehicle controls.

In operation 612, simulation system 532 may then convert the playback agent into a smart agent by initiating a planning component to control the navigation and/or driving decision-making behavior of the smart agent during the simulation. In some examples, starting the planning component may include instantiating, running, and/or configuring a planning component process based on the planning component parameters determined in operation 610, thereby causing the planning component to The component is configured to control the smart agent's navigation and/or driving decision-making behavior in a manner that more accurately reflects the driving style and/or driver personality exhibited by the playback agent.

FIG. 7 is a flow diagram of another example process 700 for converting a playback agent into a smart agent during a driving simulation in accordance with various systems and techniques described herein. In some examples, the operations of process 700 may be performed by one or more components of simulation system 532, alone or in conjunction with one or more vehicles 502. For example, the techniques described herein for determining interactions between agents in a driving simulation and converting playback agents into smart agents may be performed by agent interaction monitor 542 and/or agent converter 544 as described above. can be done.

At operation 702, simulation system 532 may determine trajectories for one or more vehicles/agents within a log-based driving simulation. A driving simulation may include multiple different agents operating within a simulation environment, including playback agents and smart agents. The trajectory of the playback agent within the simulation may be determined based on the position of the corresponding agent within the log data. Thus, for a playback agent, simulation system 532 may determine the complete trajectory of the playback agent at any time during the simulation based on the log data. In contrast, the simulated vehicles and/or other smart agents of the driving simulation may be controlled by a planning component rather than being completely based on log data. As a result, the trajectories of the simulated vehicles and smart agents may be unknown until the simulation is run, and the agent interaction monitor 542 determines the current and predicted trajectories of each smart agent in the simulation. May include functionality to track and monitor agent position, pose, velocity, acceleration, and yaw.

At operation 704, simulation system 532 may determine interactions between playback agents and non-playback agents within the driving simulation. In some examples, agent interaction monitor 542 distinguishes between playback agents and non-playback agents ( use the size and/or spatial area together with the agent trajectory to determine the likelihood or probability of interactions between agents; It's okay. In some cases, agent interaction monitor 542 may determine a set of bounding boxes associated with each playback agent and non-playback agent in the simulation, where the bounding boxes are determined based on the dimensions and orientation of the agents. Good too. The bounding box may optionally include additional safety buffers, which may be based on agent type, speed, vulnerability, and/or various other agent attributes.

In some examples, simulation system 532 may compare sets of bounding boxes associated with playback agents and non-playback agents in the driving simulation and determine whether an overlap exists at each time interval. . If, at a particular time in the simulation, there is an overlap between the bounding box of a playback agent and the bounding box of a different non-playback agent, agent interaction monitor 542 may determine that an interaction exists between the agents. .

In operation 706, the simulation system 532 determines whether an agent (e.g., a smart agent or a simulated vehicle) interacting with the playback agent is a deviation agent that has a deviation distance that meets or exceeds a deviation distance threshold. can be determined. In some examples, agent interaction monitor 542 determines the difference between the non-playback agent at the time of the interaction determined in operation 704 and the corresponding agent's position in the log data at the same relative time. Based on the position difference, a deviation distance of the non-playback agent may be determined. Agent interaction monitor 542 may then compare the deviation distance of the non-playback agent with the deviation to a deviation distance threshold. If the deviation distance of the non-playback agent with deviation does not meet or exceed the deviation distance threshold (706: NO), the process 700 returns to operation 702 and continues monitoring the agent in the driving simulation. do. However, if the deviation distance associated with the smart agent or the simulated vehicle meets or exceeds the deviation distance threshold (706: YES), then in operation 708 the simulation system 532 converts the playback agent into a smart agent. obtain.

As mentioned above, in some cases the deviation distance threshold may be set to zero, in which case there is no need to perform operation 706. However, in other cases the deviation distance thresholds are 0.1 meters, 0.2 meters, 0.3 meters, . ．． ．． , 1 meter, etc.). In some examples, the value of the deviation distance threshold may be configurable and may be adjusted up or down to produce different numbers and different timings of replay agent transformations during simulation.

Additionally, although this example uses the deviation distance to determine whether to convert a playback agent to a smart agent, in other examples, the simulation system 532 uses the yaw of a non-playback agent that has a deviation. A deviation value may be used, which may be compared to a yaw deviation threshold. In various other examples, agent interaction monitor 542 may apply deviation thresholds for distance, yaw, and/or other attributes of the agent (eg, velocity, trajectory, etc.). Such thresholds may be predetermined values, or the agent interaction monitor 542 may be configured based on the type of playback agent, the type of non-playback agent with deviations, the speed of one or both of the agents, etc. Based on this, different deviation thresholds may be determined and applied.

At act 708, based on the determination that during the simulation the playback agent interacted (or will interact) with a non-playback agent that has a deviation, the simulation system 532 converts the playback agent to a smart agent. It is possible. In some examples, the agent converter 544 controls the playback agent by executing and/or configuring the planning component 546 to control the smart agent's navigation and/or driving decision-making behavior during the driving simulation. can be transformed into a smart agent. As discussed above with reference to step 600, in some cases, planning component 546 is instantiated, executed, and/or configured based on one or more planning component parameters determined for a particular smart agent. The smart agent's navigation and/or driving decision-making behavior may thereby be controlled to more accurately reflect the playback agent's driving style and/or driver personality.

At act 710, simulation system 532 may continue running the driving simulation while determining whether the simulation is complete. Completion of a simulation may mean the end of a simulation, and/or the fact that during execution of a simulation, the simulated vehicle has performed an action consistent with a predetermined result, or has not performed an action consistent with a predetermined result. , corresponds to the fact that the vehicle control device operating the simulated vehicle has determined either to pass or fail the simulation. If the log-based simulation is in progress and has not yet completed (710: NO), the process 700 returns to operation 702 to continue monitoring the simulation and to the smart agent for additional playback agents as needed. conversion can be performed. However, once the simulation is complete (710: YES), in operation 712 the simulation system 532 may terminate the simulation process and output or store simulation results for the vehicle controller operating the simulated vehicle. As mentioned above, the simulation results may identify the behavior and/or performance of the simulated vehicle during the driving simulation, which may correspond to the behavior and/or performance of the vehicle controller being evaluated.

Processes 600 and 700 are each illustrated as a collection of blocks in a logical flow diagram that represent sequences of operations, some or all of which may be implemented in hardware, software, or a combination thereof. can be done. In the context of software, a block represents computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, perform the recited operations. do. Generally, computer-executable instructions include routines, programs, objects, components, encryption, decryption, compression, recording, data structures, etc. that perform particular functions or implement particular abstract data types. included. The order in which operations are described is not to be construed as limiting. Any number of the blocks described may be combined in any order and/or in parallel to perform the process, or alternative processes, without having to execute all of the blocks. For purposes of discussion, the processes described herein are described with reference to the frameworks, architectures, and environments described in the examples herein, but the processes may be implemented using a wide variety of other frameworks. , architecture, or environment.

(Example Clause)
A. executing a driving simulation including one or more processors and, when executed, a simulated autonomous vehicle controlled by an autonomous vehicle controller, the driving simulation being associated with a vehicle traversing an environment; and the log data is based at least in part on log data associated with a first object in the environment and a first log data associated with a second object in the environment. and controlling a first simulated object based at least in part on the first log data during a first time period in the driving simulation; controlling a second simulated object based at least in part on log data of a first interaction between the simulated autonomous vehicle and the first simulated object in the driving simulation; and determining a second time period in the driving simulation after the first time period using a first planning component based at least in part on determining the first interaction. controlling the first simulated object during the driving simulation; and controlling the first simulated object and the second simulated object during the second time period in the driving simulation. determining a second interaction between and using a second planning component based at least in part on determining the second interaction between the controlling the second simulated object during a third time period in the driving simulation; and determining a response by the autonomous vehicle controller to the driving simulation. one or more computer-readable media storing computer-executable instructions for performing operations including:

B. The system of paragraph A, wherein during the first period in the driving simulation, the simulated autonomous vehicle is controlled based at least in part on the first simulated object and the first The second simulated object is controlled to correspond to a first object in the environment and a second object in the environment, respectively, based on the log data, and the second simulated object in the driving simulation During a second time period, the simulated autonomous vehicle and the first simulated object are controlled based at least in part on the component, and the second simulated object is controlled based at least in part on the environment. during a third time period in the driving simulation, the simulated autonomous vehicle, the first simulated object, and A system in which the second simulated object is controlled based at least in part on the component.

C. The system of paragraph A, wherein the operation is to determine a deviation distance of the simulated autonomous vehicle at a time associated with the first interaction, the deviation distance being based on the log data. and the corresponding position of the simulated autonomous vehicle during the driving simulation, using the first planning element to The system further comprises: controlling the detected object based at least in part on the deviation distance.

D. The system of paragraph A, wherein the operation is to determine driving attributes associated with the first object based at least in part on the first log data, the first plan comprising: determining a driving attribute associated with the first object; The system further comprises controlling the first simulated object using the component based at least in part on the driving attribute.

E. The system of paragraph A, wherein the operation is based at least in part on third log data associated with the third object in the environment, wherein the operation is performed on a third simulated object during the driving simulation. between controlling an object and the position of the simulated autonomous vehicle at a time associated with the third interaction and the corresponding position of the third object in the third log data. and determining the third interaction between the simulated autonomous vehicle and the third simulated object in the driving simulation, the third interaction between the simulated autonomous vehicle and the third simulated object. controlling the third simulated object after determining an interaction of the system is based at least in part on the third log data to correspond to the third object in the environment.

F. The method comprises: performing a driving simulation, the driving simulation being based at least in part on log data associated with a vehicle in an environment, the log data being associated with a first object in the environment. controlling a first simulated object to correspond with the first object in the environment during a first time period; determining an interaction between the first simulated object and a second simulated object during a second time period in the driving simulation after the first time period; controlling the first simulated object using a planning component.

G. The method of paragraph F, wherein the log data includes second log data associated with a second object in the environment, and the method comprises determining the position of the second object in the second log data. and a relative position of a second simulated object in the driving simulation; and determining that the distance meets or exceeds a threshold distance. and initiating the planning component to control the first simulated object.

H. The method of paragraph F, comprising: determining a driving attribute associated with the first simulated object; determining a destination for the first simulated object; and determining the driving attribute. providing the planning component with a first parameter associated with the destination and a second parameter associated with the destination, the planning component providing the first parameter with the destination and the second parameter with the destination; and being configured to control the first simulated object in the driving simulation based at least in part.

I. The method of paragraph H, wherein the driving attributes associated with the first simulated object and the destination associated with the first simulated object are stored in the first log. A method determined based at least in part on data.

J. The method of paragraph F, comprising: controlling a third simulated object based at least in part on the log data during the second time period in the driving simulation; determining a second interaction between the first simulated object and the third simulated object; and a third time in the driving simulation after the second time period. controlling the third simulated object using the planning component during the period, wherein controlling the third simulated object using the planning component comprises: 2.

K. The method of paragraph F, comprising: controlling a third simulated object based at least in part on the log data during the second time period in the driving simulation; determining a second interaction between the first simulated object and the third simulated object, after determining the second interaction; Controlling the third simulated object is based on the log data.

L. The method of paragraph F, wherein determining the interaction includes determining a first bounding box based on a first path associated with the first simulated object at the time of the driving simulation. determining a second bounding box based on a second path associated with the second simulated object at the time of the driving simulation; and determining the overlap with the bounding box of .

M. The method of paragraph F, wherein the first simulated object includes at least one of a simulated bicycle or a simulated pedestrian.

N. one or more non-transitory computer-readable media that, when executed, performs a driving simulation, the driving simulation being based at least in part on log data associated with a vehicle in an environment; , the log data includes first log data associated with a first object in the environment; and in a first time period, the first simulated object is associated with the first object in the environment. determining an interaction between the first simulated object and a second simulated object in the driving simulation; controlling the first simulated object using a planning component during a second time period in the driving simulation after a time period of . A non-transitory computer-readable medium containing possible instructions.

O. the one or more non-transitory computer-readable media of paragraph N, wherein the log data includes second log data associated with a second object in the environment; determining a distance between a position of the second object in the log data and a relative position of the second simulated object in the driving simulation, and the distance meeting or exceeding a threshold distance; initiating the planning component to control the first simulated object based at least in part on determining the non-transitory computer-readable medium.

P. The one or more non-transitory computer-readable media of paragraph N, wherein the operations include determining driving attributes associated with the first simulated object; determining a destination for the object, and providing the planning component with a first parameter associated with the driving attribute and a second parameter associated with the destination, the planning component comprising: determining a destination for the object; further comprising: being configured to control the first simulated object in the driving simulation based at least in part on the first parameter and the second parameter. computer-readable medium.

Q. one or more non-transitory computer-readable media of paragraph P, the driving attributes associated with the first simulated object; The destination is determined based at least in part on the first log data.

R. the one or more non-transitory computer-readable media of paragraph N, wherein the operation is performed during the second time period in the driving simulation based at least in part on the log data; determining a second interaction between the first simulated object and the third simulated object in the driving simulation; controlling the third simulated object using the planning component during a third time period in the driving simulation after the two time periods; controlling the simulated object of the non-transitory computer-readable medium further comprising: based at least in part on determining the second interaction.

S. the one or more non-transitory computer-readable media of paragraph N, wherein the operation is performed during the second time period in the driving simulation based at least in part on the log data; and determining a second interaction between the first simulated object and the third simulated object in the driving simulation. , controlling the third simulated object after determining the second interaction is based on the log data on a non-transitory computer-readable medium.

T. the one or more non-transitory computer-readable media of paragraph N, wherein determining the interaction is a first path associated with the first simulated object at the time of the driving simulation; and determining a second bounding box based on a second path associated with the second simulated object at the time of the driving simulation; determining an overlap between the first bounding box and the second bounding box.

U. one or more processors; and, when executed, receiving log data associated with an environment, the log data including first log data associated with an object within the environment. identifying the object as a vehicle; and identifying the object as a first vehicle based at least in part on the first log data rather than other vehicles in the log data. determining a first driving attribute associated with a first vehicle; a simulated environment corresponding to the environment; and a first simulated vehicle corresponding to the first vehicle. performing a driving simulation including: the first simulated vehicle being controlled by a planning component; and using the planning component to perform a driving simulation including: determining a driving decision for the first simulated vehicle based at least in part on a driving attribute of the first simulated vehicle; 1. A system comprising: controlling a simulated vehicle; and one or more computer-readable media storing computer-executable instructions that cause one or more processors to perform operations.

V. The system of paragraph U, wherein the operation determines a second driving attribute associated with a second vehicle in the environment based at least in part on second log data in the log data. the second driving attribute being different from the first driving attribute; and a system further comprising:

W. The system of paragraph U, wherein determining the first driving attribute associated with the first vehicle includes: determining the first driving attribute associated with the first vehicle based on the log data; determining an associated first value and a second value associated with a second instance of driving behavior of the first vehicle; and based on the first value and the second value. , at least partially determining a distribution associated with the first driving attribute, and determining a driving decision for the first simulated vehicle includes determining a third value from the distribution. A system that includes sampling.

X. The system of paragraph U, wherein the first driving path for the first vehicle traversing the environment is determined by the planning component for the first simulated vehicle traversing the simulated environment. A system different from the determined second driving route.

Y. The system of paragraph U, wherein determining a driving decision for the first simulated vehicle includes determining a driving decision for the first simulated vehicle based at least in part on the first log data. determining a first value of a driving attribute; determining a second driving attribute associated with the driving decision; and determining a first driving attribute between the first driving attribute and the second driving attribute. determining a weight value associated with the interaction; and determining the second value of the second driving attribute based at least in part on the first value of the first driving attribute and the weight value. A system comprising: determining a value;

Z. The method comprises: receiving log data associated with an environment, the log data including first log data associated with a first object within the environment; determining behavioral attributes associated with the first object based at least in part on log data of the first object; and performing a simulation including the first simulated object controlled by the planning component. using the planning component to determine a decision of the first simulated object during the simulation, the decision determining at least in part the behavioral attributes associated with the first object; A method comprising:

A.A. The method of paragraph Z, the method comprising: determining a second behavioral attribute associated with a second object in the environment based at least in part on second log data in the log data; , the second behavioral attribute is different from the behavioral attribute, and controlling the second simulated object during the simulation based at least in part on the second behavioral attribute. How to include more.

AB. The method of paragraph AA, wherein the first simulated object is a simulated electric vehicle in the simulation, and the second simulated object is a pedestrian object in the simulation. or a non-motorized vehicle object.

A.C. The method of paragraph Z, wherein determining the behavior attribute associated with the first object includes determining the behavior attribute associated with the first instance of the behavior of the first object based on the log data. 1 and a second value associated with a second instance of the behavior of the first object; and aggregating the first value and the second value to determine the value of the first object. determining a value associated with the behavioral attribute of the first object.

A.D. The method of paragraph Z, comprising: determining, based on the log data, a destination in the environment associated with the first object; determining a simulated destination of a simulated object, wherein determining the determination of the first simulated object is based at least in part on the simulated destination; and how to further include.

A.E. The method of paragraph Z, comprising: determining second log data associated with a second object in the environment, the first object and the second object having the same object classification; and determining a second behavioral attribute associated with the second object based at least in part on the second log data, the first simulated object determining a second behavioral attribute associated with the second object. the method further comprising: determining the second behavioral attribute based at least in part on the second behavioral attribute associated with the second object.

AF. The method of paragraph Z, comprising: determining a first path of the first object across the environment based at least in part on the first log data; determining a second path of the first simulated object across a simulated environment, the first path of the first object being determined by the first simulated object; differing from the second path determined by the planning component of the selected object.

A.G. The method of paragraph Z, wherein determining the determination of the first simulated object is based at least in part on the first log data, determining a first value of a behavioral attribute; determining a weight value associated with a correlation between the behavioral attribute and a second behavioral attribute; determining the second value of the second behavioral attribute based at least in part on the weight value; and determining the second value of the second behavioral attribute based at least in part on the second value of the second behavioral attribute. How to decide and include.

A.H. one or more non-transitory computer-readable media storing instructions executable by a processor, the instructions being executed to cause the processor to receive log data associated with an environment; , the log data includes first log data associated with a first object in the environment; and based at least in part on the first log data, the log data is associated with a first object. determining behavioral attributes of the first simulated object during the simulation; determining a determination of a first object, the determination being based at least in part on the behavioral attribute associated with the first object.

A.I. one or more non-transitory computer-readable media of paragraph AH, wherein the operation is associated with a second object in the environment based at least in part on second log data in the log data; determining a second behavioral attribute that is different from the behavioral attribute, the second behavioral attribute being different from the behavioral attribute; controlling the simulated object of 2.

A.J. the one or more non-transitory computer-readable media of paragraph AH, wherein determining the behavioral attributes associated with the first object determines the behavior of the first object based on the log data; determining a first value associated with a first instance of the behavior of the first object and a second value associated with a second instance of the behavior of the first object; and determining a value associated with the behavioral attribute of the first object.

A.K. the one or more non-transitory computer-readable media of paragraph AH, wherein the operations include determining a destination within the environment associated with the first object based on the log data; determining a simulated destination of a first simulated object in the simulation based on the log data; , based at least in part on the simulated destination.

AL. one or more non-transitory computer-readable media of paragraph AH, wherein the operation is to determine second log data associated with a second object in the environment; and the second object have the same object classification; and determining a second behavioral attribute associated with the second object based at least in part on the second log data. and determining the first simulated object's determination is based at least in part on the second behavioral attribute associated with the second object. computer readable medium.

A.M. the one or more non-transitory computer-readable media of paragraph AH, wherein the operation is based at least in part on the first log data of the first object of the first object across the environment; determining, by a planning component, a second path of the first simulated object across the simulated environment of the simulation; the first path being different from the second path determined by the planning component of the first simulated object.

A.N. the one or more non-transitory computer-readable media of paragraph AH, wherein determining the determination of the first simulated object is based at least in part on the first log data; determining a first value of the behavioral attribute associated with the first object; determining a weight value associated with a correlation between the behavioral attribute and the second behavioral attribute; determining the second value of the second behavioral attribute based at least in part on the first value of the pair of behavioral attributes and the weight value; determining a determination based at least in part on a value; and a non-transitory computer-readable medium.

Although the example provisions described above are described with respect to a particular implementation, in the context of this specification, the subject matter of the example provisions is applicable to a method, device, system, computer-readable medium, and/or another implementation. It should be understood that it may be performed via Furthermore, any of Examples A-AN may be implemented alone or in combination with any one or more of the other Examples A-AN.

(Conclusion)
Although one or more examples of the techniques described herein have been described, various modifications, additions, substitutions, and equivalents thereof are included within the scope of the techniques described herein.

In the description of the examples, reference is made to the accompanying drawings, which constitute a part of the specification, and which illustrate by way of example particular examples of the claimed subject matter. It is to be understood that other examples may be used and changes or modifications may be made, such as structural changes. Such examples, changes, or modifications do not necessarily depart from the scope of the intended claimed subject matter. Although the procedures described herein may be presented in an order, the order may in some cases be changed so that certain inputs occur at different times or in a different order without changing the functionality of the systems and methods described. It may be provided in Also, the disclosed procedures may be performed in a different order. Furthermore, the various calculations described herein need not be performed in the order disclosed, and other examples using alternative orders of calculations may be readily implemented. In addition to changing the order, calculations may be decomposed into sub-calculations with the same result.

Although the subject matter has been described in language specific to structural features and/or methodological acts, the subject matter as defined in the appended claims is not necessarily limited to the specific features or acts described. I want you to understand that this is not a thing. Rather, the specific features and acts are disclosed as example forms of implementing the claims.

The components described herein represent instructions that may be stored on any type of computer-readable medium and may be implemented in software and/or hardware. All of the methods and processes described above may be embodied in and/or implemented in software code modules and/or computer-executable instructions executed by one or more computers or processors, hardware, or a combination thereof. may be automated. Part or all of the method may alternatively be embodied in specialized computer hardware.

In particular, conditional phrases such as "may," "obtain," "may," or "may" mean that a particular example It is understood that some examples may include certain features, elements, and/or steps, while other examples may indicate that certain features, elements, and/or steps are not included. Thus, such conditional expressions generally indicate that a particular feature, element, and/or step is required in some way for one or more instances, or that one or more instances , necessarily include logic for determining whether a particular feature, element, and/or step is included or executed in any particular instance, with or without user input or prompting. It doesn't mean anything.

Unless otherwise specified, a conjunction such as "at least one of X, Y, or Z" means that the item, term, etc. includes any of X, Y, or Z, or multiples of each element. This is understood to indicate that there is a possibility of a combination. Unless explicitly stated as singular, "a" means singular and plural.

The routine descriptions, elements, or blocks in the flow diagrams described herein and/or depicted in the accompanying figures represent one or more computer implementations for performing particular logical functions or elements within the routine. It should be understood that it may represent a module, segment, or portion of code that includes possible instructions. Within the scope of the examples set forth herein, elements or functionality may be deleted, substantially synchronized, or in reverse order, depending on the functionality involved, as will be understood by those skilled in the art. Alternate implementations are included, such as with additional acts or with acts omitted, or performed in a different order than that illustrated or discussed.

Many variations and modifications may be made to the examples described above, and the elements thereof are understood to be included among other acceptable embodiments. All such modifications and variations are intended to be included herein within the scope of this disclosure and protected by the following claims.

Claims (15)

A system,
one or more processors;
When executed,
performing a driving simulation, the driving simulation being based at least in part on log data associated with a vehicle in an environment, the log data being a first log associated with a first object in the environment; Containing data;
controlling a first simulated object to correspond with the first object in the environment during a first time period;
determining an interaction between the first simulated object and a second simulated object in the driving simulation;
controlling the first simulated object using a planning component during a second time period in the driving simulation after the first time period;
one or more computer-readable media storing computer-executable instructions that cause the one or more processors to perform operations comprising:
A system equipped with. The log data comprises second log data associated with a second object in the environment, and the method includes:
determining a distance between a position of the second object in the second log data and a relative position of the second simulated object in the driving simulation;
initiating the planning component to control the first simulated object based at least in part on determining that the distance meets or exceeds a threshold distance;
The system of claim 1, further comprising: The said operation is
determining driving attributes associated with the first simulated object;
determining a destination for the first simulated object;
providing a first parameter associated with the driving attribute and a second parameter associated with the destination to the planning component; configured to control the first simulated object in the driving simulation based at least in part on parameters of;
The system according to claim 1 or 2, further comprising: The driving attributes associated with the first simulated object and the destination associated with the first simulated object are based at least in part on the first log data. 4. The system of claim 3, wherein: The said operation is
controlling a third simulated object during the second time period in the driving simulation based at least in part on the log data;
determining a second interaction between the first simulated object and the third simulated object in the driving simulation;
using the planning component to control the third simulated object during a third time period in the driving simulation after the second time period, using the planning component to control the third simulated object; controlling the third simulated object is based at least in part on determining the second interaction;
5. The system according to any one of claims 1 to 4, further comprising: The said operation is
controlling a third simulated object during the second time period in the driving simulation based at least in part on the log data;
determining a second interaction between the first simulated object and the third simulated object in the driving simulation;
Furthermore,
After determining the second interaction, controlling the third simulated object is based on the log data.
A system according to any one of claims 1 to 4. Determining the interaction comprises:
determining a first bounding box based on a first path associated with the first simulated object at the time of the driving simulation;
determining a second bounding box based on a second path associated with the second simulated object at the time of the driving simulation;
determining an overlap between the first bounding box and the second bounding box;
7. A system according to any one of claims 1 to 6, comprising: A method,
performing a driving simulation, the driving simulation being based at least in part on log data associated with a vehicle in an environment, the log data being a first log associated with a first object in the environment; Containing data;
controlling a first simulated object to correspond with the first object in the environment during a first time period;
determining an interaction between the first simulated object and a second simulated object in the driving simulation;
controlling the first simulated object using a planning component during a second time period in the driving simulation after the first time period;
A method of providing. The log data comprises second log data associated with a second object in the environment, and the method includes:
determining a distance between a position of the second object in the second log data and a relative position of the second simulated object in the driving simulation;
initiating the planning component to control the first simulated object based at least in part on determining that the distance meets or exceeds a threshold distance;
9. The method of claim 8, further comprising: determining driving attributes associated with the first simulated object;
determining a destination for the first simulated object;
providing a first parameter associated with the driving attribute and a second parameter associated with the destination to the planning component; configured to control the first simulated object in the driving simulation based at least in part on parameters of;
The method according to claim 8 or 9, further comprising: The driving attributes associated with the first simulated object and the destination associated with the first simulated object are based at least in part on the first log data. 11. The method of claim 10, wherein: controlling a third simulated object during the second time period in the driving simulation based at least in part on the log data;
determining a second interaction between the first simulated object and the third simulated object in the driving simulation;
using the planning component to control the third simulated object during a third time period in the driving simulation after the second time period, using the planning component to control the third simulated object; controlling the third simulated object is based at least in part on determining the second interaction;
12. The method according to any one of claims 8 to 11, further comprising: controlling a third simulated object during the second time period in the driving simulation based at least in part on the log data;
determining a second interaction between the first simulated object and the third simulated object in the driving simulation;
Furthermore,
After determining the second interaction, controlling the third simulated object is based on the log data.
12. A method according to any one of claims 8 to 11. Determining the interaction comprises:
determining a first bounding box based on a first path associated with the first simulated object at the time of the driving simulation;
determining a second bounding box based on a second path associated with the second simulated object at the time of the driving simulation;
determining an overlap between the first bounding box and the second bounding box;
14. A method according to any one of claims 8 to 13, comprising: One or more non-transitory computer-readable media, when executed by one or more processors, can carry out the method of any one of claims 8 to 14 to the one or more processors. A non-transitory computer-readable medium containing processor-executable instructions for execution by a computer.

Images