JP7422687B2 - Occlusion awareness planning - Google Patents

Description

The present invention relates to occlusion-aware planning.

[Cross reference to related applications]
This patent application claims priority to a U.S. utility patent application entitled "Occclusion Aware Planning," serial number 16/011,436, filed on June 18, 2018; Claims priority to United States Utility Patent Application entitled "Occclusion Aware Planning and Control," filed serial number 16/011,468. Applications with serial numbers 16/011,436 and 16/011,468 are fully incorporated herein by reference.

Various methods, devices, and systems are utilized in autonomous vehicles to guide the autonomous vehicle through environments that include multiple obstacles. For example, autonomous vehicles utilize route planning methods, devices, and systems that guide the autonomous vehicle through crowded areas that may include other vehicles, buildings, pedestrians, or other objects. In some instances, vehicles, buildings, and/or objects in the environment can prevent areas of the environment from being visible to the autonomous vehicle's sensors, creating challenges when planning routes through such areas. Can be presented.

A detailed description will be given with reference to the accompanying drawings. In the drawings, the left-most digit of a reference number identifies the figure in which the reference number first appears. The use of the same reference numbers in different drawings indicates similar or identical items or features.

FIG. 1 shows a perspective view of an example occlusion grid including multiple occlusion fields in an environment, according to an embodiment of the present disclosure, and illustrates an occlusion grid for determining occlusion and non-occlusion regions of the occlusion grid due to obstacles. 1 is an environment depicting a vehicle including monitoring components; FIG. 2 is an illustrated flow diagram of an example process for determining occlusion field occupancy in an occlusion grid based on beaming LIDAR data, according to embodiments of the present disclosure. FIG. 3 is an illustrated flow diagram of an example process for determining the occupancy of an occlusion field in an occlusion grid based on projecting the occlusion field onto image data, according to an embodiment of the present disclosure. FIG. 4 depicts a block diagram of an example system for implementing the techniques described herein. FIG. 5 is an illustrated flow diagram of an example process for generating an occlusion grid based on area characteristics, according to an embodiment of the present disclosure. FIG. 6 is an illustrated flow diagram of an example process for evaluating an occlusion grid at an intersection and controlling a vehicle to cross the intersection, according to an embodiment of the present disclosure. FIG. 7 is an illustrated flow diagram of another example process for evaluating an occlusion grid at an intersection and controlling a vehicle to cross the intersection, according to an embodiment of the present disclosure. FIG. 8 is an illustrated flow diagram of an example process for determining an occlusion region associated with an object and generating a predicted trajectory of a dynamic object within the occlusion region, according to an embodiment of the present disclosure. FIG. 9 is an illustrated flow diagram of an example process for evaluating a terrain occlusion grid and controlling a vehicle to traverse a terrain obstacle, according to an embodiment of the present disclosure. FIG. 10 illustrates determining occlusion field occupancy and controlling a vehicle based on beaming LIDAR data and/or based on projection of the occlusion field onto image data, according to embodiments of the present disclosure. 3 illustrates an example process for traversing an environment. FIG. 11 is an illustration for determining an occlusion grid and controlling a vehicle to traverse an environment based on map data showing the occlusion grid or based on identification of obstacles in the environment, according to embodiments of the present disclosure. The process is shown below.

This disclosure is directed to techniques for controlling vehicles, such as autonomous vehicles, based on regions of occlusion in an environment. For example, an occlusion region can be represented by an occlusion grid, which can be stored in association with map data or dynamically generated based on obstacles in the environment. An occlusion grid may include multiple occlusion fields and may represent distinct regions of the environment, such as a drivable road surface. In some examples, the occlusion field includes occlusion conditions (e.g., indicating whether the location is visible to one or more sensors of the autonomous vehicle) and occupancy conditions (e.g., indicating whether the location is visible to a vehicle, pedestrian, (indicates whether the object is occupied by an object such as an animal). In at least some examples, the occupancy state may further include an "undetermined" state (i.e., it may not currently be known whether the occlusion field is occupied, even based on the available data). In some cases, LIDAR data and/or image data captured by sensors on the autonomous vehicle can be used to determine occlusion conditions and/or occupancy conditions. For example, LIDAR data can be beamed to determine whether a LIDAR ray passes through a region of space represented by an occlusion field. In some examples, the confidence level that the occlusion field is occupied or unoccupied depends on the first number of expected LIDAR returns associated with the occlusion field and/or on the actual number of LIDAR returns associated with the occlusion field. may be based at least in part on a second number of LIDAR returns. In some examples, the confidence level of whether the occlusion field is occupied may be based, at least in part, on LIDAR observations in the column of LIDAR returns regarding the height of the occlusion field. In some examples, occlusion field occupancy can be based on image data. For example, image data can be captured in an environment and the image data can be segmented to generate segmented image data that shows driveable areas (eg, road surfaces not occupied by objects). Location data representative of the occlusion field may be projected onto the segmented image data to determine whether the occlusion field is associated with a drivable road surface. Occlusion states and occupancy states can be determined for all occlusion fields of the occlusion grid. In some examples, if the occlusion field is all visible and unoccupied, the occlusion grid can be considered clear and the autonomous vehicle can be controlled to traverse the environment.

In some examples, LIDAR data and image data can be used to determine occlusion and occupancy conditions of an occlusion field. For example, for an occlusion field relatively close to an autonomous vehicle, the resolution of the LIDAR sensor may be sufficient to determine the occlusion and occupancy states of the occlusion field with a confidence level (also referred to as a confidence value) greater than or equal to a threshold. In some examples, image data is can be used to update or supplement decisions regarding occlusion and/or occupancy conditions using LIDAR data.

The techniques discussed herein can be used in a variety of situations. In the first situation, occlusion-based planning can be used when an autonomous vehicle is navigating through an intersection where the autonomous vehicle is performing complex maneuvers such as unprotected left turns, free right turns, and navigating through complex intersections. As can be appreciated, in such instances the oncoming vehicle may not stop at the intersection, which can present challenges in determining when it is safe to cross the intersection. In one example, an autonomous vehicle may approach an intersection and stop at a stop line (e.g., shown in map data). The occlusion grid can be accessed from the map data, and autonomous vehicles can ingest sensor data to determine the state of the occlusion field. In some instances, the autonomous vehicle detects that part of the occlusion grid is occluded (e.g., due to an obstacle such as a parked car) and/or lacks sufficient information to proceed. It can be determined that In some examples, an autonomous vehicle may gently traverse an intersection to change the location of sensors within the environment. The autonomous vehicle may continue to capture sensor data to determine the occlusion status and occupancy status of some or all of the occlusion fields. If other vehicles are crossing the intersection, the occlusion field is determined to be occupied and the autonomous vehicle may continue to wait for an appropriate amount of time to cross the intersection. If there are no other vehicles crossing the intersection and the sensors can determine the occupancy state of the field, the occlusion field is unoccupied, the occlusion grid is clear, and the autonomous vehicle may cross the intersection. As can be appreciated, occlusion-based planning can be used to traverse a variety of intersections and is not limited to particular intersections or maneuvers.

In a second situation, the autonomous vehicle may capture sensor data of the environment and determine that an object in the environment is "preventing" the sensor from sensing a portion of the environment. In some examples, occlusion grids can be dynamically associated with objects. In some instances, the autonomous vehicle may have little information about the occlusion grid initially (e.g., the occupancy of the occlusion field, especially if it is obscured by obstacles); By monitoring the area an object traverses before entering the occlusion grid, the context of the occlusion field can be inferred. For example, because the autonomous vehicle monitors the occlusion area of the occlusion grid, the autonomous vehicle can monitor any object (eg, vehicle, pedestrian, etc.) that enters the occlusion grid. After a period of time, if no objects have entered the occlusion grid, and if no objects have left the occlusion grid, the autonomous vehicle may infer that the occlusion grid is unoccupied and execute the vehicle trajectory as necessary. obtain.

In a third situation, an autonomous vehicle can capture sensor data of the environment and identify objects within the environment. An autonomous vehicle may associate an occlusion grid with an object as discussed herein. In some examples, the autonomous vehicle can also identify one or more dynamic objects, such as pedestrians, in the environment that have crossed the occlusion area. The autonomous vehicle can monitor the trajectory of a dynamic object before the object moves into the occlusion region and can generate one or more predicted trajectories associated with the object. In some examples, the one or more predicted trajectories are based on measured trajectories of the moving object and/or based on a reasonable path through the environment (e.g., crossing a crosswalk or sidewalk, a continuation of the most recent trajectory). etc.) and/or destination. Therefore, even if the dynamic object is not observed, for objects in the occlusion region, the autonomous vehicle can incorporate the predicted trajectory of the dynamic object into decisions regarding the control of the autonomous vehicle. Such predicted trajectories may later be confirmed by subsequent observations based on dynamic objects remaining in the occlusion region or based on dynamic objects leaving the occlusion region.

The fourth situation targets occlusion areas caused by topographical obstacles such as hill tops. For example, as a vehicle approaches the top of a hill (eg, apex), it may become impossible for the vehicle to see the ground on the other side of the hill top. In some examples, a topographic occlusion grid can be associated with map data representing areas of hills to facilitate traversing hills. For example, a topographic occlusion grid (and other occlusion grids discussed herein) may include an occlusion field that represents a three-dimensional region of space within the environment. As can be appreciated, the vertical portion of the occlusion field may be visible from a vehicle approaching the top of a hill, even if the vehicle is not able to see the ground portion of the occlusion field. The vehicle can capture sensor data of the environment, such as LIDAR data, beam the LIDAR data onto an occlusion field of the occlusion grid, and determine an occupancy of at least a portion of the occlusion field. Therefore, by determining that a portion of the occlusion field is clear (e.g., an area of the occlusion field between a meter or two above the ground), the autonomous vehicle determines the occlusion state of the occlusion field (e.g., occupied, unoccupied, or Undecided) can be determined. Thus, using the techniques discussed herein, autonomous vehicles can "see over" hilltops and obtain information about the environment without having to capture sensor data for all areas of the environment.

In some examples, the size of the occlusion grid stored in association with the map data can be based at least in part on some area characteristics. In the context of an intersection, area characteristics include, but are not limited to, the distance an autonomous vehicle must travel across an intersection (e.g., to avoid oncoming traffic), the speed limit for vehicles within the intersection (e.g., to avoid oncoming traffic) , a safety factor associated with the speed limit (e.g., to effectively speed up expected traffic), an acceleration level and/or average speed of the autonomous vehicle traversing the intersection distance, and the like. Such specific examples are discussed in detail below. Accordingly, the occlusion grid may be sized such that an autonomous vehicle can safely traverse the area if the occlusion grid is free of obstructions.

In some examples, the occlusion grid may include multiple occlusion fields, as described above. In some examples, the occlusion field may represent a temporal logic symbol that can be used in a temporal logic equation to evaluate or verify an autonomous vehicle's trajectory. For example, the autonomous vehicle may be prevented from starting a trajectory until the occlusion state and/or occlusion state of the occlusion field becomes "unoccupied" and "unoccluded," respectively. In some examples, the autonomous vehicle's planner system may use temporal logic, such as linear temporal logic or signal temporal logic, to evaluate the trajectory of the autonomous vehicle.

The techniques discussed herein can improve the functionality of computing devices in several ways. For example, in the context of evaluating occlusion grids, the area of the grid may be sized to ensure safe traversal of the area by autonomous vehicles without excessive resources devoted to unnecessary decisions about the environment. can. In some examples, multiple sensor modalities (LIDAR sensors, image sensors, RADAR sensors, etc.) can be used to improve the overall confidence level associated with occlusion conditions or occlusion field occupancy conditions. Improved trajectory generation can improve safety outcomes and improve rider experience (e.g., by ensuring intersections are clear before starting a trajectory, reducing the occurrence of emergency braking, turns, etc.) by). These and other improvements to computer functionality and/or user experience are discussed herein.

The techniques described herein can be implemented in several ways. Example implementations are described below with reference to the following figures. Although discussed in the context of autonomous vehicles, the methods, apparatus, and systems described herein are applicable to a variety of systems (e.g., robotic platforms) and are not limited to autonomous vehicles. In another example, this technique can be utilized in an aviation or navigational context, or in any system that uses machine vision. Additionally, the techniques described herein can be used with real data (eg, captured using sensors), simulated data (eg, generated by a simulator), or any combination of the two.

FIG. 1 is an environment 100 illustrating a perspective view of an example occlusion grid that includes multiple occlusion fields within the environment. FIG. 1 also illustrates a vehicle including an occlusion monitoring component for determining occlusion and non-occlusion areas of an occlusion grid due to obstacles, according to embodiments of the present disclosure.

As shown, environment 100 may include a vehicle 102 that includes one or more sensor systems 104 that capture data representative of environment 100 .

For purposes of illustration, vehicle 102 is described by the National Highway Traffic Safety Administration as a vehicle that does not expect constant driver (or occupant) control of the vehicle and is capable of performing all safety-critical functions for the entire trip. It may be an autonomous vehicle configured to operate in accordance with an issued Level 5 classification. In such an example, the vehicle 102 can be configured to control all functions from start to stop, including all parking functions, and thus can be unmanned. This is just an example, and the systems and methods described herein can be applied to any ground vehicle, including those that require manual control at all times by the driver, to those that are partially or fully autonomously controlled. can be incorporated into , airborne, or waterborne vehicles. Additional details associated with vehicle 102 are described below.

In at least one example, vehicle 102 can be associated with a sensor system 104 that can be positioned on vehicle 102, as described above. The sensor system 104 may include a light detection and ranging (LIDAR) sensor, a radio detection and ranging (RADAR) sensor, an ultrasound transducer, an acoustic guidance and ranging (SONAR) sensor, a position sensor (e.g., a global positioning system (GPS) ), compasses, etc.), inertial sensors (e.g. inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g. RGB, IR, intensity, depth, time of flight, etc.), wheel encoders, microphones, environmental sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.). Sensor system 104 can generate sensor data that can be utilized by vehicle computing device 106 associated with vehicle 102 .

In at least one example, vehicle computing device 106 may utilize sensor data of occlusion monitoring component 108. For example, occlusion monitoring component 108 can access occlusion data and/or map data to determine occlusion conditions and occupancy conditions of occlusion fields within an occlusion grid.

By way of example and without limitation, a vehicle 102 within environment 100 is approaching an intersection 110. In some examples, vehicle 102 can access map data and determine that occlusion grid 112 is associated with intersection 110. In some examples, intersection 110 may represent an intersection where vehicle 102 must yield to a one-way vehicle entering intersection 110.

Occlusion grid 112 may include multiple occlusion fields that separate portions of environment 100 into distinct regions. In some examples, occlusion field 114 can represent an occlusion condition (eg, an indication of whether occlusion field 114 is within sensing area 116 associated with one or more sensors of vehicle 102). As can be appreciated, environment 100 may include one or more obstacles 118, 120, and 122 that cause a portion of occlusion grid 112 to be missed from sensing area 116 of vehicle 102. Accordingly, non-occlusion areas 124 of occlusion grid 112 may represent areas of occlusion grid 112 that can be “seen” by vehicle 102. That is, vehicle 102 may capture sensor data representative of non-occlusion area 124 . Similarly, occlusion region 126 may represent an area of occlusion grid 112 that cannot be “seen” by vehicle 102. As shown in FIG. 1, the occlusion field within the occlusion region 126 is shaded gray to represent the lack of sensor data corresponding to that region, while the non-occlusion region 124 is not shaded.

In some examples, if the occlusion field is occupied by an object, the occlusion field can store additional metadata indicating the object's ID and data indicating the object's path through the occlusion grid.

Additional details for evaluating occlusion fields using LIDAR data and image data are discussed below in connection with FIGS. 2 and 3, respectively.

FIG. 2 is an illustrated flow diagram of an example process 200 for determining occlusion field occupancy in an occlusion grid based on beaming LIDAR data, according to embodiments of the present disclosure. Although discussed in the context of LIDAR data, example process 200 can be used in the context of RADAR data, SONAR data, time-of-flight image data, and the like.

At operation 202, the process can include receiving map data that includes an occlusion grid. Example 204 illustrates map data 206 that can be received in operation 202. In some examples, map data 206 may represent intersection 110 as shown in FIG. As further shown in example 204, vehicle 102 captures LIDAR data of the environment as illustrated by LIDAR ray 208. Although shown as a single line, LIDAR light beam 208 can include multiple light beams that scan the environment.

In some examples, map data 206 includes an occlusion grid 210 that includes multiple occlusion fields 216. As discussed throughout this disclosure, occlusion grid 210 can represent any size or shape of the environment. In some examples, occlusion grid 210 may include multiple occlusion fields 216, such that each occlusion field represents a separate region of the environment. In some examples, the occlusion field may represent an area that is 1 meter wide, 3 meters long, and 2 meters high, although any size may be used for the occlusion field. In some examples, an occlusion field may be considered a "meta-voxel" that represents multiple voxels in a voxelized space.

In operation 212, the process can include determining occupancy of one or more occlusion fields based at least in part on beaming the LIDAR data. In environment details 214, occlusion field 216 is shown with LIDAR rays traversing occlusion field 216. As shown in detail 214, LIDAR rays 218 and 220 are shown traversing occlusion field 216. As further shown in example 204, LIDAR ray 218 may represent a LIDAR ray without a return. This means that the LIDAR beam 218 was emitted by the LIDAR sensor and expected to traverse the occlusion field 216, but the vehicle 102 did not receive the return of the LIDAR beam 218. Such a lack of return may indicate that the occupancy state is undetermined because the LIDAR ray 218 may not have hit the surface and returned, or may have been reflected off a reflective surface away from the LIDAR sensor. . Furthermore, LIDAR ray 220 represents a LIDAR ray with a return. This means that LIDAR light 220 was emitted by the LIDAR sensor and received by vehicle 102 at a subsequent time (eg, by reflection from an object in the environment).

In some examples, LIDAR rays 208 may indicate that an occlusion field (eg, occlusion field 216) is occupied by an object. In this case, operation 212 may include determining that occlusion field 216 is occupied. In at least some examples, occupancy may be determined based on the number of LIDAR returns within a vertical column on a particular occlusion field 216 (eg, as shown in detail 226). As a non-limiting example, a LIDAR return may indicate that the area is occupied or that the LIDAR ray traversed the area before being received as a reflection by the LIDAR sensor (e.g., a return indicates that the area is unoccupied). ). In some examples, LIDAR light may traverse an area without being received by a LIDAR sensor, as discussed below.

As a non-limiting example, the vertical column associated with occlusion field 216 may include five voxels (e.g., in detail 226, voxels 216(1), 216(2), 216(3), 216(4), and 216 (5)) and a majority of them indicate that the object is located within the voxel, the occupancy state reaches the occupancy threshold and the occlusion field 216 is determined to be occupied. Ru. In other words, in some examples, the occupancy of occlusion field 216 may be based at least in part on occupancy scores associated with voxels associated with occlusion field 216. Of course, an occlusion field can be associated with any number of vertical and/or horizontal voxels.

In some examples, LIDAR rays 208 may indicate that occlusion field 216 is not occupied by an object. For example, if a majority of the voxels associated with occlusion field 216 exhibit no returns (and/or based on the number of LIDAR rays traversing the voxel and associated with returns compared to the expected number of returns) ), there is likely no object occupying the occlusion field 216, in contrast to all light rays reflecting off surfaces away from the LIDAR sensor. In some examples, decisions regarding occlusion field 216 occupancy may be associated with a confidence level. In some examples, the confidence level of the occlusion field occupancy is determined by a first number of expected LIDAR returns associated with the occlusion field and a second number of actual LIDAR returns associated with the occlusion field (e.g. , indicating an object or a non-object). By way of example and without limitation, the confidence level can be based at least in part on the ratio of the second number and the first number. Of course, any technique or heuristic can be used to determine the confidence level associated with occlusion field occupancy. In some instances where a confidence level is not exceeded, sensor data is not returned, or it is otherwise impossible to make an occupancy determination, the occupancy state associated with occlusion field 216 may be undetermined (e.g., It may be determined that there is insufficient information to determine occupancy status.

At operation 222, the process can include commanding the autonomous vehicle based at least in part on the occupancy. For example, if vehicle 102 determines that occlusion field 216 is unoccupied (and all occlusion fields of occlusion grid 210 or a threshold number of occlusion fields of occlusion grid 210 are unoccupied), then control of vehicle 102 Trajectory 224 may be followed to cross intersection 110. In some examples, operation 222 controls the autonomous vehicle (e.g., vehicle 102) if occlusion field 216 is occupied and/or if a threshold number of occlusion fields of occlusion grid 210 are occupied by objects. This may include causing the vehicle to wait without crossing the intersection 110. In some examples, the threshold may be set based on, for example, the expected maximum speed limit, the width of the lane being crossed, and the like. Such a determination may include, for example, the time it would take for the vehicle 102 to cross the intersection safely, assuming that the vehicle is proceeding through the occlusion area at the maximum speed limit (plus some buffer) toward the intersection. This can be done based on. Additional details are discussed in connection with FIG.

In some examples, operation 222 may include selecting an acceleration level and/or speed of vehicle 102. For example, the occlusion field may indicate an occupied occlusion field that effectively reduces the size of the clear portion of occlusion grid 210. In some examples, operation 222 accelerates vehicle 102 faster based on the reduced size of the clear portion of occlusion grid 210 (e.g., expressed as a distance between vehicle 102 and the occupied occlusion field). , may include selecting a higher acceleration level (e.g., "boost mode"). In some examples, operation 222 may include determining that a portion of the occlusion grid is not visible (e.g., similar to occlusion region 126) or contains an occupancy state that is otherwise undetermined. , and operation 222 may include controlling vehicle 102 to slowly move to the intersection (eg, "slow forward") to increase the visibility area of occlusion grid 210. Additional examples are discussed throughout this disclosure.

FIG. 3 is an illustrated flow diagram of an example process 300 for determining the occupancy of an occlusion field in an occlusion grid based on a projection of the occlusion field onto image data, according to an embodiment of the present disclosure.

At operation 302, the process can include receiving image data and map data. For example, image data 304 may be captured by one or more image sensors of vehicle 102. In some examples, map data 306 may be accessed from memory (eg, from vehicle computing device 106) upon determining that vehicle 102 is approaching an intersection 110 associated with map data 306.

Generally, image data 304 may include any image data captured by one or more image sensors of a vehicle. In some cases, image data 304 may include representations of obstacles 308, 310, and 312. As shown, obstacles 308, 310, and 312 may represent vehicles parked on the side of the road.

Generally, map data 306 may include an occlusion grid 314 that includes multiple occlusion fields, such as occlusion field 316. In some examples, map data 306 may include obstacles 308, 310, and 312; however, in some cases, data for obstacles 308, 310, and 312 may not be considered map data. be. That is, in some cases, map data 306 may include only static objects. In that case, obstacles 308, 310, and 312 (eg, moving objects or potential moving objects) may not be included as map data 306. Further, although vehicle 102 may or may not be considered part of map data 306, vehicle 102 is shown in map data 306 to form context and to indicate the location of vehicle 102 with respect to intersection 110. has been done.

As shown, an obstacle 312 represents traveling on the road toward the intersection 110. In some examples, obstruction 312 can occupy occlusion field 326.

At operation 318, the process may include performing semantic segmentation on the image data. In some examples, image data 304 may include, but is not limited to, drivable surfaces (e.g., surfaces not occupied by objects or obstacles), ground, vehicles, pedestrians, buildings, sidewalks, vegetation, light poles, mailboxes, etc. can be input into a segmentation algorithm to identify regions and/or objects within the image data 304, including, for example, regions and/or objects within the image data 304. In some examples, operation 318 may include segmenting image data 304 using a machine learning algorithm trained to segment and/or classify objects within image data 304.

At operation 320, the process can include determining occupancy of the one or more occlusion fields based at least in part on projecting the one or more occlusion fields onto the segmented image data. For example, segmented image data 322 with projected map data shows occlusion grid 314 and occlusion fields 316 and 326 projected onto image data 304. As further shown, segmented image data 322 (also referred to as data 322) with projected map data includes segmented obstacles 308', 310', and 310' corresponding to obstacles 308, 310, and 312, respectively. 312'. Because the occlusion field 326 is projected onto the obstacle 312' that can be determined to be a vehicle (e.g., via semantic classification), the operation 320 can determine that at least the occlusion field 326 is occupied by the obstacle 312'. .

Operation 320 may project three-dimensional occlusion grid 314 and/or occlusion field 316 onto image data 304 using any algorithm. In one example, when projecting occlusion field 316 onto an area labeled as a drivable surface, operation 320 may determine that occlusion field 316 represents an unoccupied field. In some examples, operation 320 may determine that occlusion field 326 is occupied if occlusion field 326 is projected onto object 312'. In at least some examples, additional occlusion fields may correspond to occlusion states that are pending. In other words, it may be impossible to clearly state whether an occlusion field is occupied or not, since the image of the object effectively obstructs some occlusion fields. Regardless of any of the examples described herein, for safety purposes, an undetermined state may be treated as an occlusion state.

At operation 324, the process may include commanding the autonomous vehicle (eg, vehicle 102) based at least in part on the occupancy. For example, if vehicle 102 determines that occlusion field 316 is unoccupied (and all occlusion fields of occlusion grid 314 are unoccupied), vehicle 102 may be controlled to follow a trajectory that traverses intersection 110. In some examples, if occlusion field 326 is occupied, undetermined, and/or if a threshold number of occlusion fields of occlusion grid 314 are occupied by objects, operation 324 may cause the autonomous vehicle (e.g., The vehicle 102) may be controlled to wait without crossing the intersection 110. In some instances, if the sufficiency of information is below a threshold (e.g., the number of occupied or undetermined occlusion fields is greater than or equal to the threshold, and/or the number of occlusion and invisible occlusion fields travels through the region) If the planner is unable to determine a safe trajectory for may include instructing the user to change the .

In some examples, information sufficiency is such that the number of occlusion fields that are unoccupied is greater than or equal to a threshold number, and/or the number of visible occlusion fields is greater than or equal to a certain number (e.g., the number of occlusion fields that are unoccupied is greater than or equal to a certain number) be able to respond to a decision (sufficient to exceed a certain threshold for safe planning). In some examples, the sufficiency of information may include observing the undetermined occlusion field for a period of time to increase the level of confidence that the occlusion field is unoccupied. In some examples, the sufficiency of information may be related to the distance of the intersection traveled by the autonomous vehicle and/or related to the speed, acceleration, and/or time of the autonomous vehicle traversing a portion of the intersection. The method may include determining the extent of occlusion and unoccupied areas.

In some examples, processes 200 and 300 can be executed in parallel (e.g., substantially simultaneously) and the output of each process can be compared to determine the state of one or more occlusion fields and/or the occlusion grid. It can increase the overall level of confidence about the condition. In some examples, LIDAR data may be used to determine the state of an occlusion field that is within a distance threshold to the vehicle 102, while image data may be used to determine the state of an occlusion field that is within a distance threshold from the vehicle 102. It is also possible to determine the state. Of course, the examples described herein are not intended to be limiting, and other implementations are considered within the scope of this disclosure.

FIG. 4 depicts a block diagram of an example system 400 for implementing the techniques described herein. In at least one example, system 400 can include a vehicle 402, which can correspond to vehicle 102 of FIG.

Vehicle 402 includes a vehicle computing device 404, one or more sensor systems 406, one or more emitters 408, one or more communication connections 410, at least one direct connection 412, and one or more drive modules 414. can include.

Vehicle computing device 404 may include one or more processors 416 and memory 418 communicatively coupled to one or more processors 416 . In the illustrated example, vehicle 402 is an autonomous vehicle. However, vehicle 402 may be any other type of vehicle or any other system having at least an image capture device (eg, a camera-enabled smartphone). In the illustrated example, the memory 418 of the vehicle computing device 404 includes a location component 420, a perception component 422, a planning component 424, one or more system controllers 426, one or more maps 428, a prediction component 430, an occlusion monitoring component 432, an occlusion region component 434, a ray emission component 436, a segmentation component 438, a projection component 440, an occlusion context component 442, and a velocity component 444. As shown in FIG. 4 as residing in memory 418 for illustrative purposes, a location component 420, a perception component 422, a planning component 424, one or more system controllers 426, one or more maps 428, a prediction component 430 , occlusion monitoring component 432, occlusion region component 434, light emission component 436, segmentation component 438, projection component 440, occlusion context component 442, and velocity component 444 are additionally or alternatively accessible to vehicle 402 ( For example, it is contemplated that the information may be stored in or otherwise accessible in memory remote from vehicle 402. In some examples, vehicle computing device 404 can correspond to vehicle computing device 106 of FIG.

In at least one example, position measurement component 420 receives data from sensor system 406 and determines the position and/or orientation (e.g., one or more x, y, z positions, roll, pitch, or yaw) of vehicle 402. It can include the ability to determine. For example, positioning component 420 can include and/or request/receive a map of the environment and can continually determine the position and/or orientation of the autonomous vehicle within the map. In some examples, the localization component 420 utilizes SLAM (simultaneous localization and mapping), CLAMS (simultaneous calibration, localization and mapping), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc. data, LIDAR data, RADAR data, IMU data, GPS data, wheel encoder data, etc., to accurately determine the position of an autonomous vehicle. In some examples, position measurement component 420 provides data to various components of vehicle 402 to generate trajectories and/or to store map data including occlusion grids in memory, as discussed herein. The initial position of the autonomous vehicle can be determined for deciding to search from.

In some examples, perception component 422 may include functionality to perform object detection, segmentation, and/or classification. In some examples, the perception component 422 detects the presence and/or entity type of an entity proximate to the vehicle 402 (e.g., car, pedestrian, bicycle, animal, building, tree, road surface, curb, sidewalk, unknown, etc.). can provide processed sensor data that indicates the classification of entities. In additional and/or alternative examples, the perception component 422 transmits processed sensor data indicative of one or more characteristics associated with a detected entity (e.g., a tracked object) and/or the environment in which the entity is located. can be provided. In some examples, characteristics associated with an entity include, but are not limited to, x position (global and/or local position), y position (global and/or local position), z position (global and/or local position). , direction (eg, roll, pitch, yaw), entity type (eg, classification), velocity of the entity, acceleration of the entity, range (size) of the entity, etc. Characteristics associated with the environment may include, but are not limited to, the presence of another entity within the environment, the state of another entity within the environment, time of day, day of the week, season, weather conditions, darkness/light display, etc. . In some examples, perception component 422 can operate in conjunction with segmentation component 438 to segment image data, as discussed herein.

In general, planning component 424 can determine a route that vehicle 402 will take to traverse the environment. For example, planning component 424 can determine various routes and trajectories and various levels of detail. For example, planning component 424 can determine a route to travel from a first location (eg, a current location) to a second location (eg, a target location). For purposes of this discussion, a route can be a series of waypoints for driving between two locations. By way of non-limiting example, way points include roads, intersections, Global Positioning System (GPS) coordinates, and the like. Additionally, planning component 424 can generate instructions for guiding the autonomous vehicle along at least a portion of the path from the first location to the second location. In at least one example, the planning component 424 can determine how to guide the autonomous vehicle from a first waypoint in the series of waypoints to a second waypoint in the series of waypoints. In some examples, an instruction can be a trajectory or a portion of a trajectory. In some examples, multiple trajectories can be generated substantially simultaneously (e.g., within technical tolerances) in accordance with the receding horizon technique, and one of the multiple trajectories is selected for travel of vehicle 402. .

In some examples, planning component 424 can generate one or more trajectories for vehicle 402 based at least in part on conditions of the occlusion field and/or occlusion grid, as discussed herein. In some examples, planning component 424 can evaluate one or more trajectories of vehicle 402 using temporal logic, such as linear temporal logic and/or signal temporal logic. Details of the use of temporal logic in planning component 424 are discussed in US patent application Ser. No. 15/632,147, which is incorporated herein by reference in its entirety.

In at least one example, vehicle computing device 404 can include one or more system controllers 426 that control steering, propulsion, braking, safety, emitter, communications, and other systems of vehicle 402. Can be configured to control These system controllers 426 can communicate with and/or control corresponding systems of drive module 414 and/or other components of vehicle 402 .

Memory 418 can further include one or more maps 428 that can be used by vehicle 402 as it travels through the environment. For the purposes of this description, maps can provide information about the environment such as, but not limited to, topology (such as intersections), streets, mountain ranges, roads, terrain, and the general environment in two dimensions, three dimensions, or any number of data structures modeled in N dimensions. In some examples, the map includes, but is not limited to, texture information (e.g., color information (e.g., RGB color information, lab color information, HSV/HSL color information), etc.), intensity information (e.g., LIDAR information, RADAR information), etc. information), spatial information (e.g. image data projected onto a mesh, individual "surfels" (e.g. polygons associated with individual colors and/or intensities)), reflectance information (e.g. specular reflectance information) , retroreflectivity information, BRDF information, BSSRDF information, etc.). In one example, the map may include a three-dimensional mesh of the environment. In some examples, maps can be stored in tile format and loaded into working memory as needed, such that individual tiles of the map represent separate parts of the environment. In at least one example, one or more maps 428 can include at least one map (eg, an image and/or a mesh). In some examples, vehicle 402 can be controlled based at least in part on map 428. That is, map 428 is used in conjunction with localization component 420, perception component 422, and/or planning component 424 to determine the location of vehicle 402, identify objects in the environment, and/or identify objects in the environment. A route and/or trajectory for traveling can be generated.

In some examples, one or more maps 428 can be stored on a remote computing device (such as computing device 448) that is accessible via network 446. In some examples, multiple maps 428 can be stored based on characteristics (eg, entity type, time of day, day of the week, season, etc.), for example. Storing multiple maps 428 has similar memory requirements, but can increase the speed at which data within the maps can be accessed.

In some examples, one or more maps 428 can store occlusion grids associated with individual locations within the environment. For example, as vehicle 402 traverses the environment and a map representing an area proximate to vehicle 402 is loaded into memory, one or more occlusion grids associated with the location may also be loaded into memory.

In some examples, prediction component 430 may include functionality to generate predicted trajectories of objects within the environment. For example, prediction component 430 can generate one or more predicted trajectories for vehicles, pedestrians, animals, etc. within a threshold distance from vehicle 402. In some examples, as discussed in connection with FIG. 8, the prediction component 430 can measure the footprint of the object and generate a trajectory of the object as it enters an occlusion region in the environment. Thus, even if an object is not visible to one or more sensors of vehicle 402, prediction component 430 maintains the object's trajectory in memory, even if the object is not captured in sensor data. can be extended.

In general, occlusion monitoring component 432 is capable of generating and/or accessing occlusion grid data, determining occlusion and non-occlusion areas of an occlusion grid, and determining occlusion states and/or occupancy states of occlusion fields of an occlusion grid. can include. In some examples, occlusion monitoring component 432 can correspond to occlusion monitoring component 108 of FIG. 1. As discussed herein, occlusion monitoring component 432 can receive LIDAR data, image data, map data, etc. to determine occlusion-related information in the environment. In some examples, occlusion monitoring component 432 can provide occlusion information to planning component 424 to determine when to control vehicle 402 to traverse the environment. Occlusion monitoring component 432 can provide occlusion information to prediction component 430 to generate predicted trajectories of one or more objects within an occlusion region in the environment. Additional details of occlusion monitoring component 432 are discussed in conjunction with the remaining components stored in memory 418.

Occlusion region component 434 determines portions of the occlusion grid that are occluded (e.g., similar to occlusion region 126 of FIG. 1) and portions of the occlusion grid that are non-occlusion (e.g., non-occlusion region 124 of FIG. 1). (similar to). For example, occlusion area component 434 can receive LIDAR data, image data, RADAR data, etc. to determine whether an area is “visible” from one or more sensors of vehicle 402. In some examples, occlusion region component 434 may include functionality to project sensor data onto map data to determine regions of the map where no data is located. In some examples, map data can be used to determine that there is a region "outside" the observable region, and the region can be determined to be an occlusion region. In some examples, occlusion region component 434 can dynamically generate occlusion regions based on objects in the environment.

The beam emitting component 436 may include functionality to receive the LIDAR data and beam the LIDAR data through the occlusion field to determine the occlusion status and/or occupancy status of a particular occlusion field. In some examples, the light emitting component 436 can determine the expected number of LIDAR returns associated with a particular occlusion field and can determine the actual number of LIDAR returns associated with a particular field. In some examples, the light emitting component 436 can determine a confidence level associated with the occlusion state and/or occupancy state of the occlusion field, as discussed herein. As can be appreciated, the functionality discussed herein is not limited to LIDAR data, but uses multiple sensor technologies (e.g., RADAR, SONAR, time-of-flight imagers, depth cameras, etc.) to determine occlusion conditions and/or occupancy. Can be used to determine status.

Segmentation component 438 may include functionality to receive image data and segment the image data to identify various objects and/or regions represented in the image data. For example, the segmentation component 438 identifies and categorizes image data including, but not limited to, drivable surfaces, free space (e.g., drivable surfaces), and/or non-free space, vehicles, pedestrians, buildings, vegetation, etc. can include one or more machine learning algorithms trained to segment. In some examples, segmentation component 438 can operate in conjunction with perception component 422 to perform semantic segmentation on image data and/or LIDAR data, for example.

Projection component 440 may include functionality to receive segmented image data and map data that includes an occlusion grid and occlusion field, and to project the occlusion field onto the segmented image data. In some examples, projection component 440 determines whether to project the occlusion field onto a drivable surface (e.g., a clear road), occupying area, or otherwise undetermined objects (e.g., a vehicle). By determining , the occupancy of one or more occlusion fields can be determined.

The occlusion context component 442 may include functionality to determine the occupancy of an occlusion grid, particularly an occlusion region, based on the context of the environment. As can be appreciated, portions of the occlusion grid may be occluded by objects such that vehicle 402 cannot capture sensor data in the occluded area. In some examples, occlusion context component 442 can monitor an area physically in front of the occlusion area (eg, with respect to the direction of traffic flow) and determine whether an object is entering the occlusion area. Based on the size of the occlusion region and the subsequent passage of time, the occlusion context component 442 can determine that the occlusion region is unoccupied if the object has not entered the occlusion region. As a non-limiting example, such an occlusion context component 442 may determine the presence of a vehicle approaching an intersection on a two-lane road, and based on the determination that no objects passing behind the occlusion area are detected, 442 can determine that the occlusion area is unoccupied. In some examples, the confidence level associated with determining occupancy can increase over time (eg, by accumulating additional observations). In some examples, for dynamically generated occlusion grids, the lifetime of the occlusion grid can correspond to the amount of time since the occlusion grid was instantiated. In some examples, the confidence level of occupancy of the occlusion grid can be based at least in part on the lifetime of the occlusion grid. In some examples, occlusion context component 442 can receive LIDAR data, image data, RADAR data, etc. to determine the context of the occlusion region.

Velocity component 444 may include functionality that determines the velocity and/or acceleration level of vehicle 402 while traversing the environment. In some examples, speeds and/or acceleration levels include, but are not limited to, low speeds (e.g., "creep mode"), normal speeds (e.g., normal speeds), and high speeds (e.g., "boost mode"). be able to. For example, a low speed level may be selected for a vehicle 402 entering an intersection or area without traversing the entire intersection or area. For example, and as discussed in connection with FIG. 6, before proceeding according to normal control, the vehicle 402 may be controlled to move slowly along the trajectory to "see" more area and size the occlusion area. can be reduced. For such a trajectory, velocity component 444 may select a low velocity. As another example, if the occlusion grid is clear, a normal speed may be selected for vehicle 402 traversing the environment. In some cases, if the size of the occlusion grid is reduced due to obstacles and/or based on limited line of sight in the environment, a high speed is selected to reduce the amount of time vehicle 402 can be exposed to oncoming traffic. can. In some examples, velocity component 444 can operate in conjunction with planning component 424 and/or system controller 426 to control acceleration of vehicle 402.

As can be appreciated, the components discussed herein (e.g., localization component 420, perception component 422, planning component 424, one or more system controllers 426, one or more maps 428, prediction component 430, occlusion monitoring component 432) , occlusion region component 434, ray emission component 436, segmentation component 438, projection component 440, occlusion context component 442, and velocity component 444) are described as separate for illustrative purposes. However, the operations performed by the various components can be performed in combination with or on any other components. As an example, segmentation functionality may be performed by perception component 422 (eg, rather than segmentation component 438) to reduce the amount of data transferred by the system.

In some examples, aspects of some or all of the components discussed herein may include any models, algorithms, and/or machine learning algorithms. For example, in some examples, components within memory 418 (and memory 452, discussed below) can be implemented as a neural network.

As described herein, an exemplary neural network is a biologically inspired algorithm that passes input data through a series of connected layers to produce an output. Each layer of a neural network can also include another neural network, or can include any number of layers (whether convolutional or not). As can be understood in the context of this disclosure, neural networks can utilize machine learning, which can refer to a broad class of algorithms in which outputs are generated based on learned parameters.

Although discussed in the context of neural networks, any type of machine learning can be used consistent with this disclosure. For example, without limitation, machine learning algorithms may include regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), locally estimated scarcity plot smoothing (LOESS)), instance-based algorithms (e.g. ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g. classification and regression trees), (CART), Iterative Bisection Method 3 (ID3), Chi-Square Automatic Interaction Detection (CHAID), Decision Stump, Conditional Decision Tree)), Bayesian algorithms (e.g. Naive Bayes, Gaussian Naive Bayes, Multinomial Naive Bayes, Mean 1 dependent estimator (AODE), Bayesian belief network (BNN), Bayesian network), clustering algorithms (e.g. k-means, k-median, expectation maximization (EM), hierarchical clustering), association rule learning algorithms (e.g. perceptron, backpropagation, Hopfield network, radial basis function network (RBFN)), deep learning algorithms (e.g. deep Boltzmann machine (DBM), deep brief network (DBN), convolutional neural network (CNN), stack auto encoder), dimensionality reduction algorithms (e.g. principal component analysis (PCA), principal component regression (PCR), partial least squares regression (PLSR), Summon mapping, multidimensional scaling (MDS), projective pursuit, linear discriminant analysis (LDA) , mixture discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g. boosting, bootstrap aggregation (bagging), AdaBoost, stack generalization (blending), gradient boost methods such as gradient boosting regression trees (GBRT), random forests), support vector machines (SVM), supervised learning, unsupervised learning, and semi-supervised learning.

Additional examples of architectures include neural networks such as ResNet50, ResNet101, VGG, DenseNet, PointNet, etc.

In at least one example, sensor system 406 includes a LIDAR sensor, a radar sensor, an ultrasound transducer, a sonar sensor, a position sensor (e.g., GPS, compass, etc.), an inertial sensor (e.g., an inertial measurement unit (IMU), an accelerometer, etc.). , magnetometer, gyroscope, etc.), cameras (e.g. RGB, IR, intensity, depth, time of flight, etc.), microphones, wheel encoders, environmental sensors (e.g. temperature sensor, humidity sensor, light sensor, pressure sensor, etc.), etc. can be included. Sensor system 406 may include multiple instances of each of these or other types of sensors. For example, the LIDAR sensors may include individual LIDAR sensors located at the corners, front, back, sides, and/or top of the vehicle 402. As another example, camera sensors may include multiple cameras located at various locations on the exterior and/or interior of vehicle 402. Sensor system 406 can provide input to vehicle computing device 404 . Additionally or alternatively, sensor system 406 transmits sensor data over one or more networks 446 to one or more computing devices at a particular frequency in substantially real time after a predetermined period of time has elapsed. can. In some examples, sensor system 406 can correspond to sensor system 104 of FIG.

Vehicle 402 may also include one or more emitters 408 for emitting light and/or sound, as described above. This example emitter 408 includes internal audio and visual emitters for communicating with passengers of vehicle 402. By way of example and not limitation, internal emitters include speakers, lights, signs, display screens, touch screens, tactile emitters (e.g., vibration and/or force feedback), mechanical actuators (e.g., seatbelt tensioners, seat positioners, headrest positioners). etc.). Emitter 408 in this example also includes an external emitter. By way of example and not limitation, this exemplary external emitter may include lights for signaling direction of travel or other indicators of vehicle operation (e.g., indicator lights, signs, light arrays, etc.), and acoustic beam steering technology. one or more audio emitters (eg, speakers, speaker arrays, horns, etc.) for audibly communicating with one or more pedestrians or other nearby vehicles;

Vehicle 402 may also include one or more communication connections 410 that enable communication between vehicle 402 and one or more other local or remote computing devices. For example, communication connection 410 can facilitate communication with other local computing devices on vehicle 402 and/or drive module 414. Communication connections 410 may also enable the vehicle to communicate with other nearby computing devices (eg, other nearby vehicles, traffic lights, etc.). Communications connection 410 also allows vehicle 402 to communicate with remotely operated computing devices or other remote services.

Communication connections 410 may include physical and/or logical interfaces for connecting vehicle computing device 404 to another computing device or to a network, such as network 446. For example, the communication connection 410 may support Wi-Fi-based communication such as via frequencies defined by the IEEE 802.11 standard, short-range radio frequencies such as Bluetooth, cellular communication (e.g., 2G, 3G, 4G, 4G LTE, 5G) or any suitable wired or wireless communication protocol that allows each computing device to interface with other computing devices.

In at least one example, vehicle 402 can include one or more drive modules 414. In some examples, vehicle 402 may have a single drive module 414. In at least one example, when vehicle 402 has multiple drive modules 414, individual drive modules 414 can be located at opposite ends of vehicle 402 (eg, front and rear, etc.). In at least one example, drive module 414 can include one or more sensor systems to detect conditions around drive module 414 and/or vehicle 402. By way of example and not limitation, the sensor system may include one or more wheel encoders (e.g., rotary encoders) for sensing the rotation of wheels of the drive module, inertial sensors (e.g., inertial encoders) for measuring orientation and acceleration of the drive module. measurement units, accelerometers, gyroscopes, magnetometers, etc.), cameras or other image sensors, ultrasonic sensors for acoustically detecting objects around the drive module, LIDAR sensors, RADAR sensors, etc. can. Some sensors, such as wheel encoders, may be specific to drive module 414. In some cases, a sensor system on drive module 414 can overlap or supplement a corresponding system on vehicle 402 (eg, sensor system 406).

The drive module 414 includes a high voltage battery, a motor that propels the vehicle, an inverter that converts direct current from the battery to alternating current for use in other vehicle systems, a steering system that includes a steering motor, and a steering rack (which can be electric). , brake systems including hydraulic or electric actuators, suspension systems including hydraulic and/or pneumatic components, stability control systems for brake force distribution to reduce loss of traction and maintain control, HVAC systems, lighting (e.g. lighting such as head/tail lights that illuminate the environment), and one or more other systems (e.g. cooling systems, safety systems, onboard charging systems, DC/DC converters, high voltage junctions, high voltage cables, charging systems, charging ports) and other electrical components). Additionally, drive module 414 can include a drive module controller that can receive and pre-process data from sensor systems and control operation of various vehicle systems. In some examples, the drive module controller may include one or more processors and memory communicatively coupled to the one or more processors. Memory can store one or more modules to perform various functions of drive module 414. Additionally, drive modules 414 also include one or more communication connections that enable the respective drive module to communicate with one or more other local or remote computing devices.

In at least one example, direct connections 412 can provide a physical interface for coupling one or more drive modules 414 to the body of vehicle 402. For example, direct connections 412 can enable the transfer of energy, fluids, air, data, etc. between drive module 414 and a vehicle. In some examples, direct connection 412 can further removably secure drive module 414 to the body of vehicle 402.

In at least one example, a localization component 420, a perception component 422, a planning component 424, one or more system controllers 426, one or more maps 428, a prediction component 430, an occlusion monitoring component 432, an occlusion region component 434, and a beam emitting component. Component 436, segmentation component 438, projection component 440, occlusion context component 442, and velocity component 444 can process sensor data as described above and send their respective outputs to one or more networks 446 to can be sent to computing device 448; In at least one example, a localization component 420, a perception component 422, a planning component 424, one or more system controllers 426, one or more maps 428, a prediction component 430, an occlusion monitoring component 432, an occlusion region component 434, and a beam emitting component. Component 436, segmentation component 438, projection component 440, occlusion context component 442, and velocity component 444 perform one or more computing operations on their respective outputs in near real time and at a particular frequency after a predetermined period of time. It can be sent to device 448.

In some examples, vehicle 402 can transmit sensor data to one or more computing devices 448 via network 446. In some examples, vehicle 402 can transmit raw sensor data to computing device 448. In other examples, vehicle 402 can transmit processed sensor data and/or representations of the sensor data to computing device 448. In some examples, vehicle 402 may transmit sensor data to computing device 448 at a particular frequency in substantially real time after a predetermined period of time. In some cases, vehicle 402 can send sensor data (raw or processed) to computing device 448 as one or more log files.

Computing device 448 may include a processor 450 and memory 452 that stores a map component 454.

In some examples, map component 454 can include functionality that associates an occlusion grid with a map location. In some examples, map component 454 can identify intersections where vehicles approaching the intersection do not have the right-of-way, and can generate an occlusion grid for the locations, for example. In some examples, map component 454 can identify areas of topographic occlusion caused by, for example, hill crests that may reduce visibility of vehicle 402. In some examples, as discussed in connection with FIG. 5, for example, the size of the occlusion grid can be based at least in part on area characteristics, such as intersection distance, speed limit, speed limit safety factor, and the like.

Processor 416 of vehicle 402 and processor 450 of computing device 448 may be any suitable processor capable of executing instructions to process data and perform operations as described herein. By way of example and not limitation, processors 416 and 450 may include one or more central processing units (CPUs), graphics processing units (GPUs), or other electronic data processors capable of processing electronic data and storing electronic data in registers or memory. any other device or part of a device that converts to In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices also work with processors so long as they are configured to implement encoded instructions. It can be considered as

Memories 418 and 452 are examples of non-transitory computer readable media. Memories 418 and 452 can store an operating system and one or more software applications, instructions, programs, and/or data to implement the functionality resulting from the methods and various systems described herein. In various implementations, the memory may be any suitable memory technology, such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile/flash type memory, or any other type of memory capable of storing information. It can be implemented using The architectures, systems, and individual elements described herein may include many other logical, programmatic, and physical components, including those illustrated in the accompanying drawings. Just an example relevant to the discussion.

Although FIG. 4 is depicted as a distributed system, in alternative examples, components of vehicle 402 may be associated with computing device 448 and/or components of computing device 448 may be associated with vehicle 402. This should be kept in mind. That is, vehicle 402 can perform one or more of the functions associated with computing device 448, and vice versa. Additionally, aspects of occlusion monitoring component 432 and/or prediction component 430 can be performed on any of the devices discussed herein.

2, 3, and 5-11 illustrate example processes according to embodiments of the present disclosure. These processes are illustrated as logical flow graphs, each operation representing a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the context of software, operations refer to computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are listed is not intended to be construed as limiting, and any number of the listed operations may be combined in any order and/or in parallel to implement the process. .

FIG. 5 is an illustrated flow diagram of an example process 500 for generating an occlusion grid based on region characteristics, according to an embodiment of the present disclosure. For example, some or all of process 500 can be performed by one or more components in FIG. 4, as described herein.

At operation 502, the process can include evaluating region characteristics. In some examples, area characteristics may include, but are not limited to, intersection (or area) distances (eg, width of intersections intersected), speed limits, vehicle speeds, safety factors, and the like. Example 504 shows area characteristics of intersection 110. For example, intersection 110 is associated with an intersection distance 506, which represents the distance between a stop line 508 of intersection 110 and a location associated with an exit of intersection 110. In some examples, vehicle 102 may not remain stationary within intersection 110 while attempting to cross the intersection, and therefore vehicle 102 may be controlled to travel to check line 510. In some examples, check line 510 may represent the furthest from intersection 110 that vehicle 102 can travel without impeding traffic flow. That is, once the check line 510 is crossed, the vehicle 102 may block oncoming traffic. If the vehicle 102 moves to the check line 510, the intersection distance 506 may be updated to reflect the reduced distance of the intersection 110 that the vehicle 102 crosses. In some examples, stop line 508 and/or check line 510 can be stored as map data or dynamically generated based on sensor data captured within the environment.

Intersection 110 may also be associated with a speed limit 512. As can be appreciated, a safety factor may be associated with the speed limit 512 that effectively increases the speed that a vehicle crossing the intersection 110 is expected to reach. For example, at an intersection where the speed limit 512 is 25 miles per hour, the safety factor may be an additional 15 miles per hour. That is, at least some vehicles crossing the intersection 110 are estimated to be speeding. This assumption is made in consideration of such a safety factor.

Example 504 also shows vehicle 102 associated with average speed, "creep mode", and "boost mode". As can be appreciated, the average speed may be the expected average speed of the vehicle 102 traversing the intersection distance 506. Additionally, vehicle 102 may be associated with various acceleration levels for traversing at least a portion of intersection 110 slowly or quickly.

At operation 514, the process can include generating an occlusion grid based at least in part on the region characteristics. In example 504, an occlusion grid is generated and associated with intersection 110. In some examples, occlusion grid 516 can be associated with an occlusion grid length 518 (or, more generally, a range (eg, length, width, height) of occlusion grid 516). By way of example and not limitation, occlusion grid length 518 is such that if occlusion grid 516 is clear when vehicle 102 begins its trajectory, it will cross intersection 110 associated with intersection distance 506 at the average speed of vehicle 102. The vehicle 102 can be sized so that it cannot intersect with a vehicle traveling at the speed limit 512 (and safety factor). That is, the length 518 of the occlusion grid is such that the vehicle 102 can safely cross the intersection 110 if the occlusion grid 516 is clear. As can be appreciated, generating the occlusion grid 516 based at least in part on area characteristics provides a safer outcome without allocating additional resources to the operation than is required to ensure safe travel. Improve.

In some examples, the length of the visibility area associated with an intersection may be limited or limited by the topography of the intersection. For example, the road in front of the intersection may include hairpin turns or other obstacles that reduce the size of the area that can be sensed by vehicle 102. In some examples, geographic and/or topographical information about the intersection (e.g., maximum visibility distance) is used to size the occlusion grid 516 so that the vehicle 102 has a clear area beyond what is physically possible. You can avoid situations where you would expect Accordingly, the speed and/or acceleration of the vehicle can be set to ensure that the vehicle can cross such intersections safely. Similarly, as discussed herein, a threshold associated with the size of a non-occlusion region or a threshold associated with undetermined occupancy can be adjusted based on the size of the sensed region.

FIG. 6 is an illustrated flow diagram of an example process 600 for evaluating an occlusion grid at an intersection and controlling a vehicle to traverse the intersection, according to an embodiment of the present disclosure. For example, some or all of process 600 can be performed by one or more components in FIG. 4, as described herein.

At operation 602, the process may include evaluating an occlusion grid at the intersection. In example 604, vehicle 102 is shown at stop line 606 at intersection 110. Based at least in part on map data associated with intersection 110, vehicle 102 may load occlusion grid 608 into memory. Additionally, vehicle 102 can determine stop lines 606 based on map data or generate stop lines 606 based on identifying stop lines 606 in the environment. Further, the check line 610 can be accessed from memory or determined based on identifying a shoulder or protected area of the intersection 110.

In some examples, the position of vehicle 102 with respect to one or more obstacles 612 (here, a line of parked cars) and building 632 causes vehicle 102 to associate non-occlusion region 614 with a portion of occlusion grid 608 occlusion region 616 can be associated with a portion of occlusion grid 608. As shown, the individual occlusion fields of the occlusion grid that correspond to occlusion areas 616 are shaded in gray, while non-occlusion areas 614 are not shaded. As can be appreciated, object 618 may occupy occlusion region 616, but because vehicle 102 cannot receive sensor data associated with occlusion region 616, vehicle 102 may occupy object 618 (e.g., without a corresponding occupancy state). (which could be a decision) does not exist.

In some examples, operation 602 determines that the occlusion field of non-occlusion region 614 is visible and unoccupied, while the occlusion field of occlusion region 616 is invisible and the occupancy state is unknown (e.g., undetermined). It may include determining that. In some examples, the default state of the occlusion field within occlusion region 616 may be an occupied state (or an undetermined state to be treated as an occupied state).

In operation 620, the process controls and causes the vehicle to change position based at least in part on the first evaluation of the occlusion grid (e.g., a determination that there is insufficient information to safely cross the intersection). can be included. In example 622, vehicle 102 may be controlled to move from stop line 606 to check line 610 to increase the size of non-occlusion area 614 of occlusion grid 608 (e.g., increase the size of available data from sensors on vehicle 102). by increasing the amount). Accordingly, the size of occlusion area 616 of occlusion grid 608 is reduced. As shown in example 622, a majority of occlusion grid 608 is non-occluding to vehicle 102, but vehicle 102 is now able to identify object 618 within occlusion grid 608. Accordingly, the occlusion field of occlusion grid 608 corresponding to object 618 is identified as occupied region 624 (and is shaded black for illustrative purposes). Similarly, object 618 can create "shading" with respect to the sensing area of vehicle 102, as indicated by occlusion area 634 (and as a shaded gray in example 622). Therefore, the vehicle 102 can be controlled and made to wait at the check line 610 without further entering the intersection 110. In some examples, the vehicle 102 can be controlled to slowly (e.g., below a speed threshold) advance toward the check line 610 (e.g., in "creep mode"), although any acceleration and/or speed can be used. .

At operation 626, the process can include controlling the vehicle to traverse the intersection based at least in part on the second evaluation of the occlusion grid. In example 628, object 618 has exited occlusion grid 608 and is now visible and unoccupied. Accordingly, vehicle 102 is shown traversing intersection 110 based at least in part on trajectory 630. In some examples, vehicle 102 may be controlled to accelerate through intersection 110 using normal or high acceleration (eg, "boost mode"), although any acceleration can be used.

In some examples, operations 602, 620, and/or 626 may be performed continuously by vehicle 102 to monitor environmental conditions over time. In some examples, process 600 may include evaluating the occlusion grid at a frequency of 10 Hertz (Hz) (or any frequency) to continuously evaluate the environment.

FIG. 7 is an illustrated flow diagram of another example process 700 for evaluating an occlusion grid at an intersection and controlling a vehicle to traverse the intersection, according to an embodiment of the present disclosure. For example, some or all of process 700 can be performed by one or more components in FIG. 4, as described herein.

At operation 702, the process may include evaluating an occlusion grid at the intersection. In some examples, operation 702 can include generating an occlusion grid in response to identifying obstacles in the environment. In example 704, vehicle 102 attempts to follow a trajectory 706 at intersection 110 that would result in an unprotected left turn (e.g., vehicle 102 does not have a green arrow and must yield to oncoming traffic). There is. In some examples, vehicle 102 can identify an object 708 (eg, a vehicle) in the environment, and occlusion grid 710 can be associated with object 708. As discussed herein, a vehicle can receive LIDAR data and/or image data to assess occlusion conditions and/or occupancy conditions of occlusion fields within occlusion grid 710.

In operation 712, the process may include controlling vehicle 102 to wait at the intersection. As can be appreciated, the intersection 110 may be modified based at least in part on a portion of the occlusion grid 710 that is an occlusion (an occlusion grid 710 that does not provide a sufficient amount of information to safely traverse the intersection 110). Vehicle 102 may be controlled and parked at intersection 110 based at least in part on crossing object 708 .

In operation 714, the process may include evaluating the occlusion grid as clear (or otherwise providing sufficient information to allow safe crossing of intersection 110 according to trajectory 706). can. In example 716, object 708 is shown crossing intersection 110 and proceeding outside of occlusion grid 710. In some examples, operation 714 determines that the occlusion field is visible and unoccupied (or otherwise undetermined) so that occlusion grid 710 is clear (or provides sufficient information). The determination may include capturing LIDAR data and/or image data.

At operation 718, the process may include controlling the vehicle 102 to cross the intersection based at least in part on the occlusion grid being clear. In some examples, operation 718 determines that occlusion grid 710 is clear (or that there is sufficient information to determine safe traversal of trajectory 706) before instructing vehicle 102 to traverse the environment. may include waiting a predetermined period of time (eg, 1 second, 2 seconds, etc.) for the As shown in example 716, vehicle 102 is crossing intersection 110 via track 706. Additionally, occlusion grid 710 is shown to be visible and clear from obstacles.

FIG. 8 is an illustrated flow diagram of an example process 800 for determining an occlusion region associated with an object and generating a predicted trajectory of a dynamic object within the occlusion region, according to an embodiment of the present disclosure. For example, some or all of process 800 can be performed by one or more components in FIG. 4, as described herein. For example, some or all of process 800 may be performed by vehicle computing device 404.

At operation 802, the process can include determining an occlusion region associated with the object. In example 804, vehicle 102 at intersection 110 captures LIDAR and/or image data of the environment. In some examples, operation 802 can include identifying an object 806 in the environment and an occlusion region 808 associated with object 806. In example 804, object 806 may represent a car parked near intersection 110. In some examples, operation 802 can include generating an occlusion grid (not shown) associated with object 806. In some examples, occlusion region 808 represents an occluded portion of an occlusion grid. In some examples, operation 802 can include accessing map data that includes an occlusion grid, as discussed herein.

At operation 810, the process may include identifying a moving object (eg, a pedestrian) proximate to the occlusion region 808. In some examples, operation 810 may include determining that moving object 812 (eg, a pedestrian) is within a distance threshold of occlusion region 808. In some examples, operation 810 receives LIDAR data, image data, and/or other sensor data to determine the trajectory (e.g., position, orientation, etc.) of dynamic object 812 in the environment as measured signature 814. , and velocity).

At operation 816, the process may include generating one or more predicted trajectories of the moving object. In example 804, one or more predicted trajectories 818 are shown as possible paths that pedestrian 812 may follow while in occlusion region 808 (although the predicted trajectories are not limited to occlusion region 808). In some examples, predicted trajectory 818 can be based at least in part on measured footprints 814 and/or symbols or regions in the environment. For example, the predicted trajectory may include an extension of the speed and direction of the pedestrian 812 when the pedestrian enters the occlusion area 808. In another example, the predicted trajectory may be based on an environmental destination, such as a sidewalk on the other side of an intersection. A variety of methods can be used to generate predicted trajectories. Details of generating predicted trajectories are discussed in US patent application Ser. No. 15/833,715, which is incorporated herein by reference in its entirety.

In some examples, operation 816 may include controlling vehicle 102 based at least in part on one or more predicted trajectories 818. In some examples, if the trajectory of the vehicle 102 is within a distance threshold of at least one predicted trajectory, the vehicle 102 may be controlled to wait until additional information about the moving object can be determined. In some examples, over a period of time, vehicle 102 may increase or decrease the probability associated with each predicted trajectory indicating the likelihood that pedestrian 812's path will follow one or more predicted trajectories 818. For example, as time passes and observations associated with dynamic object 812 are made or not made, a trajectory can be rejected (eg, removed) from the candidate list of predicted trajectories. In some examples, when a dynamic object 812 is observed in the environment, one or more predicted trajectories may be rejected if they do not describe or are associated with the movement of the dynamic object 812 within the occlusion region 808. .

FIG. 9 is an illustrated flow diagram of an example process 900 for evaluating a terrain occlusion grid and controlling a vehicle to traverse a terrain obstacle, according to an embodiment of the present disclosure. For example, some or all of process 900 can be performed by one or more components in FIG. 4, as described herein. For example, some or all of process 900 can be performed by vehicle computing device 404.

At operation 902, the process can include evaluating an occlusion grid. In some examples, the occlusion grid can be a topographic occlusion grid generated in response to topographic (eg, geographic) features within the environment. In example 904, occlusion grid 906 is shown in conjunction with a hilltop. As can be appreciated, as the vehicle 102 approaches a hill, there will naturally be an area at the top of the hill that is not visible to the vehicle 102 due to the height difference and angle of the ground. In one example, occlusion grid 906 is associated with non-occlusion areas 908 and occlusion areas 910. As shown, vehicle 102 emits a LIDAR beam 912 that traverses occlusion grid 906 to identify non-occlusion areas 908 and occlusion areas 910.

As can be appreciated, as the vehicle 102 approaches the top of a hill, the vehicle 102 may be unable to capture sensor data of the ground at the top of the hill. In some embodiments, this reduces the visibility of the vehicle 102 and the vehicle 102 can correspondingly reduce the speed of the vehicle 102 traveling through the environment.

However, in operation 914, the process may include determining occupancy of at least one occlusion field. For example, occlusion grid 906 includes at least one occlusion field 916. As shown in example 904, occlusion field 916 is located at the top of a hill such that LIDAR beam 912 is unable to capture ground sensor data associated with occlusion field 916. In some examples, occlusion grid 906 can be accessed via map data that includes topographical information (eg, height information) of the area indicating particular areas of occlusion or non-occlusion. Further, such map information can indicate, for example, how many data points (eg, LIDAR returns) can be expected on the surface based on the location of the vehicle 102. However, because occlusion field 916 includes or is associated with a three-dimensional spatial volume, LIDAR ray 912 may traverse a portion of occlusion field 916. In some examples, operation 914 can include determining the occupancy of occlusion field 916 based at least in part on the ray emission of LIDAR ray 912, as discussed herein. In this example, occlusion field 916 is visible from vehicle 102 (for purposes of determining occlusion conditions) and is not occupied by obstacles.

At operation 918, the process can include controlling the vehicle (eg, vehicle 102) based at least in part on the occupancy. For example, by evaluating an occlusion grid 906 that represents hills in the environment, the vehicle 102 can "see over" the top of the hill even though the sensors cannot directly capture ground data beyond the top of the hill. Accordingly, the vehicle 102 can plan its trajectory and/or determine its speed based on the occupancy of the occlusion grid 906 over the crest of the hill. As a non-limiting example, vehicle 102 may adjust one or more of speed, orientation, or position such that the information provided by occlusion grid 906 is sufficient to safely traverse a hill. .

FIG. 10 illustrates determining occlusion field occupancy and controlling a vehicle based on beaming LIDAR data and/or based on projection of the occlusion field onto image data, according to embodiments of the present disclosure. 10 illustrates an example process 1000 for traversing an environment. For example, some or all of process 1000 can be performed by one or more components in FIG. 4, as described herein. For example, some or all of process 1000 may be performed by vehicle computing device 404.

In operation 1002, the process can include accessing map data representing an environment proximate to the autonomous vehicle, where the map data includes an occlusion grid that includes an occlusion field. In some examples, operation 1002 can include determining the location of the vehicle within the environment and accessing map data that is within a distance threshold of the vehicle (eg, receding horizon). As discussed herein, in some examples, the occlusion grid may include multiple occlusion fields, each occlusion field representing a separate portion of the environment (eg, a portion of the drivable road surface). In some examples, the occlusion field is determined by an occlusion state (e.g., whether the occlusion field is visible from the vehicle) and an occlusion state (e.g., whether the occlusion field is occupied by an object in the environment or otherwise). undetermined or not).

At operation 1004, the process may include capturing sensor data using sensors on the autonomous vehicle. In some examples, operation 1004 can include capturing LIDAR data and/or image data of the environment.

In operation 1006, the process may include determining occlusion field occupancy. In some examples, operation 1006 can also include determining the visibility of the occlusion field. In some examples, operation 1006 can include determining occlusion field occupancy based at least in part on LIDAR data (e.g., operation 1008) and/or image data (e.g., operation 1010). .

At operation 1008, the process may include beaming the LIDAR data and determining occlusion field occupancy. As discussed in connection with FIG. 2 and throughout this disclosure, the LIDAR data ray emission may pass through the expected number of LIDAR returns that pass through the occlusion field (e.g., voxels in a column of voxel space above the occlusion field). and determining the actual number of LIDAR returns passing through the occlusion field relative to the total number of voxels passing over such occlusion field discussed in detail above. Of course, operation 1008 can include determining that the LIDAR data represents an object within an occlusion field, in which case the occlusion field will be occupied. The occupancy state may be unoccupied if the threshold number (or percentage) of returns indicates no returns. Otherwise, the occupancy state may be undetermined.

In operation 1010, the process may include projecting an occlusion field onto the segmented image data and determining occlusion field occupancy. As discussed in connection with FIG. 3 and throughout this disclosure, image data captured by an image sensor can be segmented to determine the location of drivable areas, vehicles, pedestrians, etc. The position of the occlusion field can be projected onto the segmented image data to determine whether the position of the occlusion field corresponds to a drivable road surface. In the corresponding case, the occlusion field is unoccupied, but if, for example, the occlusion field is projected onto a vehicle, it can be determined that the occlusion field is occupied or otherwise undetermined. A determination whether the occlusion field is occupied may be made at operation 1012.

In operation 1012, if the threshold number of occlusion fields is occupied (“yes”) or undetermined, the process continues to operation 1014. In operation 1014, the process may include determining that the occlusion grid is not clear based at least in part on the occupancy. For example, the process may return if there are not a sufficient number of unoccupied occlusion fields to determine that the vehicle can safely traverse the environment. In some examples, operation 1014 may include determining that a threshold number of occlusion fields are occupied or undetermined. In some examples, operation 1014 can include controlling a vehicle to drive into a position to view a large portion of the occlusion grid. The process may return to operation 1004 to capture additional data for determining the state of the occlusion field and thus the occlusion grid.

If the occlusion field is not occupied (“no” in operation 1012), the process continues to operation 1016. In operation 1016, the process may include determining, based at least in part on the occupancy, that the occlusion grid is clear or otherwise provides sufficient information to safely traverse the environment. can.

At operation 1018, the process can include controlling the autonomous vehicle to traverse the environment based at least in part on the occlusion grid being clear.

FIG. 11 is an illustration for determining an occlusion grid and controlling a vehicle to traverse an environment based on map data indicative of the occlusion grid or based on identification of obstacles in the environment, according to embodiments of the present disclosure. 1100 is shown. For example, some or all of process 1100 can be performed by one or more components in FIG. 4, as described herein. For example, some or all of process 1100 can be performed by vehicle computing device 404.

In operation 1102, the process can include controlling the autonomous vehicle to traverse an environment to a first location. In some cases, the first location can represent a stop line of an intersection in the environment.

At operation 1104, the process may include capturing sensor data of the environment using sensors on the autonomous vehicle. In some examples, operation 1104 can include capturing LIDAR data, image data, RADAR data, etc. in the environment.

At operation 1106, the process may include determining 1) whether the map data indicates an occlusion grid, or 2) whether an obstacle has been identified in the environment. If “no” (e.g., the map data does not indicate that there is an occlusion grid associated with the location and that there are no objects identified in the environment), the process can return to operation 1102 and may move the autonomous vehicle to a new location (operation 1102) and/or enable the autonomous vehicle to capture sensor data in the environment according to a baseline driving behavior (e.g., trajectory generation based on sensor data) (operation 1104). ). If yes in operation 1106 (eg, the map data indicates that there is an occlusion grid associated with the location or that there is an identified object in the environment), the process continues to operation 1108.

At operation 1108, the process may include determining an occlusion grid that includes a plurality of occlusion fields, where the occlusion fields indicate visibility and occupancy of corresponding areas within the environment. In some examples, the occlusion grid can be accessed from the map data, and in some examples, the occlusion grid can be dynamically generated based at least in part on the identification of the object.

In operation 1110, the process can include determining that at least a portion of the occlusion grid is an occlusion at a first time as an occlusion area and based at least in part on the sensor data. As discussed herein, the occlusion region may not be visible to one or more sensors of the autonomous vehicle at the first time.

In operation 1112, the process can include controlling the autonomous vehicle to remain at the first location or traverse the second location based at least in part on the occlusion area. An example of remaining in a first position is discussed in connection with FIG. 7, and an example of controlling the vehicle to traverse a second position is discussed in connection with FIG. For example, the second position can be a check line, allowing the autonomous vehicle to drive to the second position and see more of the occlusion grid.

In operation 1114, the process can include determining that the occlusion region (or a portion of the environment corresponding to the occlusion region) is visible at the second time based at least in part on the sensor data. . With respect to FIG. 6, the autonomous vehicle may travel to the second position so that obstacles in the environment do not prevent the sensor from capturing data for the entire occlusion grid. With respect to FIG. 7, the autonomous vehicle may wait for a moving obstacle, thereby allowing sensors to capture data representative of the occlusion grid.

In operation 1116, the process can include controlling the autonomous vehicle based at least in part on the visible occlusion region at the second time. In some examples, operation 1116 may also include determining that the occlusion grid is not occupied, indicating that the autonomous vehicle can safely traverse the environment.

[Example invention content]
A: A system comprising one or more processors and one or more computer-readable media storing instructions executable by the one or more processors, the instructions, when executed, capturing LIDAR data into a system using a LIDAR sensor on an autonomous vehicle; and accessing map data representing an environment proximate the autonomous vehicle, the map data including an occlusion grid including a plurality of occlusion fields. an occlusion field of the plurality of occlusion fields is associated with an occlusion state and an occupancy state of a portion of the environment; and emitting a portion of the LIDAR data; and determining the occlusion state, based at least in part on the occlusion field, determining sufficiency of information associated with the occlusion grid to provide a safe trajectory for the autonomous vehicle to traverse the environment. and controlling the autonomous vehicle to traverse the environment according to the safe trajectory.

B: The operations include capturing image data using an image sensor on the autonomous vehicle and performing semantic segmentation on at least a portion of the image data to identify at least a drivable road surface in the environment. generating segmented image data that identifies; projecting the occlusion field onto the segmented image data; determining that the occlusion field is projected onto the drivable road surface; The system of paragraph A, further comprising controlling the autonomous vehicle to traverse the environment based at least in part on the occlusion field projected onto a drivable surface.

C: as described in paragraph A or B, wherein the plurality of occlusion fields individualize portions of a drivable road surface in the environment, and the occlusion state of the occlusion fields indicates visibility of the occlusion field by the LIDAR sensor. system.

D: the operation is based at least in part on a first number of LIDAR returns associated with the occlusion field and a second number of individualized regions associated with the occlusion field; The system of any of paragraphs A through C, further comprising determining a confidence value associated with the occupancy state of an occlusion field.

E: a first ratio of the first number to the second number is less than a threshold indicating a high reliability of the unoccupied state of the occlusion field; The system of paragraph D, wherein the second ratio of equal to or greater than the threshold indicates high confidence in the occupied state of the occlusion field.

F: capturing sensor data using a sensor on a robot platform; determining a position of the robot platform based at least in part on the sensor data; and an environment proximate the robot platform at the position. accessing map data representing an occlusion grid, the step of determining an occlusion state of an occlusion field of the occlusion grid; and determining the sufficiency of information based at least in part on the occlusion field. and controlling the robot platform based at least in part on the occupancy of the occlusion grid. Method.

G: capturing LIDAR data using a LIDAR sensor on the robot platform as the sensor data; and emitting a portion of the LIDAR data to determine the occupancy state of the occlusion field. , the occupancy state includes one of an occupied state, an unoccupied state, or an undetermined state; and emitting a portion of the LIDAR data to determine an occlusion state of the occlusion field. the occlusion state includes one of being visible or occluding; and controlling the robotic platform to traverse the environment based at least in part on the occlusion grid being clear. and, the sufficiency of the information is related to at least one of the size of the occlusion grid or the speed limit associated with the occlusion grid, the occlusion field of the occlusion grid being visible to the LIDAR sensor. The method of paragraph F, wherein the method is determined based at least in part on the number of .

H: determining a first number of expected LIDAR returns associated with the area above the occlusion field and a first number of actual LIDAR returns associated with the area above the occlusion field; determining a second number associated with the occupancy of the occlusion field based at least in part on the first number of expected LIDAR returns and the second number of actual LIDAR returns; The method of paragraph G, comprising: determining a confidence value; and determining that the confidence value is greater than or equal to a confidence threshold.

I: using an image sensor on the robot platform to capture image data as the sensor data; and performing segmentation on at least a portion of the image data to identify drivable road surfaces in the environment. projecting the occlusion field onto the segmented image data; determining to project the occlusion field onto the drivable road surface; and projecting the occlusion field onto the drivable road surface. 3. The method of any of paragraphs F through H, further comprising controlling the robot platform to traverse the environment based at least in part on the occlusion field.

J: the robotic platform includes an autonomous vehicle, the environment proximate the autonomous vehicle includes an intersection, and the method is associated with the intersection, including determining a distance of the intersection traversed by the autonomous vehicle. Paragraphs F through F further comprising: determining a speed limit; and determining a range of the occlusion grid based at least in part on the distance of the intersection, the speed limit, and a safety factor. The method according to any one of I.

K: The method of paragraph J, further comprising determining the range of the occlusion grid based at least in part on an expected time for the autonomous vehicle to traverse the distance of the intersection.

L: the occlusion field includes a temporal logic symbol, the method includes verifying a trajectory based at least in part on evaluation of a temporal logic expression including the temporal logic symbol; 3. The method of any of paragraphs F through K, further comprising controlling the robotic platform to traverse the environment based on the method.

M: the occlusion grid includes a plurality of occlusion fields, and the method determines whether a threshold number of the plurality of occlusion fields are visible to the sensor and unoccupied before controlling the robot platform to traverse the environment. The method according to any of paragraphs F to L, further comprising the step of determining that.

N: determining that a portion of the occlusion grid is not visible from the sensor; and controlling the robot platform to a position that increases the portion of the occlusion grid that is visible from the sensor. The method according to any one of paragraphs F to M, further comprising:

O: when executed, causing one or more processors to receive sensor data captured using sensors on a robot platform; and determining a position of the robot platform based at least in part on the sensor data. accessing map data representing an environment proximate to the robot platform at the location, the map data including an occlusion grid; and determining the occupancy of an occlusion field of the occlusion grid. , determining sufficiency of information based at least in part on the occlusion field to determine a trajectory that traverses a region of the environment proximate to the occlusion grid; controlling the robot platform based on the robot platform.

P: the sensor data is LIDAR data, the sensor is a LIDAR sensor, and the operation includes beam emitting a portion of the LIDAR data to determine the occupancy state and occlusion state of the occlusion field; determining sufficiency of the information associated with the occlusion grid based at least in part on an occupancy state and the occlusion state; and controlling the robot platform based at least in part on the sufficiency of the information. and traversing the environment.

Q: The beam emission includes determining a first number of expected LIDAR returns associated with the area on the occlusion field and a second number of actual LIDAR returns associated with the area on the occlusion field. and a confidence value associated with the occupancy state of the occlusion field based at least in part on the first number of expected LIDAR returns and the second number of actual LIDAR returns. and determining that the confidence value is greater than or equal to a confidence threshold, wherein the confidence level being greater than or equal to the confidence threshold indicates that the occlusion field is unoccupied. A non-transitory computer-readable medium as described in paragraph P.

R: using an image sensor on the robot platform to capture image data as the sensor data and performing segmentation on at least a portion of the image data to identify driveable road surfaces in the environment; the non-transitory computer of any of paragraphs O through Q, the non-transitory computer of any of paragraphs O through Q determining to generate image data that has been segmented, project the occlusion field onto the segmented image data, and project the occlusion field onto the drivable road surface; readable medium.

S: The operation includes determining that a portion of the occlusion grid is not visible from the sensor, and controlling the robot platform to move the entire occlusion grid to a position where it is visible from the sensor. The non-transitory computer-readable medium of any of paragraphs O through R, further comprising: a speed of the robot platform below a speed threshold.

T: the sensor data is first sensor data, the operation includes receiving image data captured by an image sensor on the robot platform as the first sensor data; receiving LIDAR data captured by a LIDAR sensor as second sensor data; and determining the occupancy state of the occlusion field based at least in part on the first sensor data and the second sensor data. The non-transitory computer-readable medium of any of paragraphs O through S, further comprising the step of determining an associated confidence value.

U: A system comprising one or more processors and one or more computer-readable media storing instructions executable by the one or more processors, the instructions, when executed, controlling an autonomous vehicle to traverse a first location in an environment; and capturing at least LIDAR data or image data of the environment as sensor data using sensors of the autonomous vehicle; determining an occlusion grid including a plurality of occlusion fields based at least in part on accessing data or identifying obstacles in the environment, wherein the occlusion field of the occlusion grid is in the environment; indicating an occlusion state and an occupancy state of a corresponding region; as an occlusion region and based at least in part on the sensor data, at least a first portion of the occlusion grid is in occlusion at a first time; determining that at least a second portion of the occlusion grid is non-occluding at the first time as a non-occlusion area and based at least in part on the sensor data; determining a confidence level associated with safely traversing a trajectory through the environment based at least in part on the extent and occupancy of the portions of 2; controlling the autonomous vehicle to remain at the first location or traverse to a second location based on the occlusion grid at a second time based at least in part on the sensor data; determining that a third portion is visible and unoccupied; and at least partially based on the third portion of the occlusion grid being visible and unoccupied at the second time. A system that causes operations to be performed, including steps to control.

V: the operation determines that the autonomous vehicle is approaching an intersection at which the autonomous vehicle is required to yield to other traffic; the obstacle is associated with a dynamic occlusion grid; The system of paragraph U further comprising one or more of the following: determining a predicted trajectory of a dynamic obstacle traversed within the occlusion grid; or determining the occlusion grid based on a topographical occlusion area. .

W: the sensor data includes the LIDAR data, the sensor is a LIDAR sensor, and the operation beams a portion of the LIDAR data to determine the occlusion state and the occupancy state of the occlusion field. The system according to paragraph U or paragraph V, further comprising .

X: the sensor data is image data, the sensor is an image sensor, and the operation performs a classification on at least a portion of the image data to identify a drivable road surface in the environment; projecting the occlusion field onto the segmented image data; determining that the occlusion field is projected onto the drivable road surface; The system of any of paragraphs U through W, further comprising controlling the autonomous vehicle to traverse the environment based at least in part on the projected occlusion field.

Y: said operation determines a first number of non-occlusion and non-occupied occlusion fields of said plurality of occlusion fields, and said first number of occlusion fields associated with safe traversal of said environment. determining that the first number is less than a threshold number; and controlling the autonomous vehicle to remain in the first position or to cause the autonomous vehicle to remain in the first position based at least in part on the first number being less than the threshold number. 2. The system according to any of paragraphs U through X, further comprising the step of traversing to position 2.

Z: controlling an autonomous vehicle to traverse an environment to a first location; determining an occlusion grid based at least in part on accessing map data or identifying obstacles in the environment; determining that at least a first portion of the occlusion grid is occluded at a first time as a region; and at least a second portion of the occlusion grid is visible at the first time as a non-occlusion region; determining an occupancy state of the non-occlusion region; and controlling the autonomous vehicle based at least in part on the extent of the non-occlusion region and the occupancy state of the non-occlusion region. remaining at a first position or traversing to a second position; determining that a third portion of the occlusion grid is non-occluding and unoccupied at a second time; controlling the autonomous vehicle based at least in part on the third portion of the occlusion grid being unoccluded and unoccupied at a time of .

AA: determining that the autonomous vehicle is approaching an intersection in an environment, the autonomous vehicle lacking right-of-way at the intersection; The method of paragraph Z, wherein the stop line is associated with the intersection, and the second location is a check line within the intersection at least partially beyond the stop line.

AB: controlling the autonomous vehicle to traverse from the first location to the second location at a first speed less than a threshold speed; and determining to capture sensor data at the second location. and determining, based at least in part on the sensor data, that the third portion is non-occluding and unoccupied at the second time, and a range of the third portion is greater than or equal to a threshold. and controlling the autonomous vehicle to cross the intersection from the second location at a second speed that is greater than or equal to the speed threshold.

AC: the obstacle is a dynamic obstacle, the method comprises: identifying the dynamic obstacle in the environment; associating the occlusion grid with the dynamic obstacle; and the non-occlusion area. controlling the autonomous vehicle to remain in the first position at the first time based at least in part on the extent of the area and the occupancy of the non-occlusion region; The method of any of paragraphs Z through AB, further comprising controlling the autonomous vehicle based at least in part on the third portion of an occlusion grid being non-occluding and non-occupying.

AD: identifying the obstacle in the environment; associating the occlusion grid with the obstacle; identifying a dynamic object in the environment; and motion of the dynamic object in the environment. capturing data; determining that the dynamic object has traversed the occlusion region; determining a predicted trajectory of the dynamic object within the occlusion region; and determining the predicted trajectory of the dynamic object within the occlusion region. controlling the autonomous vehicle based at least in part on the method of any of paragraphs Z through AC.

AE: determining that the autonomous vehicle is within a distance threshold to a terrain occlusion area; and determining the occlusion based at least in part on the map data associated with the terrain occlusion area as a terrain occlusion grid. The method of any of paragraphs Z to AD, further comprising the step of determining a grid.

AF: capturing LIDAR data using a LIDAR sensor on the autonomous vehicle and emitting a portion of the LIDAR data to determine an occlusion state and an occupancy state of an occlusion field; The method of any of paragraphs Z to AE, further comprising the step of representing a portion of a grid.

AG: capturing image data using an image sensor on the autonomous vehicle; and performing segmentation on at least a portion of the image data to identify drivable surfaces in the environment. generating image data; projecting an occlusion field onto the segmented image data, the occlusion field representing a portion of the occlusion grid; and the step of projecting an occlusion field onto the segmented image data; and controlling the autonomous vehicle to traverse the environment based at least in part on the occlusion field being projected onto the drivable surface. The method described in any of paragraphs AF.

AH: selected based at least in part on a first range of the portion for traversing within the environment and a second range of the third portion of the occlusion grid that is non-occluding and non-occupying. The method of any of paragraphs Z through AG, further comprising controlling the autonomous vehicle to traverse the portion of the environment based on the speed determined.

AI: determining a time period associated with the occlusion region; determining that no objects entered the occlusion region during the time period; and based at least in part on a speed limit associated with the occlusion region. determining a confidence level associated with the occlusion region occupancy based at least in part on the time period.

AJ: When executed, causing one or more processors to control a robot platform to traverse to a first location in an environment, and determining a position of the robot platform with respect to a map, the map comprising: determining an occlusion grid based at least in part on: including map data; and accessing the map data associated with the location; or identifying obstacles in an environment proximate the robot platform; determining that at least a first portion of the occlusion grid is occluded at a first time as an occlusion region; and at least a second portion of the occlusion grid at the first time as a non-occlusion region; determining an occupancy state of the non-occlusion region, based at least in part on the extent of the non-occlusion region and the occupancy state of the non-occlusion region; controlling a platform to remain at the first location or traverse to a second location; and determining at a second time that a third portion of the occlusion grid is non-occluding and non-occupying. , controlling the robotic platform based at least in part on the third portion being non-occluding and non-occupying at the second time. computer-readable medium.

AK: the operation further comprises determining that the robotic platform is approaching an intersection in an environment at which the robotic platform needs to yield to another vehicle, and the first location is associated with the intersection. the non-transitory computer-readable medium of paragraph AJ, wherein the second location is a check line within the intersection at least partially beyond the stop line.

AL: the operations include capturing LIDAR data using a LIDAR sensor on the robot platform and beaming a portion of the LIDAR data to determine occlusion field occupancy; an occlusion field representing a portion of the occlusion grid; and determining that the occlusion grid is visible and unoccupied based at least in part on the occupancy percentage. non-transitory computer-readable medium as described in .

AM: the operation includes capturing image data using an image sensor on the robot platform and performing segmentation on at least a portion of the image data to identify drivable surfaces in the environment. generating segmented image data; projecting an occlusion field onto the segmented image data, the occlusion field representing a portion of the occlusion grid; and and controlling the robotic platform to traverse the environment based at least in part on the occlusion field projected onto the drivable road surface. A non-transitory computer-readable medium according to any of paragraphs AJ to AL.

AN: the operation determines a period of time associated with the occlusion region; determines that no objects entered the occlusion region during the period; and determines a confidence level associated with a safe trajectory. The non-transitory computer-readable medium of any of paragraphs AJ to AM, further comprising the step of:

Although the example invention subject matter described above is described with respect to one particular implementation, in the context of this document, the example invention subject matter also relates to methods, devices, systems and/or computer readable media, and/or other implementations. It should be understood that it can be implemented via

Conclusion Having described one or more examples of the technology described herein, various modifications, additions, substitutions, and equivalents thereof fall within the scope of the technology described herein.

The illustrative description refers to the accompanying drawings, which form a part of the specification, and which illustrate by way of example specific examples of the claimed subject matter. It is to be understood that other examples may be used and modifications or substitutions may be made, such as structural changes. Such illustrations, modifications, or substitutions do not necessarily depart from the scope of the intended claimed subject matter. Although the steps herein may be presented in a particular order, the order may be changed in some cases to provide certain inputs at different times or in a different order without changing the functionality of the systems and methods described. . The disclosed procedures can also be performed in different orders. Furthermore, the various calculations herein need not be performed in the order disclosed, and other examples using alternative orders of calculations can be easily implemented. In addition to reordering, calculations can also be decomposed into sub-calculations with the same result.

Claims (15)

one or more processors;
one or more computer-readable media storing instructions executable by the one or more processors;
A system comprising: the instructions, when executed, causing the system to receive sensor data from a sensor on a vehicle;
receiving map data representing an environment proximate to the vehicle, the map data including an occlusion grid including a plurality of occlusion fields, the occlusion field of the plurality of occlusion fields being an occlusion; an occlusion state indicative of the part of the environment associated with the environment; and an occlusion state indicative of the part of the environment being occupied;
determining the occupancy state and the occlusion state of the occlusion field based at least in part on a portion of the sensor data;
an amount of information associated with the occlusion grid based at least in part on the occlusion field ;
the number of unoccupied occlusion fields is greater than or equal to a first threshold number;
the number of visible occlusion fields is greater than or equal to a second threshold number;
observing an undetermined occlusion field for a threshold period, or
determining the extent of non-occlusion and unoccupied areas relative to the distance of the intersection;
determining based at least in part on one or more of the
determining a safe trajectory for the vehicle traversing the environment based at least in part on the amount of information ;
controlling the vehicle to traverse the environment according to the safe trajectory;
A system that performs operations involving. the sensor is a LIDAR sensor, the sensor data includes LIDAR data, and the operation includes:
receiving image data from an image sensor on the vehicle;
performing semantic segmentation on at least a portion of the image data to produce segmented image data that identifies at least a drivable road surface within the environment;
associating the occlusion field with the segmented image data;
determining that the occlusion field is associated with the drivable road surface;
controlling the vehicle to traverse the environment based at least in part on the occlusion field associated with the drivable surface;
The system of claim 1, further comprising: 3. The system of claim 1 or 2, wherein the plurality of occlusion fields individualize portions of a driveable road surface in the environment, and wherein the occlusion state of the occlusion fields is indicative of the visibility of the occlusion field by the sensor. The operation is
a first number of LIDAR returns associated with the occlusion field;
a second number of individualized regions associated with the occlusion field;
3. The system of claim 1 or 2, further comprising determining a confidence value associated with the occupancy of the occlusion field based at least in part on the occlusion field. A first ratio of the first number to the second number is less than a threshold, indicating a high reliability of the unoccupied state of the occlusion field; 5. The system of claim 4, wherein a ratio of 2 equal to or greater than the threshold indicates a high reliability of the occupied state of the occlusion field. one or more processors;
one or more computer-readable media storing instructions executable by the one or more processors, the method comprising:
receiving sensor data captured using sensors on the robot platform;
determining a position of the robotic platform based at least in part on the sensor data;
receiving map data representing an environment proximate the robot platform at the location, the map data including an occlusion grid;
determining an occupancy state of an occlusion field of the occlusion grid, the step of:
an amount of information associated with the occlusion field based at least in part on the occlusion field ;
the number of unoccupied occlusion fields is greater than or equal to a first threshold number;
the number of visible occlusion fields is greater than or equal to a second threshold number;
observing an undetermined occlusion field for a threshold period, or
determining the extent of non-occlusion and unoccupied areas relative to the distance of the intersection;
determining based at least in part on one or more of the
determining a trajectory that traverses a region of the environment proximate to the occlusion grid based at least in part on the amount of information ;
controlling the robotic platform based at least in part on the occupancy of the occlusion grid;
How to prepare. receiving LIDAR data captured using a LIDAR sensor on the robot platform as the sensor data;
determining the occupancy state of the occlusion field based at least in part on associating a portion of the LIDAR data with an occupancy field, the occupancy state being an occupied state, an unoccupied state, or an unoccupied state; a step including one of the decision states;
determining an occlusion state of the occlusion field based at least in part on the portion of the LIDAR data, the occlusion state comprising one of being visible or occluded;
controlling the robotic platform to traverse the environment based at least in part on the occlusion state and the occupancy state;
Furthermore,
The amount of information is related at least in part to the number of occlusion fields of the occlusion grid that are visible to the LIDAR sensor, in relation to at least one of the size of the occlusion grid or a speed limit associated with the occlusion grid. further determined based on,
The method according to claim 6. determining at least one of the occupancy state or the occlusion state ,
determining a first number of expected LIDAR returns associated with a region above the occlusion field;
determining a second number of actual LIDAR returns associated with the area above the occlusion field;
determining a confidence value associated with the occupancy state of the occlusion field based at least in part on the first number of expected LIDAR returns and the second number of actual LIDAR returns;
determining that the confidence value is greater than or equal to a confidence threshold;
8. The method according to claim 7, comprising: receiving image data captured using an image sensor on the robot platform as the sensor data;
segmenting at least a portion of the image data to produce segmented image data that identifies drivable road surfaces within the environment;
associating the occlusion field with the segmented image data;
determining that the occlusion field is associated with the drivable surface;
controlling the robotic platform to traverse the environment based at least in part on the occlusion field associated with the drivable surface;
7. The method of claim 6, further comprising: the robotic platform includes an autonomous vehicle, the environment proximate the autonomous vehicle includes an intersection, and the method includes:
determining a distance associated with the intersection traversed by the autonomous vehicle;
determining a speed limit associated with the intersection;
the distance associated with the intersection;
the speed limit;
safety factor and
determining a range of the occlusion grid based at least in part on
7. The method of claim 6, further comprising: 11. The method of claim 10, further comprising determining the extent of the occlusion grid based at least in part on an expected time for the autonomous vehicle to traverse the distance associated with the intersection. the occlusion field includes a temporal logic symbol, and the method is based at least in part on evaluating a temporal logic expression including the temporal logic symbol;
controlling the robotic platform to traverse the environment based at least in part on the trajectory;
7. The method of claim 6, further comprising: the occlusion grid includes a plurality of occlusion fields, and the method determines that a threshold number of the plurality of occlusion fields are visible and unoccupied from the sensor before the method controls the robotic platform to traverse the environment. ,
7. The method of claim 6, further comprising: determining that a portion of the occlusion grid is not visible to the sensor;
controlling the robot platform to move the portion of the occlusion grid visible from the sensor into a position;
7. The method of claim 6, further comprising: 15. A computer program comprising coded instructions which, when executed on a computer, performs a method according to any one of claims 6 to 14.

Images