JP2024510058A - Collision avoidance using object contours - Google Patents

Abstract

A collision avoidance technique using object outlines will be explained. A trajectory associated with the vehicle may be received. Sensor data can be received from sensors associated with the vehicle. A boundary contour can be determined and associated with the object represented by the sensor data. Based on the trajectory, a simulated position of the vehicle can be determined. Furthermore, the predicted position of the boundary contour can be determined. Based on the simulated vehicle position and the predicted position of the boundary contour, a distance between the vehicle and the object may be determined. Actions can be performed based on the distance between the vehicle and the object.

Description

The present invention relates to collision avoidance using object contours.

This patent application is a priority application of U.S. Utility Patent Application No. 17/138,710, filed on December 30, 2020, and U.S. Utility Patent Application No. 17/138,751, filed on December 30, 2020. claim rights. Application numbers 17/138,710 and 17/138,751 are fully incorporated herein by reference.

Vehicles often encounter objects as they drive through the environment. These objects may include other vehicles, pedestrians, animals, etc. When attempting to safely pass a particular object, a vehicle may determine the object's location, size, and shape. In some situations, a vehicle may determine the size and shape of an object based on a bounding box. However, bounding boxes may not provide an accurate representation of objects. This obstruction can impair safe navigation of a vehicle passing the object.

The detailed description will be explained with reference to the accompanying drawings. In the drawings, the left-most digit of a reference number identifies the drawing in which the reference number first appears. The use of the same reference numbers in different drawings indicates similar or identical components or features.
FIG. 3 illustrates an example implementation for determining a bounding contour associated with an object and using the bounding contour to check a vehicle's trajectory for potential collision with the object, according to examples of the present disclosure. 2 is a flowchart illustrating an example process for determining a boundary contour associated with an object based on sensor data associated with the object, according to examples of the present disclosure. 2 is a flowchart illustrating an example process for using bounding boxes and bounding contours to identify potential collisions with objects, according to examples of the present disclosure. FIG. 3 is a diagram illustrating an example lidar point cloud associated with an object and a boundary shape that defines the lidar point cloud, according to examples of the present disclosure. FIG. 3 is a diagram illustrating an example lidar point cloud associated with an object and a boundary shape that defines the lidar point cloud, according to examples of the present disclosure. 2 is a flowchart illustrating an example process for generating a convex hull associated with a lidar point cloud and creating a boundary contour based on the convex hull, according to examples of the present disclosure. FIG. 7 illustrates an example process for replacing a bounding edge of a convex hull with two shorter segments, according to examples of the present disclosure. FIG. 3 illustrates another example process for replacing a bounding edge of a convex hull with two shorter segments, according to examples of the present disclosure. FIG. 7 illustrates an example convex hull with lidar points on a boundary edge and an example stray point located away from a majority of the lidar points, according to examples of the present disclosure. FIG. 7 illustrates an example convex hull with lidar points on a boundary edge and an example stray point located away from a majority of the lidar points, according to examples of the present disclosure. 2 is a flowchart illustrating an example process for determining a distance between a vehicle and an object based on a bounding contour of the object, according to examples of the present disclosure. FIG. 3 is a diagram illustrating an example vehicle trajectory and boundary contour of an object when determining the distance between the vehicle and the boundary contour, according to examples of the present disclosure. FIG. 3 is a diagram illustrating an example vehicle trajectory and boundary contour of an object when determining the distance between the vehicle and the boundary contour, according to examples of the present disclosure. FIG. 3 is a diagram illustrating an example bounding box, bounding contour, and axis-aligned bounding box, according to examples of the present disclosure. FIG. 3 is a diagram illustrating an example bounding box, bounding contour, and axis-aligned bounding box, according to examples of the present disclosure. FIG. 3 is a diagram illustrating an example bounding box, bounding contour, and axis-aligned bounding box, according to examples of the present disclosure. FIG. 1 illustrates an example system for implementing the techniques described herein.

The present disclosure is directed to techniques for determining boundary contours associated with objects and using those boundary contours for actions such as collision avoidance. In some examples, a lidar sensor may receive lidar data associated with an object. Lidar data may be projected onto a two-dimensional plane, and a convex hull may be identified based on the projected lidar data. The bounding edges associated with the convex hull can be removed to determine a bounding contour that represents a close-fitting perimeter of the object. Boundary contours may be used, for example, in a vehicle planning component to determine whether a vehicle is likely to collide with an object represented by the boundary contour.

In some examples, when replacing the longest bounding edge of a convex hull, the longest bounding edge can have a first endpoint and a second endpoint. Interior points within the convex hull may be identified. In some examples, the longest bounding edge may be replaced with 1) a first segment based on a first endpoint and an interior point, and 2) a second segment based on an interior point and a second endpoint. . An updated convex hull may be determined based on the first segment and the second segment. The process of successively replacing the longest boundary edge can produce a boundary contour that closely represents the perimeter of the associated object.

In some examples, the sensor data can include lidar data and can include a variety of lidar points. In connection with projecting lidar data into a two-dimensional representation, some of all lidar points can be relocated to the closest position on a predetermined grid. If any two or more lidar points share the same grid location, some or all of the duplicate lidar points may be removed so that one lidar point remains for each grid location. In some examples, the predetermined grid locations may have a spacing of one centimeter, although other dimensions are contemplated herein.

In some examples, the system may determine whether a vehicle is likely to collide with an object based on a boundary contour associated with the object. If a potential collision is detected, the system may determine a vehicle action to avoid collision with the object and initiate that action to attempt to avoid the collision. In other examples, the action may be determined based on a bounding contour associated with the object. For example, the action may include at least one of controlling a vehicle, determining a static object, determining a dynamic object, or updating map data.

In some examples, bounding boxes associated with objects identified in the lidar point cloud data may be identified. The size of the bounding box is expanded to include a larger amount of lidar point cloud data. Lidar points within the expanded bounding box can be captured and provide increased lidar data associated with the object. A bounding shape associated with all lidar points within the expanded bounding box may be created to better represent the contour of the object.

In some examples, the system may receive a trajectory associated with the vehicle and receive sensor data from sensors associated with the vehicle. A boundary contour may be determined, the boundary contour being associated with the object represented in the sensor data. Based on the trajectory associated with the vehicle, the system may determine a simulated position of the vehicle. Furthermore, a predicted position of the boundary contour may be determined. Based on the simulated position of the vehicle and the predicted position of the boundary contour, a distance between the vehicle and the object may be determined. Actions can be performed based on the distance between the vehicle and the object.

In some examples, the bounding contours described herein may be polygons that closely identify the perimeter of an object. Actions performed based on the distance between the vehicle and the object may include validating the trajectory or invalidating the trajectory. In some examples, the distance between the vehicle and the object may be determined using ray casting. For example, ray casting may extend from the side of the vehicle (or from a point associated with the trajectory) toward the bounding contour (eg, vertically). Collisions can be detected based on some boundary contour edges through which the ray passes. For example, if a ray passes through an even number of boundary contour edges, no collision has occurred. However, if the ray passes through an odd number of boundary contour edges, a collision will occur between the vehicle and the object.

In some examples, axis-aligned bounding boxes can be used to quickly identify potential collisions, such as between a vehicle and an object. For example, an axis-aligned bounding box may be associated with a bounding contour. A second distance between the vehicle and the object may be determined based on the axis-aligned bounding box. In some examples, the axis-aligned bounding box is aligned along the longitudinal axis of the vehicle.

The techniques described herein can improve the functionality of a vehicle's computing device in many ways. The described technique provides an improved representation of the shape of an object. As described herein, a bounding contour provides a polygon that closely follows the perimeter of an object. Bounding contours are not limited to four sides like bounding boxes. Therefore, the boundary contour can be adjusted to all shapes of the object to provide an accurate representation of the object. Close-fitting boundary contours support more accurate determination of potential collisions, distances between vehicles and objects, and other actions associated with objects.

The techniques described herein may also provide improved functionality for a vehicle's computing device by determining boundary contours faster and with fewer computing resources. Compared to existing techniques such as Delaunay triangulation, the described systems and methods may simplify the creation of boundary contours without the need to create diverse triangles that cover all points in a set of points. The Delaunay triangulation process, which requires creating a variety of triangles within a set of points, can be time consuming and require significant computing resources. The systems and methods described herein may simplify the process by eliminating the need to calculate triangles at a set of points.

The techniques described herein can be implemented in many ways. Example implementations are provided below with reference to the drawings below. Although described in the context of autonomous vehicles, the methods, apparatus, and systems described herein can be applied to a variety of systems and are not limited to autonomous vehicles. In another example, the technology can be utilized with any type of vehicle, robotic system, or any system that uses sensor data. Additionally, the techniques described herein may be applied to real data (e.g., captured using a sensor), simulated data (e.g., generated by a simulator), or any combination of the two. It can be used with

FIG. 1 illustrates an environment 122 and an environment 132 for determining a bounding contour associated with an object and using the bounding contour to check the trajectory of a vehicle for possible collision with the object, according to examples of the present disclosure. 1 shows a pictorial flow diagram of an example process 100 associated with. In some examples, determining a boundary contour associated with an object may be performed by a vehicle computing device within a vehicle, as described herein. Further, in some examples, a vehicle computing device may be used to check the vehicle's trajectory for potential collisions with objects based on the boundary contours.

As shown in FIG. 1, process 100 includes an act 120 of determining a boundary contour associated with an object proximate to a vehicle. Process 100 further includes an act of checking 130 the trajectory of the vehicle for potential collisions with the object using the bounding contour associated with the object.

As shown in environment 122, example image 102 depicts a scene in which object 104 may operate. In example image 102, object 104 is shown as a vehicle. In other examples, object 104 may be any type of static or dynamic object. For example, static objects may include trees, buildings, signs, traffic lights, etc. Exemplary dynamic objects may include vehicles, pedestrians, animals, etc. In some examples, the object 104 may be moving at certain times (e.g., a moving vehicle) and stationary at other times (e.g., a parked vehicle or a vehicle stopped at a traffic light). good.

FIG. 1 also shows an example of a boundary contour 106 associated with an object 104 within the image 102. Boundary contour 106 may be created using, for example, the systems and methods described herein. Boundary contour 106 may be associated with the outer perimeter of object 104. As described herein, boundary contour 106 may be used to avoid collisions with object 104 and maintain a safe distance from the object. In some examples, bounding contour 106 may be used in combination with a bounding box (e.g., a rectangular box associated with object 104), both of which may provide information associated with object 104. .

In some examples, the bounding contour 106 may provide a better representation of the exact shape of the object than the bounding box. In a particular example, the bounding box may be a rectangle with four sides. This basic rectangular shape may not accurately represent the object because the bounding box may be expanded to include all parts of the object. For example, in the situation of a vehicle with large side mirrors, the bounding box expanded to fit the mirror may be significantly larger than the actual vehicle. Similarly, if the bounding box is sized to fit the body of a vehicle with large side mirrors, the bounding box may not encompass the side mirrors. In contrast, the bounding contour 106 may be a polygon created to closely follow the outer edges of the object, regardless of the object's shape or irregularities. This close-fitting polygon may provide a more accurate representation of the object's shape than a rectangular bounding box. In one or more examples, the polygon representing bounding contour 106 may have more than five outer edges, also referred to herein as "bounding edges."

In some examples, bounding contour 106 may be used when checking a vehicle's trajectory for potential collisions with object 104. As shown in environment 132, trajectory 110 may define a planned driving path for vehicle 112 through environment 132. In FIG. 1, a boundary contour 114 associated with an object is displayed for a trajectory 110 and a vehicle 112. Boundary contour 114 may be associated with any object type, such as another vehicle, pedestrian, animal, building, street sign, tree, etc. As described herein, the described systems and methods may use ray casting to determine the distance between vehicle 112 and boundary contour 114, as illustrated by ray 116. Based on the distance between vehicle 112 and boundary contour 114, vehicle 112 may take one or more actions to avoid collision with an object associated with boundary contour 114. For example, vehicle 112 may take actions such as steering, braking, sounding the horn, alerting vehicle occupants, etc. depending on the distance from the object and the potential risk associated with the type of object.

As described herein, some examples include determining a boundary contour associated with an object, checking the trajectory of the vehicle for potential collisions with the object, and using a vehicle computing device within the vehicle. may be used to perform other activities. Additional details regarding the vehicle are described herein.

FIG. 2 is a flowchart illustrating an example process 200 for determining a boundary contour associated with an object based on sensor data associated with the object, according to examples of the present disclosure. The operations described herein with respect to process 200 may be performed by one or more vehicle computing devices as described herein.

As an example, process 200 is depicted as a logical flow graph, each operation of which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the case of software, the operations may represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined (in any order and/or in parallel) to implement process 200. or omitted). In some examples, the various branches represent alternative implementations that may be used individually or in combination with other operations described herein.

At act 202, the process may include receiving image data that includes an object. In some examples, image data may be captured by an image sensor (eg, a camera) associated with a vehicle. At act 204, the process may include determining a bounding box associated with the object based on the image data. In some examples, a three-dimensional bounding box may be determined based on image data. Exemplary techniques for determining three-dimensional bounding boxes are disclosed in U.S. Patent Application No. 15/814,870, entitled "Pose Determination from Contact Point," filed November 16, 2017, and published in 2019. No. 16/408,414, filed May 9, entitled "Image-Based Depth Data and Bounding Boxes," both of which are available in their entirety for all purposes , incorporated herein by reference. However, the bounding box can be determined based on other sensor data such as lidar data, time-of-flight data, etc. In some examples, process 200 may omit acts 202 and 204.

In some examples, the bounding box may be determined based on lidar data that may be captured by a lidar sensor, as described herein. In this example, acts 202 and 204 may be replaced with an act of determining a bounding box based on lidar data associated with the object.

At act 206, the process may receive lidar data associated with the object. Lidar data may be received from lidar sensors associated with a vehicle, structure, or any other item. In some examples, lidar data includes a point cloud that includes multiple lidar points identified by lidar sensors. At act 208, the process may determine a point cloud associated with the object. A point cloud associated with an object may represent a sub-region (eg, a region of interest) of the overall point cloud specifically associated with the object. In some examples, the subset of lidar points may be determined based on instance segmentation that identifies objects.

In operation 210, the point cloud associated with the object may be projected onto a two-dimensional plane, such as a substantially horizontal plane. At act 212, the process may determine a boundary contour associated with the object based on the point cloud projected onto the two-dimensional plane. In some examples, the bounding contour is provided by a vehicle planning component (e.g., the planning illustrated in FIG. 12) for use in comparing the bounding contour to a planned vehicle trajectory, as described herein. component 1232).

In one or more examples, the boundary contour may be expanded into a three-dimensional representation. For example, a two-dimensional boundary contour may be defined in the xy plane. A two-dimensional boundary contour can be extended to a three-dimensional representation by extending the boundary contour along the x-axis perpendicular to the xy plane. In some examples, a two-dimensional boundary contour may be expanded to a three-dimensional representation using a convolutional neural network (CNN).

At act 214, the process may determine whether the object is a dynamic object, such as a moving vehicle, pedestrian, animal, etc. If the object is not a dynamic object, the process may proceed to operation 218 to select another object. In situations where the object is static (e.g., not dynamic), the boundary contour may not change at regular intervals, and therefore the boundary contour determined in operation 212 may not require updating. be.

If the object is dynamic, the process may continue from act 214 to act 216, where the process may receive updated lidar data associated with the object. In some examples, if an object is dynamic, its boundary contour may change as the object moves. In this situation, it may be advantageous to update lidar data to determine updated boundary contours. For example, the shape of an articulated bus may change over time as the articulated bus passes corners in the environment. In some examples, the shape of the convex hull can be predicted based on a predicted trajectory associated with the object. After receiving the updated lidar data in operation 216 (or a prediction for the object), the process may return to operation 206 to determine updated lidar data associated with the object.

FIG. 3 is a flowchart illustrating an example process 300 for using bounding boxes and bounding contours to identify potential collisions with objects, according to examples of the present disclosure. The operations described herein with respect to process 300 may be performed by one or more vehicle computing devices as described herein.

By way of example, process 300 is depicted as a logical flow graph, each operation of which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the case of software, the operations may represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined (in any order and/or in parallel) to implement process 300. or omitted). In some examples, the various branches represent alternative implementations that may be used individually or in combination with other operations described herein.

At act 302, the process may include determining a bounding box associated with the object. In some examples, the bounding box may be determined based on an image of the object, lidar data associated with the object, etc. In some examples, the bounding box determined in act 302 may be an axis-aligned bounding box based on the bounding contour determined in act 304. Additional examples of axis-aligned bounding boxes are described in this disclosure as well as in FIGS. 11A-11C.

At act 304, the process may determine a boundary contour associated with the object. For example, the systems and methods described herein may be used to determine boundary contours associated with objects. Additional examples of determining boundary contours are described in FIGS. 6, 7, 8A, and 8B and throughout this disclosure.

At act 306, the process may determine whether there is a potential collision with the object based on the bounding box. In some examples, operation 306 may determine the likelihood of a collision between the object performing process 300 and the vehicle. If operation 306 determines that there is no probability of collision based on the bounding box (or the probability is less than a threshold), the process ends at operation 314, which indicates "no probability of collision." Indicates the condition. In some examples, a "no collision possible" condition may be communicated to one or more systems, such as the vehicle, other vehicles, other systems, objects, etc.

If act 306 determines that there is a likelihood of a collision based on the bounding box (eg, the likelihood is greater than a threshold), the process continues to act 308. At act 308, the process may determine whether there is a potential collision with the object based on the boundary contour. In some examples, operation 308 may determine the likelihood of a collision between the object performing process 300 and the vehicle. If operation 308 determines that there is no likelihood of collision based on the boundary contour (or the likelihood is less than a threshold), the process ends at operation 314, which indicates "no likelihood of collision." Indicates the condition.

If act 308 determines that there is a likelihood of a collision based on the boundary contour (eg, the likelihood is greater than a threshold), the process continues to act 310.

In some examples, the bounding boxes described in acts 302 and 306 may have box/cube shapes or other relatively simple geometric shapes. If a potential collision is detected (at act 306), a more accurate representation of the object (eg, a boundary contour) may be used to verify or disprove the detected potential collision at act 306. This approach may reduce the use of computing resources by first checking for collisions with bounding boxes, which is a simple calculation. More complex calculations associated with bounding contours predict that unless a collision with the bounding box is detected or that the bounding box is within a threshold distance of the trajectory (and/or the vehicle's simulated position along the trajectory) It may not be executed unless it is done.

At act 310, the process may determine vehicle actions to avoid a collision with an object. For example, vehicle actions may include steering, braking, honking the horn, warning vehicle occupants, and the like. As described herein, the vehicle action determined in operation 310 may vary depending on the distance from the object and the potential risk associated with the type of object. In act 312, the vehicle actions determined in act 310 are initiated to avoid collision with the object.

4A and 4B are diagrams illustrating an example lidar point cloud 402 associated with an object and a boundary shape 410 that defines the lidar point cloud, according to examples of the present disclosure. In some examples, the lidar point cloud 402 shown in FIG. 4A may be captured by a lidar sensor associated with a vehicle. As described herein, lidar point cloud 402 may be used to create a boundary contour associated with an object. FIG. 4A shows a bounding box 404 that may be associated with some or all of the points in lidar point cloud 402. In some examples, bounding box 404 may represent the general shape of the vehicle, but may include portions of the vehicle that extend beyond the general shape (e.g., mirrors and other objects that extend outward from the body of the vehicle). items) may not be included.

FIG. 4B shows a boundary shape 410 that may be generated based on the lidar point cloud 402 shown in FIG. 4A. As shown in FIG. 4B, a portion of bounding shape 410 extends outside of bounding box 404 shown in FIG. 4A. Therefore, the bounding shape 410 associated with the lidar point cloud may be a better representation of the actual shape of the vehicle than the bounding box 404. When determining whether a vehicle will collide with an object, the exact shape of the object can be important for identifying the likelihood of a collision. The systems and methods described herein describe the creation of boundary contours that define the perimeter of an object. In some examples, each bounding contour may be a polygon that closely identifies the perimeter of the object.

In some examples, the size of bounding box 404 may be expanded to improve the likelihood of capturing all lidar points associated with the object. As shown in FIG. 4A, a portion of lidar point cloud 402 extends outside of bounding box 404. By expanding the size of bounding box 404, some or all of the lidar points located outside bounding box 404 may be captured in bounding shape 410. Therefore, rather than limiting object information to lidar points contained within bounding box 404, the bounding shape 410 associated with the object can be improved by including lidar points located outside of bounding box 404. can provide expression.

In some examples, the bounding box may be expanded by a threshold amount based on the size of the object, object type, etc. In some examples, bounding box 404 may be expanded in size by 0.4 meters. In some implementations, this expansion allows bounding shape 410 to include portions of objects that protrude beyond bounding box 404, such as vehicle side mirrors, pedestrian arms, etc. In other examples, bounding box 404 may expand in size by any amount based on the type of objects encountered, safety requirements, desired safety margins, etc. In one or more examples, bounding box 404 is extended to all four sides to capture lidar points located outside of either side of bounding box 404.

FIG. 5 is a flowchart illustrating an example process 500 for generating a convex hull associated with a lidar point cloud and creating a boundary contour based on the convex hull, according to examples of the present disclosure. The operations described herein with respect to process 500 may be performed by one or more vehicle computing devices as described herein.

As an example, process 500 is depicted as a logical flow graph, each operation of which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the case of software, the operations may represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined (in any order and/or in parallel) to implement process 500. or omitted). In some examples, the various branches represent alternative implementations that may be used individually or in combination with other operations described herein.

At operation 502, the process may include receiving lidar data associated with the environment proximate the vehicle. In some examples, lidar data may include one or more objects that may be important to vehicle navigation or collision avoidance systems.

At act 504, the process may determine a point cloud associated with the object identified within the lidar data. For example, the object may be within or near the vehicle's planned path. In some examples, the point cloud associated with the object may be part of the lidar data received in operation 502.

At act 506, the process may project the point cloud associated with the object onto a two-dimensional plane. At operation 508, the process may generate a convex hull associated with the point cloud. In some examples, the convex hull may include multiple bounding edges associated with the point cloud. In one or more examples, the convex hull may be generated using known algorithms for computing convex hulls for a finite set of points. In some examples, the convex hull may have more than five bounding edges.

In some examples, points associated with a point cloud or other grouping of points may be relocated to the closest position on a predetermined grid. For example, a given grid may have uniform spacing, such as one centimeter between grid points. Each point in this point cloud can be relocated to the nearest grid point. In some examples, two or more points may be placed on the same grid point. In this situation, all duplicate points of the same grid point are removed, thereby leaving one point at the grid point.

In some instances, aligning the points in a point cloud to a predetermined grid may simplify the creation of boundary contours associated with the point cloud, since fewer points may be represented in the grid. . This simplification may reduce the time and computing resources required to generate boundary contours from point clouds.

At act 510, the process may determine the longest bounding edge associated with the convex hull. The longest bounding edge can have a first endpoint and a second endpoint.

In act 512, the process may replace the longest border edge with two shorter border edges. In some examples, the first short border edge may include a first endpoint and the second short border edge may include a second endpoint. In some implementations, replacing the longest bounding edge may include determining an interior point (eg, an interior lidar point) inside the convex hull. The first short boundary edge may be based on the first endpoint and the interior point. The second short boundary edge may be based on the interior point and the second endpoint. In some examples, operation 512 can be based at least in part on the angle formed by the two shorter bounding edges. Additional examples of operations 512 are provided in connection with FIGS. 6 and 7 and throughout this disclosure.

At act 514, the process may determine whether the next longest border edge is too short to remove. In some examples, the next longest border edge may be too short to remove if the length of the next longest border edge is less than half the length of the first border edge replaced in operation 512. be. In alternative examples, other parameters may be used to determine when the next longest border edge is too short to be removed. For example, other parameters may be associated with the updated convex hull, such as the number of boundary edges of the convex hull that meet or exceed a first threshold, the length of the longest remaining boundary edge that is less than a second threshold, or the length of the longest remaining boundary edge that is less than a second threshold. the number of interior points determined is less than a third threshold.

If the next longest border edge is not too short to remove, the process returns to operation 512 to replace the next longest border edge.

If the next longest boundary edge is too short to remove, the process proceeds to operation 516, where the process may create a boundary contour associated with the object. In some examples, the boundary contour is based on the boundary edge of a convex hull associated with the point cloud. In some examples, a boundary contour associated with an object may be used to determine the likelihood of a collision between the vehicle and the object.

In some examples, each point is a vector of one or more triangles. After creating a plurality of triangles that encompass all points in the series of points, various edges may be removed to define the outer boundary associated with the series of points. However, this technique can be time consuming because it requires the generation and processing of triangles encompassing all points in the set of points in an attempt to define the outer boundary. The systems and methods described herein may provide faster techniques that use fewer computing resources to generate boundary contours.

FIG. 6 is a diagram illustrating an example process 600 for replacing a bounding edge of a convex hull with two shorter segments, according to examples of the present disclosure. As shown in FIG. 6, polygon 620 may include any number of bounding edges and any number of points (eg, lidar points) contained within the bounding edges. In the example of FIG. 6, polygon 620 includes multiple points. Some of these points are labeled as 602-1, 602-2, 602-3 and 602-4. In some examples, point 602 may be located inside a polygon (eg, points 602-1, 602-3, and 602-4) or on a boundary of a polygon (eg, point 602-2). An exemplary bounding edge 604 of polygon 620 is shown in FIG.

In some embodiments, all points associated with a polygon can be on the boundary of the polygon or inside the polygon. As described herein, one or more bounding edges of a polygon may be replaced to create a tighter bounding contour. In one or more examples, the resulting polygon (after replacing the bounding edges) may maintain a simple polygon shape. For example, the resulting polygon may not have any loops, holes, or self-intersecting edges. The examples described herein illustrate the removal of bounding edges in a way that maintains a simple polygon shape.

As shown in FIG. 6, polygon 630 is similar to polygon 320 described above. Based on polygon 630, the systems and methods described herein select the longest bounding edge to replace with two shorter bounding edges. In one example, the longest bordering edge may be bordering edge 606 having a first endpoint 602-5 and a second endpoint 602-6. Endpoints 602-5 and 602-6 are points associated with the boundaries of polygon 630.

In some examples, pairs of shorter border edges may be identified as possible candidates to replace the longest border edge 606. As shown in polygon 630, the first pair of shorter bounding edges are shown as bounding edges 608 and 610. For example, the first short bounding edge 608 extends from point 602-5 (one of the endpoints of the longest bounding edge 606) to point 602-3 (shown as polygon 620). In this example, point 602-3 is an internal point inside the polygon. As shown in polygon 630, the second short bounding edge 610 extends from point 602-6 (another endpoint of longest bounding edge 606) to point 602-3 (shown in polygon 620). Thus, the first pair of shorter bounding edges 608, 610 replaces the longest bounding edge 606 by connecting with point 602-3.

In the example of FIG. 6, the second pair of shorter border edges are shown as border edges 612 and 614. For example, the first short bounding edge 612 extends from point 602-5 (one of the endpoints of the longest bounding edge 606) to point 602-4 (shown as polygon 620). In this example, point 602-4 is an interior point inside the polygon. As shown in polygon 630, the second short bounding edge 614 extends from point 602-6 (another endpoint of longest bounding edge 606) to point 602-4 (shown in polygon 620). In this situation, the second pair of shorter bounding edges 612, 614 replace the longest bounding edge 606 by connecting with point 602-4.

Therefore, the two candidates to replace the longest border edge 606 are the first pair 608, 610 and the second pair 612, 614. In some examples, to select a particular candidate, the described systems and methods may identify the angle between the two shorter bounding edges associated with each pair. In one or more examples, the selected pair of shorter bounding edges is the pair that has the largest angle between the two shorter bounding edges. In the example of FIG. 6, the first pair of shorter bounding edges 608, 610 have an angle of 616. As shown in FIG. 6, the second pair of shorter bounding edges 612, 614 have an angle of 618. In this example, angle 616 is greater than angle 618. Therefore, the first pair of shorter border edges 608, 610 is selected to replace the longest border edge 606.

As shown in FIG. 6, polygon 640 is similar to polygons 620 and 630 described above. Referring to polygon 640, the longest bounding edge 606 has been replaced by two shorter bounding edges 608 and 610. As shown in polygon 640, the new polygon is smaller based on the replacement of longest bounding edge 606. Because the shorter bounding edges 608 and 610 share the point 602-3 that was previously located within the polygon, the integrity of the polygon is maintained while tightening the convex hull. In some examples, shorter bounding edges 608 and 610 replace longest bounding edge 606 without removing any points.

In some examples, the described systems and methods repeat the process 600 described with respect to FIG. 6 to continue removing the longest border edges. As described above with respect to FIG. 5, the process may be repeated until the next longest border edge is too short.

FIG. 7 is a diagram illustrating another example process 700 for replacing a bounding edge of a convex hull with two shorter segments, according to examples of the present disclosure. As shown in FIG. 7, polygon 710 is similar to the polygon described with respect to FIG. Polygon 710 represents a polygon with new bounding edges 608 and 610, as described above. In some examples, border edge 610 is determined to be the next longest border edge to be replaced. As shown in polygon 710 of FIG. 7, a pair of shorter bounding edges 704 and 706 is identified as a candidate to replace the current longest bounding edge 610. In this example, the angles associated with shorter bounding edges 704 and 706 are the same as the angles associated with other shorter bounding edge candidates. In some examples, if the two angles are the same, the described systems and methods may randomly select one of the candidates. In the example of FIG. 7, the current longest bounding edge 610 is replaced with new shorter bounding edges 706 and 708, as shown in polygon 720. In example polygon 720, new short bounding edge 706 has endpoints 702-3 and 702-4, and new short bounding edge 708 has endpoints 702-4 and 602-6.

8A and 8B are diagrams illustrating an example convex hull with lidar points on the boundary edge and example stray points located away from a majority of the lidar points, according to examples of the present disclosure. In the example of FIG. 8A, the convex hull is shown as having a plurality of lidar points on the boundary of the convex hull. For example, points 802-1, 802-2, and 802-3 are located on the convex hull boundary. As shown in FIG. 8A, the proposed new boundary edge 804 extends from points 802-1 to 802-3. In some examples, this proposed new boundary edge 804 is not allowed because the new edge 804 would remove point 802-2 from the boundary. In this situation, point 802-2 is outside the convex hull. In one or more examples, this type of border edge replacement is not allowed because all points may be required to be within the convex hull or may be associated with the boundaries of the convex hull.

In the example of FIG. 8B, the exemplary stray point 810 is located away from the majority of the lidar points. In some examples, point 810 is maintained as part of the convex hull polygon. In one or more examples, point 810 is not removed from the polygon, and the edges that place point 810 outside the polygon are not replaced (as described with respect to FIG. 8A).

In some examples, border edges are replaced sequentially to ensure that simple polygons are maintained after each border edge replacement. In particular examples, the systems and methods described herein may be used with convex hulls representing static or dynamic objects.

FIG. 9 is a flowchart illustrating an example process 900 for determining a distance between a vehicle and an object based on a bounding contour of the object, according to examples of the present disclosure. The operations described herein with respect to process 900 may be performed by one or more vehicle computing devices as described herein.

As an example, process 900 is depicted as a logical flow graph, each operation of which represents a sequence of operations that can be implemented in hardware, software, or a combination thereof. In the case of software, the operations may represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations. Generally, computer-executable instructions may include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types. The order in which the operations are described is not intended to be construed as a limitation, and any number of the described operations may be combined (in any order and/or in parallel) to implement process 900. or omitted). In some examples, the various branches represent alternative implementations that may be used individually or in combination with other operations described herein.

At operation 902, the process may include receiving a trajectory associated with a vehicle. A trajectory may be a planned route for a vehicle to follow toward a destination. At operation 904, the process may receive sensor data from sensors associated with the vehicle. In some examples, the sensor may be a lidar sensor and the sensor data may be lidar data, which may include a lidar point cloud.

At operation 906, the process may determine a boundary contour associated with the object represented in the sensor data. In some examples, the boundary contour may be determined based on a lidar point cloud associated with the object.

At operation 908, the process may determine a simulated position of the vehicle based on a trajectory associated with the vehicle. In some examples, the vehicle's position may be simulated at different points along the trajectory to determine the likelihood of collision with an obstacle at those different points. By identifying a potential collision along the trajectory at a future point in time, the vehicle may have extra time to take one or more actions that reduce or eliminate the possibility of a collision. In some examples, a collision may be detected laterally from a pose that the vehicle may occupy at a future time. For example, multiple poses can be predicted that the vehicle may occupy depending on its current trajectory. Along each of these poses, lateral collision detection techniques can be applied according to the disclosed techniques. Similar techniques can be applied to multiple projected trajectories/poses to determine the predicted likely future of the vehicle along multiple paths and/or in response to different future decision possibilities. be able to explain the poses of

At operation 910, the process may determine the predicted location of the boundary contour. In some examples, the simulated position of the vehicle at different points along the trajectory (e.g., future times) may be compared to the predicted position of the boundary contour at the same future time. In some examples, operation 910 can include receiving a predicted trajectory of an object associated with the boundary contour, and in some examples, the predicted position can be based on the predicted trajectory. can. In some examples, operation 910 can include determining a predicted orientation of the boundary contour along the predicted trajectory.

At act 912, the process may determine the distance between the vehicle and the object at a future time. In some examples, the distance may be determined based on the simulated position of the vehicle and the predicted position of a boundary contour representing the object. The distance between the vehicle and the boundary contour may help determine the likelihood of a collision between the vehicle and the object.

At act 914, the process may perform an action based on the distance between the vehicle and the object. In some examples, an action may be initiated if the distance between the vehicle and the object is less than a threshold. In some examples, the threshold may be based on vehicle and/or object speed, object classification, map data associated with the environment, object behavior, etc. As described herein, the action may include a driving maneuver that reduces or eliminates the possibility of a collision between the vehicle and the object.

At operation 916, the process may (later) determine an updated vehicle position on the trajectory. In some examples, the process may determine the simulated vehicle position based on a later time and return to operation 908 to determine the likelihood of a collision at a later time.

10A and 10B are diagrams illustrating an example vehicle trajectory 1002 and a bounding contour 1004 of an object when determining the distance between the vehicle and the bounding contour 1004, according to examples of the present disclosure. In some examples, vehicle trajectory 1002 identifies the vehicle's planned navigation path through the environment. As shown in FIGS. 10A and 10B, boundary contour 1004 is located proximate vehicle track 1002. As described herein, boundary contour 1004 may be associated with any type of object, such as a vehicle, pedestrian, animal, building, road sign, tree, etc.

FIG. 10B shows a particular time when vehicle 1006 is located at a particular location on vehicle trajectory 1002. In the example of FIG. 10B, it may be desirable to determine the distance between vehicle 1006 and the object defined by bounding contour 1004. This distance may be important for determining whether vehicle 1006 can be safely guided past boundary contour 1004 when following vehicle trajectory 1002. In some embodiments, the distance between vehicle 1006 and the object defined by bounding contour 1004 may be determined by raycasting, as shown by raytracing 1008.

Based on the distance between vehicle 1006 and boundary contour 1004, vehicle 1006 may take one or more actions to avoid collision with an object associated with boundary contour 1004. For example, the vehicle 1006 may take actions such as steering, braking, sounding the horn, alerting vehicle occupants, etc. depending on the distance from the object and the potential risk associated with the type of object. In some examples, certain types of objects, such as fast-moving vehicles, may be at greater risk of colliding with vehicle 1006 than stationary objects, such as trees or traffic signs.

In some examples, vehicle 1006 and boundary contour 1004 are tracked along vehicle trajectory 1002 to determine future collisions between vehicle 1006 and objects as vehicle 1006 moves along vehicle trajectory 1002. can be propagated. In some implementations, boundary contour 1004 may be determined at multiple points along vehicle trajectory 1002. The position of vehicle 1006 may be determined at those same points along vehicle trajectory 1002 using ray casting, as described herein. These points along vehicle trajectory 1002 may be particular points in space and time.

In some examples, the systems and methods described herein may determine whether vehicle 1006 can be safely guided past the object associated with boundary contour 1004. Even in the absence of a collision, in some examples there may be a minimum distance threshold (eg, a safety margin) to ensure safe navigation of the vehicle 1006 past the object.

In some examples, the first boundary may be used to detect a potential collision between the vehicle and an external object. For example, the first boundary may be associated with a bounding box/cube or other relatively simple geometric shape. If a potential collision is detected, a more accurate (and possibly more processor intensive) boundary technique, such as the disclosed technique, can be used. More accurate bounding techniques can then be used to verify or disprove potential collisions detected using bounding boxes (or other shapes).

As shown in FIG. 10B, ray trace 1008 may extend vertically from the side of vehicle 1006 and pass through boundary contour 1004. In some examples, when determining whether a collision may occur between the vehicle 1006 and the object associated with the boundary contour 1004, the systems and methods described herein may detect the boundary contour by ray tracing 1008. Determine the number of boundaries intersected within contour 1004. In a particular example, if ray trace 1008 crosses an even number of boundaries as it passes through boundary contour 1004, there is no collision between vehicle 1006 and the object. In the example of FIG. 10B, ray trace 1008 intersects two boundaries of bounding contour 1004 (eg, once as the ray enters bounding contour 1004 and once as the ray exits bounding contour 1004). Because the number of boundaries is even, vehicle 1006 is less likely to collide with an object. This can be confirmed by FIG. 10B, which shows the separation between the vehicle 1006 and the boundary contour 1004. In some examples, if ray trace 1008 crosses an odd number of boundaries, it means that ray trace 1008 occurred within boundary contour 1004. If ray trace 1008 occurs within boundary contour 1004, it is an indication of a collision (eg, at least a portion of vehicle 1006 and the object share the same space at the same time).

In the example of FIG. 10B, ray trace 1008 is shown extending perpendicularly from the side of vehicle 1006. In other examples, ray trace 1008 may extend in any direction from any portion of vehicle 1006. For example, ray trace 1008 may extend longitudinally from the front or rear of vehicle 1006 to determine the distance between vehicle 1006 and an object in front of or behind vehicle 1006.

11A-11C are diagrams illustrating example bounding boxes, bounding contours, and axis-aligned bounding boxes, according to examples of the present disclosure. In the example of FIG. 11A, three representations of the object are shown: a bounding box 1102, a bounding contour 1104, and an axis-aligned bounding box (AABB) 1106. The example object depicted in FIG. 11A may be a vehicle. In some examples, bounding box 1102 is a rectangle that represents a significant portion of an object (ie, a vehicle). As shown in FIG. 11A, the long sides of bounding box 1102 may represent the sides of the vehicle. However, the long sides of bounding box 1102 may not include side mirrors extending from each side of the vehicle. Therefore, bounding box 1102 may not accurately represent all parts of a vehicle or other object.

In the example of FIG. 11A, boundary contour 1104 may provide a more accurate representation of the vehicle. For example, boundary contour 1104 identifies the vehicle's two side mirrors, which may be important to consider when determining the likelihood of a collision between the vehicle and another object. As mentioned above, the vehicle's side mirrors are not represented within bounding box 1102, which means that the systems and methods described herein are not aware of side mirrors based solely on bounding box 1102. This may increase the possibility of collisions.

FIG. 11A also includes an axis alignment bounding box (AABB) 1106. AABB (1106) is a bounding box that can be aligned with a particular axis, such as the xy axis, the longitudinal axis of a vehicle or other object. In the example of FIG. 11A, AABB (1106) is aligned on the xy axis. In some examples, AABB (1106) is used to perform an early check regarding a potential collision between the vehicle represented by AABB (1106) and another object. For example, early validation using AABB (1106) may use a dynamic AABB tree for dynamic contours. A dynamic AABB tree may enable efficient searching for contours with AABBs that overlap with the AABB of the query contour (e.g., a vehicle contour to check for possible collisions with other objects).

In the example of FIG. 11B, environment 1110 includes a first vehicle (or object) 1114 and a second vehicle (or object) 1118. In this example, the first vehicle 1114 has an associated AABB (1112) and the second vehicle 1118 has an associated AABB (1116). As shown in FIG. 11B, AABB (1112) and AABB (1116) are aligned with the local universal transverse Mercator (UTM). Therefore, AABB (1112) and AABB (1116) are aligned with the horizontal and vertical axes shown in FIG. 11B. In the example of FIG. 11B, AABB(1112) and AABB(1116) overlap each other, which is useful for detecting a potential collision between the first vehicle 1114 and the second vehicle 1118. May trigger collision avoidance systems or algorithms. However, the example shown in FIG. 11B does not represent a collision situation. Instead, in the example of FIG. 11B, there is a suitable spacing between the first vehicle 1114 and the second vehicle 1118. A potential collision between two vehicles 1114 and 1118 may be indicated due to regions of AABB(1112) and AABB(1116) not accurately representing first vehicle 1114 and second vehicle 1118, respectively. .

In the example of FIG. 11C, environment 1120 includes a first vehicle (or object) 1124 and a second vehicle (or object) 1128. In this example, first vehicle 1124 has an associated AABB (1122) and second vehicle 1128 has an associated AABB (1126). As shown in FIG. 11C, AABB (1122) and AABB (1126) are aligned with the yaw of vehicles 1124 and 1128. This alignment with the yaw of vehicles 1124 and 1128 avoids duplication of AABB (1122) and 1126. By aligning A ABBs (1122) and 1126 to the vehicle's yaw, AABBs (1122) and 1126 provide a more accurate representation of the vehicle's contour. These "closer" AABBs (1122) and 1126 do not overlap, which is a more accurate representation of environment 1120. In the example of FIG. 11C, the collision avoidance system or algorithm may not detect a potential collision between the first vehicle 1124 and the second vehicle 1128.

In some examples, a vehicle planning and/or prediction component uses one or more bounding boxes, bounding contours, and AABBs to determine a vehicle trajectory, guide the vehicle along the vehicle trajectory, and Potential collisions between the vehicle and other objects may be predicted and actions (e.g., vehicle actions) may be initiated based on the potential collision. In one or more examples, the systems and methods described herein first perform a coarse analysis to predict a collision or determine whether the object maintains a threshold distance from the vehicle. It is possible. This threshold distance may also be referred to as a safety radius associated with the vehicle. This coarse analysis may be performed using AABB or bounding boxes, for example. This coarse analysis can be performed quickly with minimal computing resources. If a coarse analysis identifies a potential collision, or if the vehicle and object cannot maintain a threshold distance from each other, the described system and method can perform a more detailed analysis using, for example, boundary contours and ray tracing. It can be executed. In some examples, vehicle planning may be performed from both the front vehicle axle and the rear vehicle axle.

In some examples, collision confirmation may be limited to time or distance. For example, collision confirmation may be time limited to the next few seconds. In a dynamic environment, collision checking in the distant future can be inaccurate due to multiple dynamic objects moving within the environment. Therefore, to save computing resources and generate accurate data, collision checking may be limited to a few seconds into the future.

Additionally, collision confirmation may be limited to a certain distance from the vehicle. For example, a vehicle moving along a trajectory may relate to objects that are within a certain distance from the vehicle or the vehicle's trajectory. In some implementations, objects far from the vehicle and the vehicle's trajectory may be ignored until they move within a threshold distance for analysis. Ignoring these objects saves computing resources and focuses those computing resources on objects that are high priority based on their proximity to the vehicle or vehicle trajectory.

In some examples, the systems and methods described herein may identify collisions between a vehicle trajectory and a boundary contour associated with an object. In one or more examples, an AABB tree can be created, with all bounding contours aligned to the vehicle's yaw (as shown in FIG. 11C). Vehicle trajectories may be added to the AABB tree. In some examples, the AABB tree can be queried for collision candidates. In some examples, the systems and methods described may query the AABB tree based on a particular time period.

In some examples, AABB trees or other techniques may be used to determine potential collisions using AABB or other relative course representations. In response to detecting potential collisions using bounding boxes, a more accurate representation may be determined via bounding contours, as disclosed herein. For example, a course representation may indicate a first probability of a potential collision in a particular scenario. In response to the first probability meeting the threshold, a second determination of potential collisions can be made using a more accurate representation. The second determination may include determining a second probability of a potential collision in the same scenario, where the second probability may be different than the first probability. The second probability may, for example, indicate that a collision is less likely than the first probability. Using a more precise representation can be more complex than using a coarser representation. Using the two-step approach described reduces the overall complexity for the system to determine potential conflicts.

In some examples, if a collision is detected, the described systems and methods override the vehicle's current trajectory or change the accuracy of the trajectory based on the time and distance of the collision from the vehicle. obtain. Similarly, the systems and methods may override or change the vehicle's current trajectory if the safety margin is compromised.

FIG. 12 is a diagram illustrating an example system 1200 for implementing the techniques described herein. Vehicle 1202 includes one or more vehicle computing devices 1204, one or more sensor systems 1206, one or more emitters 1208, one or more communication connections 1210, at least one direct connection 1212, and One or more drive systems 1214 may be included.

In some examples, the system 1200 describes a vehicle that is capable of performing all safety-critical functions during the entire journey without the driver (or occupants) being expected to control the vehicle at any time. It may be implemented as a vehicle, such as an autonomous vehicle, configured to operate according to the Level 5 classification issued by the United States National Highway Traffic Safety Administration. In such instances, the vehicle can be unmanned because it can be configured to control all start-stop functions, including all parking functions. This is by way of example only, and the systems and methods described herein can range from vehicles that must be manually controlled at all times by the driver to vehicles that are partially or fully autonomously controlled. It can be incorporated into any land, air, or water vehicle, including vehicles ranging from

Vehicle computing device 1204 may include one or more processors 1216 and memory 1218 communicatively coupled with one or more processors 1216. In the illustrated example, vehicle 1202 is an autonomous vehicle. However, vehicle 1202 may be any other type of vehicle. In the illustrated example, memory 1218 of vehicle computing device 1204 stores a location component 1220, a perception component 1222, one or more maps 1224, and one or more system controllers 1226. Although shown in FIG. 12 as residing in memory 1218 for exemplary purposes, location component 1220, perception component 1222, one or more maps 1224, and one or more system controllers 1226 are , additionally or alternatively, may be accessible to vehicle 1202 (eg, may be remotely stored).

In at least one illustration, location component 1220 is configured to determine the position and/or orientation (e.g., one or more of x-position, y-position, z-position, roll, pitch, or yaw) of vehicle 1202. may include the ability to receive data from sensor system 1206 . For example, location component 1220 may include and/or request/receive a map of the environment and may continually determine the autonomous vehicle's position and/or orientation within the map. In some examples, the localization component 1220 utilizes SLAM (simultaneous localization and mapping), CLAMS (calibration, localization and mapping simultaneously), relative SLAM, bundle adjustment, nonlinear least squares optimization, etc. The system can receive image data, lidar data, radar data, IMU data, GPS data, wheel encoder data, etc., and accurately determine the position of the autonomous vehicle. In some examples, location component 1220 provides data to various components of vehicle 1202 to generate a trajectory and/or to generate map data or as described herein. An initial position of the autonomous vehicle for receiving may be determined.

In some examples, perception component 1222 may include functionality to perform object detection, segmentation, and/or classification. In some examples, the perception component 1222 detects the presence and/or object type of an object proximate to the vehicle 1202 (e.g., car, pedestrian, cyclist, building, tree, road surface, curb, sidewalk, unknown object, etc.). Processed sensor data may be provided that indicates a classification of the object. In additional or alternative examples, the perception component 1222 is a processed sensor that is indicative of one or more characteristics associated with a detected object (e.g., a tracked object) and/or the environment in which the object is located. Data may be provided. In some examples, the properties associated with an object include x-position (global and/or local position), y-position (global and/or local position), z-position (global and/or local position), orientation. (e.g., roll, pitch, yaw), object type (e.g., classification), object velocity, object acceleration, object extent (size), and the like. Characteristics associated with the environment may include, but are not limited to, the presence of another object in the environment, the state of another object in the environment, time of day, day of the week, season, weather conditions, darkness/light indications, etc. .

In some examples, perception component 1222 may include a boundary contour component 1228 that may create and manage any number of boundary contours, as described herein. In particular examples, boundary contour component 1228 identifies boundary contours associated with one or more objects.

Memory 1218 may further include one or more maps 1224 that may be used by vehicle 1202 to navigate within the environment. In some examples, the map includes, but is not limited to, texture information (e.g., color information (e.g., RGB color information, Lab color information, HSV/HSL color information), etc.), intensity information (e.g., LIDAR information, RADAR information, etc.), spatial information (e.g. image data projected onto a mesh, individual "surfels" (e.g. polygons associated with individual colors and/or intensities)), reflectivity information (e.g. , specularity information, retroreflectivity information, BRDF information, BSSRDF information, etc.). In some examples, the map may include a three-dimensional mesh of the environment. In some examples, the map may be stored in tile format, such that each tile of the map represents a separate portion of the environment, and may be loaded into working memory as needed, as described herein. be. In at least one example, one or more maps 1224 may include at least one map (eg, an image and/or a mesh). In some examples, one or more maps 1224 may be stored on a remote computing device (such as computing device 1242) that is accessible via network 1240. In some examples, various maps 1224 may be stored based on characteristics (eg, object type, time of day, day of the week, season of the year, etc.), for example. Storing a variety of maps 1224 may have similar memory requirements, but increases the speed at which data in the maps can be accessed.

In at least one example, vehicle computing device 1204 may include one or more system controllers 1226 that control steering, propulsion, braking, safety, emitters, communications, and other systems of vehicle 1202. may be configured to control. These system controllers 1226 may communicate with and/or control corresponding systems of drive system 1214 and/or other components of vehicle 1202.

In some examples, memory 1218 may include a prediction component 1230 that predicts various activities and conditions proximate vehicle 1202. In at least one example, prediction component 1230 may predict the likelihood of a collision with an object or other vehicle activity or situation.

In some examples, memory 1218 may include a planning component 1232 that plans various vehicle activities, plans a vehicle trajectory to reach a destination, and the like. In at least one example, planning component 1232 may include a collision confirmation component 1234 that may identify potential collisions along the vehicle's trajectory. As described herein, the collision confirmation component 1234 uses the boundary contours to determine the distance (and probability of collision) between the vehicle 1202 and the boundary contours associated with objects near objects in the vehicle's trajectory. ) can be determined.

In some examples, some or all aspects of the components described herein may include any models, algorithms, and/or machine learning algorithms. For example, in some illustrations, components within memory 1218 (and memory 1246, described below) may be implemented as a neural network.

As described herein, an exemplary neural network is an algorithm that passes input data through a series of connected layers to generate an output. Each layer in a neural network may also include another neural network, or any number of layers (convolutional or not). As understood in the context of this disclosure, neural networks may utilize machine learning, which may refer to a broad class of algorithms in which outputs are generated based on learned parameters.

Although described in the context of neural networks, any type of machine learning may be used consistent with this disclosure. For example, machine learning algorithms include, but are not limited to, regression algorithms (e.g., ordinary least squares regression (OLSR), linear regression, logistic regression, stepwise regression, multivariate adaptive regression splines (MARS), local estimation scatter figure smoothing (LOESS)), instance-based algorithms (e.g. ridge regression, least absolute shrinkage and selection operator (LASSO), elastic net, least angle regression (LARS)), decision tree algorithms (e.g. classification and regression trees (CART), Iterative Dichotomizer 3 (ID3), Chi-Square Automatic Interaction Detection (CHAID), Decision Stamp, Conditional Decision Tree), Bayesian algorithms (e.g. Naive Bayes, Gaussian Naive Bayes, Multinomial Naive Bayes, Mean 1 Dependence Estimator (AODE), Bayesian Belief Network (BNN), Bayesian Network), Clustering Algorithms (e.g. k-means, k-median, Expectation Maximization (EM), Hierarchical Clustering), Association Rule Learning Algorithms (e.g. Perceptron) , backpropagation, Hopfield network, radial basis function network (RBFN)), deep learning algorithms (e.g., deep Boltzmann machine (DBM), deep belief network (DBN), convolutional neural network (CNN), stacked autoencoder), Dimensionality reduction techniques (e.g., Principal Component Analysis (PCA), Principal Component Regression (PCR), Partial Least Squares Regression (PLSR), Summon mapping, Multidimensional Scaling (MDS), Projection Pursuit, Linear Discriminant Analysis (LDA) , mixture discriminant analysis (MDA), quadratic discriminant analysis (QDA), flexible discriminant analysis (FDA)), ensemble algorithms (e.g. boosting, bootstrap aggregation (bagging), AdaBoost, stacked generalization (blending), gradient boosting) machine (GBM), gradient-boosted regression tree (GBRT), random forest), SVM (support vector machine), supervised learning, unsupervised learning, semi-supervised learning, etc.

Additional examples of architectures include neural networks such as ResNet50, ResNetl01, VGG, DenseNet, PointNet, etc.

In at least one example, sensor system 1206 includes lidar sensors, radar sensors, ultrasound transducers, sonar sensors, position sensors (e.g., GPS, compass, etc.), inertial sensors (e.g., inertial measurement units (IMUs), accelerometers, etc.). magnetometer, gyroscope, etc.), image sensors (e.g. RGB, IR, intensity, depth, etc.), time-of-flight sensors, audio sensors, wheel encoders, environmental sensors (temperature sensors, humidity sensors, light sensors, pressure sensors, etc.) ), and others. Sensor system 1206 may include multiple instances of each of these or other types of sensors. For example, the lidar sensors may include individual lidar sensors located at the corners, front, rear, sides, and/or top of the vehicle 1202. As another example, the image sensors may include various cameras located at various locations outside and/or inside the vehicle 1202. Sensor system 1206 may provide input to vehicle computing device 1204. Additionally or alternatively, sensor system 1206 transmits sensor data via one or more networks 1240 to one or more computers at a specified frequency, such as in near real time, after a predetermined period of time. 1242 .

Vehicle 1202 may also include one or more emitters 1208 for emitting light and/or sound, as described above. Emitters 1208 in this example include internal audio and visual emitters that communicate with the occupants of vehicle 1202. By way of non-limiting example, internal emitters include speakers, lights, signage, display screens, touch screens, tactile emitters (e.g. vibration and/or force feedback), mechanical actuators (e.g. seatbelt tensioners, seat positioners, headrest positioners). etc.). Emitter 1208 in this example also includes an external emitter. By way of example, the external emitters in this example include lights that signal heading or other indicators of vehicle movement (e.g., indicator lights, signs, light arrays, etc.), and pedestrians, or acoustic beam steering technology. One or more audio emitters (e.g., speakers, speaker arrays, horns, etc.) that acoustically communicate with a plurality of other nearby vehicles.

Vehicle 1202 may also include one or more communication connections 1210 that enable communication between vehicle 1202 and one or more other local or remote computing devices. For example, communication connection 1210 can facilitate communication with other local computing devices and/or drive systems 1214 on vehicle 1202. Communication connections 1210 may also enable the vehicle to communicate with other nearby computing devices (eg, other nearby vehicles, traffic lights, etc.). Communications connection 1210 also allows vehicle 1202 to communicate with remotely operated computing devices or other remote services.

Communication connections 1210 may include physical and/or logical interfaces for connecting vehicle computing device 1204 to another computing device or a network, such as network 1240. For example, the communication connection unit 1210 uses a frequency defined by the IEEE 802.11 standard, a short range wireless frequency such as Bluetooth, a Wi - Fi-based communications, or any suitable wired or wireless communication protocol that allows each computing device to interface with other computing devices.

In at least one example, vehicle 1202 may include one or more drive systems 1214. In some examples, vehicle 1202 may have a single drive system 1214. In at least one example, when vehicle 1202 has multiple drive systems 1214, individual drive systems 1214 may be located at opposite ends of vehicle 1202 (eg, front and rear, etc.). In at least one example, drive system 1214 may include one or more sensor systems to detect conditions surrounding drive system 1214 and/or vehicle 1202. By way of example and not limitation, the sensor system may include one or more wheel encoders (e.g., rotary encoders) for sensing rotation of the wheels of the drive system, and for measuring orientation and acceleration of the drive system. inertial sensors (e.g. inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.), cameras or other image sensors, ultrasonic sensors for acoustically detecting objects in the surrounding conditions of the drive system, lidar sensors, May include radar sensors, etc. Some sensors, such as wheel encoders, may be unique to drive system 1214. In some cases, sensor systems in drive system 1214 may overlap or complement corresponding systems in vehicle 1202 (eg, sensor system 1206).

Drive system 1214 is a steering system that includes a high voltage battery, a motor that propels the vehicle, an inverter that converts direct current from the battery to alternating current for use in other vehicle systems, a steering motor, and a steering rack (which can be electrically powered). , brake systems that include hydraulic or electric actuators, suspension systems that include hydraulic and/or pneumatic components, stability control systems for brake force distribution to reduce loss of traction and maintain control, HVAC systems, lighting (e.g. external lighting such as headlights/taillights) and one or more other systems (e.g. cooling systems, safety systems, onboard charging systems, DC/DC converters, high voltage junctions, high voltage cables, charging systems) , other electrical components such as charging ports). Additionally, drive system 1214 may include a drive system controller to control the operation of various vehicle systems that can receive and preprocess data from the sensor system. In some examples, a drive system controller may include one or more processors and memory communicatively coupled to the one or more processors. Memory may store one or more components for performing various functions of drive system 1214. Additionally, drive systems 1214 also include one or more communication connections that enable the respective drive system to communicate with one or more other local or remote computing devices.

In at least one example, direct connection 1212 may use the body of vehicle 1202 to provide a physical interface to couple to one or more drive systems 1214. For example, direct connections 1212 can enable energy, fluid, air, data, etc. to be transferred between drive system 1214 and a vehicle. In some examples, direct connection 1212 can further removably secure drive system 1214 to the body of vehicle 1202.

In some examples, vehicle 1202 may transmit sensor data to one or more computing devices 1242 via network 1240. In some examples, vehicle 1202 may transmit raw sensor data to computing device 1242. In other illustrations, vehicle 1202 may transmit processed sensor data and/or representations of sensor data to computing device 1242. In some examples, vehicle 1202 may transmit sensor data to computing device 1242 at a certain frequency, such as in near real time after a predetermined period of time. In some instances, vehicle 1202 may transmit sensor data (raw or processed) to computing device 1242 as one or more log files.

Computing device 1242 may include a processor 1244 and memory 1246 that stores training components 1248. In some examples, training component 1248 may include training data generated by a simulator.

Processor 1216 of vehicle 1202 and processor 1244 of computing device 1242 may be any suitable processors that can process data and execute instructions to perform the operations described herein. By way of example, processors 1216 and 1244 may include one or more central processing units (CPUs), graphics processing units (GPUs), or other electronic data processors that can process electronic data and store the electronic data in registers or memory. It may include any other device or part of a device to convert. In some examples, integrated circuits (e.g., ASICs, etc.), gate arrays (e.g., FPGAs, etc.), and other hardware devices can also be used as long as they are configured to execute encoded instructions. It may be considered a processor.

Memories 1218 and 1246 are examples of non-transitory computer readable media. Memories 1218 and 1246 may store an operating system and one or more software applications, instructions, programs, and/or data to implement the functionality resulting from the methods and various systems described herein. I can do it. In various implementations, the memory is a suitable memory such as static random access memory (SRAM), synchronous dynamic RAM (SDRAM), non-volatile/flash type memory, or any other type of memory capable of storing information. may be implemented using technology. The architectures, systems, and individual elements described herein may include numerous other logical, programmatic, and physical components, only those illustrated in the accompanying drawings. These are merely examples in connection with the description herein.

In some cases, memories 1218 and 1246 may include at least working memory and storage memory. For example, working memory may be a limited amount of high speed memory (eg, cache memory) used to store data manipulated by processors 1216 and 1244. In some cases, memories 1218 and 1246 may include storage memory, which may be a relatively large capacity, slow memory used for long-term storage of data. In some instances, processors 1216 and 1244 may not directly manipulate data stored in storage memory, as described herein, and the data may be transferred to working memory to perform operations on the data. may need to be loaded into

While FIG. 12 is illustrated as a distributed system, in alternative illustrations, components of vehicle 1202 may be associated with computing device 1242 and/or components of computing device 1242 may be associated with vehicle 1202. It should be noted that the That is, vehicle 1202 may perform one or more of the functions associated with computing device 1242, and vice versa.

Example clause A. A system comprising one or more processors and one or more non-transitory computer-readable media storing instructions executable by the one or more processors, the instructions comprising: , a system is configured to receive a trajectory associated with an autonomous vehicle traversing an environment, receive sensor data from a sensor associated with the autonomous vehicle, and, based on the sensor data, an object represented by the sensor data. determining a boundary contour associated with the boundary contour, and determining a predicted position and orientation of the boundary contour based on the position of the autonomous vehicle along the trajectory, and a predicted position and orientation based on ray casting. A system for performing operations comprising: determining a distance between a location in and a boundary contour; and controlling an autonomous vehicle based on the distance.

B. The system of paragraph A, wherein the bounding contour is a polygon representing a convex hull associated with the object.

C. The system of paragraph B, wherein the operations further include connecting at least two points of sensor data, and the polygon is determined based on connecting the at least two points.

D. The system of any one of paragraphs A-C, wherein the distance is a first distance, and the operations further include determining a second distance based on an axis-aligned bounding box corresponding to the object.

E. The system of any one of paragraphs A-D, wherein the operations further include comparing the distance to a threshold, the threshold being based on a speed of the autonomous vehicle and an object type associated with the object.

F. A method comprising: receiving a trajectory associated with a vehicle; receiving sensor data from a sensor associated with the vehicle; and determining a boundary contour associated with an object represented by the sensor data. , determining a predicted position of the vehicle based on the trajectory; determining a predicted position of a boundary contour associated with the object; and a simulated position of the vehicle and a predicted position of the boundary contour. and performing an action based on the distance between the vehicle and the object.

G. The method of paragraph F, wherein the boundary contour is a polygon representing a convex hull associated with the object as determined by the connection of at least two points of the sensor data.

H. The method of paragraph F or G, wherein the action includes validating the trajectory and/or invalidating the trajectory.

I. A method according to any one of paragraphs F to H, wherein determining the distance between the vehicle and the object is based at least in part on ray casting.

J. The method of any one of paragraphs F-I, further comprising determining a predicted orientation of the boundary contour.

K. The method of paragraph J, wherein the predicted direction is based at least in part on a predicted trajectory associated with the object.

L. The method of any one of paragraphs F-K, wherein the sensor is a lidar sensor and the sensor data includes lidar data.

M. The method of any one of paragraphs F-L, further comprising representing the boundary contour as a representation of a plurality of connected points.

N. The method of any one of paragraphs F-M, wherein the distance is a first distance, and the method further comprises determining a second distance based on the axis-aligned bounding box.

O. The method of any one of paragraphs F-N, further comprising comparing the distance to a threshold, the threshold being based on at least one of a speed of the vehicle or an object type associated with the object.

P. The method of any one of paragraphs F-O, further comprising determining an axis-aligned bounding box associated with the bounding contour based at least in part on the orientation of the vehicle.

Q. 5. The method of any one of paragraphs F-P, wherein the bounding contour associated with the object includes at least five bounding edges.

R. one or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform operations, including: receiving a trajectory associated with a vehicle; receiving sensor data from an associated sensor; determining a boundary contour associated with an object represented by the sensor data; and determining a simulated position of the vehicle based on the trajectory. determining a predicted position of a boundary contour associated with the object; and determining a distance between the vehicle and the object based on the simulated position of the vehicle and the predicted position of the boundary contour. , and performing an action based on a distance between the vehicle and the object.

S. The one or more non-transitory computer-readable media of paragraph R, wherein the boundary contour is a polygon representing a convex hull associated with an object determined by a connection of at least two points of sensor data.

T. The operations further include determining an axis-aligned bounding box associated with the bounding contour and determining a second distance between the vehicle and the object based on the axis-aligned bounding box. One or more non-transitory computer-readable media as described in paragraph R or S.

U. A system comprising one or more processors and one or more non-transitory computer-readable media storing instructions executable by the one or more processors, the instructions comprising: , the system is configured to receive lidar data associated with the object, project the lidar data onto a two-dimensional plane as projected lidar data, and generate a first convex hull based on the projected lidar data. , the first convex hull includes a plurality of bounding edges, and determining a longest bounding edge, the longest bounding edge being a first endpoint and a second endpoint. determining an interior point inside a first convex hull with endpoints of and determining an updated convex hull based on the first segment and the second segment.

V. The updated convex hull represents the bounding contour associated with the object, the action is determining an action based on the bounding contour associated with the object, the action is controlling the vehicle, the static The system of paragraph U, comprising at least one of determining an object, determining a dynamic object, or updating map data.

W. Determining the longest bounding edge includes identifying a plurality of bounding edges associated with the first convex hull, and determining the longest bounding edge of the plurality of bounding edges associated with the first convex hull. The system of paragraph U or V, comprising: determining .

X. Identifying a bounding box associated with an object identified in the lidar data, increasing the size of the bounding box as an expanded bounding box, and capturing additional lidar data within the expanded bounding box The system of any one of paragraphs U-W, further comprising:

Y. A method comprising: receiving sensor data associated with an object; and determining a first convex hull based on the sensor data, the first convex hull including a plurality of bounding edges. , determining a longest bounding edge, the longest bounding edge having a first endpoint and a second endpoint, determining an interior point inside a first convex hull; replacing the border edge with a first segment based on the first endpoint and the interior point and a second segment based on the interior point and the second endpoint; and determining a second convex hull.

Z. The sensor data associated with the object includes a plurality of lidar points, and the method includes the steps of: projecting the plurality of lidar points onto a two-dimensional plane; The method of paragraph Y further comprising the step of repositioning.

A.A. determining a second interior point inside the first convex hull; determining a third segment based on the first endpoint and the second interior point; and determining the second interior point and the second interior point. The method of paragraph Y or Z, further comprising determining a fourth segment based on the second endpoint.

AB. determining a first angle associated with the first segment and the second segment; determining a second angle associated with the third segment and the fourth segment; and determining the largest of the second angles, the step of replacing the longest bounding edge being responsive to determining that the first angle is greater than the second angle. in response to determining that the second angle is greater than the first angle; The method of paragraph AA, further comprising at least one of replacing a segment of.

A.C. The method according to any one of paragraphs Y-AB, further comprising determining a three-dimensional object based on the second convex hull.

A.D. The second convex hull represents a boundary contour associated with the object, and the method determines a probability associated with a potential collision between the vehicle and the object based on the boundary contour associated with the object. The method of any one of paragraphs Y-AC, further comprising steps.

A.E. the second convex hull represents a bounding contour associated with the object, and the method includes determining an action based on the bounding contour associated with the object, the action comprising: controlling a vehicle; The method of any one of paragraphs Y-AD, comprising at least one of determining a static object, determining a dynamic object, or updating map data.

AF. identifying a bounding box associated with an object identified in the sensor data; expanding the size of the bounding box as an expanded bounding box; and all sensor data points within the expanded bounding box. The method of any one of paragraphs Y-AE, further comprising: capturing.

A.G. determining a next longest border edge; determining that the length of the next longest border edge is less than a threshold length; and based at least in part on the length being less than a threshold length. The method according to any one of paragraphs Y to AF, further comprising the step of outputting the second convex hull.

A.H. The method of paragraph AG, further comprising determining a threshold length based at least in part on the longest border edge.

A.I. the number of edges of the convex hull meets or exceeds a first threshold; and the length of the longest remaining bounding edge is less than a second threshold; or The method of any one of paragraphs Y-AH, further comprising outputting a second convex hull based at least in part on the number of interior points being less than a third threshold.

A.J. The method of any one of paragraphs Y-AI, further comprising determining a plurality of lidar points based on instance segmentation.

A.K. The method of any one of paragraphs Y-AJ, wherein the first convex hull includes at least five bounding edges.

AL. one or more non-transitory computer-readable media storing instructions that, when executed, cause one or more processors to perform operations, including: receiving sensor data associated with an object; determining a first convex hull based on data, the first convex hull including a plurality of boundary edges, and determining a longest boundary edge, the longest boundary edge being a first convex hull; determining an interior point within a first convex hull, having a first endpoint and a second endpoint, a first segment based on the first endpoint and the interior point; replacing one or more non-convex hulls with a second segment based on the interior points and the second segment; and determining a second convex hull based on the first segment and the second segment. Transient computer-readable medium.

A.M. The sensor data associated with the object includes a plurality of lidar points, and the operations include projecting the plurality of lidar points onto a two-dimensional plane, and locating a point of the plurality of lidar points to the nearest position on a predetermined grid. one or more non-transitory computer-readable media as described in paragraph AL, further comprising: relocating to one or more non-transitory computer-readable media as described in paragraph AL.

A.N. The operation consists of identifying the bounding box associated with the identified object in the sensor data, expanding the size of the bounding box as an expanded bounding box, and adding all sensors within the expanded bounding box. one or more non-transitory computer-readable media as described in paragraph AL or AM, further comprising capturing data points.

While the example provisions described above are described with respect to one particular implementation, in the context of this specification, the subject matter of the example provisions is applicable to methods, devices, systems, computer-readable media, and/or It should be understood that it may also be implemented via another implementation. Furthermore, any of examples A through AN may be implemented alone or in combination with any of one or more other examples A through AN.

Conclusion While one or more examples of the technology described herein have been described, various modifications, additions, substitutions, and equivalents thereof are within the scope of the technology described herein. included. In the description of the examples, reference is made to the accompanying drawings, which form a part of the specification, and which illustrate by way of illustration certain examples of the claimed subject matter. It is to be understood that other examples may be used and modifications or substitutions may be made, including structural changes. Such examples, modifications, or variations do not necessarily depart from the intended scope of the claimed subject matter. While the steps herein may be presented in a particular order, in some cases certain inputs may be provided at different times or in a different order without changing the functionality of the systems and methods described. You can change the order so that The disclosed procedures can also be performed in different orders. Furthermore, the various calculations described herein need not be performed in the order disclosed, and other examples using alternative orders of calculations may be readily implemented. In addition to reordering, calculations can also be decomposed into sub-calculations with the same result.

Claims (15)

A method,
receiving a trajectory associated with the vehicle;
receiving sensor data from a sensor associated with the vehicle;
determining a boundary contour associated with an object represented in the sensor data;
determining a predicted position of the vehicle based on the trajectory;
determining a predicted location of the boundary contour associated with the object;
determining a distance between the vehicle and the object based on the simulated position of the vehicle and the predicted position of the boundary contour;
performing an action based on the distance between the vehicle and the object;
How to prepare. 2. The method of claim 1, wherein the boundary contour is a polygon representing a convex hull associated with the object determined by a connection of at least two points of the sensor data. 3. The method of claim 1 or 2, wherein the action comprises validating the trajectory or invalidating the trajectory. 4. A method according to any preceding claim, wherein determining the distance between the vehicle and the object is based at least in part on ray casting. 5. A method according to any preceding claim, further comprising the step of determining a predicted orientation of the boundary contour. 6. The method of claim 5, wherein the predicted direction is based at least in part on a predicted trajectory associated with the object. 7. A method according to any preceding claim, wherein the sensor is a lidar sensor and the sensor data comprises lidar data. 8. A method according to any preceding claim, further comprising representing the boundary contour as a representation of a plurality of connected points. 9. A method according to any preceding claim, wherein the distance is a first distance, and the method further comprises determining a second distance based on an axis-aligned bounding box. 10. Any one of claims 1 to 9, further comprising the step of comparing the distance to a threshold value, the threshold value being based on at least one of a speed of the vehicle or an object type associated with the object. The method described in. 11. A method according to any preceding claim, further comprising determining an axis-aligned bounding box associated with the bounding contour based at least in part on the direction of the vehicle. 12. A method according to any preceding claim, wherein the bounding contour associated with the object comprises at least five bounding edges. One or more non-transitory computers according to any one of claims 1 to 12, storing instructions that, when executed by one or more processors, cause one or more computing devices to execute. readable medium. A system,
one or more processors; and one or more non-transitory computer-readable media storing instructions that cause the one or more processors to perform operations.
and the operation is
receiving a trajectory associated with the vehicle;
receiving sensor data from a sensor associated with the vehicle;
determining a boundary contour associated with an object represented in the sensor data;
determining a simulated position of the vehicle based on the trajectory;
determining a predicted position of the boundary contour associated with the object;
determining a distance between the vehicle and the object based on the simulated position of the vehicle and the predicted position of the boundary contour; and determining the distance between the vehicle and the object. performing an action based on the distance; and
system containing. The operations include: determining an axis-aligned bounding box associated with the bounding contour; and determining a second distance between the vehicle and the object based on the axis-aligned bounding box. 15. The system of claim 14, further comprising:

Images