JP2023508276A - map containing covariances at multiresolution voxels - Google Patents

Abstract

Techniques for representing a scene or map based on statistics of captured environmental data are described herein. In some cases, data (eg, covariance data, mean data, etc.) may be stored as a multi-resolution voxel space that includes multiple semantic layers. In some cases, individual semantic layers may contain multiple voxel grids with different resolutions. Multiple resolution voxel spaces may be merged to produce a combined scene based on the covariance of detected voxels at one or more resolutions.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS This application claims priority to U.S. Application No. 16/722,598, filed Dec. 20, 2019 and entitled "MAPS COMPRISING COVARIANCES IN MULTI-RESOLUTION VOXELS." , which is incorporated herein by reference in its entirety.

Data can be captured in the environment and represented as a map of the environment. As is often the case, the map can be used by a vehicle navigating the environment and the map can be used for a variety of purposes. In some cases the environment can be represented as a two-dimensional map, while in other cases the environment can be represented as a three-dimensional map. Furthermore, surfaces in the environment are often represented using multiple polygons or triangles.

The detailed description is described with reference to the accompanying drawings. In the drawings, the left-most digit(s) of a reference number identifies the drawing in which the reference number first appears. The use of the same reference numbers in different drawings indicates similar or identical components or features.

FIG. 4 is an example diagram illustrating an example architecture of the multi-resolution voxel space described herein; FIG. 4 is an exemplary pictorial diagram illustrating an exemplary resolution of the semantic layer of the multi-resolution voxel space described herein; 3 is an exemplary diagram illustrating the first resolution for the multi-resolution voxel space of FIG. 2 described herein; FIG. 3 is an exemplary diagram illustrating a second resolution for the multi-resolution voxel space of FIG. 2 described herein; FIG. 3 is an exemplary diagram illustrating a third resolution for the multi-resolution voxel space of FIG. 2 described herein; FIG. FIG. 2 is an example process flow diagram illustrating an example data flow of a system configured to align data representing a physical environment as described herein with map data; FIG. 4 is an exemplary flow diagram illustrating exemplary processing associated with generating a multi-resolution voxel space as described herein; FIG. 4 is another flow diagram illustrating an exemplary process for aligning a target multi-resolution voxel space with a reference multi-resolution voxel space as described herein; 1 is a block diagram of an exemplary system for implementing the multi-resolution voxel spatial alignment system described herein; FIG. 5 is a pictorial illustration of the example multi-resolution voxel space of FIGS. 2-4 compared to the point cloud representation of the captured data described herein; FIG.

The techniques described herein map data including a multi-resolution voxel space containing voxels that store spatial means, covariances, and weights of point distributions of data representing a physical environment. is directed to determining and/or using the Map data may include multiple voxel grids or voxel layers that represent the physical environment at different resolutions or physical distances. As an example, each voxel layer may represent the physical environment at twice the resolution of the advancing layer. That is, voxels in a first layer may represent a first volume (e.g., 10 cm x 10 cm x 10 cm), while voxels in a second layer may represent a second volume (e.g., 20 cm x 20 cm x 20 cm). may represent Data associated with a voxel in a multi-resolution voxel space may be represented as multiple covariance ellipsoids. An ellipsoidal representation of covariance may be generated based on the calculated mean and covariance values of the data points associated with individual voxels. In some cases, the voxel data can be associated with semantic information, such as, for example, classification and/or segmentation information, and the data associated with a particular classification is a unique multi-resolution It is possible to relate to voxel space. In the example just described, each voxel covariance semantic layer represents data points associated with a particular semantic class (e.g. trees, vehicles, buildings, etc.) as covariance ellipsoids. may contain.

In some cases, map data represented by a multi-resolution voxel space may be generated from data points representing the physical environment, such as the output of a lidar (light detection and ranging) system, for example. As an example, the system may receive multiple lidar points or lidar data represented as a point cloud. The system bases the lidar points at least in part on a local frame of reference of the vehicle (e.g., the system captures the lidar points) in a voxel grid having a first resolution (e.g., voxels in a multi-resolution voxel space). grid with the maximum number of voxels). The system may determine statistical data associated with each voxel, such as, for example, mean values, centroids, covariances, and the like of the collected data. The system may then merge or otherwise combine the voxels (or data associated with the voxels) of the lower resolution grid when generating the higher level voxel grid. For example, a voxel (or data associated with a voxel) within a three-dimensional neighborhood (e.g., the number of voxels in the x, y, and z directions associated with the physical space of the higher level voxel) in a lower resolution grid ) may be merged when forming the next higher level voxel grid. In one particular example, voxels within a neighborhood are merged by taking a weighted sum of individual Gaussian distributions for each voxel of the lower resolution grid. In some cases, merging voxels at a lower resolution to form a higher resolution grid is not only computationally cheaper, but also more efficient to help localize lidar data by reference frame. Allows for lower resolution grids.

In some implementations, the system utilizes multi-resolution voxel space to align multiple scans of the physical environment to generate maps and scenes of the physical environment, as well as localization of vehicles within the maps or scenes. can help As an example, when a multi-resolution voxel space (e.g., a target multi-resolution voxel space) is generated for a particular scan or dataset representing a physical environment, the system converts the generated multi-resolution voxel space into a representation of the scene. It may be aligned with a multi-resolution voxel space (eg, a reference multi-resolution voxel space). In some cases, alignment may be performed by substantially simultaneously finding correspondences between voxels at each resolution of the reference multi-resolution voxel space and the target multi-resolution voxel space. For example, the system may, for each voxel of a specific resolution in the target multiresolution voxel space, within a threshold distance containing the average target point at the corresponding specific resolution of the reference multiresolution voxel space to the occupied voxel, or It may search among voxels within a threshold number of voxels (eg, a neighborhood of voxels). In examples involving semantic layers, the system includes, for each voxel of each semantic layer-specific resolution in the target multi-resolution voxel space, the average target point at the corresponding semantic layer-specific resolution in the reference multi-resolution voxel space. A neighborhood of voxels may be searched.

Of the voxels identified within the neighborhood of the reference multiresolution voxel space, the system may select voxels with centroids close to those of the target multiresolution voxel space. The system may then average the distribution of the selected voxels in the reference multiresolution voxel space with the voxels of the target covariance stack. The system may then perform principal component analysis on the combined covariance matrix and select the eigenvalue (eg, the smallest eigenvalue) as the matched normal vector for the two voxels. Next, the system determines a residual (or error, etc.) for each of the matched voxels, which, in at least some examples, may be based at least in part on the matched normal vectors above, followed by , we may perform an optimization on all the above residuals. The optimization may minimize the distance between pairs of voxel centroids above. In this way, a merged voxel representing two voxels may be placed within a grid of positions that accurately represent the covariances (eg, of the associated data) and weights of both of the original voxels. Of course, there are applications that do not require the above voxel merging. As a non-limiting example, relative transformations between two voxel spaces are commonly used for localization without merging voxels.

Even though each layer may be merged substantially simultaneously during alignment, coarser resolutions (e.g., resolutions corresponding to larger voxels) may result in matching before finer resolutions. Thus, matching at the coarser resolution brings the two multi-resolution voxel spaces into closer alignment so that the finer resolution can begin matching and complete the alignment process. can help In some cases, by merging the captured sensor data into a multi-resolution voxel space representing the environment, the vehicle can be more accurate and/or It may be possible to initialize or localize locations in the environment more quickly. Additionally, storing voxels in a multi-resolution voxel space may improve processing speed and throughput by storing data in a more easily indexable/searchable manner. For example, if coarser resolution is acceptable for a practical task, coarser layers may be loaded into memory to reduce the amount of data accessed and processed for the desired operation.

In some cases, a multi-resolution voxel space may represent the environment more accurately than conventional systems because each layer of space provides a different resolution of detail about the environment. Therefore, in some situations, having access to a more detailed representation of the physical environment may improve the overall safety of an autonomous vehicle.

FIG. 1 is an exemplary diagram illustrating an exemplary architecture 100 of the multi-resolution voxel space 102 described herein. In the present example, multi-resolution voxel space 102 is formed from multiple semantic layers illustrated as semantic layers 104 , 106 and 108 . Each of the semantic layers 104-108 may represent data for a specific semantic class or type. As an example, the first semantic layer 104 may contain data representing trees, while the second semantic layer 106 may contain data representing buildings. Thus, a multi-resolution voxel space 102 that includes multiple semantic layers 104-108 can represent the data from each semantic layer 104-108 as a full picture or map of the physical environment illustrated below with respect to FIGS. be. In some cases, some applications may require identification or awareness of only specific semantic classes, while other applications may require detailed understanding of the entire physical environment. By segmenting the multi-resolution voxel space 102 into semantic layers 104-108, each application can speed up processing in some applications by processing only the appropriate class or type of data corresponding to the environment. may cause

Additionally, each of the semantic layers 104-108 may include one or more voxel grids, illustrated as voxel covariance grids 110, 112, and 114. FIG. Each of the voxel covariance grids 110-114 represents the same semantic data of the corresponding semantic layers 104-108, but at different resolutions. As an example, a first voxel covariance grid of plurality of grids 110 may have voxels having a size of approximately 25 centimeters, while a second voxel covariance grid of plurality of grids 110 may have a size of approximately 16 centimeters. has a voxel with a size of . Thus, each voxel covariance grid in each of the plurality of grids 110-114 may have a different resolution or coarseness to aid in alignment and processing of data representing the physical environment. For example, some applications may require only a coarse, general understanding of the physical environment, while other applications may require a detailed understanding of the physical environment, and each application may require a voxel grid. , processing at a desired or suitable resolution may improve processing speed in some applications.

In some examples, data associated with voxels of voxel covariance grids 110-114 in multi-resolution voxel space 102, such as those illustrated below with respect to FIGS. It may be represented by voxels that store weights representing point distributions. In some cases, the voxels of grids 110-114 may be visually presented as covariance ellipsoids. The covariance ellipsoid may be based at least in part on the shape parameter of the eigenvalue ratios of each voxel.

In the illustrated example, three semantic layers 104-108 and three sets of voxel covariance grids 110-114 are shown. However, it should be understood that the multi-resolution voxel space 102 may contain any number of semantic layers, and that each semantic layer may contain any number of voxel covariance grids. In some implementations, the number of voxel covariance grids may be the same for each semantic layer, while in other implementations the number of voxel covariance grids in each semantic layer may differ. As an example, some semantic classes, e.g. foliage (or pedestrians), may require more additional fine resolution voxel covariance grids than other semantic classes, e.g. buildings, Therefore, the semantic layer representing the pedestrian class may contain more voxel covariance grids than the semantic layer representing the building class.

FIG. 2 is an exemplary illustrative pictorial diagram 200 illustrating exemplary resolutions 202, 204, and 206 of the semantic layers of the multi-resolution voxel space 208 described herein. In the current example, resolution is shown in two dimensions for illustrative purposes only, and it should be understood that any number of dimensions may be used (eg, three dimensions representing a three-dimensional physical space in the real world, etc.). . In the current example, voxels within first neighborhood 210 at first resolution 202 are combined to form voxels 212 at second resolution 204 . Similarly, voxels within first neighborhood 214 of second resolution 204 are combined to form voxels 216 of third resolution 206 . Third resolution voxels 216, discussed below, are formed based on a weighted sum of individual Gaussian distributions from each of voxels 218 and 220 in neighborhood 214 to yield a single higher resolution voxel. may generate. Determining the weighted sum of the individual Gaussian distributions is computationally cheap in terms of processing resources and time, so building the multi-resolution voxel space 208 is faster and less expensive than conventional systems. It should be understood that it may be done by processing resources.

In the current example, a two-dimensional 2x2 neighborhood is shown. However, the multi-resolution voxel space can be formed as a three-dimensional voxel grid representing physical space, and the neighborhoods can be of various uniform sizes, e.g. 2x2x2, 3x3x3, 5x5x5, or e.g. It should be understood that they may have non-uniform sizes, such as 4x3x4, 5x1x3, and so on. In one specific example, the neighborhood may have a 2x2x2 voxel size, with each higher resolution layer having half the number of voxels of the proceeding lower layer.

3-5 are exemplary diagrams illustrating multiple resolutions 202, 204, and 206 for the multi-resolution voxel space 208 of FIG. 2 described herein. In the current example, each of the semantic layers of multi-resolution voxel space 208 is shown to produce a picture or map of the physical environment. As an example, multi-resolution voxel space 208 may be formed by merging or aligning multiple lidar scans of the physical environment captured by the autonomous vehicle. In the current example, multi-resolution voxel space 208 may be zoomed in or out to show the physical environment at different resolutions 202, 204, and 206. FIG. As an example, resolution 202 shows voxels at a first or finest resolution. Thus, resolution 202 of multi-resolution voxel space 208 contains more voxels than each of resolutions 204 or 206, and also contains the most detailed representation of the physical environment. Each progressive resolution 204 or 206 represents the physical environment with subsequent coarser resolution voxels. As an example, each voxel in the multi-resolution voxel space of resolution 202 may represent an area of 25 cm, while each voxel in the multi-resolution voxel space of resolution 206 may represent an area of 16 m.

In some cases, voxels associated with a particular semantic layer are colored or textured to visually distinguish voxels associated with two semantic layers from each other when viewing the multi-resolution voxel space 208. Sometimes. Further, the data associated with each voxel is represented as an ellipsoid of covariance having a shape based at least in part on the voxel's eigenvalue ratios, shape parameters, and spatial statistics, thus the data illustrated in FIGS. However, it should be noted that they have shapes that substantially represent the real-world shapes of the corresponding objects.

In some examples, each higher resolution 300-500 of the multi-resolution voxel space 102 may have half as many voxels as the preceding lower-level resolution 200-400. As an example, if the resolution 300 has voxels approximately 4 meters in size, the resolution 400 voxels may be approximately 8 meters in size (eg, twice the size of the resolution 300 voxels). However, in other examples, the size and/or number of voxels at each resolution 200-500 may have other mathematical and/or arbitrary relationships.

In the current example, different semantic classes are shown based on ellipse patterns or color differences. By way of example, ellipsoid 302 may correspond to foliage, ellipsoid 304 may correspond to walls, structures, or buildings, and ellipsoid 306 may correspond to ground cover, such as grass. I have something to do.

FIGS. 6-8 are flow diagrams illustrating exemplary processing associated with the multi-resolution voxel space of FIGS. 1-5. Processes are illustrated as collections of blocks in logical flow diagrams that represent sequences of some or all actions that may be implemented in hardware, software, or a combination thereof. In the context of software, blocks represent computer-executable instructions stored on one or more computer-readable media that, when executed by one or more processors, perform the recited acts. Generally, computer-executable instructions include routines, programs, objects, components, data structures, etc. that perform particular functions or implement particular abstract data types.

The order of operations described should not be interpreted as a limitation. Any number of the illustrated blocks may be processed or alternately combined in any order and/or in parallel, and not all blocks need be executed. For purposes of discussion, the processes herein are described with reference to the frameworks, architectures, and environments described in the examples herein, but the processes can be implemented in a wide variety of other It may be implemented in a framework, architecture, or environment.

FIG. 6 is an example process flow diagram 600 illustrating an example data flow for a system configured to align data representing a physical environment with a scene as described herein. In the illustrated example, the system may be configured such that the scene, as well as the data representing the environment, are stored as a multi-resolution voxel space. The multi-resolution voxel space described above may have multiple semantic layers, each semantic layer containing multiple voxel grids representing voxels as ellipsoids of covariance at different resolutions.

In one specific example, a sensor system 602, such as a lidar, radar, sonar, infrared, camera, or other image capture device, may capture data representing the physical environment surrounding the system. In some cases, the captured data can be multiple data points 604, such as a point cloud generated from the output of a lidar scan. In the example just described, data points 604 may be received by multi-resolution voxel space generation component 606 .

Multi-resolution voxel space generation component 606 may be configured to generate target multi-resolution voxel space 608 from data points 604 . In some cases, multi-resolution voxel space generation component 606 may process data points via classification and/or segmentation techniques. By way of example, multi-resolution voxel space generation component 606 assigns types or classes to data points using one or more neural networks (e.g., deep neural networks, convolutional neural networks, etc.), regression techniques, etc. to generate data Points 604 may be identified and classified by semantic labels. In some cases, semantic labels may include classes or entity types such as, for example, vehicle, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, unknown. In additional and/or alternative examples, semantic labels may include one or more characteristics associated with data points 604 . For example, properties include, but are not limited to, x position (global and/or local position), y position (global and/or local position), z position (global and/or local position), orientation (e.g., roll, pitch, yaw), entity type (eg, classification), entity velocity, entity acceleration, rate of change of velocity and/or acceleration, entity range (magnitude), and the like.

In some examples, generating the target multi-resolution voxel space 608 involves associating data associated with static objects (e.g., buildings, trees, foliage, etc.) to the target multi-resolution voxel space 608, whereas dynamic It may include filtering data associated with objects (eg, representing pedestrians, vehicles, etc.).

In alternative implementations, data points 604 may be output with semantic labels attached by a perception pipeline or component. By way of example, data points 604 may be received as part of a stray object state representation output by a perception component, the details of which are incorporated herein by reference in their entirety, US Application Ser. 549,694.

In the current example, multiresolution voxel space generation component 606 converts semantically labeled data points 604 into semantic vectors of target multiresolution voxel space 608 with corresponding semantic labels (eg, trees, buildings, pedestrians, etc.). may be assigned to layers. As an example, the multi-resolution voxel space generation component 606 projects the data points 604 into a common frame of reference, and then converts the data points 604 in the common frame of reference into an appropriate point cloud associated with the corresponding semantic class. may be multiplexed into For each point cloud, a multi-resolution voxel space generation component 606 may then assign each data point 604 to a voxel of the finest resolution voxel grid (eg, base voxel grid) of each semantic layer. In some particular cases, a multi-resolution voxel space may be a single layer that stores multiple statistics, including semantic classes for each of the voxels.

Once each of the data points 604 for the corresponding group has been assigned to a voxel, the multi-resolution voxel space generation component 606 computes spatial statistics (e.g., spatial mean, covariance, and the weight or number of data points 604 assigned to the voxel). In one specific example, spatial statistics for specific voxels may be calculated using Welford's Online Algorithm.

Once the base or finest resolution voxel grid of the semantic layer is complete, the multi-resolution voxel space generation component 606 may iteratively or recursively generate each of the next higher resolution voxel grids of the semantic layer. . As an example, the multi-resolution voxel space generation component 606 utilizes a preceding lower resolution grid (gazing at the base or finest resolution grid) and merges data associated with voxels within a 2x2x2 neighborhood. , may form the next higher level voxel grid. In one particular example, voxels within a neighborhood of the lower resolution voxel grid are merged by taking a weighted sum of the individual Gaussian distributions of each voxel within the neighborhood. Thus, the voxel grids in the semantic layer of the multiresolution voxel space are described in more detail above with respect to FIGS. 1-5, with each higher resolution grid containing fewer voxels than the preceding lower resolution grid. A multi-resolution pyramid may be formed. In one particular example, each preceding lower resolution grid in the semantic layer may have four times as many voxels as the next higher resolution grid.

Once the target multi-resolution voxel space 608 is generated from the data points 604, the target multi-resolution voxel space 608 is aligned with the reference multi-resolution voxel space 610 (eg, the multi-resolution voxel space representing the scene). By way of example, in the illustrated example, the multi-resolution voxel space alignment component 612 may align the newly generated target multi-resolution voxel space 608 with the reference multi-resolution voxel space 610 or the target multi-resolution voxel space 608 and the reference multi-resolution voxel space 610 . To align the target multiresolution voxel space 608 with the reference multiresolution voxel space 610, the multiresolution voxel space alignment component 612 substantially simultaneously for each semantic layer and each resolution of the target multiresolution voxel space 608: We may take each voxel and determine the average target point at the corresponding resolution and semantic layers of the reference multi-resolution voxel space 610 . A multi-resolution voxel space alignment component 612 may then determine a 2x2x2 neighborhood of the corresponding resolution and semantic layer voxel grids in the reference multi-resolution voxel space 610 and identify whether the neighboring voxels are occupied. Next, a multiresolution voxel space alignment component 612 selects the voxel with the closest centroid to the voxel from the target multiresolution voxel space 608 and calculates the distribution of the selected voxels and the distribution of voxels from the target voxel. to average Next, a multi-resolution voxel space alignment component 612 performs a principal component analysis on the combined covariance matrix of the selected voxels and voxels from the target, and minimizes The target multiresolution voxel space 608 may be more closely aligned with the reference multiresolution voxel space 610 by selecting the eigenvalues of . In some cases, optimization is performed on the matched voxels to improve the overall alignment between the reference multiresolution voxel space and the target multiresolution voxel space and/or without limitation Relative transformations (eg, used for localization) may be determined that involve non-linear optimization (eg, non-linear least-squares optimization). As an example, a gradient descent technique, such as the Gauss-Newton technique discussed below, may be utilized.

In the Gauss-Newton technique, the matching, matched residual, between the first voxel i in the target multiresolution voxel space 608 and the second voxel j in the reference multiresolution voxel space 610 is computed as Sometimes.

however,

is the matched normal vector, μ _i , is the mean of voxel i, and λ ₀ is the smallest eigenvalue of the matching covariance matrix. As alluded to above, matching normal vectors are computed from the smallest eigenvectors of the weighted sums of the corresponding voxel covariance matrices. Each residual weight, z _ij , is reweighted according to the M-estimator framework (eg, using the Cauchy loss function). Then the Jacobian of the matching error ij with respect to the transformation between the reference grid and the target grid

is given by
J _ij =[Rn _ij ×μ _j ×Rn _ij ] ^T
Next, the multi-resolution voxel space alignment component 612 may compute the total gradient and approximate Hessian for each matching ij as follows.

The Gauss-Newton optimization is computed as follows.
HδT=-g
Further, the multi-resolution voxel space alignment component 612 may compute the delta transform by modeling it as an element of SO(3)×R ³ , and the updated alignment transform is given by:
xn ⁺¹ =[exp(δR) ^Rnδp + ^pn ] ^T
where exp() is the SO(3) exponential map. It will be appreciated that the transforms given above may be applied to the entire multi-resolution voxel space in further iterations of the optimization, and the final iteration may involve transformations between the two voxel spaces. should.

The alignment process continues until the two multiresolution voxel spaces 608 and 610 are aligned within a tolerance or threshold, or until a predetermined number of iterations (eg, voxel merges) are completed. After each adjustment of space 608, iterations may continue. Thus, during alignment, coarser resolutions (e.g., resolutions corresponding to larger voxels) begin to match finer resolutions beyond a tolerance or threshold to complete the alignment process. As can be done, it may result in earlier matching that brings the two multi-resolution voxel spaces 608 and 610 into closer alignment. However, in some implementations, operations are performed across all layers and/or semantic classes substantially simultaneously with a single data transformation determined to align some or all of the various voxel spaces. Sometimes.

In one specific example, the multi-resolution voxel space alignment component 612 utilizes only the highest or coarsest resolution of each semantic layer in the first iteration to initialize alignment prior to additional iterations. There is In some cases, each additional iteration may introduce another finer resolution to the alignment process. The perfectly aligned multiresolution voxel space 614 may then be output by a multiresolution voxel space alignment component 612 and used as the next reference multiresolution voxel space 610 .

FIG. 7 is an example flow diagram illustrating an example process 700 associated with constructing a multi-resolution voxel space as described herein. A multi-resolution voxel space, as discussed above, may include multiple voxel grids or voxel layers that represent the physical environment at different resolutions or physical distances. As an example, each voxel layer may represent the physical environment at twice the resolution of the advancing layer (eg, 1 foot, 2 feet, 4 feet, etc.). In some cases, the multi-resolution voxel space may be separated into multiple semantic layers, each semantic layer containing multiple voxel grids of different resolutions.

At 702, a multi-resolution voxel space generation component may receive data representing a physical environment. For example, a multi-resolution voxel space may be generated from data points representing the physical environment, eg, the output of a lidar system. In other examples, the data may include the output of radar, sonar, infrared, cameras, or other image/data capture devices. In some examples, the multi-resolution voxel space generation component may assign a semantic class to each data point. By way of example, in one particular example, the assignment of semantic classes to data points is set forth in US Application Serial No. 15/820,245, which is incorporated herein by reference in its entirety.

At 704, the multi-resolution voxel space generation component generates a semantic point cloud from the data representing the physical environment. For example, a multi-resolution voxel space generation component may project data points from data representing the physical environment onto a common frame.

By way of example, a multi-resolution voxel space generation component or another component may apply classification and/or segmentation techniques to data points to assign mantic classes. In some examples, one or more neural networks (e.g., deep neural networks, convolutional neural networks, etc.), regression techniques, etc. may be used to identify and classify data points by semantic class. In some cases, semantic classes may include classes or entity types such as, for example, vehicle, pedestrian, cyclist, animal, building, tree, road surface, curb, sidewalk, unknown.

At 706, the multi-resolution voxel space generation component may generate a voxel covariance grid for each semantic class for the first resolution of the multi-resolution voxel space. In some examples, the multiresolution voxel space generation component can assign data points to corresponding voxels of a matching semantic layer of the multiresolution voxel space to generate each of the first resolution grids. be. Once the data points are assigned to voxels in the semantic layer, the multi-resolution voxel space generation component may determine voxel space statistics, such as the mean and covariance for each voxel. In some cases, the multi-resolution voxel space generation component may start with the finest resolution layer and generate each subsequent coarser layer when forming the multi-resolution voxel space.

At 708, the multi-resolution voxel space generation component determines whether there are additional resolutions to generate. For example, the multi-resolution voxel space generation component may determine whether the resolution is greater than the resolution threshold and/or whether the number of layers is greater than the layer threshold. If there are additional resolutions, process 700 proceeds to 710 . However, if there are no additional resolutions to generate, process 700 proceeds to 712 .

At 710, the multi-resolution voxel space generation component may generate a voxel covariance grid for each semantic class for the next higher resolution. Each higher resolution grid may be formed based at least in part on a lower resolution grid by merging voxels of the lower grid. As an example, the multi-resolution voxel space generation component takes a neighborhood of voxels (e.g., 2x2x2 groups, etc.) from a lower resolution grid in the semantic layer and generates an individual distribution (e.g., Gaussian distribution) of weights from each of the voxels in the neighborhood. A weighted sum may be computed to produce a single higher resolution voxel. Thus, each higher resolution has fewer voxels than the lower resolution grid, and the multi-resolution voxel space may form a multi-resolution voxel pyramid.

At 712, the multi-resolution voxel space generation component may smooth the resulting multi-resolution voxel space. For example, the multiresolution voxel space generation component may convolve the voxels of the multiresolution voxel space with a Gaussian kernel to reduce noise in the normal estimates of the voxels. In addition, when lidar is used to collect data representing the physical environment, the multi-resolution voxel space generation component finds that these data points have erroneously determined normals and insufficient statistical information. Therefore, voxels may be removed when they correspond to observations below the threshold (eg, when a single lidar beam is observed).

At 714, the multi-resolution voxel space generation component may reduce voxels with weights greater than the maximum weight to the maximum weight and remove voxels with weights less than the minimum weight. In some cases, by applying a maximum and minimum weight range to the voxels, the multi-resolution voxel space maintains a more uniform sample density, and voxels close to the system (e.g., autonomous vehicle) are shown below with respect to FIG. It may prevent the described alignment process from causing disruption. In another example, the multi-resolution voxel space is stored as a hash of each voxel's location in the three-dimensional space, and then the hash is indexed to provide quick memory access (e.g., voxel hashing ) to include a lookup table. In this way, only the desired portion of the multi-resolution voxel space is loaded into memory, and accessing the multi-resolution voxel space may be done using fewer processing resources.

FIG. 8 is another flow diagram illustrating an exemplary process 800 for aligning a target multi-resolution voxel space with a reference multi-resolution voxel space as described herein. By way of example, the multiresolution voxel space alignment component described above utilizes a multiresolution voxel space structure to align multiple scans of a physical environment, e.g., a target multiresolution voxel space and a reference multiresolution voxel space. I have something to do. By way of example, when a multi-resolution voxel space (e.g., a target multi-resolution voxel space) is generated for a particular scan or dataset representing a physical environment, the multi-resolution voxel space alignment component transforms the multi-resolution voxel space into It may be aligned with a multi-resolution voxel space representing the scene (eg, a reference multi-resolution voxel space).

At 802, a multi-resolution voxel space alignment component may receive a target multi-resolution voxel space aligned with a reference multi-resolution voxel space representing a scene. In some cases, a reference multi-resolution voxel space may be maintained by the system and updated with each new scan of the environment to predetermine object detection and tracking.

At 804, a multi-resolution voxel space alignment component may determine a voxel correspondence between the target multi-resolution voxel space and the reference multi-resolution voxel space. In some examples, the correspondence may be per semantic layer and per resolution. Moreover, correspondences may be determined substantially simultaneously for each resolution of each semantic layer. As an example, at 804, for each voxel of a unique resolution in the target multiresolution voxel space, the multiresolution voxel space alignment component aligns 2x2x2 of the voxels containing the average target point at the corresponding unique resolution of the reference multiresolution voxel space. It may search the neighborhood. A multi-resolution voxel space alignment component may then select a voxel from the 2x2x2 neighborhood with the closest centroid to the voxel in the target multi-resolution voxel space.

At 806, the multi-resolution voxel spatial alignment component may reweight the corresponding voxels. As an example, a multi-resolution voxel spatial alignment component may compute a weighted average of data contained in two corresponding voxels (eg, a target voxel and a selected voxel). As an example, a combined covariance may be calculated. Once the aggregated covariance is determined, the multiresolution voxel space alignment component performs a principal component analysis (such as eigenvalue decomposition) on the combined covariance matrix of the two corresponding voxels and finds the smallest eigenvalue by the matched method May be selected as a line vector. A residual (or error) for each voxel may be computed proportional to the difference between the matched normal vector and/or the mean (or centroid) of the corresponding voxel, over the transformation between the two frames. Optimization may be done to minimize the residual. Thus, during alignment, coarser resolutions (eg, resolutions corresponding to larger voxels) may result in matching before finer resolutions. Thus, matching at the coarser resolution divides the two multi-resolution voxel spaces so that the finer resolution can begin matching as described above with respect to FIG. 6 and complete the alignment process. , leading to a closer alignment.

At 808, the multi-resolution voxel spatial alignment component may determine whether the number of iterations has been completed. For example, the system may cap or limit the processing time associated with aligning two multi-resolution voxel spaces, including the maximum number of iterations of the alignment process. If the number of iterations has been completed, process 800 proceeds to 812 , otherwise process 800 proceeds to 810 .

At 810, a multi-resolution voxel space alignment component may determine whether the re-weighted average for the combined multi-resolution voxel space is below a tolerance threshold. If the re-weighted average is below the tolerance threshold, process 800 proceeds to 812 , otherwise process 800 returns to 804 . Systems may include tolerances that set requirements on how well two multi-resolution voxel spaces should be aligned for a particular use. As an example, some applications may require only a coarse understanding of the physical environment, while others, such as autonomous vehicles, may require a more precise and detailed understanding.

At 812, a multi-resolution voxel space alignment component may reduce and/or determine the amount of uncertainty in the alignment of the target multi-resolution voxel space and the reference multi-resolution voxel space. For example, after optimization (eg, the non-linear optimization described above), the multi-resolution voxel spatial alignment component may propagate measurement noise to the aligned voxels. In one particular example, discussed in more detail below, the multi-resolution voxel space alignment component may determine the alignment uncertainty model according to a Gaussian distribution with zero mean and zero covariance. As an example, the multi-resolution voxel space alignment component may model normal random variables xN(μ _x , Σ _x ) such that each step is computed as follows.
x=Cz
where C=(J ^T WJ) ⁻¹ J ^T W (J and W represent the same determined Jacobian and weights, C represents the weighted pseudo-Hamiltonian), z˜N(0, δ _z ² I ) is the residual.

The covariance of x may then be determined by propagating the residual noise and expanding:
_Σx = _CΣzC ^T
Σ _x = (J ^T WJ) ^-1 J ^T σ _z ² W ² J (J ^T WJ) ^-1
In some cases, the residual noise may be calculated incrementally and then the matrices J ^T WJ and σ _z ² J ^T W ² J may be collected for each voxel. In one specific example, the multi-resolution voxel spatial alignment component further includes each residual

We also derive the isotropic variance of , and the mean of each voxel is

where σ _p ² is the isotropic Gaussian noise at each point observation and W _i are the voxel weights. The residual covariance may then be computed as follows.
σ _z ² I=E[zz ^T ]
The term involving E[zz ^T ] vanishes due to the assumed independence of voxel means. therefore,

however,

The multiresolution voxel spatial alignment component then computes the resulting covariance matrix can be further regularized.

At 814, a multi-resolution voxel space alignment component may output an aligned multi-resolution voxel space (which may include measurement uncertainty). By way of example, the aligned multi-resolution voxel space may be provided to another system, such as, for example, an autonomous vehicle planning system or a perception system. In other cases, the aligned multi-resolution voxel space may be sent via one or more networks to a remote system or device, such as a cloud-based computing system, for example. In other examples, the multi-resolution voxel space alignment component may output localization data or transformation data between a target multi-resolution voxel space associated with the position of the vehicle and a reference multi-resolution voxel space with respect to the physical environment. . In some examples, the reference multi-resolution voxel space may be pre-generated by a cloud-based computing system and sent to the vehicle before the vehicle begins moving. In some cases, the cloud-based system ma uses data collected during operation (e.g., merge target multi-resolution voxel space) from multiple vehicles to generate reference multi-resolution voxel Update space. Further, in some examples, the vehicle may be equipped to update the reference multi-resolution voxel space in an off-line manner (e.g., when parked or otherwise not in an otherwise nimble situation). There is

FIG. 9 illustrates an example system for implementing the techniques described herein, in accordance with aspects of this disclosure. In some examples, a system may include one or more of the features, processing resources, components, and/or functionality of the aspects described herein with reference to FIGS. 1-8. In some aspects noted above, the system may include an autonomous vehicle.

FIG. 9 is a block diagram of an exemplary system 900 for implementing the multi-resolution voxel spatial alignment system described herein. In the aspect just described, the system 900 may include a vehicle computing device 904, one or more sensor systems 906, one or more communication connections 908, and one or more drive systems 91010. is.

Vehicle computing device 904 may include one or more processors 912 (or processing resources) and computer readable media 914 communicatively coupled with one or more processors 912 . In the illustrated example, vehicle 902 is an autonomous vehicle, however, vehicle 902 may be any other type of vehicle or any other system (eg, robotic system, camera-enabled smart phone, etc.). would be possible. In the illustrated example, computer readable medium 914 of vehicle computing device 904 implements multi-resolution voxel space generation component 916, multi-resolution voxel space alignment component 918, planning component 920, and perception component 922 associated with the autonomous vehicle. Other systems store similarly. Additionally, computer readable media 914 may also store sensor data 924 and multi-resolution voxel space 926 . In some implementations, the system may be able to access data stored on computer readable media as well, in addition or alternatively, to vehicle 902 (e.g., other computer readable media remote from vehicle 902). stored in or otherwise accessible).

A multi-resolution voxel space generation component 916 may generate a multi-resolution voxel space from data points representing a physical environment, such as the output of a lidar system, for example. In some cases, the multi-resolution voxel space generation component 916 may receive lidar data represented as a plurality of lidar points or point clouds. A multi-resolution voxel space generation component 916 may assign lidar points to voxels of the first base resolution voxel grid. Next, the multi-resolution voxel space generation component 916 may merge the voxels of the lower resolution grid when generating the higher level voxel grid. For example, the multi-resolution voxel space generation component 916 may merge voxels within a neighborhood (eg, a 2x2x2 neighborhood, etc.) in a lower resolution grid when forming the next higher level voxel grid.

In one specific example, the multi-resolution voxel space generation component 1016 generates blocks via a collision-free hash table with pointers implemented as offsets that allow them to be moved or relocated in memory. A multi-resolution voxel space may be generated as a mappable contiguous block of memory accessible by In some cases, memory blocks are represented as tiles with headers, indices (eg, hash tables), and voxel arrays. The indices may be separated by layer and/or resolution. The voxel array may be a single array, or multiple arrays arranged by resolution (e.g., first semantic layer first resolution grid, second semantic layer first resolution grid, third semantic layer Grid of the first resolution of the layer,...). In a voxel array, each element can be a voxel and a key to the spatial position of the voxel. In some cases, the header may include a stack identifier, version number, resolution number, semantic label number, total number of layers, offset, and the like. The index can be a sparse hash table that relates hash values to offsets in memory blocks. In addition, the index may also contain the salt value used to salt the input to the particular table just mentioned, and the prime value used in the first round of modulus calculation.

In some examples, the multi-resolution voxel space alignment component 1018 may align two multi-resolution voxel spaces (eg, a target multi-resolution voxel space and a reference multi-resolution voxel space). In some cases, the multi-resolution voxel space alignment component 918 may find correspondence between voxels in the reference multi-resolution voxel space and the target multi-resolution voxel space. A multi-resolution voxel space alignment component 918 , for each voxel of a unique resolution in the target multi-resolution voxel space, aligns the voxel's three dimensions (e.g., 2x2x2, 3x3x3, 5x5x5, etc.) correspondences may be found by searching the neighborhood. Of the voxels identified within the neighborhood, the multiresolution voxel space alignment component 918 may select voxels with centroids that are close to voxels in the target multiresolution voxel space. A multiresolution voxel space alignment component 918 may then average the distribution of the selected voxels in the reference multiresolution voxel space with the voxels of the target covariance stack. A multi-resolution voxel spatial alignment component 1018 may then perform principal component analysis on the combined covariance matrices and select the smallest eigenvalue as the matched normal vector for the two voxels.

A planning component 920 may determine a path for the vehicle 902 to follow to traverse through the physical environment. For example, planning component 920 may determine different routes and trajectories and different levels of detail. For example, planning component 920 may determine a travel route from a current location to a target location. For the purposes of this discussion, a route may be a sequence of waypoints to travel between two locations.

In some implementations, the prediction component 922 generates, e.g., pose, velocity , trajectory, velocity, yaw, yaw rate, roll, roll rate, pitch, pitch rate, position, acceleration, or other properties, and/or or may be configured to predict a property or condition in the future.

Additionally, vehicle 902 may also include communication connection(s) 908 that enable communication between vehicle 902 and other local or remote computing device(s). By way of example, communication connection(s) 908 may facilitate communication with other local computing device(s) of vehicle 902 and/or with drive system(s) 910 . Additionally, communication connection(s) 908 may allow vehicle 902 to communicate with other nearby computing device(s) (eg, other nearby vehicles, traffic lights, etc.). Additionally, communication connection(s) 908 also allow vehicle 902 to communicate with a remote teleoperated computing device or other remote service.

Communication connection(s) 908 connect vehicle computing device 904 to another computing device (eg, computing device(s) 930) and/or to a network, such as network(s) 928. may include physical and/or logical interfaces for For example, the communication connection(s) 908 may be, for example, frequencies defined by the IEEE 802.11 standard, short range wireless frequencies such as BLUETOOTH, cellular communications (eg 2G, 3G, 4G, 4G LTE, 5G), or via any suitable wired or wireless communication protocol that allows each computing device to interface with other computing device(s). Sometimes. In some examples, communication connection 908 of vehicle 902 may transmit or send multi-resolution voxel space 926 to computing device(s) 930 .

In at least one example, the sensor system(s) 906 may include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, location sensors (eg, GPS, compass, etc.), inertial sensors (eg, inertial measurement unit (IMU), accelerometer, magnetometer, gyroscope, etc.), camera (e.g. RGB, IR, intensity, depth, time of flight, etc.), microphone, wheel encoder, environmental sensor (e.g. temperature sensor, humidity sensor, light sensors, pressure sensors, etc.), and one or more time of flight (ToF) sensors, and the like. Sensor system(s) 906 may include multiple instances of each of the just mentioned or other types of sensors. By way of example, the lidar sensors may include individual lidar sensors located on the corners, front, back, sides, and/or top of vehicle 902 . As another example, camera sensors may include multiple cameras positioned at various locations around the exterior and/or interior of vehicle 902 . Sensor system(s) 906 may provide input to vehicle computing device 904 . Additionally or alternatively, the sensor system(s) 906 may transmit sensor data over one or more networks 928 at specific frequencies, over a predetermined period of time, in near real-time. , etc., to one or more computing device(s) 930 .

In at least one example, vehicle 902 may include one or more drive systems 910 . In some examples, vehicle 902 may have a single drive module 910 . In at least one example, if the vehicle 902 has multiple drive systems 910, individual drive systems 910 can be located at opposite ends of the vehicle 902 (eg, front and rear, etc.). In at least one example, drive system(s) 910 includes one or more sensors that detect the conditions of drive system(s) 910 and/or conditions surrounding vehicle 902, as described above. A sensor system 906 can be included. By way of example and not limitation, the sensor system(s) 906 may include one or more wheel encoders (e.g., rotary encoders) that sense the rotation of the wheels of the drive module, inertial sensors (e.g., rotary encoders) that measure orientation and acceleration of the drive system. inertial measurement units, accelerometers, gyroscopes, magnetometers, etc.), cameras or other image sensors, ultrasonic sensors for auditory detection of objects around the drive system, lidar sensors, radar sensors, etc. It is possible. Some sensors, such as wheel encoders, may be unique to drive system(s) 910 . In some cases, sensor system(s) 906 in drive system(s) 910 may overlap or supplement corresponding systems in vehicle 902 .

In at least one example, the components described herein can process sensor data 924 as described above, via one or more network(s) 928, respectively may be sent to one or more computing device(s) 930. In at least one example, the components described herein transmit their respective outputs to one or more computers at a particular frequency, after a predetermined period of time, in near real time, etc. device(s) 930 .

In some examples, vehicle 902 may send sensor data to one or more computing device(s) 930 via network(s) 928 . In some examples, vehicle 902 may send raw sensor data 924 or processed multi-resolution voxel space 926 to computing device(s) 930 . In other examples, vehicle 902 can send processed sensor data 924 and/or representations of sensor data (eg, object perception tracks) to computing device(s) 930 . be. In some examples, vehicle 902 may send sensor data 924 to computing device(s) 930 at a specific frequency, after a predetermined period of time, such as in near real time. be. In some cases, vehicle 902 may send sensor data (raw or processed) to computing device(s) 930 .

Computing system(s) 930 implements processor(s) 932 , multi-resolution voxel space generation component 936 , multi-resolution voxel space alignment component 938 to process sensor data 940 and multi-resolution voxel space 942 received from vehicle 902 . may also include computer readable media 934 for storage. In some examples, multi-resolution voxel space generation component 936 and multi-resolution voxel space alignment component 938 generate multi-resolution voxel space 942 or multi-resolution voxels generated from data captured by multiple vehicles 902 . Space 942 may be configured to align to form more complete scenes of different physical environments and/or connect different scenes together as a signal augmenting physical environment. In some cases, multi-resolution voxel space generation component 936 and/or multi-resolution voxel space alignment component 938 may be used for machine learning and/or future code testing of one or more data from sensor data 924 . May be configured to generate a model.

Processor(s) 912 of vehicle 902 and processor(s) 932 of computing device(s) 930 are capable of processing data and executing instructions to perform the operations described herein. It can be any suitable processor. By way of example and not limitation, processor(s) 912 and 932 may be one or more CPUs (central processing units), GPUs (graphics processing units), or processors that process electronic data and store the electronic data in registers and /or may include any other device or portion of a device that converts to other electronic data that can be stored on a computer readable medium. In some examples, integrated circuits (eg, ASICs, etc.), gate arrays (eg, FPGAs, etc.), and other hardware devices are also considered, so long as they are configured to implement the encoded instructions. It can also be a processor with

Computer-readable media 914 and 934 are examples of non-transitory computer-readable media. Computer readable media 914 and 934 include an operating system and one or more software applications, instructions, programs, and/or software for implementing the methods and functions ascribed to various systems described herein. It is possible to store data. In various implementations, the computer-readable medium can be any suitable computer-readable medium technology, such as SRAM (static RAM), SDRAM (synchronous DRAM), non-volatile/flash memory, or the ability to store information. It can be implemented with some other type of memory. The architectures, systems, and individual elements described herein can include many other logical, programmatic, and physical components, illustrated in the accompanying figures , are merely examples relevant to the description herein.

As can be understood, the components described herein are described as partitioned for purposes of illustration. However, the actions performed by various components may be combined or performed on any other components.

While FIG. 9 is illustrated as a distributed system, in alternative examples, components of vehicle 902 can be associated with computing device(s) 930 and/or computing device(s). It should be noted that components at 930 can be associated with vehicle 902 . That is, vehicle 902 may perform one or more functions associated with computing device(s) 930 and vice versa.

FIG. 10 is a pictorial diagram 1000 for an example multi-resolution voxel space, such as the multi-resolution voxel space 208 of FIGS. 2-4 compared to a point cloud representation 1008 of the captured data described herein. is. Both the illustrated multi-resolution voxel space 208 and point cloud representation 1008 correspond to real-world physical locations or spaces.

Exemplary Clause A. including a lidar sensor, one or more processors, and one or more non-transitory computer readable media storing instructions executable by the one or more processors, the instructions being executed , having the system receive data representing a physical environment from a lidar sensor; determining a first semantic class associated with a first portion of the data; determining a semantic class of 2; and associating the first portion of the data with voxels in the first voxel of the first voxel grid, the first voxel grid being in the first semantic layer of the target multiresolution voxel space. associating a second portion of the data with voxels of the second voxel of the second voxel grid, the second voxel grid being associated with a second semantic layer of the target multiresolution voxel space; and the second voxel grid are associated with a first resolution; and merging the first set of adjacent voxels of the first voxel grid to obtain a first set of voxel grids associated with the first semantic layer. forming voxels of a voxel grid of three, wherein the third voxel grid is associated with a second resolution lower than the first resolution; and merging a second set of adjacent voxels of the second voxel grid. to form voxels of a fourth voxel grid associated with the second semantic layer, wherein the fourth voxel grid is associated with the second resolution.

B. An operation merges a third set of adjacent voxels of the third voxel grid to form voxels of a fifth voxel grid associated with the first semantic layer, the fifth voxel grid being associated with the first semantic layer. and merging a fourth set of adjacent voxels of the fourth voxel grid to form a sixth voxel grid associated with the second semantic layer. Forming voxels, the sixth voxel grid having a third resolution.

C. Correlating the first portion of data includes determining that the number of observations in the first portion of data is greater than or equal to a threshold number of observations and determining a mean value of the first portion of data. and determining the covariance of the first portion of the data, and associating the mean and covariance with the first voxel.

D. The operations include receiving a reference multiresolution voxel space, determining a voxel correspondence between a target voxel in the target multiresolution voxel space and a reference voxel in the reference multiresolution voxel space; voxels and reference voxels comprising the same resolution; determining a combined voxel weighted statistic representing the target voxels and the reference voxels; The system of Section C, further comprising determining a transform between the voxel space and the target multi-resolution voxel space and controlling the autonomous vehicle based at least in part on the transform.

E. receiving sensor data from a sensor; associating at least a first portion of the sensor data with a first voxel of a first voxel grid in multi-resolution voxel space, the first voxel having a first semantic classification; ) and associated with the first resolution; and at least a second portion of the sensor data with a second voxel of a second voxel grid in multiresolution voxel space, the second voxel being associated with the first semantic classification and determining a third voxel associated with a second resolution lower than the first resolution based at least in part on being associated with the first resolution and the first voxel and the second voxel; A method comprising associating a third voxel with a first semantic classification and controlling an autonomous vehicle based at least in part on the multi-resolution voxel space.

F. determining a first semantic classification associated with the first portion of the data; determining a second semantic classification associated with the third portion of the data; and associating the third portion of the data with the third voxel of the multi-resolution voxel space based on the objective.

G. Associating the first portion of data includes determining a first mean of the first portion of data; determining a first covariance of the first portion of data; associating a mean and a first covariance with the first voxels; determining a second mean value for the second portion of the data; and determining a second covariance for the second portion of the data. and associating the second mean and the second covariance with the second voxel.

H. Determining a third voxel comprises determining a weighted average of a first average of the first voxels and a second average of the second voxels; a first covariance of the first voxels; determining a weighted average of a second covariance of the second voxels; weighted average of the first average and the second average and weighted average of the first covariance and the second covariance; to the third voxel.

I. receiving a reference multiresolution voxel space; determining voxel correspondence between a first voxel and a reference voxel of the reference multiresolution voxel space, the reference voxel having a first resolution; and determining a weighted statistic of the combined voxels representing the voxels and the reference voxels, and determining a transformation between the multiresolution voxel space and the reference multiresolution voxel space based at least in part on the weighted average statistic. and wherein controlling the autonomous vehicle is based at least in part on the transformation.

J. The method of Section I, wherein the voxel correspondence is based on at least a distance between a first centroid associated with the reference multiresolution voxel space and a second centroid associated with the target multiresolution voxel space.

K. The method of Section I, wherein the weighted statistics include weighted covariances.

L. The method of Section I, wherein determining the transform further comprises determining measurement uncertainty based at least in part on modeling the alignment as a Gaussian distribution.

M. The method of Section I, wherein determining the transform includes determining a weighted average of the covariance of the first voxel and the covariance of the reference voxels; and determining the smallest eigenvector of the weighted average.

N. The method of term E, wherein the first voxel and the second voxel are adjacent within the first resolution.

O.D. When executed, cause one or more processors to receive sensor data from sensors associated with the vehicle, associate a first portion of the data with a first voxel of a first grid in voxel space; a first portion of data having a first semantic class; determining a first weighted statistic associated with the first portion of data; associating with a second voxel of the first grid; determining a second weighted statistic associated with a second portion of the data, the second portion of the data having a first semantic class; and a third weighted statistic associated with a third voxel of a second grid in voxel space based at least in part on the first weighted statistic and the second weighted statistic determining that the first grid is associated with a first resolution having fewer voxels than a second resolution associated with the second grid; and controlling the vehicle based at least in part on the voxel space. A non-transitory computer-readable medium storing instructions for performing actions including:

P. The operation associates a first portion of data and a second portion of data with a first semantic layer of voxel space, wherein the first semantic layer corresponds to a first semantic class; associating the portion with a third voxel of the first grid in voxel space, the third portion of the data having a second semantic class; The non-transitory computer-readable medium of term O comprising an association, the second semantic layer corresponding to the second semantic class.

Q. A first semantic class is a non-transitory computer-readable medium of term O including pedestrians, vehicles, buildings, animals, or foliage.

R. The first weighted statistic comprises the first mean and the first covariance of the first portion of the data, and the second weighted statistic comprises the second mean and the first covariance of the second portion of the data. A non-transient computer-readable medium of term O comprising a second covariance.

S. A third weighted statistic is determined by determining a weighted average of the first and second averages; determining a weighted average of the first and second covariances; of the term O, determined at least in part based on a weighted average of the one's average and the second average and associating the weighted average of the first covariance and the second covariance with the third voxel A non-transitory computer-readable medium.

T. The non-transitory computer-readable medium of Term O, wherein the operations further include determining the location of the vehicle within the physical environment based at least in part on the voxel space and the multi-resolution voxel space.

U.S.A. one or more processors and one or more non-transitory computer-readable media storing instructions executable by the one or more processors, the instructions being executed to cause a system to: Receiving data; relating the data to a target multiresolution voxel space; receiving a reference multiresolution voxel space; , determining that the target voxel and the reference voxel are associated with the same resolution; determining a weighted statistic associated with the combined voxels representing the target voxel and the reference voxel; A system for performing an action including determining a transformation based in part and controlling an autonomous vehicle based at least in part on the transformation.

V. A system of terms U, wherein the weighted statistic is the weighted covariance matrix.

W. The system of term U, wherein the operations further include performing a principal component analysis on the weighted mean covariance matrix and determining a minimum eigenvector of the principal component analysis, wherein determining the transformation is further based on the minimum eigenvector. .

X. Determining a target voxel in the target multiresolution voxel space associated with the reference voxel in the reference multiresolution voxel space and determining a weighted statistic associated with the combined voxel representing the target voxel and the reference voxel is performed by determining the voxel , wherein each pair of voxels comprises a voxel in the target multiresolution voxel space and a voxel in the reference multiresolution voxel space.

Y. receive map data including a first voxel space, the first voxel space comprising a first layer associated with a first resolution and a second layer associated with a second resolution different than the first resolution; receiving sensor data from sensors associated with the vehicle; and associating the sensor data with a second voxel space, the second voxel space being associated with the first resolution. having a first layer and a second layer associated with a second resolution; and a first aggregation based at least in part on the first voxel space and the second voxel space. determining the aggregated voxel data; determining a transformation between the first voxel space and the second voxel space based at least in part on the first aggregated voxel data; determining the location of the vehicle in the physical environment based at least in part.

Z. Determining the first aggregated voxel data includes for the first voxel in the first voxel space a second voxel having a centroid within a specified distance of the centroid of the first voxel. identifying a set of voxels in space; selecting a second voxel of the set of voxels, the second voxel having a centroid closest to the centroid of the first voxel; and determining a weighted average of the covariance and the covariance of the reference voxels.

AA. The method of term Z, wherein the first voxel and the second voxel include the same semantic class.

AB. Determining the first aggregated voxel data includes determining a minimum eigenvector of the weighted average and determining a normal vector representing the first aggregated voxel data based at least in part on the minimum eigenvector. The method of term Z, further comprising: determining.

AC. The method of Section AB, wherein reweighting the first aggregated voxel data includes applying an m-estimator framework.

AD. Determining the transform comprises determining a residual based at least in part on the minimum eigenvalue and rotating or translating between the target voxel space and the reference voxel space based at least in part on the residual. and determining one or more of

AE. The method of Section AD, further comprising determining the uncertainty associated with the alignment based at least in part on modeling the distribution.

AF. The method of term Z, wherein the transform indicates a difference in one or more of position or orientation between the first voxel space and the second voxel space.

AG. The method of term Z, wherein the vehicle is an autonomous vehicle and the method further comprises controlling the autonomous vehicle based at least in part on the location of the autonomous vehicle in the physical environment.

AH. When executed, cause one or more processors to receive a target multiresolution voxel space; receive a reference multiresolution voxel space; associated with a first reference voxel in voxel space, wherein the first target voxel and the first reference voxel share a first resolution; determining a weighted statistic of 1; determining that a second target voxel in the target multiresolution voxel space is associated with a second reference voxel in the reference multiresolution voxel space; of the second combined voxels share a second resolution, the second resolution being different than the first resolution, and a second combined voxel representing the second target voxel and the second reference voxel. determining a weighted statistic of 2 and between the target multiresolution voxel space and the reference multiresolution voxel space based at least in part on the first weighted statistic and the second weighted statistic; A non-transitory computer-readable medium storing instructions for performing actions including determining a transformation.

AI. The target multiresolution voxel space includes a first set of voxels associated with the first classification and a second set of voxels associated with the second classification. possible medium.

AJ. Determining that a first target voxel in the target multiresolution voxel space is associated with a first reference voxel in the reference multiresolution voxel space involves determining, for the first target voxel, the centroid of the first target voxel. identifying a set of voxels in the reference multiresolution voxel space that have centroids within the specified distance; determining a first reference voxel from the set. The non-transitory computer-readable medium of Section AH.

AK. determining that the first target voxel and the first reference voxel correspond based at least in part on a distance between the centroid of the first target voxel and the centroid of the first reference voxel. A non-transitory computer readable medium of AH.

AL. The non-temporal computer readable medium of term AH, wherein the first weighted statistic is the weighted mean covariance.

AM. Determining a transformation includes performing a principal component analysis on the first weighted statistic, determining a minimum eigenvalue of the principal component analysis, and determining a residual based at least in part on the minimum eigenvalue. and, as transformations, determining one or more of rotations or translations between the target multiresolution map and the reference multiresolution map that optimize the residuals. computer readable medium.

AN. further comprising applying one or more of a gradient descent technique or a non-linear optimization technique that minimizes a value based at least in part on the residuals, wherein the transformation is one of translation or rotation or The non-transitory computer-readable medium of section AM, including a plurality.

Although the example clauses described above are described with respect to certain specific implementations, in the context of this document it is further understood that the content of the example clauses may include methods, devices, systems, computer-readable media, and/or or via another implementation. Additionally, any of Examples A-AN may be implemented alone or in combination with any other one or more of Examples A-AN.

As can be finally understood, the components described herein are described as partitioned for purposes of illustration. However, the actions performed by various components may be combined or performed on any other components. Furthermore, it should be understood that components or steps described with respect to one example or implementation may be used in conjunction with components or steps of other examples. For example, the components and instructions of Figure 9 may utilize the processes and flows of Figures 1-8.

Having described one or more examples of the techniques described herein, various alternatives, additions, permutations and equivalents are included within the scope of the techniques described herein.

In the description of the examples, reference is made to the accompanying drawings, which form a part, and which illustrate, by way of illustration, certain examples of the claimed subject matter. It is understood that other examples can be used and that modifications or alternatives, such as structural changes, can be made. The above examples, modifications, or alternatives are not necessarily departures from the scope of the intended and claimed subject matter. While it is possible for the steps herein to be given in a certain order, in some cases the order is such that certain inputs are Or they can be modified so that they are provided in separate orders. Additionally, the disclosed procedures could be performed in a different order. Additionally, the various computations described herein need not be performed in the order disclosed, and other examples using alternate orderings for the computations could be readily implemented. . In addition to being reordered, in some cases a computation could also be decomposed into sub-computations with identical results.

Claims (15)

Receiving map data including a first voxel space, the first voxel space having a first layer associated with a first resolution and a second layer different from the first resolution. a second layer associated with resolution;
receiving sensor data from sensors associated with the vehicle;
Associating the sensor data with a second voxel space, the second voxel space comprising a first layer associated with the first resolution and a second layer associated with the second resolution. and a layer of
determining first aggregated voxel data based at least in part on the first voxel space and the second voxel space;
determining a transformation between the first voxel space and the second voxel space based at least in part on the first aggregated voxel data;
determining a location of the vehicle in the physical environment based at least in part on the transformation. Determining the first aggregated voxel data comprises:
for a first voxel in the first voxel space, identifying a set of voxels in the second voxel space having a centroid within a specified distance of the centroid of the first voxel;
selecting a second voxel of said set of voxels, said second voxel having a centroid closest to said centroid of said first voxel;
2. The method of claim 1, comprising determining a weighted average of the covariance of the first voxels and the covariance of the reference voxels. 3. A method according to claim 1 or 2, wherein said first voxel and said second voxel contain the same semantic class. Determining the first aggregated voxel data comprises:
determining the smallest eigenvector of the weighted average;
4. The method of any one of claims 1-3, further comprising: determining a normal vector representing the first aggregated voxel data based at least in part on the smallest eigenvector. the method of. A method according to any preceding claim, wherein re-weighting the first aggregated voxel data comprises applying an m-estimator framework. Determining the transform includes:
determining a residual based at least in part on the smallest eigenvalue;
determining one or more of a rotation or translation between the target voxel space and a reference voxel space based at least in part on the residual. 5. The method of any one of 4. 7. The method of any one of claims 1-6, further comprising determining an uncertainty associated with the alignment based at least in part on modeling a distribution. 8. A method as claimed in any one of claims 1 to 7, wherein the transform indicates a difference in one or more of position or orientation between the first voxel space and the second voxel space. described method. The vehicle is an autonomous vehicle, and the method includes:
9. The method of any one of claims 1-8, further comprising: controlling the autonomous vehicle based at least in part on the location of the autonomous vehicle in the physical environment. A computer program product comprising coded instructions which, when executed on a computer, implements the method of any one of claims 1-9. a system,
one or more processors;
one or more non-transitory computer-readable media storing instructions executable by the one or more processors, the instructions, when executed, causing the system to:
receiving a target multi-resolution voxel space;
receiving a reference multi-resolution voxel space;
determining that a first target voxel of the target multiresolution voxel space is associated with a first reference voxel of the reference multiresolution voxel space, wherein the first target voxel and the first reference voxel; share the first resolution; and
determining a first weighted statistic of the first target voxel and the first reference voxel;
determining that a second target voxel of the target multiresolution voxel space is associated with a second reference voxel of the reference multiresolution voxel space, wherein the second target voxel and the second reference voxel; share a second resolution, said second resolution being different than said first resolution;
determining a second weighted statistic of a second combined voxel representing the second target voxel and the second reference voxel;
determining a transformation between the target multiresolution voxel space and a reference multiresolution voxel space based at least in part on the first weighted statistic and the second weighted statistic. A system characterized by performing 12. The method of claim 11, wherein the target multiresolution voxel space includes a first set of voxels associated with a first classification and a second set of voxels associated with a second classification. System as described. Determining that the first target voxel of the target multiresolution voxel space is associated with the first reference voxel of the reference multiresolution voxel space includes:
For a first target voxel, identifying a set of voxels in the reference multiresolution voxel space having a centroid within a specified distance of the centroid of the first target voxel;
determining the first reference voxel from the set of voxels based on the distance of the centroid of the first reference voxel to the centroid of the first target voxel. System according to claim 11 or 12. The operation determines that the first target voxel and the first reference voxel correspond based at least in part on a distance between the centroid of the first target voxel and the centroid of the first reference voxel. 14. The system of any one of claims 1-13, further comprising: Determining the transform includes:
performing principal component analysis on the first weighted statistic;
determining the smallest eigenvalue of the principal component analysis;
determining a residual based at least in part on the smallest eigenvalue;
determining as the transformation one or more of a rotation or translation between the target multi-resolution map and a reference multi-resolution map that optimizes the residual;
The operations further include applying one or more of a gradient descent technique or a non-linear optimization technique that minimizes a value based at least in part on the residuals, and wherein the transform is a translation or a rotation 15. The system of any one of claims 11-14, comprising one or more of:

Images