EP4728244A2 - Performing map localization and updating map localization in an autonomous driving environment - Google Patents

Abstract

Systems and methods for performing map localization for use with autonomous vehicles (AVs) and updating map localization in an autonomous driving environment are provided. The method may comprise, using a localizer, receiving one or more sensor inputs from at least one of: a perception module; a local pose module, and a prior map; and, using the perception inputs, generating, using the localizer, a filtered estimate of a morphing of the prior map, generating a morphed prior map. The morphed prior map may match a perceived reality around a current local pose.

Description

Attorney Docket No.345278.22001 PERFORMING MAP LOCALIZATION AND UPDATING MAP LOCALIZATION IN AN AUTONOMOUS DRIVING ENVIRONMENT CROSS-REFERENCE TO RELATED APPLICATION(S) [0001] This application claims priority under 35 U.S.C. §119(e) to U.S. Provisional Patent Application No. 63/508,730, filed June 16, 2023. The foregoing application is incorporated by reference herein in its entirety. BACKGROUND Field of the Disclosure [0002] Embodiments of the present disclosure relate to performing map localization for use with autonomous vehicles (AVs) and updating map localization in an autonomous driving environment. Description of the Related Art [0003] To safely and efficiently navigate an environment and prevent possible collision, self-driving or otherwise autonomous vehicles (AVs) require the ability to detect one or more lanes of traffic, objects, obstacles, and/or road surface conditions within an environment of the AV. To detect these objects and potential hazards, AVs are often equipped with one or more types of environmental sensing technologies, such as, e.g., photographic imaging systems and technologies (e.g., cameras), radio detection and ranging (RADAR) systems and technologies, and Light Detection and Ranging (LiDAR) systems and technologies, among other suitable environmental sensing technologies. SUMMARY [0004] According to an object of the present disclosure, a method for performing map localization for use with one or more autonomous vehicles (AVs) and updating map localization in an autonomous driving environment is provided. The method may comprise, using a localizer, receiving one or more sensor inputs from at least one of: a perception module; a local pose module, and a prior map; and, using the perception inputs, generating, using the localizer, a filtered estimate Attorney Docket No.345278.22001 of a morphing of the prior map, generating a morphed prior map. The morphed prior map may match a perceived reality around a current local pose. [0005] According to various embodiments, the morphing of the prior map may comprise adjusting a number of vertices and local coordinates of the number of vertices in polylines in each of one or more lane segments. [0006] According to various embodiments, the morphing of the prior map may comprise performing one or more topological changes involving changes to a graph connectivity of the prior map. [0007] According to various embodiments, the method may further comprise, using the localizer, determining an uncertainty score of the prior map. [0008] According to various embodiments, the method may further comprise generating a performance score of the localizer. [0009] According to various embodiments, the generating the performance score of the localizer may comprise comparing the morphed prior map with the perceived reality. [0010] According to an object of the present disclosure, a method is provided. The method may comprise performing map localization for use with one or more Avs. The performing may comprise, using a localizer, receiving one or more sensor inputs from at least one of: a perception module; a local pose module, and a prior map. The method may comprise updating map localization in an autonomous driving environment. The updating may comprise, using the one or more sensor inputs from the perception module, generating, using the localizer, a filtered estimate of a morphing of the prior map, generating a morphed prior map. The morphed prior map may match a perceived reality around a current local pose. [0011] According to various embodiments, the updating may comprise morphing the prior art map. [0012] According to various embodiments, the morphing the prior map may comprise adjusting a number of vertices and local coordinates of the number of vertices in polylines in each of one or more lane segments. [0013] According to various embodiments, the morphing the prior map may comprise performing one or more topological changes. [0014] According to various embodiments, the one or more topological changes may comprise changes to a graph connectivity of the prior map. Attorney Docket No.345278.22001 [0015] According to various embodiments, the method may comprise, using the localizer, determining an uncertainty score of the prior map. [0016] According to various embodiments, the method may comprise generating a performance score of the localizer. [0017] According to various embodiments, the generating the performance score of the localizer may comprise comparing the morphed prior map with the perceived reality. [0018] According to an object of the present disclosure, a system is provided. The system may comprise a localizer, comprising a processor and a memory. The memory may be configured to store instructions that, when executed by the processor, are configured to cause the processor to perform map localization for use with one or more AVs and update map localization in an autonomous driving environment. The performing may comprise, using a localizer, receiving one or more sensor inputs from at least one of: a perception module; a local pose module, and a prior map. The updating may comprise, using the one or more sensor inputs from the perception module, generating, using the localizer, a filtered estimate of a morphing of the prior map, generating a morphed prior map. The morphed prior map may match a perceived reality around a current local pose. [0019] According to various embodiments, the instructions, when executed by the processor, may be further configured to cause the processor to morph the prior art map. [0020] According to various embodiments, the morphing the prior map may comprise adjusting a number of vertices and local coordinates of the number of vertices in polylines in each of one or more lane segments. [0021] According to various embodiments, the morphing the prior map may comprise performing one or more topological changes. [0022] According to various embodiments, the one or more topological changes may comprise changes to a graph connectivity of the prior map. [0023] According to various embodiments, the instructions, when executed by the processor, may be further configured to cause the processor to, using the localizer, determine an uncertainty score of the prior map. [0024] According to various embodiments, the instructions, when executed by the processor, may be further configured to cause the processor to generate a performance score of the localizer. Attorney Docket No.345278.22001 [0025] According to various embodiments, the generating the performance score of the localizer may comprise comparing the morphed prior map with the perceived reality. [0026] According to various embodiments, the system may comprise an AV. BRIEF DESCRIPTION OF THE DRAWINGS [0027] The accompanying drawings, which are included to provide a further understanding of the disclosure and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the disclosure and together with the description serve to explain the principle of the disclosure. In the drawings: [0028] FIGs. 1-9 illustrate images of autonomous vehicle (AV) sensor (e.g., camera sensor) fields of view, according to various embodiments of the present disclosure; [0029] FIG. 10 illustrates an example system for map updating, according to various embodiments of the present disclosure; [0030] FIG.11 illustrates an example of map-updating software architecture, according to various embodiments of the present disclosure; [0031] FIG. 12 illustrates example elements of a computing device, according to various embodiments of the present disclosure; and [0032] FIG.13 shows example architecture of a vehicle, according to various embodiments of the present disclosure. DETAILED DESCRIPTION [0033] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. As used herein, the singular forms “a,” “an” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. These terms are merely intended to distinguish one component from another component, and the terms do not limit the nature, sequence or order of the constituent components. It will be further understood that the terms “comprises” and/or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and/or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. As used herein, the term “and/or” includes any and all combinations of one or more of the associated listed items. Throughout the Attorney Docket No.345278.22001 specification, unless explicitly described to the contrary, the word “comprise” and variations such as “comprises” or “comprising” will be understood to imply the inclusion of stated elements but not the exclusion of any other elements. In addition, the terms “unit”, “-er”, “-or”, and “module” described in the specification mean units for processing at least one function and operation, and can be implemented by hardware components or software components and combinations thereof. [0034] In this document, when terms such as “first” and “second” are used to modify a noun, such use is simply intended to distinguish one item from another, and is not intended to require a sequential order unless specifically stated. In addition, terms of relative position such as “vertical” and “horizontal”, or “front” and “rear”, when used, are intended to be relative to each other and need not be absolute, and only refer to one possible position of the device associated with those terms depending on the device’s orientation. [0035] An “electronic device” or a “computing device” refers to a device that includes a processor and memory. Each device may have its own processor and/or memory, or the processor and/or memory may be shared with other devices as in a virtual machine or container arrangement. The memory may contain or receive programming instructions that, when executed by the processor, cause the electronic device to perform one or more operations according to the programming instructions. [0036] The terms “memory,” “memory device,” “computer-readable storage medium,” “data store,” “data storage facility” and the like each refer to a non-transitory device on which computer-readable data, programming instructions or both are stored. Except where specifically stated otherwise, the terms “memory,” “memory device,” “computer-readable storage medium,” “data store,” “data storage facility” and the like are intended to include single device embodiments, embodiments in which multiple memory devices together or collectively store a set of data or instructions, as well as individual sectors within such devices. [0037] The terms “processor” and “processing device” refer to a hardware component of an electronic device that is configured to execute programming instructions. Except where specifically stated otherwise, the singular term “processor” or “processing device” is intended to include both single-processing device embodiments and embodiments in which multiple processing devices together or collectively perform a process. [0038] The term “module” refers to a set of computer-readable programming instructions, as executed by a processor, that cause the processor to perform a specified function. Attorney Docket No.345278.22001 [0039] The term “vehicle,” or other similar terms, refers to any motor vehicles, powered by any suitable power source, capable of transporting one or more passengers and/or cargo. The term “vehicle” includes, but is not limited to, autonomous vehicles (i.e., vehicles not requiring a human operator and/or requiring limited operation by a human operator, either onboard or remotely), automobiles (e.g., cars, trucks, sports utility vehicles, vans, buses, commercial vehicles, class 8 trucks etc.), boats, drones, trains, and the like. [0040] Although exemplary embodiment is described as using a plurality of units to perform the exemplary process, it is understood that the exemplary processes may also be performed by one or plurality of modules. Additionally, it is understood that the term controller/control unit refers to a hardware device that includes a memory and a processor and is specifically programmed to execute the processes described herein. The memory is configured to store the modules and the processor is specifically configured to execute said modules to perform one or more processes which are described further below. [0041] Further, the control logic of the present disclosure may be embodied as non- transitory computer readable media on a computer readable medium containing executable programming instructions executed by a processor, controller, or the like. Examples of computer readable media include, but are not limited to, ROM, RAM, compact disc (CD)-ROMs, magnetic tapes, floppy disks, flash drives, smart cards and optical data storage devices. The computer readable medium can also be distributed in network-coupled computer systems so that the computer readable media may be stored and executed in a distributed fashion such as, e.g., by a telematics server or a Controller Area Network (CAN). [0042] Unless specifically stated or obvious from context, as used herein, the term “about” is understood as within a range of normal tolerance in the art, for example within 2 standard deviations of the mean. About can be understood as within 10%, 9%, 8%, 7%, 6%, 5%, 4%, 3%, 2%, 1%, 0.5%, 0.1%, 0.05%, or 0.01% of the stated value. [0043] Hereinafter, some embodiments of the present disclosure will be described in detail with reference to the drawings. In the drawings, the same reference numerals will be used throughout to designate the same or equivalent elements. In addition, a detailed description of well-known features or functions will be ruled out in order not to unnecessarily obscure the gist of the present disclosure. Attorney Docket No.345278.22001 [0044] Hereinafter, systems and methods for performing map localization for use with autonomous vehicles (AVs) and updating map localization in an autonomous driving environment, according to embodiments of the present disclosure, will be described with reference to the accompanying drawings. [0045] Map localization is the process of locating a position on a map where a self-driving or autonomous vehicle (AV) is within a previously defined (published) map. A localizer may attempt to place the AV within the published map, but may also be robust enough to account for one or more changes from a prior map to an actual local environment. Within a self-driving environment, lane segments may be used to identify driving lanes. A lane segment is a pair of polylines in which each polyline, of the pair of polylines, has an orientation (i.e., a beginning and an end). One polyline may be denoted left, and one polyline may be denoted right. The map, then, may be a directed graph in which each node is a lane segment, and directed edges between nodes may indicate connectivity from one lane segment to another. [0046] Geometric information in the map such as, e.g., the vertices of the polylines, may be encoded using coordinates of a local frame. Lane segments may also be annotated with semantic information such as, e.g., speed limits, construction status, lane boundary type, road surface type, lane purpose, etc. Conceivably, the map could also convey additional information, such as, e.g., locations of signs, overpasses, truckport destinations, etc. These could be communicated in a list where each item has coordinates in the local frame. The map, as defined here, may be the output of the localizer. A static stored map computed from previous drives may be the prior map. Geometric data in the prior map may be recorded using global coordinates (e.g., latitude, longitude, altitude, etc.). The map and/or the detections within the environment may be used by a planner to plan a path of the AV. The planner may require: the map generated by the localizer to be a faithful geometric representation of the actual implied lanes of traffic at and near the AV; the map to pass one or more map validation checks regarding structure, orientation, and topological connectivity; the region covered by the map to allow for reasonable future planning of AV motion, for both the AV’s own trajectory and also to predict the motion of other detected actors; and/or the uncertainty in the map to be expressed. The uncertainty may be expressed for a position of each vertex in the lane segment polylines, or more succinctly as a function of range from the AV, for example. [0047] There are many examples of changes which may occur within a local environment. For example, according to various embodiments, a localizer may be configured to provide a map Attorney Docket No.345278.22001 that contains faithful lane segments for each of four implied lanes of travel. For example, according to various embodiments, based on the actual environment shown in the image depicted in FIG.1, the map may be configured to indicate: that the AV is currently in the rightmost of four lanes; that the lane immediately to the left of the AV will divide into two lanes, yielding five lanes in total; that the two rightmost lanes will exit the highway; that the remaining three leftmost lanes will continue on the highway; that the opposing traffic also has three lanes of travel; that the concrete curb to the right and the concrete barrier to the left form road boundaries; and/or that the shoulder (up to the road boundary) is either considered a ‘lane of travel’ or explicitly marked as a shoulder. [0048] According to various embodiments, positional errors of the lane segments near the AV up to a range of 50 meters ahead of the AV should be less than 10 centimeters. It is noted, however, that other error ranges and/or quantities may be incorporated while maintaining the spirit and functionality of the present disclosure. According to various embodiments, positional errors may increase by 10 centimeters for every additional 50 meters in range in the direction of travel. It is noted, however, that other rates of positional error increase may be incorporated while maintaining the spirit and functionality of the present disclosure. [0049] According to various embodiments, the localizer may be configured to allow the AV to continue driving forward even when the prior map is wrong. For example, a canonical test case may be to ensure that the localizer provides a local map that matches reality when the prior map has an incorrect kink in the direction of the lanes. According to various embodiments, in this second example, a long stretch of highway may have newly painted lines that are shifted laterally to the right from their original locations (indicated by faded lane markings) (as shown, e.g., in the images depicted in FIGs. 2-3). According to various embodiments, the localizer may be configured to determine which lines are indicative of the newly painted lines and which are indicative of the faded lane markings (as shown, e.g., in the image depicted in FIG.3). [0050] According to various embodiments, traffic may be routed from one side of a highway to the other side of the highway, as shown, e.g., in the image depicted in FIG.4. This is considered routine highway construction practice. Complications may comprise, e.g.: lanes delineated by cones in spite of strong painted lines delineating a lane in a different direction; the lane that takes traffic across to the other side of the highway moves down the highway on a regular basis so that the map from one day is entirely incorrect the next; while traveling on the opposite Attorney Docket No.345278.22001 side of the highway, there are local exit offramps that in principle take the AV back to the prior map, but nevertheless should not be followed; and/or orange signs with black arrows may be used to indicate that the AV should turn to travel off the new lane, but in other places the same sign is used to indicate an exit that the AV should not follow; among other complications. [0051] According to various embodiments, as shown, e.g., in the image depicted in FIG. 5, a location of a detour lane has moved. The map may show that the detour lane existed previously, but now the detour ramp has been moved further down the highway. Construction may cause the highway to be closed, as indicated by reflective signs straight ahead, as shown, e.g., in the image depicted in FIG.5. The implied lanes of traffic may now exit the highway to the right onto a frontage road. [0052] According to various embodiments, as shown, e.g., in the image depicted in FIG. 6, new asphalt may leave few visible traces of implied lanes, yet the implied lane may still well be understood by human drivers. [0053] According to various embodiments, as shown, e.g., in the image depicted in FIG. 7, although there may be clearly painted lines, inference of intended lane of travel may be inferred by observing one or more cones. Visual cues may be taken in aggregate when determining an implied lane of travel. Note that the correct solution may not simply be an average of “paint cues” and “cone cues.” [0054] According to various embodiments, as shown, e.g., in the image depicted in FIG. 8, what used to be the rightmost lane of the highway may now be used as an exit, and traffic may now be routed to the left across the median to the opposite side of the highway. [0055] According to various embodiments, as shown, e.g., in the image depicted in FIG. 9, not only may surrounding semi-trailer trucks prevent direct sight of lane markings, their bright lights and exhaust fumes may degrade a camera image. The AV may be stopped still in this traffic for many minutes (perhaps an hour). According to various embodiments, the localizer may be configured to remain healthy in this crawling traffic scenario. [0056] In general, an AV may likely encounter situations where perception is highly degraded, and cases where the prior map is vastly incorrect. Moreover, these may not be independent events. In construction zones, e.g., these events may be correlated. [0057] Degraded perception and errors in the prior map may not be rare long tail events. These events may occur multiple times every day and may be routinely navigated by humans. The Attorney Docket No.345278.22001 AV may not drive forward on the assumption that the prior map is correct, and the localizer may first use perception outputs to generate the perceived local map. [0058] The information in the prior map may be helpful, but alone it may not provide sufficient information for the AV to drive. The prior map may be helpful to improve average performance, but to approach human-level driving performance, estimation of the implied lanes of travel is fundamentally a perception activity. When the implied lane ahead cannot be perceived, the AV may fall back. To reduce the number of fallbacks, perception of the implied lane ahead of the AV may be proportionately more consistent and accurate. The proposed localization system may fail (causing the AV to veer across lanes or stop on the highway, for example) if perception detections across all modalities, and taken in aggregate, are consistently biased. [0059] Perception outputs may comprise: one or more detections of painted lane lines ahead of an AV; one or more drivable surfaces ahead of the AV in the form of a segmentation of a ground plane; a detection of one or more individual road furniture elements; one or more nominal perceived lanes of travel ahead of the AV as intimated by cones, barriers, nominal vehicle widths, oil drips, rubber tire marks, chevron signs, rumble strips, flares, etc.; and/or other suitable perception outputs. According to various embodiments, the one or more individual road furniture elements may comprise: 3-Dimensional (3D) cuboid (or point) detection of one or more cones as a “cones” class; 3D cuboid detection of one or more barrels as a “barrels” class; 3D cuboid detection of one or more barriers as a “barriers” class; 3D cuboid detection of one or more construction signs as a “construction signs” class with high recall on all construction sign subclasses (e.g. right lane closed signs) that require navigation; 3D cuboid detection of one or more digital signs as a “digital signs” class; and/or other suitable road furniture elements. According to various embodiments, the detection of the one or more digital signs may comprise reading of text on the one or more digital signs. [0060] According to various embodiments, the localizer may be configured to take one or more inputs from the perception, local pose, and the prior map. According to various embodiments, a location identification system (e.g., a global positioning system (GPS) device) may be required at startup. According to various embodiments, the GPS may be optional and/or provide intermittent input when the AV is performing in autonomy. Long tunnels and bridges, for instance, inhibit GPS. Attorney Docket No.345278.22001 [0061] Using the perception inputs as detections, the localizer may be configured to generate a filtered estimate of a morphing of the prior map, creating a morphed prior map. The morphed prior map should match the perceived reality around current local pose. The local map should be valid in a region that allows the planner to correctly plan its motion and predict the motion of one or more other actors. [0062] According to various embodiments, a morphing of the prior map may principally be an adjustment to a number of vertices and local coordinates of the vertices in the polylines in each lane segment. A morphing may also comprise one or more topological changes involving changes to graph connectivity of the prior map to allow for unmapped exits or onramps, unexpected lane closures, lane openings, etc. According to various embodiments, in certain cases, when the prior map is perfect (e.g., perfect according to a minimum calculated standard) and/or approximately perfect, and the detections are accurate, and the morphing is trivial, the prior map may match reality with only a rigid transformation. [0063] According to various embodiments, an optimal morphing may be chosen so as to best explain and agree with some and/or all perceived detections received from perception. [0064] According to various embodiments, far away from a truck, where no detections have yet been obtained, the localizer may be configured to simply return the existing prior map. In the absence of new data, this may be the most likely proposition for the lanes of travel. According to various embodiments, the localizer may maintain a state and use some or all of a detection history (e.g., through filtering and/or batch estimation) to maximize a signal, and reject outlier detections. According to various embodiments, the localizer may return uncertainty in the map. Vertices of lane segments that have been accurately detected many times may have lower uncertainty, whereas vertices in regions that have not been observed may be more uncertain. [0065] Since reality (e.g., an actual implied lane of travel) is static, the output of the localizer may be only needed at a frequency sufficient to ensure adequate coverage of, and agreement with, the lanes ahead of the AV so that the planner may perform its functions. According to various embodiments, the performance of the localizer may be measured by comparing a generated map with reality in a region pertinent to the planner from a bird’s-eye-view perspective. [0066] According to various embodiments, map modification may be performed. According to various embodiments, during map modification, a nearly global map may induce a Attorney Docket No.345278.22001 set of ^{^^} in a 2-Dimensional (2D) local frame, where ^^^^ is a unique polyline ID, and the index ^^^^ enumerates the node(s) in the polyline, and ^^^^ _^ ^{^^} ^^{^^} ^_^ ∈ ^^^^² is a position of node ^^^^ in polynomial ^^^^. [0067] According to various embodiments, current polylines� ^^^^ _^ ^{^^} ^^{^^} ^_^� may have uncertainties � ^^^^ _^ ^{^^} ^^{^^} ^_^�, precomputed node headings�ℎ ^{^} ^^{^^} ^^{^} ^_^�, and a precomputed list of four neighboring nodes for each node, comprising left� ^^^^ _^ ^{^^} ^^{^^} ^_^�, right� ^^^^ _^ ^{^^} ^^{^^} ^_^�, previous� ^^^^ _^ ^{^^} ^^{^^} ^_^�, and next� ^^^^ _^ ^{^^} ^^{^^} ^_^�. According to various embodiments, each of the left, right, previous, and next operations may return a (potentially empty) vector of neighboring nodes. [0068] Give segmentation measurements,� ^^^^ _^ ^{^} ^^{^^} ^^{^} ^�, for each node, ^^^^ _^ ^{^^} ^^{^^} ^_^ , weights may be chosen in R. The weights may comprise, e.g., ^^^^ _{^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^}, ^^^^ _{^^^^ ^^^^ ^^^^ ^^^^ ^^^^ℎ ^^^^ ^^^^ ^^^^}, ^^^^ _{^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ℎ}, and ^^^^_{ℎ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^}. According to various embodiments, the goal more optimal corrections � ^^^^ _^ ^{^^} ^^{^^} ^_^� to the polylines so that the corrected polylines, ^^�^^ _^ ^{^^} ^^{^^} ^_^ , may be found via Equation 1. ^^�^^ _^ ^{^^} ^^{^^} ^_^ = ^^^^ _^ ^{^^} ^^{^^} ^_^ + ^^^^�ℎ ^{^} ^^{^^} ^^{^} ^_^� ∙ ^^^^ _^ ^{^^} ^^{^^} ^_^ Equation 1 [0069] Where ^^^^�ℎ _^ ^{^^} ^^{^^} ^_^� applies the correction,� ^^^^ _^ ^{^} ^^{^^} ^^{^} ^�, perpendicular to the heading of the node positioned at ^^^^ _^ ^{^^} ^^{^^} ^_^ . Then, according to various embodiments, ^^^^ _^ ^{^^} ^^{^^} ^_^ may be chosen to minimize the following objectives, according to Equation 2. ^ ^{^^} ^^{^^} ^_{^ ^} ^{^} ^^{^^} ^^{^} ^ _^ ^{^} ^^{^^} ^^{^ 2} ^ ^{^^^^+1 ^ 2 2} ^^^^ = ∙� ^^^^ − ^^^^� ^^^^ ^^^^ ^_^^^ ^{+ ^^^^ ^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ℎ ^^^^ ^^^^ ^^^^ ∙� ^^^^ ^^^^ − ^^^^ ^^^^� + ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ℎ ∙� ^^^^ ^^^^ − ^^^^ ^^^^ ^ ^^ ^^^ ^^�} + _{^^^^ ∙ ^^^ ^^^^ ^^^^ 2 ℎ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^ ^^^^� ^^^^ ^^^^��} Equation 2 Attorney Docket No.345278.22001 [0070] Taking all the corrections, ^^^^ _^ ^{^^} ^^{^^} ^_^ , in a single vector, the objective function is modelled using the following quadratic: ^^^^ ^{^^^^} ^^^^ ^^^^ + ^^^^ ^{^^^^} ^^^^ + ^^^^, with ^^^^ symmetric. The problem is modelled such that ^^^^ > 0, and a unique global minimum exists. A necessary condition for ^^^^ to be a minimizer is that the gradient must be zero: 2 ^^^^ ^^^^ + ^^^^ = 0. Solving the linear system, 2 ^^^^ ^^^^ = − ^^^^, yields a candidate colution for ^^^^. [0071] In generating ^^^^ and ^^^^, four terms are to be optimizing. For history, the new correction must be weighted by the previous best estimate for the correction: ^^^^_{ℎ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^} ∙ ∑ ^^^^ _^ ^{^^} ^^{^^} ^_^� ^^^^ _^ ^{^^} ^^{^^} ^_^�² . For segmentation, the new correction should try to place ^^�^^ on regions of high segmentation value: ^^^^ _{^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^} ∙ ∑� ^^^^ _^ ^{^} ^^{^^} ^^{^} ^ − ^^^^ _^ ^{^} ^^{^^} ^^{^} ^�² . For smoothness, the new corrections out to vary smoothly: _{^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ℎ ^^^^ ^^^^ ^^^^ ∙} ^∑ _{� ^^^^ ^} ^{^} ^^{^^} ^^{^+} ^ ¹ − ^^^^ _^ ^{^} ^^{^^} ^^{^} ^�² . For width, the width between the polylines ought to be reasonable, 2 and not differ too much from the initial width: ^^^^ _{^^^^ ^^^^ ^^^^ ^^^^ℎ} ∙ ^∑� ^^^^ _^ ^{^} ^^{^^} ^^{^} ^ − ^^^^ _^^^^ ^ ^^ ^^^ ^_^� . [0072] This method and architecture of map modification allows for handling of certain complexities within the modification of a map. The parameterization of morphing, discontinuity of morphing, correctness, detection and perception are all complex computations addressed within [0073] The process of morphing the prior map to match the detections, requires computing a diffeomorphism from the plane space of diffeomorphisms is an infinite-dimensional function space. Search over this space is computationally prohibitive. Implementation of a practicable solution will necessitate a discretization or finite parameterization of the morphing that is sufficiently elastic to describe all necessary morphings, while also being computationally feasible. Cases may be encountered where the morphing from the prior map to the optimal map that explains the detections is not continuous, in which new lanes appear or a lane splits into an exit. Searching over these discontinuous options may otherwise introduce a discrete combinatorial optimization problem that is likely non-trivial to relax. [0074] According to various embodiments, the performance of the localizer may be adjudicated by comparing the generated map with reality (the actual lanes around the AV). Lateral morphing may, in principle, match reality without having to move the AV longitudinally down the highway (the optimizer may, according to various embodiments, match the exact same patch of three lanes in the map to every segment of a three-lane highway for hundreds of miles). A Attorney Docket No.345278.22001 requirement placed on the localizer may include that global accuracy is valid to within 20 meters, or penalize drifting away from true global position as measured using GPS. It is noted, however, that other suitable requirements may be placed on localizer, while maintaining the spirit and functionality of the present disclosure. According to various embodiments, global localization may be handled by reporting raw GPS data to the planner. [0075] Implementing the localizer as an optimization solver may require computation, and perhaps a significant amount of computation. Consideration should be given to requiring the localizer to pulse only once per set amount of time (e.g., once per second), if accuracy requirements are met and the coverage of the map is sufficient for the planner. As accuracy degrades, pulse times may increase for the localizer to improve accuracy. The localization architecture may also improve optimization and efficiency (an optimization scheme for determining the optimal state estimate). Possible choices may comprise filtering, batch estimation, factor graphs, and/or other suitable choices. [0076] According to various embodiments, the localizer may be configured to receive one or more detections and/or cues about an implied lane from multiple detectors. Finding the optimal map estimate is not simply minimization of a squared-error norm across these detectors. Rather, the optimal estimate may likely arise after reasoning about the various elements and their interactions. For example, a mechanism may be configured to decide that paint (regardless of how bright) is irrelevant to the implied lane, and that, actually, a line of barrels now defines the lane. According to various embodiments, that mechanism may also reason that a line of barrels stored along the edge of the roadway do not imply a lane, but are simply there for storage during the day. [0077] For another example, consider an orange construction sign featuring a long black arrow. This sign may be used to indicate that all traffic must exit the highway to cross the median onto the other side of the highway. However, in construction zones, this same sign may also be placed at exits where it is optional for traffic to exit the highway. The localizer may need to reason about the intent of the sign, not simply its presence. A further nuance of this required reasoning is that the inference needs to be conducted over both space and time. For example, the correct lane may be inferred by considering the import of a sign depicting a lane shift that is placed a quarter mile before the actual shift in the road. [0078] For a local map to be more usable by the planner, invisible lane segments may also be inferred and included with the lane segments. This may require a dedicated gore point detector Attorney Docket No.345278.22001 that alerts the localizer to the presence of a lane merge or split. Then, these lane segments may be spliced into the existing lane segments. [0079] Turning to the localization architecture, the localizer may be configured to perform an estimation process in which perception detections are used to estimate a local map that minimizes error from reality. In the estimation setting, the prior map may serve as an uncertain prior estimate of the local map state. This prior estimate should be updated using detections to ensure that the map matches reality. [0080] According to various embodiments, a set of perception detections obtained up to time ^^^^ _^^^^ by ^^^^ _^^^^may be denoted as ^{ ^^^^₁, … , ^^^^ _^^^^ ^}. Let ^^^^ _^^^^ be the map returned by the localizer at time ^^^^ _^^^^. Then, the localizer may seek to return the map estimate ^^^^⁽ ^^^^ _^^^^| ^^^^ _^^^^ ⁾ that minimizes the error from the true map implied by reality at time ^^^^ _^^^^. [0081] Where detections are sparse, the use of the prior map will, on average, reduce error from reality. Following the Bayesian framework for estimation, the key governing equation is Bayes’ Rule: ^^^^( ^^^^| ^^^^ _^^^^) = ^{^^^^( ^^^^ ^^^^| ^^^^ ^^^^)∙ ^^^^( ^^^^ ^^^^| ^^^^ ^^^^−1)} ^_{^^^( ^^^^ ^^^^ ^^^^−1)} , where the normalizing constant, ^^^^( ^^^^ _^^^^| ^^^^ _^^^^−1) = _{∫ ^^^^} ⁽ _{^^^^ ^^^^| ^^^^ ^^^^} ⁾ _{∙ ^^^^} ⁽ _{^^^^ ^^^^| ^^^^ ^^^^−1} ⁾ _{^^^^ ^^^^ ^^^^, ^^^^ ^^^^| ^^^^ , which is the detection (or measurement)} [0082] The localization architecture, then, may include choices for state representation, detection models, and/or the optimization framework applicable generally or applicable to specific operational design domains. The state representation may define the parameterization of the map. Different sensor configurations and different detections. optimization frameworks based on the map defined as a tuple M := (L, E, A), where L is a set of lane elements. Each lane element may contain a Local lane element ID, Left polyline, Right polyline, Optional ID for subsequent element, Optional ID for preceding element, Optional ID for left lane element that shares a polyline, and Optional ID for right lane element that shares a polyline. Each polyline may contain an ordered list of (vertex, uncertainty) pairs, where the coordinates of the vertex may be given in the local frame, and uncertainty expresses the uncertainty of those coordinates in the lateral direction. E may be a set of directed edges that defines travel connectivity from one lane element to another. A may be a set of undirected edges that defines adjacency between lane elements. Attorney Docket No.345278.22001 [0084] As a first estimate, the localizer may be configured to generate M by using the existing prior map. The polylines in M may be populated by adjusting the polylines in the prior map to match detections. This is done by editing the vertex coordinates and recording the uncertainty. The localizer may also be configured to generate new lane segments based on observations that require a different topology of the graph. The system for map updating, then, may be depicted as shown, e.g., as system 1000 in FIG.10, and, within the architecture of a self- driving vehicle software structure, the software architecture may be depicted, e.g., as software architecture 1100 in FIG.11. [0085] According to various embodiments, according to system 1000, one or more perceived lane detections, one or more perceived drivable surfaces, one or more perceived corridors ahead, one or more prior maps, M_0, and/or one or more local poses may be input into a localizer. According to various embodiments, the localizer may comprise a computing device (e.g., computing device 1200 of FIG.12). [0086] The localizer may be configured to use the prior map, M_0, with initial uncertainty, P_0, and noisy detections, Z, with uncertainty, R, to determine a P(M|Z) map such that M minimized one or more differences from reality, according to Equation 3, where M is estimated so as to minimize the error from reality. Via the Localizer, the P(M|Z) map with uncertainty may be determined. ^^^^ = ^^^^ ^^^^ ^^^^ ^^^^ ^^^^ ^^^^‖ ^^^^ − ℎ( ^^^^)‖_ ^^^^^2 + ‖ ^^^^ − ^^^^ _^^^^‖_{ ^^^^_0}^2 Equation 3 [0087] Referring now to FIG. 4, a block diagram of example non-updating software architecture 1100 is provided. [0088] According to various embodiments, the software may be configured to perform one or more functions 1102 in order to generate an estimated map 1104. This estimated map 1104 may be used in the process of suggesting one or more changes to the vehicle. For example, the changes may comprise a change in direction or speed of the vehicle. Accordingly, the vehicle may be configured to implement the suggestion if it is operating in an autonomous mode. Attorney Docket No.345278.22001 [0089] According to various embodiments, the estimated map 1104 comprises a map state. The map state may comprise one or more components of the map such as, e.g., a local pose (e.g., local_pose 1130), a coordinate pair (e.g., coordinate_pair 1132), a translation of a 2-dimensional estimate (e.g., translation_2DEstimate 1134), one or more construction hulls (e.g., construction_hulls 1136), the estimated map (e.g., estimated_map 1104), and a current region map (e.g., current_region_map 1138). The map state may be updated 1106 with, e.g., measurement frames. [0090] According to various embodiments, measurement frames are measurements of the environment that are incorporated into the map state estimate by way of an update step. An example measurement is the detection of road interior by the front left long-range camera (there are many environmental features detected and many sensors with detectors making such measurement observations). This measurement may be used to update the state of the road surface and lanes in the map state estimate, by way of a data-fitting update step: a state estimate that may approximately best match past measurement observations (“measurement frames”) and the new measurement frame is produced. [0091] This updating may comprise refreshing an estimated map state 1108 (e.g., by refocusing the estimated map state, pruning segments, etc.), updating an estimated map state 1110 (e.g., extending, updating, initiating, etc. the estimated map state), validating a map structure 1112 (e.g., parsing segments, parsing structure, etc.), and/or via one or more other suitable means, while maintaining the spirit and functionality of the present disclosure. [0092] According to various embodiments, construction hulls may be detected 1114 (e.g., using convex hull creation), and this detection may influence, or be influenced by, the refreshed estimated map state. According to various embodiments, device tracks may be channeled 1142 in the detection of the construction hulls 1114. [0093] According to various embodiments, channelizing devices may be construction cones and barrels. They are produced in 3D by a neural network. Convex hulls may be produced by a LiDAR geometric clustering model that groups lidar points together (“obstacle detection”: things that cannot be hit, etc.). These may be both used as measurements to determine where the vehicle should drive. Attorney Docket No.345278.22001 [0094] According to various embodiments, the estimated map state may be updated based on the incorporation of one or more elements such as, e.g., lane segmentation 1116, road interior 1118, road edge 1120, lane centerlines 1122, and/or other suitable elements. [0095] According to various embodiments, one or more external data points may be incorporated into the determination of the map state. For example, a local pose 1124, a global pose 1126, and a prior map 1128 may be incorporated into the determination of the map state. These external data points may be incorporated into the generation of the map state 1140. [0096] According to various embodiments, the local pose 1124 may be incorporated (local_pose 1130) into the map state 1140. The local pose 1130 may be used in the refreshing 1108 if the estimated heat map. The local pose 1130 and the global pose 1126 may be used to update 1144 the coordinate pair, in order to generate coordinate pair 1132. The coordinate pair 1132, the prior map 1128, and/or the estimated map 1104 may be used to localize the information to a global map 1146. The coordinate pair 1132, the translation of the 2-dimensional estimate 1134, the construction hulls 1136, the estimated map 1104, and the prior map 1128 may be used to generate 1148 a region map, which may be used to generate the current region map 1138. [0097] According to various embodiments, the region map may be published 1150. According to various embodiments, the regional map may be used in the determination of one or more actions by a vehicle. The one or more actions may comprise a change in speed of the vehicle, a change of direction of the vehicle, and/or one or more other suitable actions. [0098] Referring now to FIG.12, an illustration of an example architecture for a computing device 1200 is provided. [0099] The hardware architecture of FIG.12 represents one example implementation of a representative computing device configured to one or more methods and means for performing map localization for use with AVs and updating map localization in an autonomous driving environment, as described herein. As such, the computing device 1200 of FIG.12 implements at least a portion of the method(s), system(s) (e.g., system 1000 of FIG. 10), and software architecture(s) (e.g., architecture 1100 of FIG.11) described herein. [0100] Some or all components of the computing device 1200 may be implemented as hardware, software and/or a combination of hardware and software. The hardware comprises, but is not limited to, one or more electronic circuits. The electronic circuits may comprise, but are not limited to, passive components (e.g., resistors and capacitors) and/or active components (e.g., Attorney Docket No.345278.22001 amplifiers and/or microprocessors). The passive and/or active components may be adapted to, arranged to and/or programmed to perform one or more of the methodologies, procedures, or functions described herein. [0101] As shown in FIG.12, the computing device 1200 comprises a user interface 1202, a Central Processing Unit (“CPU”) 1206, a system bus 1210, a memory 1212 connected to and accessible by other portions of computing device 1200 through system bus 1210, and hardware entities 1214 connected to system bus 1210. The user interface can include input devices and output devices, which facilitate user-software interactions for controlling operations of the computing device 1200. The input devices include, but are not limited to, a physical and/or touch keyboard 1250. The input devices can be connected to the computing device 1200 via a wired or wireless connection (e.g., a Bluetooth® connection). The output devices include, but are not limited to, a speaker 1252, a display 1254, and/or light emitting diodes 1256. [0102] At least some of the hardware entities 1214 perform actions involving access to and use of memory 1212, which can be a Random Access Memory (RAM), a disk driver and/or a Compact Disc Read Only Memory (CD-ROM), among other suitable memory types. Hardware entities 1214 can include a disk drive unit 1216 comprising a computer-readable storage medium 1218 on which is stored one or more sets of instructions 1220 (e.g., programming instructions such as, but not limited to, software code) configured to implement one or more of the methodologies, procedures, or functions described herein. The instructions 1220 can also reside, completely or at least partially, within the memory 1212 and/or within the CPU 1206 during execution thereof by the computing device 1200. The memory 1212 and the CPU 1206 also can constitute machine-readable media. The term “machine-readable media”, as used here, refers to a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of instructions 1220. The term “machine- readable media”, as used here, also refers to any medium that is capable of storing, encoding or carrying a set of instructions 1220 for execution by the computing device 1200 and that cause the computing device 1200 to perform any one or more of the methodologies of the present disclosure. According to various embodiments, one or more computer applications 1224 may be stored on the memory 1212. [0103] Referring now to FIG.13, example vehicle system architecture 1300 for a vehicle is provided, in accordance with various embodiments of the present disclosure. Attorney Docket No.345278.22001 [0104] As shown in FIG. 13, the vehicle system architecture 1300 includes an engine, motor or propulsive device (e.g., a thruster) 1302 and various sensors 1304-1318 for measuring various parameters of the vehicle system architecture 1300. In gas-powered or hybrid vehicles having a fuel-powered engine, the sensors 1304-1318 may comprise, for example, an engine temperature sensor 1304, a battery voltage sensor 1306, an engine Rotations Per Minute (RPM) sensor 1308, and/or a throttle position sensor 1310. If the vehicle is an electric or hybrid vehicle, then the vehicle may have an electric motor, and accordingly will have sensors such as a battery monitoring system 1312 (to measure current, voltage and/or temperature of the battery), motor current 1314 and voltage 1316 sensors, and motor position sensors such as resolvers and encoders 1318. [0105] Operational parameter sensors that are common to both types of vehicles include, for example: a position sensor 1334 such as an accelerometer, gyroscope and/or inertial measurement unit; a speed sensor 1336; and/or an odometer sensor 1338. The vehicle system architecture 1300 also may have a clock 1342 that the system uses to determine vehicle time during operation. The clock 1342 may be encoded into the vehicle on-board computing device 1320 (e.g., computing device 1200), it may be a separate device, or multiple clocks may be available. [0106] The vehicle system architecture 1300 also may comprise various sensors that operate to gather information about the environment in which the vehicle is traveling. These sensors may comprise, for example: a location sensor 1344 (for example, a Global Positioning System (GPS) device); object detection sensors such as one or more cameras 1346; a LiDAR sensor system 1348; and/or a radar and/or a sonar system 1350. The sensors also may comprise environmental sensors 1352 such as a precipitation sensor and/or ambient temperature sensor. The object detection sensors may enable the vehicle system architecture 1300 to detect objects that are within a given distance range of the vehicle 1300 in any direction, while the environmental sensors 1352 collect data about environmental conditions within the vehicle's area of travel. [0107] During operations, information is communicated from the sensors to an on-board computing device 1320. The on-board computing device 1320 may be configured to analyze the data captured by the sensors and/or data received from data providers, and may be configured to optionally control operations of the vehicle system architecture 1300 based on results of the analysis. For example, the on-board computing device 1320 may be configured to control: braking via a brake controller 1322; direction via a steering controller 1324; speed and acceleration via a Attorney Docket No.345278.22001 throttle controller 1326 (in a gas-powered vehicle) or a motor speed controller 1328 (such as a current level controller in an electric vehicle); a differential gear controller 1330 (in vehicles with transmissions); and/or other controllers. [0108] Geographic location information may be communicated from the location sensor 1344 to the on-board computing device 1320, which may then access a map of the environment that corresponds to the location information to determine known fixed features of the environment such as streets, buildings, stop signs and/or stop/go signals. Captured images from the cameras 1346 and/or object detection information captured from sensors such as LiDAR 1348 is communicated from those sensors to the on-board computing device 1320. The object detection information and/or captured images are processed by the on-board computing device 1320 to detect objects in proximity to the vehicle. Any known or to be known technique for making an object detection based on sensor data and/or captured images may be used in the embodiments disclosed in this document. [0109] The features and functions described above, as well as alternatives, may be combined into many other different systems or applications. Various alternatives, modifications, variations or improvements may be made by those skilled in the art, each of which is also intended to be encompassed by the disclosed embodiments.

Claims

Attorney Docket No.345278.22001 WHAT IS CLAIMED IS: 1. A method, comprising: performing map localization for use with one or more autonomous vehicles (AVs), the performing comprising: using a localizer, receiving one or more sensor inputs from at least one of: a perception module; a local pose module, and a prior map; and updating map localization in an autonomous driving environment, the updating comprising: using the one or more sensor inputs from the perception module, generating, using the localizer, a filtered estimate of a morphing of the prior map, generating a morphed prior map, wherein the morphed prior map matches a perceived reality around a current local pose. 2. The method of claim 1, wherein the updating comprises morphing the prior art map. 3. The method of claim 2, wherein the morphing the prior map comprises adjusting a number of vertices and local coordinates of the number of vertices in polylines in each of one or more lane segments. 4. The method of claim 2, wherein the morphing the prior map comprises performing one or more topological changes. 5. The method of claim 4, wherein the one or more topological changes comprise changes to a graph connectivity of the prior map. 6. The method of claim 1, further comprising, using the localizer, determining an uncertainty score of the prior map. Attorney Docket No.345278.22001 7. The method of claim 1, further comprising generating a performance score of the localizer. 8. The method of claim 7, wherein the generating the performance score of the localizer comprises comparing the morphed prior map with the perceived reality. 9. A system, comprising: a localizer, comprising: a processor; and a memory, wherein the memory is configured to store instructions that, when executed by the processor, are configured to cause the processor to: perform map localization for use with one or more autonomous vehicles (AVs), the performing comprising: using a localizer, receiving one or more sensor inputs from at least one of: a perception module; a local pose module, and a prior map; and update map localization in an autonomous driving environment, the updating comprising: using the one or more sensor inputs from the perception module, generating, using the localizer, a filtered estimate of a morphing of the prior map, generating a morphed prior map, wherein the morphed prior map matches a perceived reality around a current local pose. 10. The system of claim 9, wherein the instructions, when executed by the processor, are further configured to cause the processor to morph the prior art map. Attorney Docket No.345278.22001 11. The system of claim 10, wherein the morphing the prior map comprises adjusting a number of vertices and local coordinates of the number of vertices in polylines in each of one or more lane segments. 12. The system of claim 10, wherein the morphing the prior map comprises performing one or more topological changes. 13. The system of claim 12, wherein the one or more topological changes comprise changes to a graph connectivity of the prior map. 14. The system of claim 9, wherein the instructions, when executed by the processor, are further configured to cause the processor to, using the localizer, determine an uncertainty score of the prior map. 15. The system of claim 9, wherein the instructions, when executed by the processor, are further configured to cause the processor to generate a performance score of the localizer. 16. The system of claim 15, wherein the generating the performance score of the localizer comprises comparing the morphed prior map with the perceived reality. 17. The system of claim 9, further comprising an AV.