EP4728245A2 - System and methods for av maps using satellite images - Google Patents

Abstract

Disclosed are methods, systems and non-transitory computer readable memory for AV maps using satellite images. For instance, a method may include: obtain at least one satellite image; annotate the at least one satellite image; convert the annotated map into a trajectory and a sparse map; sense an environment and update the sparse map and the trajectory; and execute the trajectory through the environment.

Description

System and Methods for AV Maps using Satellite Images

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

[0001] This invention was made with Government support under Agreement No. W15QKN-22- 9-5026 awarded by USACC-NJ. The Government has certain rights in the invention.

CROSS-REFERENCE TO RELATED APPLICATION(S)

[0002] This application claims the benefit of priority of U.S. Provisional Application No. 63/508,868, filed June 16, 2023, entitled “Systems and Methods For AV Maps Using Satellite Images.”

TECHNICAL FIELD

[0003] Various aspects of the present disclosure relate generally to autonomous vehicles (AVs) and, more particularly, to systems and methods for AV maps using satellite images.

BACKGROUND

[0004] The trucking industry transports a significant portion of raw materials and finished goods through roadways around the world and a significant portion of military supply chains within a theater of operations. Developments in technology, such as those associated with autonomous driving using AVs, have contributed to many improvements within the industry to increase productivity and safety of such operations.

[0005] Generally, AVs use pre-collected mapping data to help navigate the environment. For instance, high density locations, frequently traveled areas or frequently traveled roadways may be scanned using radar, lidar, or cameras, and reference maps may be generated based on the scan data. AVs may use the reference maps to navigate from location to location within the reference map.

[0006] However, collecting data for mapping may not always be possible due to various constraints. In some cases, an area or portion of roadway may not have been scanned yet to generate a reference map due to a lack of resources or time to scan a particular area or roadway, or the area may be difficult to access. Additionally, in some domains, such as in off- grid or foreign country domains, scanning may not be feasible before an AV is deployed.

[0007] Additionally, data collections may encounter challenges that are difficult to predict or mitigate. For instance, AVs (whether scanning vehicles or deployed AVs) may encounter adverse weather conditions (e.g., interfering with scanning vehicle’s sensors or deployed AV sensors) or platform issues (e.g., connectivity to receive new reference maps).

[0008] Thus, the advent of autonomous vehicles (AVs) has revolutionized the transportation industry, offering the potential for safer, more efficient, and more convenient travel. A cornerstone of AV technology is the ability to accurately perceive and understand the environment in which the vehicle operates. This is typically achieved through a combination of onboard sensors and pre-existing maps that provide a detailed representation of the world around the vehicle.

[0009] Traditionally, these maps are high-definition (HD) maps that contain a wealth of detailed information about the driving environment, including the precise location of road boundaries, lane lines, traffic signs, and other navigational elements. While HD maps offer a high level of detail, they are also data-intensive, requiring substantial computational resources to generate, update, and use at run time. This can slow down the map generation process and limit the speed at which map updates can be transmitted to the vehicle, which can be a disadvantage in dynamic environments where road conditions can change rapidly. Further, this can require significant onboard processing to utilize HD maps in navigation and vehicle control.

[0010] Another challenge with traditional mapping approaches is the complexity of the map creation process. Typically, this involves the use of specialized map editing tools that require a high level of expertise to operate. This can make the process time-consuming and prone to errors, particularly when dealing with large-scale maps that cover extensive geographic areas. [0011] Furthermore, traditional mapping approaches often rely on a static map that is updated infrequently. This can be problematic in situations where the road layout or conditions change rapidly, such as in construction zones or areas affected by natural disasters. In such cases, the static map may not accurately reflect the current state of the environment, potentially leading to navigation errors.

[0012] The creation, maintenance, and use of HD maps are data-intensive endeavors. They necessitate the collection of vast amounts of data, often gathered through specialized survey vehicles equipped with an array of sensors such as high-resolution cameras, LiDAR, and GPS systems. The data collected is then processed and compiled into a multi-dimensional map that can be interpreted by the AV's onboard computer systems.

[0013] Due to their complexity and the level of detail they contain, HD maps require substantial computational resources not just for their initial generation but also for ongoing updates and use. These updates are imperative to reflect the ever-changing nature of real-world driving environments, such as alterations due to construction activities, roadworks, or changes in traffic regulations. However, the sheer volume of data and the processing power demanded to update or use HD maps can introduce latency in the map generation and use processes, potentially hindering the ability of AVs to receive timely map updates or perform efficient, timely operation at run time.

[0014] This latency can be particularly disadvantageous in dynamic environments where road conditions and layouts are subject to rapid and unpredictable changes. For instance, in the case of construction zones or after natural disasters, the static nature of traditional HD maps may fail to provide an accurate depiction of the altered state of the roads, leading to potential navigation challenges for AVs. [0015] Thus, breaking the dependency on previously collected data (e.g., scanning data or reference maps) for runtime mapping may be a significant challenge.

[0016] The present disclosure is directed to overcoming one or more of these above-referenced challenges.

SUMMARY OF THE DISCLOSURE

[0017] According to certain aspects of the disclosure, systems, methods, and computer readable memory are disclosed for systems and methods for AV maps using satellite images. [0018] In some cases, a system for AV maps based on satellite images may include: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations. The operations may include: obtain at least one satellite image; obtain annotations for the at least one satellite image to obtain at least one annotated satellite image; convert the at least one annotated satellite image into a trajectory and a sparse map; sense an environment and update the sparse map and the trajectory; and execute the trajectory through the environment.

[0019] In some cases, method for creating maps for autonomous vehicle (AV) navigation based on satellite images may include: obtaining, by at least one processor, at least one satellite image; obtaining, by the at least one processor, annotations for the at least one satellite image to generate at least one annotated satellite image; converting, by the at least one processor, the at least one annotated satellite image into a trajectory and a sparse map; sensing, by the at least one processor, an environment and updating the sparse map and the trajectory based on sensed environmental data; and executing, by the at least one processor, the trajectory through the environment based on the updated sparse map and trajectory.

[0020] In some cases, a non-transitory computer-readable medium (CRM) comprising instructions that, when executed by at least one processor of an autonomous vehicle (AV) system, cause the at least one processor to perform operations. The operations may include: obtaining at least one satellite image; obtaining annotations for the at least one satellite image to obtain at least one annotated satellite image; converting the at least one annotated satellite image into a trajectory and a sparse map; sensing an environment and updating the sparse map and the trajectory; and executing the trajectory through the environment.

[0021] Additional objects and advantages of the disclosed technology will be set forth in part in the description that follows, and in part will be apparent from the description, or may be learned by practice of the disclosed technology.

[0022] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosed technology, as claimed. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary aspects and together with the description, serve to explain the principles of the disclosed technology.

[0024] FIG. 1 depicts an example environment for AV maps using satellite images.

[0025] FIG. 2 depicts an AV system for AVs.

[0026] FIGS. 3A and 3B depicts a data flow diagram schematically showing features of AV maps using satellite images.

[0027] FIG. 4 depicts a flowchart of a routine for AV maps using satellite images.

[0028] FIGS. 5A, 5B, and 50 depict graphical user interfaces for AV maps using satellite images in a satellite image interface.

[0029] FIGS. 6A and 6B depict graphical user interfaces for AV maps using satellite images in a map editor interface.

[0030] FIGS. 7A-7K depict graphical user interfaces for AV maps using satellite images in a mission editor interface.

[0031] FIG. 8 depicts an example system that may execute techniques presented herein.

DETAILED DESCRIPTION

[0032] Various aspects of the present disclosure relate generally to AVs. In general, the present disclosure is directed to methods and systems for AV maps using satellite images.

[0033] In some cases, a user may input trajectories on Satellite Images. The system may convert the inputs (e.g., annotations) and the satellite image (and any associated) data into a sparse map and a trajectory. For instance, the system may receive user annotations via user devices and/or map editing tools.

[0034] Once a trajectory is available, the system may (e.g., a mapping pipeline or map updated 80) may convert the annotated satellite image into a sparse map with start/destination points for an AV to execute.

[0035] In some cases, any information from satellite images may be used to determine drivable area properties. For example, when roads are present, the number of lanes, expected width of lanes, edges of drivable surfaces due to any barriers, etc. In some cases, when properties are unknown from the satellite image, the system may populate defaults for segments or features and be updated by real-time map sensor data as the AV traverses the environment. For instance, the system may be configured to proceed from a starting point and default to two way traffic conditions (so as to create two lanes in opposite directions), until/if the environmental data indicates otherwise.

[0036] In some cases, systems and methods of the present disclosure may be used for creating and updating maps for autonomous vehicles, referred to as a map creation workflow. This workflow leverages satellite imagery and user annotations to generate sparse maps, which are lightweight and efficient representations of the driving environment. These maps may contain the minimum amount of information that an autonomous vehicle requires to navigate effectively, thereby reducing computational overhead, increasing efficiency, and reducing development time of maps.

[0037] The map creation workflow allows for the definition of missions and the tracing of routes directly on the satellite image. The user can add annotations for route and attributes such as speed limit and road type to the map, which are then validated by a pre-validation service. If validated, the map is sent to the vehicle where it is converted into a format that the vehicle's autonomy stack can understand and is patched into the existing map.

[0038] A notable feature of the map creation workflow is the concept of "stitching" together different map segments. This involves a geometric process of aligning the endpoints of new map segments with the corresponding endpoints of existing segments, and a topological process of connecting pointers in both segments so they are connected. This sophisticated method allows for the seamless integration of new map segments into an existing map framework, enabling the map to evolve in response to the changing environment, and, in some cases, use of the AVs off standard roadways.

[0039] The map creation workflow may find applications in various scenarios where rapid deployment of autonomous vehicles is paramount, such as long-haul trucking, military operations, and emergency response. The ability to quickly generate and disseminate maps, and to adapt to sudden changes in road conditions, may provide a decisive advantage in these contexts.

[0040] Furthermore, the map creation workflow incorporates a user-friendly interface with built- in validations that provide immediate feedback during the map annotation process. These validations take into account the physical capabilities of the vehicle and the characteristics of the road or area, ensuring that the created maps are not just geometrically and topologically accurate, but also practical and safe for the vehicle to navigate.

[0041] In summary, the map creation workflow provides a streamlined approach to map generation for autonomous vehicles, enabling faster map generation, reduced computational load, and accelerated delivery of map updates. This may enhance the operational flexibility and responsiveness of autonomous vehicles in dynamic and unpredictable environments.

[0042] Thus, methods and systems of the present disclosure may be improvements to computer technology and/or AV technology.

Environment

[0043] FIG. 1 shows an example environment 100 for AV maps using satellite images. The environment 100 may include user device(s) 105, network(s) 110, AV(s) 115, an AV server 120, an AV data structure 125, and a satellite image server 130. The user device(s) 105 may include an editor interface 105A, a first validation module 105B, and a convertor module 105C. While the convertor module 105C is depicted on the user device(s) 105, the convertor module 105C may instead, or in addition, be on the AV(s) 115 and/or the AV server 120.

[0044] The user device(s) 105 may (hereinafter "user device 105" for ease of reference) may be a personal computing device, such as a cell phone, a tablet, a laptop, or a desktop computer. In some cases, the user device 105 may be an extended reality (XR) device, such as a virtual reality device, an argument reality device, a mixed reality device, and the like. In some cases, the user device 105 may be associated with a user (e.g., a driver or passenger of an AV). The user may have a user account associated with the user device 105 that uniquely identifies the user within the AV server 120. Additional features of the user device 105 and interactions with other devices are described below.

[0045] The network(s) 110 may include one or more local networks, private networks, enterprise networks, public networks (such as the internet), cellular networks, and satellite networks, to connect the various devices in the environment 100. Generally, the various devices of the environment 100 may communicate over network(s) 110 using, e.g., network communication standards that connect endpoints corresponding to the various devices of the environment 100. [0046] The AV(s) 115 (“AV 115” for ease of reference) may be autonomous vehicles that comprises a truck and a semi-trailer. Alternatively, the AV 115 may be a car, a truck, a trailer, a cart, a snowmobile, a tank, a bulldozer, a tractor, a van, a bus, a motorcycle, a scooter, or a steamroller. The AV 115 may comprise a land vehicle. The AV 115 may have a forward side, a right side, a left side, and a rear side. The forward side may be defined as the forward, or main, direction of travel of the AV 115. The right side may be defined from the point of view of the AV 115, or as 90 degrees clockwise from the forward direction when viewed from above.

[0047] A semi-trailer truck, also known as a semi-truck, a semi, a tractor trailer, a big rig or an eighteen-wheeler, is the combination of a tractor unit carriage and one or more semi-trailers that are configured to contain a freight.

[0048] An AV 115, also known as a self-driving vehicle or driverless vehicle, is a vehicle that is capable of sensing its environment and moving with little or no human input. AVs 115 may employ a variety of sensors to perceive their surroundings, whereby advanced control systems interpret sensory information to identify appropriate navigation paths, as well as obstacles and relevant signage. The AV 115 may comprise a fully autonomous vehicle or a semi-autonomous vehicle.

[0049] The AV server 120 may coordinate data and/or instructions between various devices of the environment, such as the operations center 88 (see FIG. 2). The AV server 120 may also manage data stored and provided from the AV data structure 125. The AV data structure 125 may store and manage relevant data for user device(s) 105, relevant data for AV(s) 115, satellite images, and reference maps. [0050] The satellite image server 130 may coordinate and obtain satellite images from reconnaissance satellites. The satellite image server 130 may provide satellite images to the AV server 120 based on requests.

[0051] In some aspects, the user device(s) 105 may receive user inputs to obtain annotations for at least one satellite image. The user inputs may be received by the editor interface 105A to facilitate the annotations for the at least one satellite image. The annotations may include, for example, the designation of a starting point and a destination point on the satellite image, the tracing of a route, and the addition of attributes such as speed limit and road type.

[0052] Once the annotations are made, the first validation module 105B may validate the annotations based on a set of predefined heuristics and validation rules. These rules may reflect the physical capabilities of the AV(s) 115, such as maximum speed and turning radius, as well as the characteristics of the road, like whether it is paved or unpaved. If the annotations pass the validation checks, the convertor module 105C may convert the annotated satellite image into a sparse map.

[0053] The sparse map may be populated with features generated based on the annotated satellite image. These features may be output from machine learning models that detect and label portions of the annotated satellite image. The features may include, for example, road boundaries, lane lines, and other navigational elements that are indispensable for AV navigation.

[0054] The sparse map, along with the trajectory derived from the annotations, may then be sent to the AV(s) 115 via the network(s) 110. The AV(s) 115 may include a stitching module 115A and existing sparse maps 115B. The stitching module 115A may stitch the received sparse map to the existing sparse maps 115B to create an extended sparse map of the environment.

[0055] In some cases, the AV server 120 may include a second validation module 120A and may be connected to an AV data structure 125. The second validation module 120A may provide further validations and/or feedback on the annotations. If the annotations pass the validation checks of the second validation module 120A, the AV server 120 may transmit the trajectory and the sparse map to the AV(s) 115 for execution of the trajectory through the environment.

[0056] The satellite image server 130, connected to the network(s) 110, may provide the satellite images used in the map creation workflow. The satellite image server 130 may enable the exchange of satellite images and map data within the system, facilitating the creation and updating of the sparse maps.

[0057] In summary, the interaction and data flow between the user device(s) 105, the AV(s) 115, the AV server 120, and the satellite image server 130 in the system for autonomous vehicle maps based on satellite images may enable the rapid creation and updating of sparse maps, enhancing the operational flexibility and responsiveness of autonomous vehicles in dynamic and unpredictable environments.

[0058] In some aspects, the user device(s) 105 may be used to annotate the satellite image obtained from the satellite image server 130. The annotations may be performed through the editor interface 105A, which may provide a user-friendly platform for interacting with the satellite image. The annotations may include designating a starting point and a destination point on the satellite image. These points may represent the beginning and end of a mission or route that an AV 115 is intended to follow.

[0059] In some cases, the annotations may also include designating a set of segment points on the satellite image. These segment points may define the path or corridor that the AV 115 is expected to traverse. The segment points may be strategically placed to guide the AV 115 along a specific route, such as a road or a trail, and may take into account various factors such as road conditions, traffic patterns, and the physical capabilities of the AV 115.

[0060] Once the starting point, destination point, and segment points have been annotated on the satellite image, the convertor module 105C may automatically generate a vehicle corridor based on the set of segment points. The vehicle corridor may represent a drivable path for the AV 115, delineated by the segment points. The vehicle corridor may be defined in such a way that it provides a safe and efficient route for the AV 115, taking into account the vehicle's capabilities and the characteristics of the road or terrain.

[0061] In addition to generating the vehicle corridor, the convertor module 105C may also automatically generate a trajectory for the AV 115. The trajectory may be a path that the AV 115 is expected to follow from the starting point to the destination point. The trajectory may be generated based on the vehicle corridor and the starting and destination points. The trajectory may take into account the vehicle's capabilities, the road conditions, and any other relevant factors, ensuring that the AV 115 can navigate the route safely and efficiently.

[0062] In some aspects, the trajectory and the vehicle corridor may be represented as a series of waypoints or nodes, each associated with a specific geographic location on the satellite image. The waypoints may be connected by edges, which represent the path that the AV 115 is expected to follow. The waypoints and edges may collectively form a graph or network that represents the sparse map of the environment 100.

[0063] In some cases, the annotations, the vehicle corridor, and the trajectory may be validated by the first validation module 105B before they are sent to the AV 115. The validation may ensure that the annotations are accurate and feasible, and that the vehicle corridor and trajectory are safe and efficient for the AV 115 to navigate. If the validation is successful, the annotated satellite image, the vehicle corridor, and the trajectory may be converted into a format that the AV 115 can understand and use for navigation.

[0064] In some aspects, the convertor module 105C may process the annotated satellite image to generate a sparse map. This conversion process may involve identifying and delineating drivable and non-drivable areas within the satellite image. Drivable areas may include roads, lanes, and other navigable paths that an AV 115 can traverse. Non-drivable areas may include buildings, bodies of water, and other obstacles or terrain features that the AV 115 cannot traverse. The delineation of drivable and non-drivable areas may be based on the annotations provided by the user, as well as additional information extracted from the satellite image, such as color, texture, and pattern information.

[0065] In some cases, the convertor module 105C may determine drivable area properties from the satellite image. These properties may include, for example, the number of lanes and the expected width of lanes. The number of lanes may be determined based on the width of the road and the typical width of a lane. The expected width of lanes may be estimated based on standard lane widths for the type of road depicted in the satellite image. These drivable area properties may provide valuable information for the AV 115 to navigate the environment 100 effectively and safely.

[0066] In some aspects, the convertor module 105C may populate default values for unknown drivable area properties until updated by real-time sensor data from the environment 100. For instance, if the satellite image does not provide sufficient information to determine the surface type of a road, the convertor module 105C may assign a default surface type, such as asphalt. This default value may be used by the AV 115 for initial navigation until it can collect sensor data that provides more accurate information about the road surface.

[0067] In some cases, the sparse map generated by the convertor module 105C may be updated with environmental data collected by sensors onboard the AV 115 executing the trajectory. These sensors may include, for example, cameras, lidar sensors, radar sensors, and other types of sensors capable of detecting and measuring features of the environment 100. The environmental data collected by the sensors may include information about road conditions, traffic patterns, weather conditions, and other relevant factors. This data may be used to update the sparse map in real-time, allowing the AV 115 to adapt to changes in the environment 100 and navigate more effectively and safely.

[0068] In some aspects, the environmental data collected by the AV 115 and/or new sparse map features based on the environmental data may be cached for subsequent transmission to a server, such as the AV server 120. This caching may allow for efficient data transmission and storage, reducing the computational load on the AV 115 and the network(s) 110. The cached data may be transmitted to the AV server 120 at regular intervals, or when the AV 115 is within range of a network access point. The AV server 120 may then integrate the cached data with existing map data to update the sparse maps used by other AVs in the environment 100.

[0069] In some aspects, the convertor module 105C within the user device(s) 105 plays a central role in the map creation workflow. The convertor module 105C is configured to process the annotated satellite image and generate a sparse map. This conversion process may involve applying image processing techniques and pattern recognition algorithms to the annotated satellite image. These techniques and algorithms may be used to identify and delineate features within the satellite image that are relevant for autonomous navigation, such as roads, lanes, and other navigable paths.

[0070] In some cases, the convertor module 105C may employ a combination of edge detection, object recognition, and semantic segmentation to distinguish between different types of terrain and infrastructure elements depicted in the annotated satellite image. Edge detection may be used to identify boundaries between different types of terrain, such as the edge of a road or the boundary of a building. Object recognition may be used to identify specific objects within the image, such as vehicles, traffic signs, or trees. Semantic segmentation may be used to classify each pixel in the image according to its semantic meaning, such as whether it represents a road, a building, or a tree.

[0071] In some aspects, the convertor module 105C may generate a topological representation and a geometric representation of the environment 100 based on the annotated satellite image. The topological representation may include nodes and edges connecting the nodes, where the nodes correspond to segments of the environment 100. The geometric representation may include the physical dimensions and shapes of the segments, such as the width and length of a road or the shape of a building.

[0072] In some cases, the convertor module 105C may generate these representations for any segments of the environment 100 that have not previously been mapped. This may allow the map creation workflow to continuously update and expand the sparse map as new areas of the environment 100 are explored and annotated.

[0073] In summary, the convertor module 105C provides a powerful tool for processing annotated satellite images and generating sparse maps. By applying advanced image processing techniques and pattern recognition algorithms, the convertor module 105C can extract valuable information from the satellite images and convert it into a format that is useful for autonomous navigation. This may enhance the efficiency and accuracy of the map creation workflow, enabling rapid deployment of autonomous vehicles in a wide range of environments. [0074] In some aspects, the map creation workflow may involve integrating a newly created sparse map with an existing sparse map stored in the AV(s) 115. This integration process may be facilitated by the stitching module 115A within the AV(s) 115. The stitching module 115A may be configured to stitch the new sparse map to the existing sparse maps 115B to create an extended sparse map of the environment 100. This extended sparse map may provide a more comprehensive and up-to-date representation of the environment 100, enabling the AV(s) 115 to navigate more effectively and safely.

[0075] In some cases, the stitching process may involve aligning and merging overlapping features from the new sparse map and the existing sparse maps 115B. This may ensure the continuity and accuracy of the extended sparse map. For instance, if the new sparse map includes a segment of a road that overlaps with a segment in the existing sparse maps 115B, the stitching module 115A may align the overlapping segments and merge them into a single segment in the extended sparse map. This may prevent any discontinuities or inconsistencies in the map that could potentially confuse the AV's navigation system.

[0076] In some aspects, the stitching module 115A may utilize geo-referencing techniques to accurately overlay the new sparse map onto the existing sparse maps 115B. These georeferencing techniques may involve matching geographic coordinates from the new sparse map with corresponding coordinates in the existing sparse maps 115B. This may ensure that the new sparse map is accurately positioned and oriented relative to the existing sparse maps 115B, providing a seamless and coherent map representation.

[0077] In some cases, the stitching process may also involve topological stitching of the new sparse map to the existing sparse maps 115B. This may include integrating topological information from the new sparse map with topological information from the existing sparse maps 115B. The topological information may include, for example, the connections between different segments of the map, which reflect the actual navigable paths that an AV can take. By integrating this topological information, the stitching module 115A may maintain navigational continuity and logical connectivity within the extended sparse map.

[0078] In some aspects, the topological stitching may account for a hierarchical structure of road and/or offroad vehicle pathways. This hierarchical structure may include a set of nodes and edges connecting individual nodes of the set of nodes, where the nodes correspond to segments of vehicle corridors. The segments of the vehicle corridors may be defined by geometric boundaries and features, such as the number of lanes, direction of travel, and road type. By accounting for this hierarchical structure, the stitching module 115A may ensure that topological relationships, including intersections, merges, and splits, are accurately represented in the extended sparse map. This may enable the AV(s) 115 to navigate the environment 100 more effectively and safely, even as the environment changes over time.

[0079] In some aspects, the stitching module 115A within the AV(s) 115 may be configured to stitch a newly created sparse map to the existing sparse maps 115B. This stitching process may involve a combination of geometric and topological adjustments to ensure a seamless integration of the new map data into the existing map framework.

[0080] The stitching process may involve a hierarchical structure of vehicle corridors. This hierarchical structure may include a set of nodes and edges connecting individual nodes of the set of nodes, where the nodes correspond to segments of vehicle corridors. The segments of the vehicle corridors may be defined by geometric boundaries and features, including a number of lanes, direction of travel, and road type. This hierarchical structure may provide a structured and flexible framework for integrating new map segments into the existing sparse maps 115B. [0081] In some cases, the stitching process may involve selecting a first node of the new sparse map that is closest to the existing sparse maps 115B and selecting a second node of the existing sparse maps 115B. The stitching module 115A may then connect the first node to the second node with a new edge, thereby integrating the new sparse map into the existing sparse maps 115B. This node selection and connection process may ensure that the new map segments are accurately and logically integrated into the existing map framework.

[0082] In some aspects, the stitching process may also involve geometric stitching of the new sparse map to the existing sparse maps 115B. This geometric stitching may involve aligning and adjusting geometric features from the new sparse map with corresponding geometric features from the existing sparse maps 115B. This alignment and adjustment process may ensure that there is a continuous and coherent path for the AV(s) 115 to follow, without abrupt discontinuities or misalignments that could confuse the AV's navigation system.

[0083] In some cases, the stitching module 115A may utilize geometric transformation algorithms to correct for discrepancies in scale, rotation, and translation between the new sparse map and the existing sparse maps 115B. These geometric transformation algorithms may ensure that the new sparse map is accurately positioned and oriented relative to the existing sparse maps 115B, providing a seamless and coherent map representation.

[0084] In summary, the stitching module 115A within the AV(s) 115 may provide a powerful tool for integrating new sparse maps into the existing sparse maps 115B. By combining geometric and topological adjustments with a hierarchical structure of vehicle corridors, the stitching module 115A may ensure a seamless and accurate integration of new map data into the existing map framework. This may enhance the adaptability and responsiveness of the AV(s) 115 in dynamic and unpredictable environments.

[0085] In some aspects, the editor interface 105A and the first validation module 105B within the user device(s) 105 play a central role in the map creation workflow. The editor interface 105A may provide a user-friendly platform for interacting with the satellite image obtained from the satellite image server 130. The user may use the editor interface 105A to annotate the satellite image, designating a starting point, a destination point, and a set of segment points that define the path or corridor that an AV 115 is expected to traverse.

[0086] Once the annotations are made, the first validation module 105B may validate the annotations based on a set of predefined heuristics and validation rules. These heuristics may be derived from simulations of different vehicle types on various road or offroad conditions. For instance, the simulations may include parameters such as vehicle weight, suspension characteristics, tire or track traction, and environmental conditions to define an operational envelope for vehicle type and road type. The operational envelope may define the heuristics used by the first validation module 105B for validating the annotations.

[0087] In some cases, the first validation module 105B may select specific heuristics for validation based on a type of vehicle that will execute the trajectory and/or based on a type of road or terrain indicated by the editor interface 105A or depicted in the satellite image. For example, if the vehicle is a heavy-duty truck and the road is a highway, the first validation module 105B may select heuristics that reflect the capabilities and limitations of a heavy-duty truck on a highway. This may ensure that the annotations, and the resulting trajectory and vehicle corridor, are feasible given the specific vehicle type and road conditions.

[0088] In some aspects, the first validation module 105B may be further configured to recommend modifications to the annotations if the trajectory does not satisfy the validation conditions of the first validation module 105B. The validation conditions may indicate whether a trajectory is deemed infeasible or suboptimal for an intended vehicle type or driving conditions. For instance, if a user attempts to annotate a turn that is too sharp for the vehicle's actuation limits at the designated speed, the first validation module 105B may flag this in real-time, prompting the user to adjust the annotation accordingly. This immediate feedback loop may ensure that the user can make corrections on the fly, optimizing the map for the vehicle's navigation system without the delay of back-and-forth validation checks with the AV server 120. [0089] In summary, the editor interface 105A and the first validation module 105B within the user device(s) 105 provide a powerful tool for annotating satellite images and validating the annotations. By leveraging heuristics derived from simulations of different vehicle types on various road or offroad conditions, the first validation module 105B may ensure that the annotations, and the resulting trajectory and vehicle corridor, are feasible and safe for the AV(s) 115 to navigate. This may enhance the efficiency and accuracy of the map creation workflow, enabling rapid deployment of autonomous vehicles in a wide range of environments.

[0090] In some aspects, the AV server 120 may include a second validation module 120A that provides additional validations and feedback on the annotations made on the satellite image. The second validation module 120A may utilize a larger set of validations than the first validation module 105B in the user device(s) 105. These additional validations may include, for example, more detailed checks on the feasibility of the trajectory, the compatibility of the trajectory with the intended vehicle type, and the suitability of the trajectory for the expected driving conditions. [0091] The validation conditions used by the second validation module 120A may indicate whether a trajectory is deemed infeasible or suboptimal for an intended vehicle type or driving conditions. For instance, if the trajectory involves a steep uphill climb and the intended vehicle is a heavy-duty truck with a full load, the second validation module 120A may deem the trajectory infeasible due to the truck's limited climbing ability. Similarly, if the trajectory involves a sharp turn at high speed and the driving conditions are wet or icy, the second validation module 120A may deem the trajectory suboptimal due to the increased risk of skidding.

[0092] In some cases, the second validation module 120A may provide feedback to a user through the editor interface 105A on the user device(s) 105. This feedback may include recommendations for altering the trajectory to improve drivability, safety, and/or efficiency. For example, the second validation module 120A may recommend reducing the speed at a sharp turn, rerouting the trajectory to avoid a steep uphill climb, or adding a waypoint to guide the AV 115 through a complex intersection. These recommendations may be based on heuristics derived from simulations of different vehicle types on various road or offroad conditions, ensuring that the trajectory is optimized for the specific capabilities and limitations of the AV 115. [0093] In some aspects, the user device(s) 105 may be connected to an autonomous vehicle or a network connected to the autonomous vehicle in an operations center. The user device(s) 105 may be configured to transmit the annotated satellite image directly to the AV 115. The AV 115 may host the first validation module 105B, which is configured to validate the annotations. If the annotations pass the validation checks of the first validation module 105B, the AV 115 may proceed with the execution of the trajectory.

[0094] In some cases, the user device(s) 105 may instead or in addition transmit the annotated satellite image to the AV server 120 for further validations by the second validation module 120A. If the annotations pass the validation checks of the second validation module 120A, the AV server 120 may transmit the trajectory and the sparse map to the AV 115 for execution of the trajectory through the environment 100. This two-tier validation process, involving both the first validation module 105B in the AV 115 and the second validation module 120A in the AV server 120, may provide a robust and comprehensive check on the annotations, ensuring that the trajectory is feasible, safe, and efficient for the AV 115 to navigate.

[0095] In some aspects, the map creation workflow may prompt a user to input new annotations on the satellite image upon determining that an autonomous vehicle may not continue on a current trajectory. This prompting may be based on a prediction that the trajectory will lead to an area without pre-existing map data or where the pre-existing map data is determined to be unreliable. For instance, if the AV(s) 115 encounters an unexpected obstacle or a sudden change in road conditions that were not accounted for in the initial mission plan, the user may be prompted to annotate a new route on the satellite image. This new route may then be validated and sent to the AV(s) 115, which can process the updated information and adjust its trajectory accordingly. This capability to dynamically manage the AV's route, path or trajectory underscores the transformative potential of the map creation workflow in enhancing the operational flexibility and responsiveness of AVs in unpredictable environments.

[0096] In some aspects, the map creation workflow may involve user intervention when the AV(s) 115 encounters unexpected obstacles or discrepancies between the sensed environment and the sparse map. This intervention may be facilitated by the user device(s) 105, which may be configured to prompt the user to input new annotations on the satellite image.

[0097] In some cases, the prompting may be triggered by the AV(s) 115 encountering an unexpected obstacle or condition that indicates a trajectory or route change. For instance, if the AV(s) 115 encounters a roadblock or a detour that was not accounted for in the initial (or extended) sparse map, the user device(s) 105 may prompt the user to input new annotations to adjust the trajectory. This may allow the AV(s) 115 to navigate around the obstacle or follow a detour, ensuring that it can continue its mission without interruption. [0098] In some aspects, the user may be prompted through the editor interface 105A on the user device(s) 105. The editor interface 105A may be configured to display the satellite image and receive the annotations from the user. The user may use the editor interface 105A to trace out a new route on the satellite image, add attributes like speed limit and road type, and, optionally, define a new mission. This direct interaction with the map data may allow the user to quickly and intuitively adjust the trajectory in response to unexpected obstacles or changes in road conditions.

[0099] In some cases, the system may generate a request for user intervention when sensors onboard the AV(s) 115 detect a discrepancy between a sensed version of the environment 100 and the sparse map. For instance, if the sensors detect a new road that is not included in the sparse map, or if they detect a change in road conditions that is not reflected in the sparse map, the system may prompt the user to input new annotations to update the sparse map. This realtime feedback loop may enable the system to continuously update and refine the sparse map, ensuring that it accurately represents the current state of the environment 100.

[0100] In some aspects, the user may input the annotations to provide adjustments to the trajectory to avoid an area causing the AV(s) 115 to be unable to continue on the current trajectory. For example, if the AV(s) 115 encounters a flooded road that it cannot traverse, the user may annotate a detour on the satellite image that routes the AV(s) 115 around the flooded area. These new annotations may then be validated by the first validation module 105B and the second validation module 120A, converted into a sparse map by the convertor module 105C, and sent to the AV(s) 115 for execution. This may allow the AV(s) 115 to quickly and efficiently adapt to changes in the environment 100, enhancing its operational flexibility and responsiveness.

[0101] In some aspects, the user device(s) 105 may provide the user with options for alternative trajectories based on the satellite image and real-time data from the AV(s) 115. For instance, if the AV(s) 115 encounters an unexpected obstacle or a sudden change in road conditions, the user device(s) 105 may display the satellite image with the current trajectory and a set of alternative trajectories that avoid the obstacle or account for the change in road conditions. The user may then select one of the alternative trajectories, annotate the selected trajectory on the satellite image, and send the annotated satellite image to the AV(s) 115 for execution.

[0102] In some cases, the user device(s) 105 may update the sparse map with the new annotations and recalculate a new trajectory for the AV(s) 115 to execute. The convertor module 105C within the user device(s) 105 may process the annotated satellite image to generate a new sparse map that includes the alternative trajectory. The new sparse map may then be validated by the first validation module 105B and the second validation module 120A, and if validated, sent to the AV(s) 115 for execution.

[0103] In some aspects, the AV(s) 115 may stitch the new sparse map to the existing sparse maps 115B to form an extended sparse map of the environment 100. The stitching module 115A within the AV(s) 115 may align and merge overlapping features from the new sparse map and the existing sparse maps 115B, ensuring the continuity and accuracy of the extended sparse map. The stitching module 115A may also integrate topological information from the new sparse map with topological information from the existing sparse maps 115B, maintaining navigational continuity and logical connectivity within the extended sparse map.

[0104] In some cases, the AV(s) 115 may update the extended sparse map with environmental data collected by sensors onboard the AV(s) 115 executing the trajectory. This environmental data may include changes in road conditions, new construction, or detected obstacles. The updated extended sparse map may then be cached for subsequent transmission to the AV server 120.

[0105] In some aspects, the AV server 120 may receive and integrate the updated extended sparse map with existing sparse maps stored in an AV data structure 125. This integration may involve aligning and merging overlapping features from the updated extended sparse map and the existing sparse maps, ensuring the continuity and accuracy of the overall map data. The updated map data may then be made available to other AVs in the environment 100, enabling them to navigate more effectively and safely in the updated environment.

[0106] In some aspects, the AV server 120 may be configured to update tiles within a regional or global map database based on the updated extended sparse map to form updated tiles. The tiles may be portions of the regional or global map database that correspond to specific geographic areas. For instance, if the AV(s) 115 has explored a new area of the environment 100 that was not previously mapped, the AV server 120 may create a new tile for this area based on the updated extended sparse map. Similarly, if the AV(s) 115 has collected new environmental data for an area that was previously mapped, the AV server 120 may update the corresponding tile with this new data.

[0107] In some cases, the updated tiles may be made available to other AVs in the environment 100 for use in future navigation. For instance, if an AV is planning a mission that involves traversing an area for which an updated tile is available, the AV may download this tile from the AV server 120 and incorporate it into its own sparse map. This may allow the AV to benefit from the latest environmental data collected by other AVs, enhancing its ability to navigate the environment 100 effectively and safely.

[0108] In some aspects, the updated tiles may reflect the latest sensed environmental changes and/or the annotations made by the user. For example, if the AV(s) 115 has detected a new road construction or a change in road conditions, this information may be included in the updated tile. Similarly, if the user has annotated a new route or added new attributes to the map, these annotations may be reflected in the updated tile. This may ensure that the updated tiles provide a comprehensive and up-to-date representation of the environment 100, enabling the AVs to adapt to changes in the environment 100 and navigate more effectively and safely. [0109] In some cases, the AV(s) 115 may execute the trajectory based on the updated extended sparse map. The execution of the trajectory may involve dynamically adjusting the trajectory in response to updated environmental features and conditions. For instance, if the AV(s) 115 detects a new obstacle or a change in road conditions while executing the trajectory, it may adjust its path or speed to navigate around the obstacle or accommodate the change in conditions. This dynamic adjustment of the trajectory may enable the AV(s) 115 to respond in real-time to changes in the environment 100, enhancing its operational flexibility and responsiveness.

[0110] In some aspects, the technology may be used to automatically generate maps, potentially speeding up the process by two orders of magnitude. For instance, the AV(s) 115 may use machine learning algorithms to automatically detect and label features in the satellite images, such as roads, lanes, and other navigable paths. These features may then be incorporated into the sparse map, reducing the amount of manual annotation that is typically involved in map creation. This automatic map generation may enable the rapid deployment of AVs in the environment 100, potentially speeding up the process by two orders of magnitude. [0111] The following outlines examples of the present disclosure.

[0112] Example 1 : map creation workflow

[0113] In this example, the map creation workflow is used to create a sparse map for an autonomous vehicle (AV) that is tasked with navigating a long-haul trucking route. The user begins by obtaining a satellite image of the route from the satellite image server. The user then uses the editor interface on their device to annotate the satellite image, designating a starting point at the truck's current location and a destination point at the truck's intended destination. [0114] The user also traces out the route on the satellite image, designating a set of segment points that define the path the truck is expected to follow. These segment points may be placed to guide the truck along the highway, taking into account factors such as road conditions, traffic patterns, and the truck's physical capabilities.

[0115] Once the annotations are made, the user's device sends the annotated satellite image to a pre-validation service. This service checks the annotations against a set of predefined heuristics and validation rules, which reflect the physical capabilities of the truck and the characteristics of the highway. If the annotations pass the validation checks, the annotated satellite image is converted into a sparse map and a trajectory for the truck to follow.

[0116] The sparse map is then sent to the truck, where it is patched into the existing map. The truck's onboard systems use the sparse map and the trajectory to navigate the route, continuously monitoring the truck's position relative to the waypoints and making real-time adjustments to the driving strategy as it progresses along the route.

[0117] Example 2: Stitching Process [0118] In this example, the stitching process is used to integrate a new map segment into an existing sparse map. The new map segment is created when the user annotates a detour on the satellite image, in response to a roadblock on the AV’s original route.

[0119] The stitching process begins with the geometric alignment of the endpoints of the new map segment with the corresponding endpoints of the existing map segments. This alignment ensures that there is a continuous and coherent path for the AV to follow, without abrupt discontinuities or misalignments.

[0120] The stitching process also involves topological stitching, which connects pointers in both the new and existing map segments so the pathway is traversable in logic of the AV. This ensures that the relationships between different segments of the map are logically consistent, reflecting the actual navigable paths the AV can take.

[0121] Once the stitching process is complete, the updated sparse map is sent to the AV, where it is patched into the existing map. The AV’s onboard systems use the updated sparse map to navigate the detour, continuously monitoring the truck's position relative to the waypoints and making real-time adjustments to the driving strategy as it progresses along the detour.

[0122] Example 3: Real-Time Map Updates

[0123] In this example, the sparse map is updated in real-time as the AV navigates the route. The AV’s onboard sensors continuously monitor the environment, collecting data about road conditions, traffic patterns, and other relevant factors.

[0124] This environmental data is used to update the sparse map in real-time, allowing the AV to adapt to changes in the environment. For instance, if the sensors detect a new road construction that was not accounted for in the initial sparse map, the AV’s onboard systems can adjust the trajectory to navigate around the construction site.

[0125] The updated sparse map is also cached for subsequent transmission to a server, where it is integrated with existing map data to update the sparse maps used by other AVs. This allows other AVs to benefit from the latest environmental data collected by the AV, enhancing their ability to navigate the environment effectively and safely.

AV System

[0126] FIG. 2 depicts an AV system 200 for AVs. The features of AV system 200 for AVs of FIG. 2 may apply to FIGS. 1, 3A-3B, 4, 5A-5C, 6A-6B, and 7A-7K.

[0127] An AV system 200 for a vehicle (e.g., an AV 115) may include a sensor module 10, a perception module 12, a planning module 14, a control module 16, a mapping module 18 and a remote driving module 20. The sensing module 10 may include a plurality of sensor types for sensing the environment around the vehicle. The sensor module 10 sends the data from the sensors to the perception module 12. The perception module 12 ingests the sensor data to detect the environmental conditions around the vehicle. This may include the presence of nearby vehicles, the driving surface, flora and fauna, or other objects that may be on or near the road surface. The planning module 14 receives outputs of detected objects and obstacles from the perception module 12 and map information from the mapping module 18 to create a driving plan for the vehicle. The driving plan from the planning module 14 is passed to the control module 16 where the plan is interpreted into control signals for the vehicle, including a throttle signal, brake signal, and steering control signal. The remote driving module 20 may be attached through a gateway 22 to a remote driver 26. The remote driver module 20 may take input from the sensor module 10 and input from the controller 16 for remote operation of the vehicle from the remote driver 26.

[0128] Other inputs to the system 200 may include commands from an operations center 88 that may be made in response to remote monitoring of the vehicle, or from requests made from the operations center 88 related to higher level requests of the total environment (e.g., rerouting for different pickups or responding to traffic, maintenance and service requests). Trailer information may also be input into the planning module 14 in order to create the driving plan. Certain processes, such as calibrations 22, local pose and health monitoring 26 may interact with many components in the system 200 as necessary to validate, monitor and inform performance of the vehicle.

[0129] The sensor module 10 may include inertial measurement units (IMU) and GPS encoders 24, cameras 30, lidars 32, and radars 34. The IMlls and GPS encoders 24 may define the pose (position and orientation of the vehicle) as well as the kinematic properties of the vehicle (speed, acceleration). These properties may be used within all modules of the vehicle to perform certain processes of the system 200 to continue to perform the driving tasks. The sensors capture environmental data from the surroundings of the vehicle. The number and position of the sensors on the vehicle may vary based on vehicle type and shape, as well as expected driving conditions. For example, at higher speeds, the environment may be sensed at greater distances. The field of view for the sensors should be capable of sensing for the driving domain for which the vehicle plans to operate.

[0130] The perception module 12 detects elements in the environment of the vehicle by processing the images or data received from the sensor module 10. The detectors may include 2D object detectors 42, 3D object detectors 44, sensor detectors 46 (for example, a radar detector) and obstacle detectors 48. Outputs from the detectors may be used in the perception module 12, either individually or in combination with other detector output or sensor output, to generate additional perception features of the environment. These features may include ground plane features such as lane and road lane detectors 40, 2D to 3D models 50, and sensor fusion and tracking module 54. Output from the sensors and detectors may also be used in the localizer 58 to determine the precise position of the vehicle relative to the map and the environment. For example, the localizer 58 may determine precisely which lane on the road the vehicle is in. [0131] Object detectors receive the sensor data in order to classify objects in the environment as well as bound those objects to a certain position within the environment. The detectors may use machine learning methods to classify objects within the sensor data. The 2D detectors 42 may primarily use the data from the cameras 30 to identify objects from the camera images. 3D detectors 44 may use lidar and radar data in addition to the camera data to classify and locate objects in 3D space around the vehicle. Sensor detectors 46 may use output from a specific sensor to classify and locate objects in the environment. For each detector type, a number of detectors may exist. For example, the detectors may have individual detectors for vehicle classes, pedestrian classes, bridge classes, or any other class where the data sets are large enough to create specific classes. The detectors may also have specific detectors for a single sensor alone or a combination of sensors. For example, a detector may exist from a left side (driver’s side) forward facing camera to detect vehicles, and a detector may exist from a left side rear facing detector to also detect vehicles. As vehicles look different from the front or behind, these detectors may be trained differently to determine what objects are vehicles. Similarly, a right side camera, which may capture more off road features like animals. Thus, each detector type 42-46, may have many different detectors trained to specific classes as well as trained to specific sensor inputs.

[0132] Obstacle detector 48 identifies perceived obstacles in the environment, but may not classify the obstacles as specific objects. The obstacle detector 48, then, may capture uncommon obstacles (like debris) that may be on the driving surface. As the object detectors 42-46, unable to classify a certain object without enough previous experience with the object in the training sets, may not be able to identify the objects on the road, the obstacle detector 48 may capture all of the unidentified returns as obstacles, which can then be passed into the planning module 14 as obstacles.

[0133] The sensor fusion and tracking module 54 may take all of the object detector outputs to create a single perceived environment around the vehicle. Many of the detectors will detect the same objects in the environment. The sensor fusion may combine those detections to create a single track for an object. That track may also persist with subsequent detections as time passes and the vehicle and objects around the vehicle move through the environment.

[0134] The planning module 14 uses the perception output and map information from the mapping module 18 to plan the path for the vehicle through the environment. The planning module 14 includes a construction zone handler 64, a routing handler 62, and a trajectory planner 66. The trajectory planner includes a path option generator 72, a path decision 74, and a path to trajectory module 76. The construction zone handler 64 may identify changes to the road because of new or completed construction on the roadway. The construction zone handler 64 may send updates to the mapping module 18 to update the map information for future vehicles that may enter that part of the map. Information from the construction zone handler 64 may also be sent to the trajectory planner 66 so that the drivable zones of the roadway can be identified relative to the construction zones. The router 62 includes the information for all the routing instructions for the vehicle trip so that the trajectory planner may be informed of upcoming lane changes for road switches.

[0135] The trajectory planner 66 receives the environmental information and road information from the perception module 12 and generates possible paths through the environment toward the goal of following the routing information in the path option generator 72. The path decision module 74 may analyze the paths that are generated in the generator 72 to determine which paths are within the capabilities of the vehicle’s performance, and then from those pick the path that fits the goals of the route and is safe within the vehicle’s capabilities. The path to trajectory module 76 then converts the path information to a trajectory model for a given period of time going forward. For example, the trajectory may be projected for the next 15 seconds, so that the trajectory sent to the control module 16 will receive coordinates for where the trajectory planner 66 wants the truck to be for the next 15 seconds.

[0136] Control module 16 receives the trajectory information from the planning module 14 to generate control signals for the vehicle. The control signals may include a steering angle, a throttle control, and a brake control. The control module 16 calculates the values of these signals based upon the physical coordinates and the times generated in the planning module 14. The control module 16 has maximum and minimum values for each of the signals and maintains the signals within those thresholds.

[0137] The mapping module 18 holds the map for all roads the vehicles may proceed. As new roads are mapped, a map updater 80 updates the map in the map update process 84 and stores the map in the map store 82. The map update process 84 may also update existing maps through data received from the current vehicle data such as information from the construction handler 64. Thus, the map updater 84 may update existing maps or add additional maps to the map store 82.

[0138] The remote driver module 20 is connected through a gateway 92 to the truck. The gateway 92 includes the software necessary to interpret the data streams that are being passed to and from the station and the truck. The gateway also may connect the operational center 88 to the remote station 90. The operational center 88 can pass information to the remote station 90 for display on the remote station 90, or may communicate via microphone and speakers. Similar to the planning module 14 of the truck software, the remote station 90 passes a trajectory to the control module 16 which gives specific locations and times along the trajectory for the truck to proceed to during the next time interval. The control module 16 may then process the information the same as if the trajectory was sent from the planning module 14.

[0139] When the remote driver module 20 takes control of the vehicle from the AV system, a switch in the control module may switch the input from the AV to the remote station so that the control module responds to the signal from the remote station. The vehicle may also be driven by a driver in the vehicle. Thus, the switch in the control module may pass control signals from the AV system, the remote station, or may not pass any signal so that the mechanical controls within the vehicle are passing the control signals to the actuation components in the vehicle. The control module also may mix the control signals to account for perceptions from the AV stack and input form the remote driver by combining the trajectories of the two signals and choosing an optimal path from both driving systems.

[0140] The operational center 88 may pass information about the vehicle and load to the remote station, and also pass that information to the vehicle for use in the software within the vehicle. This may include instructions for the map module for the route for the vehicle, and other information such as vehicle weight and planned stops on the route.

Data Flow

[0141] FIGS. 3A and 3B depicts a data flow diagrams 300A and 300B schematically showing features of AV maps using satellite images. The features of AV maps using satellite images of FIGS. 3A and 3B may apply to FIGS. 1 , 2, 4, 5A-5C, 6A-6B, and 7A-7K. The data flow diagrams 300A and 300B may enables devices of environment 100 to perform operations 0302 through 0320.

[0142] In operation 0302, the user device 105 may determine its location (or the location of the AV) and determine to pull reference maps and/or satellite images based on that location. Reference maps may be existing sparse maps for a region.

[0143] In operation 0304, the user device 105 may transmit a pull request for satellite images.

[0144] In operation 0306, the AV server 120 may obtain relevant satellite images or request relevant satellite images from satellite image server 130.

[0145] In operation 0308, the AV server 120 may transmit a pull request for relevant satellite images from the satellite image server 130.

[0146] In operation 0310, the satellite image server 130 may retrieve relevant satellite images (e.g., based on the location in operation 0302).

[0147] In operation 0312, the satellite image server 130 may transmit relevant satellite images. [0148] In operation 0314, the AV server 120 may transmit reference maps and/or relevant satellite images to the user device 105.

[0149] In operation 0316, the user device 105 may determine to initiate a satellite image process (e.g., based on connectivity issue, a standard user interface, etc.), receive user inputs for annotation(s), and generate annotated satellite images (see FIGS. 5A-5C, 6A-6B, and 7A- 7K).

[0150] In operation 0318, the user device 105 may transmit annotated satellite images to a convertor module 1050. The convertor module 1050 may be onboard the user device 105, the AV server 120, or the AV 115.

[0151] In operation 0320, the convertor module 1050 may convert annotated satellite images to trajectory/sparse map. For instance, satellite images and associated data (e.g., topographical data) may be processed via machine learning models (e.g., NNs) to identify relevant roadway features (e.g., roads, not road, infrastructure, gradient, and the like). The features may be geolocated in a map to thereby provide a base layer that may be updated over time as the AV 115 traverses the environment. The convertor module 105C may generate a trajectory by following roadways and/or user deviations away from roadways to follow a contour of the earth (if not on a roadway) or the contour of the roadway, and the like.

[0152] In operation 0322, the convertor module 105C may transmit trajectory/sparse map the AV 115.

[0153] In operation 0324, the AV 115 may process the trajectory/sparse map (e.g., upload it to map store 82), obtain environment data from sensors (e.g., sensor module 10), update the sparse map (e.g., map updater 80), and (if needed) the trajectory (e.g., trajectory planner 66), and execute the trajectory (e.g., control module 16).

[0154] In operation 0326, the AV 115 may cache and transmit environment data to the AV server 120.

[0155] In operation 0328, the AV server 120 may update reference maps based on the new environment data and/or detected objects, etc.

[0156] In some aspects, the system may obtain at least one satellite image. The user device(s) 105 may determine its location or the location of the AV(s) 115 and decide to pull reference maps and/or satellite images based on that location. The user device(s) 105 may then transmit a pull request for satellite images. The AV server 120 may obtain relevant satellite images or request relevant satellite images from the satellite image server 130. The satellite image server 130 may retrieve relevant satellite images based on the location determined by the user device(s) 105 and transmit these images to the AV server 120. The AV server 120 may then transmit reference maps and/or relevant satellite images to the user device(s) 105.

[0157] In some cases, the user device(s) 105 may determine to initiate a satellite image process, receive user inputs for annotations, and generate annotated satellite images. The user inputs may be received through an editor interface 105A on the user device(s) 105. The editor interface 105A may facilitate the annotations for the at least one satellite image. The user inputs may include designating a starting point and a destination point on the satellite image. The user device(s) 105 may then transmit the annotated satellite images to a convertor module 105C.

The convertor module 105C may be onboard the user device(s) 105, the AV server 120, or the AV(s) 115.

[0158] In some aspects, the user device(s) 105 may be configured to receive user inputs for obtaining the annotations for the at least one satellite image. The user inputs may be received through an editor interface 105A on the user device(s) 105. The editor interface 105A may facilitate the annotations for the at least one satellite image. The user inputs may include designating a starting point and a destination point on the satellite image. The user device(s) 105 may then transmit the annotated satellite images to a convertor module 105C. The convertor module 105C may be onboard the user device(s) 105, the AV server 120, or the AV(s) 115.

[0159] In some aspects, the operations of the system may include converting the at least one annotated satellite image into a trajectory and a sparse map. This conversion process may be performed by a convertor module 105C. The convertor module 105C may be configured to process the at least one annotated satellite image and generate the sparse map. In some cases, the convertor module 105C may apply image processing techniques and pattern recognition algorithms to identify and label portions of the at least one annotated satellite image.

[0160] In some aspects, the sparse map may be populated with features generated based on the at least one annotated satellite image. These features may be output from machine learning models that detect and label portions of the at least one annotated satellite image. The machine learning models may be trained to recognize and classify various types of features relevant to autonomous vehicle navigation, such as road boundaries, lane markings, traffic signs, and other navigational elements.

[0161] In some cases, the system may determine drivable area properties from the at least one satellite image, including a number of lanes and expected width of lanes. This information may be used to define the drivable area within the sparse map and to guide the trajectory planning process. The drivable area properties may be determined based on the annotations provided by the user, as well as any additional information extracted from the satellite image, such as the presence of road markings or the width of the road.

[0162] In some aspects, the system may populate default values for unknown drivable area properties until updated by real-time sensor data from the environment. For instance, if the satellite image does not provide sufficient detail to determine the exact number of lanes or the width of each lane, the system may assume a default number of lanes or a default lane width. These default values may be selected based on typical road standards or based on the type of road (e.g., highway, residential street, etc.) indicated in the annotations.

[0163] In some cases, the system may update the sparse map with environmental data collected by sensors onboard an AV(s) 115 executing the trajectory. The AV(s) 115 may be equipped with a variety of sensors, such as cameras, lidar, radar, and other sensors, which can collect data about the environment as the AV(s) 115 navigates along the trajectory. This environmental data may include information about road conditions, traffic, weather, and other factors that could affect the AV's navigation. The system may use this environmental data to update the features and properties of the sparse map, thereby ensuring that the sparse map remains accurate and up-to-date as the environment changes.

[0164] In some cases, the operations of the system may include converting the at least one annotated satellite image into a trajectory and a sparse map. This conversion process may be performed by a convertor module 105C. The convertor module 105C may be configured to process the at least one annotated satellite image and generate the sparse map. In some aspects, the convertor module 105C may apply image processing techniques and pattern recognition algorithms to identify and label portions of the at least one annotated satellite image. [0165] In some aspects, the convertor module 105C may employ a combination of edge detection, object recognition, and semantic segmentation to distinguish between different types of terrain and infrastructure elements depicted in the at least one annotated satellite image. For instance, the convertor module 105C may identify drivable areas, such as roads or paths, and non-drivable areas, such as buildings, trees, or bodies of water, within the satellite image. This identification process may involve analyzing the color, texture, shape, and other visual characteristics of different regions within the image to classify them into different categories. [0166] In some cases, the convertor module 105C may delineate the drivable areas within the satellite image by drawing boundaries around them or marking them with specific colors or symbols. These delineated drivable areas may then be incorporated into the sparse map as drivable corridors for the AV(s) 115. The convertor module 105C may also assign attributes to these drivable corridors based on the annotations provided by the user, such as the designated speed limit, road type, and other navigational instructions.

[0167] In some aspects, the convertor module 105C may transmit the trajectory and the sparse map to the AV(s) 115. The AV(s) 115 may process the trajectory and the sparse map, upload them to the map store 82, and use them to navigate through the environment 100. As the AV(s) 115 navigates, it may collect environmental data using its onboard sensor module 10. This environmental data may include information about the current road conditions, traffic, weather, and other factors that could affect the AV's navigation.

[0168] In some cases, the AV(s) 115 may update the sparse map and the trajectory based on the collected environmental data using the map updater 80 and the trajectory planner 66. For instance, if the AV(s) 115 encounters a new road that was not included in the original sparse map, it may add this road to the sparse map. Similarly, if the AV(s) 115 detects a change in the road conditions, such as a road closure or a new traffic pattern, it may adjust the trajectory accordingly.

[0169] In some aspects, the AV(s) 115 may cache the collected environmental data and/or the updated sparse map features for subsequent transmission to the AV server 120. This transmission may occur periodically, such as at the end of each mission, or in real-time, as the environmental data is collected and the sparse map is updated. The AV server 120 may receive this data and integrate it with the existing sparse maps stored in the AV data structure 125, thereby keeping the AV maps up-to-date with the latest environmental changes.

[0170] In some aspects, the convertor module 105C may generate a topological representation and geometric representation of the environment based on the at least one annotated satellite image for any segments that have not previously been mapped. The topological representation may include nodes and edges connecting the nodes, and the nodes may correspond to segments. This topological representation may provide a structured framework for the sparse map, defining the navigable paths within the environment.

[0171] In some cases, the system may perform operations to stitch the sparse map to an existing sparse map to create an extended sparse map of the environment. This stitching process may involve aligning and merging overlapping features from the sparse map and the existing sparse map to ensure continuity and accuracy of the extended sparse map. For instance, the stitching module 115A within the AV(s) 115 may be configured to align the endpoints of new map segments with the corresponding endpoints of existing segments in the existing sparse maps 115B. This alignment ensures that there is a continuous and coherent path for the AV(s) 115 to follow, without abrupt discontinuities or misalignments that could confuse the vehicle's navigation system.

[0172] In some aspects, the stitching process may involve performing topological stitching of the sparse map to the existing sparse map. This topological stitching may include integration of topological information from the sparse map with topological information from the existing sparse map to maintain navigational continuity and logical connectivity within the extended sparse map. For example, if a new road intersects with an existing road at a right angle, the stitching process would involve creating a node at the intersection point and defining the possible paths from that node in all directions that a vehicle could travel. This topological stitching ensures that the relationships between different segments of the map are logically consistent, reflecting the actual navigable paths an AV(s) 115 can take.

[0173] In some cases, the stitching process may be underpinned by a set of heuristics and validations that take into account the physical capabilities and limitations of the AV(s) 115, as well as the characteristics of the road. These heuristics help to ensure that the stitched map is not just geometrically and topologically accurate, but also practical and safe for the AV(s) 115 to navigate. For instance, the stitching module 115A may be configured to validate the stitched map against a set of predefined rules and constraints, such as maximum speed, turning radius, and road type. If the stitched map does not meet these validation criteria, the stitching module 115A may adjust the map accordingly, ensuring that the AV(s) 115 can safely and effectively navigate the environment.

[0174] In summary, the stitching process is a dynamic and ongoing process that allows for the map to evolve in response to the changing environment, ensuring that the AV(s) 115 has the information it requires to make informed and safe navigation decisions. This adaptability is a cornerstone of the sparse mapping approach, which prioritizes the inclusion of the minimum amount of information necessary for effective autonomous navigation, thereby reducing computational overhead and increasing efficiency.

[0175] In some aspects, the AV(s) 115 may update the sparse map with environmental data collected by sensors onboard the AV(s) 115 executing the trajectory. The AV(s) 115 may be equipped with a variety of sensors, such as cameras, lidar, radar, and other sensors, which can collect data about the environment as the AV(s) 115 navigates along the trajectory. This environmental data may include information about road conditions, traffic, weather, and other factors that could affect the AV's navigation. The system may use this environmental data to update the features and properties of the sparse map, thereby ensuring that the sparse map remains accurate and up-to-date as the environment changes.

[0176] In some cases, the AV(s) 115 may cache the collected environmental data and/or the updated sparse map features for subsequent transmission to the AV server 120. This transmission may occur periodically, such as at the end of each mission, or in real-time, as the environmental data is collected and the sparse map is updated. The AV server 120 may receive this data and integrate it with the existing sparse maps stored in the AV data structure 125, thereby keeping the AV maps up-to-date with the latest environmental changes.

[0177] In some aspects, the system may perform operations to stitch the sparse map to an existing sparse map to create an extended sparse map of the environment. This stitching process may involve aligning and merging overlapping features from the sparse map and the existing sparse map to ensure continuity and accuracy of the extended sparse map. For instance, the stitching module 115A within the AV(s) 115 may be configured to align the endpoints of new map segments with the corresponding endpoints of existing segments in the existing sparse maps 115B. This alignment ensures that there is a continuous and coherent path for the AV(s) 115 to follow, without abrupt discontinuities or misalignments that could confuse the vehicle's navigation system.

[0178] In some cases, the stitching process may involve performing topological stitching of the sparse map to the existing sparse map. This topological stitching may include integration of topological information from the sparse map with topological information from the existing sparse map to maintain navigational continuity and logical connectivity within the extended sparse map. For example, if a new road intersects with an existing road at a right angle, the stitching process would involve creating a node at the intersection point and defining the possible paths from that node in all directions that a vehicle could travel. This topological stitching ensures that the relationships between different segments of the map are logically consistent, reflecting the actual navigable paths an AV(s) 115 can take.

[0179] In some aspects, the topological stitching accounts for a hierarchical structure of road and/or offroad vehicle pathways, ensuring that topological relationships, including intersections, merges, and splits, are accurately represented in the extended sparse map. The hierarchical structure includes a set of nodes and edges connecting individual nodes of the set of nodes, and the set of nodes correspond to segments of vehicle corridors. The segments of the vehicle corridors are defined by geometric boundaries and features, including a number of lanes, direction of travel, and road type. Stitching the sparse map to the existing sparse map includes selecting a first node of the sparse map closest to the existing sparse map and selecting a second node of the existing sparse map, and connecting the first node to the second node with a new edge.

[0180] In some cases, the stitching process may involve performing geometric stitching of the sparse map to the existing sparse map. This geometric stitching includes alignment and adjustment of geometric features from the sparse map with corresponding geometric features from the existing sparse map to create a continuous map representation. The geometric stitching utilizes geometric transformation algorithms to correct for discrepancies in scale, rotation, and translation between the sparse map and the existing sparse map. This ensures that the extended sparse map accurately represents the physical layout of the environment, providing a reliable basis for the AV(s) 115 to navigate.

[0181] In some aspects, the system may perform operations to update the sparse map with new annotations. These annotations may be provided by a user through the editor interface 105A on the user device(s) 105. The user may input these annotations in response to changes in the environment 100 or to adjust the trajectory of the AV(s) 115. For instance, if the AV(s) 115 encounters an unexpected obstacle or a sudden change in road conditions that were not accounted for in the initial mission plan, the user may quickly annotate a new route on the satellite image. This new route is then validated and sent to the AV(s) 115, which can process the updated information and adjust its trajectory accordingly.

[0182] In some cases, the convertor module 105C may play a central role in the process of updating the sparse map with new annotations. The convertor module 105C may be configured to process the annotated satellite images and convert them into a format that can be integrated into the sparse map. This conversion process may involve applying image processing techniques and pattern recognition algorithms to identify and label portions of the annotated satellite images. The convertor module 105C may also generate a topological representation and geometric representation of the environment based on the annotated satellite images for any segments that have not previously been mapped.

[0183] In some aspects, the geometric stitching process may utilize geometric transformation algorithms to correct for discrepancies in scale, rotation, and translation between the sparse map and the existing sparse maps 115B. This ensures that the extended sparse map accurately represents the physical layout of the environment, providing a reliable basis for the AV(s) 115 to navigate.

[0184] In some cases, the system may generate a request for user intervention when sensors of the AV(s) 115 detect a discrepancy between a sensed version of the environment and the sparse map. This may occur, for example, when the AV(s) 115 encounters an unexpected obstacle or a sudden change in road conditions that were not accounted for in the initial mission plan. In response to this request, the user may input new annotations on the satellite image through the editor interface 105A on the user device(s) 105. These new annotations may provide adjustments to the trajectory to avoid the area causing the discrepancy. [0185] In some aspects, the system may update the sparse map with the new annotations and recalculate a new trajectory for the AV(s) 115 to execute. This updated sparse map and trajectory may then be used by the AV(s) 115 to navigate through the environment 100. The updated sparse map may provide the AV(s) 115 with the latest information about the environment, allowing it to make informed and safe navigation decisions.

[0186] In some cases, the system may perform operations to stitch the updated sparse map to an existing sparse map to form an extended sparse map of the environment. This stitching process may involve aligning and merging overlapping features from the updated sparse map and the existing sparse map to ensure continuity and accuracy of the extended sparse map. The stitching module 115A within the AV(s) 115 may be configured to perform this stitching process, utilizing geometric transformation algorithms to correct for discrepancies in scale, rotation, and translation between the updated sparse map and the existing sparse map.

[0187] In some aspects, the system may update the extended sparse map with environmental data collected by sensors onboard the AV(s) 115 executing the trajectory. This environmental data may include information about changes in road conditions, new construction, or detected obstacles. The system may use this environmental data to further update the features and properties of the extended sparse map, thereby ensuring that the extended sparse map remains accurate and up-to-date as the environment changes.

[0188] In some cases, the AV(s) 115 may execute the trajectory based on the updated extended sparse map. The execution of the trajectory may include dynamically adjusting the trajectory in response to updated environmental features and conditions. This adaptability allows the AV(s) 115 to navigate safely and efficiently through the environment, even in the face of unexpected changes or obstacles.

Routine

[0189] FIG. 4 depicts a flowchart of a routine 400 for AV maps using satellite images. The routine 400 for AV maps using satellite images of FIG. 4 may apply to FIGS. 1 , 2, 3A-3B, 5A-5C, 6A-6B, and 7A-7K.

[0190] The system may perform a routine 400 for creating and updating autonomous vehicle (AV) maps based on satellite images. The routine 400 may start at block 402, where the system may obtain at least one satellite image. This satellite image may be obtained from a satellite image server 130 or other sources of geographic imagery. The satellite image may provide a high-resolution view of a particular geographic area, which can serve as a basis for creating a map for an AV(s) 115.

[0191] At block 404, the system may obtain annotations for the at least one satellite image to obtain at least one annotated satellite image. These annotations may be provided by a user through an editor interface 105A on a user device(s) 105. The user may input these annotations in response to changes in the environment 100 or to adjust the trajectory of the AV(s) 115. For instance, if the AV(s) 115 encounters an unexpected obstacle or a sudden change in road conditions that were not accounted for in the initial mission plan, the user may quickly annotate a new route on the satellite image. This new route is then validated and sent to the AV(s) 115, which can process the updated information and adjust its trajectory accordingly.

[0192] At block 406, the system may convert the at least one annotated satellite image into a trajectory and a sparse map. This conversion process may be performed by a convertor module 105C. The convertor module 105C may be configured to process the at least one annotated satellite image and generate the sparse map. In some aspects, the convertor module 105C may apply image processing techniques and pattern recognition algorithms to identify and label portions of the at least one annotated satellite image.

[0193] At block 408, the system may sense an environment and update the sparse map and the trajectory. The AV(s) 115 may be equipped with a variety of sensors, such as cameras, lidar, radar, and other sensors, which can collect data about the environment as the AV(s) 115 navigates along the trajectory. This environmental data may include information about road conditions, traffic, weather, and other factors that could affect the AV's navigation. The system may use this environmental data to update the features and properties of the sparse map, thereby ensuring that the sparse map remains accurate and up-to-date as the environment changes.

[0194] In some aspects, the system may perform operations to stitch the updated sparse map to an existing sparse map to form an extended sparse map of the environment. This stitching process may involve aligning and merging overlapping features from the updated sparse map and the existing sparse map to ensure continuity and accuracy of the extended sparse map. The stitching module 115A within the AV(s) 115 may be configured to perform this stitching process, utilizing geometric transformation algorithms to correct for discrepancies in scale, rotation, and translation between the updated sparse map and the existing sparse map.

[0195] In some cases, the system may update the extended sparse map with environmental data collected by sensors onboard the AV(s) 115 executing the trajectory. This environmental data may include information about changes in road conditions, new construction, or detected obstacles. The system may use this environmental data to further update the features and properties of the extended sparse map, thereby ensuring that the extended sparse map remains accurate and up-to-date as the environment changes.

[0196] In some aspects, the AV(s) 115 may execute the trajectory based on the updated extended sparse map. The execution of the trajectory may include dynamically adjusting the trajectory in response to updated environmental features and conditions. This adaptability allows the AV(s) 115 to navigate safely and efficiently through the environment, even in the face of unexpected changes or obstacles.

[0197] In some aspects, the user device(s) 105 may be configured to receive user inputs for obtaining the annotations for the at least one satellite image. These user inputs may be received through an editor interface 105A on the user device(s) 105. The editor interface 105A may facilitate the annotations for the at least one satellite image. The user inputs may include designating a starting point and a destination point on the satellite image. The user device(s) 105 may then transmit the annotated satellite images to a convertor module 105C. The convertor module 105C may be onboard the user device(s) 105, the AV server 120, or the AV(s) 115.

[0198] In some cases, the editor interface 105A may include a validation module configured to validate the annotations on the at least one satellite image. This validation module may use heuristics derived from simulations of different vehicle types on various road or offroad conditions. For instance, the validation module may select specific heuristics for validation based on the type of vehicle that will execute the trajectory and/or based on the type of road or terrain indicated by the editor interface 105A or depicted in the at least one satellite image. The simulations may include parameters such as vehicle weight, suspension characteristics, tire or track traction, and environmental conditions to define an operational envelope for vehicle type and road type, and the operational envelope defining the heuristics.

[0199] In some aspects, the validation module may provide feedback to a user through the editor interface 105A. The feedback may include recommendations for altering the trajectory. These recommendations may be selected to improve drivability, safety, and/or efficiency based on the heuristics. For instance, if a user attempts to annotate a turn that is too sharp for the vehicle's actuation limits at the designated speed, the interface will flag this in real-time, prompting the user to adjust the annotation accordingly. This immediate feedback loop ensures that the user can make corrections on the fly, optimizing the map for the vehicle's navigation system without the delay of back-and-forth validation checks.

[0200] In some cases, the system may prompt a user to input the annotations on the at least one satellite image upon determining that an AV(s) 115 may not continue on a current trajectory. This prompting may be based on a prediction that the trajectory will lead to an area without preexisting map data or where the pre-existing map data is determined to be unreliable. For instance, if the AV(s) 115 encounters an unexpected obstacle or a sudden change in road conditions that were not accounted for in the initial mission plan, the user may quickly annotate a new route on the satellite image. This new route is then validated and sent to the AV(s) 115, which can process the updated information and adjust its trajectory accordingly.

[0201] In some aspects, the system may generate a request for user intervention when sensors of the AV(s) 115 detect a discrepancy between a sensed version of the environment 100 and the sparse map. This discrepancy may be triggered by the AV(s) 115 encountering an unexpected obstacle or a sudden change in road conditions that were not accounted for in the initial mission plan. In response to this request, a user may input new annotations on the satellite image through the editor interface 105A on the user device(s) 105. These new annotations may provide adjustments to the trajectory to avoid the area causing the discrepancy.

[0202] In some cases, the user device(s) 105 may be configured to receive user inputs for obtaining the annotations for the at least one satellite image. These user inputs may be received through an editor interface 105A on the user device(s) 105. The editor interface 105A may facilitate the annotations for the at least one satellite image. The user inputs may include designating a starting point and a destination point on the satellite image. The user device(s) 105 may then transmit the annotated satellite images to a convertor module 105C. The convertor module 105C may be onboard the user device(s) 105, the AV server 120, or the AV(s) 115.

[0203] In some aspects, the system may provide the user with options for alternative trajectories based on the at least one satellite image and real-time data from the AV(s) 115. These alternative trajectories may be generated by the system based on the current location of the AV(s) 115, the destination point, and the detected obstacles or changes in road conditions. The user may select one of these alternative trajectories and input annotations to adjust the trajectory accordingly.

[0204] In some cases, the system may update the sparse map with the new annotations and recalculate a new trajectory for the AV(s) 115 to execute. This updated sparse map and trajectory may then be used by the AV(s) 115 to navigate through the environment 100. The updated sparse map may provide the AV(s) 115 with the latest information about the environment, allowing it to make informed and safe navigation decisions.

Satellite Image Interface

[0205] FIGS. 5A, 5B, and 5C depict graphical user interfaces 500A, 500B, and 500C for AV maps using satellite images in a satellite image interface. The graphical user interfaces 500A- 500C for AV maps using satellite images in a satellite image interface of FIGS. 5A-5C may apply to FIGS. 1, 2, 3A-3B, 4, 6A-6B, and 7A-7K.

[0206] In graphical user interface 500A, the system may depict an example of a satellite image 502, e.g., displayed on a user device 105.

[0207] In graphical user interface 500B, the system may, on the user device, display an updated graphical user interface that has a starting point 504. The user may input the starting point 504 or the system may populate a location of the AV 115 on the satellite layer.

[0208] In graphical user interface 500C, the system may generate a trajectory 508 (from the starting point), to a destination point 506. For instance, the trajectory 508 may be input by several user inputs, a user dragging a user input across the display, or the system may automatically generate the trajectory between the two points (e.g. directly, or using nearest roads, or combinations thereof).

[0209] In some cases, the starting point and destination may be imported directly. [0210] In some cases, the trajectory may be deformed to match a road geometry (if within a threshold distance of a roadway), e.g., in the satellite layer.

[0211] In some cases, lanes (and respective boundaries) are generated and then imported to the sparse map. For instance, a detector on the SAT image may detect roadways and/or lanes on roadways and the like. To import to the sparse map, geo-locations of the features may be indexed to the map coordinate system.

[0212] In some cases, the system may determine defaults when actual properties are unknown. For instance, certain defaults may include a default regarding the width/directionality of the drivable area to include at a minimum one lane in each direction. When the AV 115 perceives the environment, the system may update the sparse map to include more or different features over time.

[0213] In some cases, the system may obtain topology of the sparse map and user the topology to determine relationships (e.g., extracting neighboring and predecessor/successor relationships).

[0214] In some cases, the spare map may be validated (e.g., based on a user in the loop). [0215] The map editing tool may be part of the editor interface 105A on the user device(s) 105. The satellite image 502 may provide a high-resolution view of a particular geographic area, which can serve as a basis for creating a map for an AV(s) 115. The map editing tool may provide additional functions that facilitate the map creation process. For instance, the map editing tool may include features such as zoom in/out, pan, rotate, and other image manipulation functions that allow the user to view different parts of the satellite image 502 in detail.

[0216] In some cases, the map editing tool may also include annotation tools that allow the user to mark up the satellite image 502 with various types of annotations. These annotations may include designating a starting point, a destination point, and any intermediate waypoints or segments on the satellite image 502. The user may input these annotations by clicking or tapping on specific locations on the satellite image 502, or by drawing lines or shapes on the image. The map editing tool may also include text input fields, dropdown menus, checkboxes, or other user interface elements that allow the user to input additional information or instructions associated with the annotations.

[0217] In some aspects, the map editing tool may include a "snap to grid" function that automatically aligns the annotations with the underlying grid of the satellite image 502. This function may facilitate the accurate placement of annotations, particularly in cases where the annotations correspond to specific geographic coordinates or features on the satellite image 502. The "snap to grid" function may be particularly useful when annotating roadways or other linear features on the satellite image 502, as it can ensure that the annotations follow the exact path of the roadways.

[0218] In some cases, the map editing tool may include a validation function that checks the annotations for errors or inconsistencies. This validation function may be part of the first validation module 105B on the user device(s) 105. The validation function may use heuristics derived from simulations of different vehicle types on various road or offroad conditions to validate the annotations. If the validation function detects any issues with the annotations, it may provide feedback to the user through the graphical user interface 500A, prompting the user to correct the annotations.

[0219] In some aspects, the map editing tool may include a preview function that allows the user to view a simulated version of the trajectory and sparse map generated based on the annotations. This preview function may provide the user with a visual confirmation of the trajectory and sparse map before they are finalized and sent to the AV(s) 115. The preview function may also allow the user to make any adjustments to the annotations, ensuring that the trajectory and sparse map accurately reflect the intended path for the AV(s) 115.

[0220] In some cases, the map editing tool may include a save function that allows the user to save the annotated satellite image 502 for future reference or modification. The save function may store the annotated satellite image 502 in a local or cloud-based storage system, allowing the user to access and modify the annotated satellite image 502 at a later time. This save function may be particularly useful in scenarios where the map creation process is complex and time-consuming, or where multiple users are collaborating on the same map.

[0221] In some aspects, the user device(s) 105 may be configured to receive user inputs for obtaining the annotations for the at least one satellite image. These user inputs may be received through an editor interface 105A on the user device(s) 105. The editor interface 105A may facilitate the annotations for the at least one satellite image. The user inputs may include designating a starting point 504 on the satellite image. The user device(s) 105 may then transmit the annotated satellite images to a convertor module 105C. The convertor module 105C may be onboard the user device(s) 105, the AV server 120, or the AV(s) 115.

[0222] In some cases, the starting point 504 may be selected by the user by clicking on a specific location on the satellite image displayed on the editor interface 105A. The starting point 504 may represent the location where the AV(s) 115 is expected to begin its mission. The starting point 504 may be represented as a specific geographic coordinate, such as a latitude and longitude, or as a pixel coordinate on the satellite image.

[0223] In some aspects, the starting point 504 may be used as the initial point for generating the trajectory for the AV(s) 115. The trajectory may be a path that the AV(s) 115 is expected to follow during its mission. The trajectory may be generated based on the annotations provided by the user, including the starting point 504, a destination point, and any intermediate waypoints or segments.

[0224] In some cases, the convertor module 105C may convert the annotated satellite image, including the starting point 504, into a sparse map for the AV(s) 115. The sparse map may be a simplified representation of the environment that includes the minimum amount of information that the AV(s) 115 requires to navigate effectively. The sparse map may include features such as road boundaries, lane lines, and other navigational elements that are derived from the annotated satellite image.

[0225] In some aspects, the starting point 504 may be used as a reference point for aligning the sparse map with the existing sparse maps 115B in the AV(s) 115. This alignment may ensure that the sparse map is correctly positioned within the overall map framework of the AV(s) 115, allowing for accurate navigation.

[0226] In some cases, the starting point 504 may also be used as a reference point for stitching together different map segments. The stitching process may involve aligning the endpoints of new map segments with the corresponding endpoints of existing segments, including the starting point 504. This alignment ensures that there is a continuous and coherent path for the AV(s) 115 to follow, without abrupt discontinuities or misalignments that could confuse the vehicle's navigation system.

[0227] In some aspects, the system may automatically generate a trajectory 508 between a starting point 504 and a destination point 506 based on the annotations provided by the user. The trajectory 508 may be a path that the AV(s) 115 is expected to follow during its mission. The trajectory 508 may be generated by the convertor module 105C, which may process the annotated satellite image and generate the trajectory 508. The trajectory 508 may be represented as a series of waypoints or segments that the AV(s) 115 is expected to traverse in a specific order.

[0228] In some cases, the trajectory 508 may be generated based on a vehicle corridor and the starting point 504 and the destination point 506. The vehicle corridor may be a drivable area within the environment 100 that is suitable for the AV(s) 115 to navigate. The vehicle corridor may be defined by the user through the editor interface 105A on the user device(s) 105. The user may input annotations to define the boundaries of the vehicle corridor, such as the edges of a road or the boundaries of a lane. The convertor module 105C may then generate the trajectory 508 by following the vehicle corridor from the starting point 504 to the destination point 506.

[0229] In some aspects, the SAT image annotations may enable the AV(s) 115 to deviate from its current planned trajectory based on dynamic updates. These dynamic updates may be provided by the user through the editor interface 105A on the user device(s) 105. The user may input new annotations on the satellite image in response to changes in the environment 100 or to adjust the trajectory of the AV(s) 115. For instance, if the AV(s) 115 encounters an unexpected obstacle or a sudden change in road conditions that were not accounted for in the initial mission plan, the user may quickly annotate a new route on the satellite image. This new route is then validated and sent to the AV(s) 115, which can process the updated information and adjust its trajectory accordingly.

[0230] In some cases, the system may update the sparse map with the new annotations and recalculate a new trajectory for the AV(s) 115 to execute. This updated sparse map and trajectory may then be used by the AV(s) 115 to navigate through the environment 100. The updated sparse map may provide the AV(s) 115 with the latest information about the environment, allowing it to make informed and safe navigation decisions. The updated trajectory may provide the AV(s) 115 with a new path to follow, allowing it to avoid obstacles or navigate through changed road conditions.

[0231] In some aspects, the system may perform operations to stitch the updated sparse map to an existing sparse map to form an extended sparse map of the environment. This stitching process may involve aligning and merging overlapping features from the updated sparse map and the existing sparse map to ensure continuity and accuracy of the extended sparse map. The stitching module 115A within the AV(s) 115 may be configured to perform this stitching process, utilizing geometric transformation algorithms to correct for discrepancies in scale, rotation, and translation between the updated sparse map and the existing sparse map.

[0232] In some cases, the AV(s) 115 may execute the trajectory based on the updated extended sparse map. The execution of the trajectory may include dynamically adjusting the trajectory in response to updated environmental features and conditions. This adaptability allows the AV(s) 115 to navigate safely and efficiently through the environment, even in the face of unexpected changes or obstacles.

[0233] In some aspects, the system may generate a trajectory 508 from a starting point 504 to a destination point 506 using the graphical user interface 500C. The trajectory 508 may be a path that the AV(s) 115 is expected to follow during its mission. The trajectory 508 may be generated based on the annotations provided by the user, including the starting point 504, a destination point 506, and any intermediate waypoints or segments.

[0234] In some cases, the trajectory 508 may "snap to road" during the generation process of the trajectory, to make trajectories align to road surfaces. This may be particularly useful when the trajectory 508 is being generated for an AV(s) 115 that is expected to navigate primarily on roadways. The "snap to road" function may ensure that the trajectory 508 accurately follows the path of the road, even if the user's annotations are not perfectly aligned with the road. This function may be implemented using a variety of techniques, such as image processing algorithms that detect the boundaries of the road in the satellite image, or machine learning models that have been trained to recognize road features.

[0235] In some aspects, the system may determine association driving surfaces, such as adjacent roads that are not driven by the vehicle. For instance, these may include overpasses and ramps, where other vehicles may be exiting or entering the road. Adding those driving surfaces to the map may allow the perception and planning modules the ability to detect and label other vehicles on other surfaces. For example, a car on the overpass may be identified on the overpass instead of assuming that it will intersect with (e.g., crash down to) the driving surface in front of the AV(s) 115.

[0236] In some cases, the spare map may be validated (e.g., based on a user in the loop). The validation process may involve checking the sparse map against a set of predefined rules or heuristics to ensure that it is accurate and feasible for the AV(s) 115 to navigate. These rules or heuristics may be based on the physical capabilities of the AV(s) 115, such as its maximum speed, turning radius, and sensor range, as well as the characteristics of the environment, such as road type, traffic conditions, and weather. If the sparse map does not meet these validation criteria, the system may provide feedback to the user, prompting them to adjust the annotations or the trajectory 508 accordingly. This validation process may help to ensure that the sparse map is not just geometrically and topologically accurate, but also practical and safe for the AV(s) 115 to navigate.

Map Editor Interface

[0237] FIGS. 6A and 6B depict graphical user interfaces 600A and 600B for AV maps using satellite images in a map editor interface. The graphical user interfaces 600A-600B for AV maps using satellite images in a map editor interface of FIGS. 6A-6B may apply to FIGS. 1 , 2, 3A-3B, 4, 5A-5C, and 7A-7K.

[0238] In graphical user interface 600A, the system may display a satellite image 602 in a map editing tool. The map editing tool may provide additional functions, such as snap to grid, and the like.

[0239] In graphical user interface 600B, the system may generate, from a given starting point, a trajectory 604 up to a desired destination. For instance, the trajectory may “snap to road” during the generation process of the trajectory, to make trajectories align to road surfaces.

[0240] In some cases, the system may determine association driving surfaces, such as adjacent roads that are not driven by the vehicle. For instance, these may include overpasses and ramps, where other vehicles may be exiting or entering the road. Adding those driving surfaces to the map may allow the perception and planning modules the ability to detect and label other vehicles on other surfaces. For example, a car on the overpass may be identified on the overpass instead of assuming that it will intersect with (e.g., crash down to) the driving surface in front of the AV 115.

[0241] In some cases, the spare map may be validated (e.g., based on a user in the loop).

Mission Editor Interface

[0242] FIGS. 7A-7K depict graphical user interfaces 700A-700K for AV maps using satellite images in a mission editor interface.

[0243] Referring to FIG. 7A, in some aspects, the system may display a graphical user interface 700A with a mission editor interface 702 on the user device(s) 105. The mission editor interface 702 may be part of the editor interface 105A and may be configured to facilitate the creation and modification of mission routes and waypoints for the AV(s) 115. The mission editor interface 702 may include a mission creation panel 704, which may provide various tools and options for defining the mission. For instance, the mission creation panel 704 may include an add segment button 704A and an add waypoint button 704B, which allow the user to add new segments and waypoints to the mission route.

[0244] In some cases, the mission editor interface 702 may display a detailed map 706 of the environment 100. The detailed map 706 may be based on the satellite image 502 and may include various features and landmarks that are relevant to the mission. The user may interact with the detailed map 706 to define the mission route, such as by tracing out the route on the map or by placing waypoints at specific locations. The mission editor interface 702 may also allow the user to add annotations to the map, such as labels, notes, or symbols, to provide additional information or instructions for the AV(s) 115.

[0245] In some aspects, the mission editor interface 702 may include a validation module configured to validate the mission route and waypoints. This validation module may use heuristics derived from simulations of different vehicle types on various road or offroad conditions. For instance, the validation module may check if the mission route is feasible given the physical capabilities of the AV(s) 115, such as its maximum speed, turning radius, and sensor range. The validation module may also check if the waypoints are placed at feasible locations, such as on drivable areas or at safe distances from obstacles. If the validation module detects any issues with the mission route or waypoints, it may provide feedback to the user through the mission editor interface 702, prompting the user to adjust the mission route or waypoints accordingly.

[0246] In some cases, the mission editor interface 702 may provide real-time feedback to the user during the map annotation process. For instance, if a user attempts to annotate a turn that is too sharp for the vehicle's actuation limits at the designated speed, the interface will flag this in real-time, prompting the user to adjust the annotation accordingly. This immediate feedback loop ensures that the user can make corrections on the fly, optimizing the map for the vehicle's navigation system without the delay of back-and-forth validation checks.

[0247] In some aspects, the mission editor interface 702 may include a save function that allows the user to save the mission route and waypoints for future reference or modification. The save function may store the mission route and waypoints in a local or cloud-based storage system, allowing the user to access and modify the mission route and waypoints at a later time. This save function may be particularly useful in scenarios where the mission planning process is complex and time-consuming, or where multiple users are collaborating on the same mission. [0248] In some cases, the mission editor interface 702 may include a preview function that allows the user to view a simulated version of the mission route and waypoints before they are finalized and sent to the AV(s) 115. The preview function may provide the user with a visual confirmation of the mission route and waypoints, allowing the user to make any last-minute adjustments if necessary. The preview function may also allow the user to visualize how the AV(s) 115 will navigate the mission route, providing a better understanding of the mission dynamics and potential challenges. [0249] Referring to FIG. 7B, in some aspects, the system may display a segment definition panel 708 on the graphical user interface 700B. The segment definition panel 708 may be part of the editor interface 105A on the user device(s) 105. The segment definition panel 708 may facilitate the creation of corridor boundaries 712 based on a plurality of annotations 710, including a first annotation 710A, a second annotation 710B, and a third annotation 710C. These annotations 710 may include designating a set of segment points on the at least one satellite image. The user may input these annotations 710 by clicking or tapping on specific locations on the satellite image, or by drawing lines or shapes on the image.

[0250] In some cases, the system may automatically generate a vehicle corridor based on the set of segment points. The vehicle corridor may be a drivable area within the environment 100 that is suitable for the AV(s) 115 to navigate. The vehicle corridor may be defined by the user through the editor interface 105A on the user device(s) 105. The user may input annotations to define the boundaries of the vehicle corridor, such as the edges of a road or the boundaries of a lane. The convertor module 105C may then generate the trajectory by following the vehicle corridor from the starting point 504 to the destination point 506.

[0251] In some aspects, the segment definition panel 708 may also include corridor settings 714, which allow the user to adjust various parameters of the vehicle corridor. These corridor settings 714 may include a corridor type setting 714A, a speed limit setting 714B, and a corridor width setting 714C. The user may input these settings based on the characteristics of the road or terrain, the capabilities of the AV(s) 115, and the mission objectives. For instance, the user may set a lower speed limit for a narrow or winding road, or a higher speed limit for a wide and straight highway. The user may also adjust the corridor width to accommodate the size of the AV(s) 115 and the traffic conditions.

[0252] In some cases, the system may validate the corridor settings 714 using a validation module. This validation module may use heuristics derived from simulations of different vehicle types on various road or offroad conditions. For instance, the validation module may check if the corridor width is sufficient for the AV(s) 115 to navigate safely, or if the speed limit is within the operational limits of the AV(s) 115. If the validation module detects any issues with the corridor settings 714, it may provide feedback to the user through the segment definition panel 708, prompting the user to adjust the corridor settings 714 accordingly.

[0253] In some aspects, the system may update the sparse map with the vehicle corridor and the corridor settings 714. The convertor module 105C may process the vehicle corridor and the corridor settings 714, and integrate them into the sparse map. This updated sparse map may then be used by the AV(s) 115 to navigate through the environment 100. The updated sparse map may provide the AV(s) 115 with the latest information about the environment, allowing it to make informed and safe navigation decisions.

[0254] Referring to FIG. 7C, in some aspects, the system may display an updated mission panel 704 on the graphical user interface 700C. The updated mission panel 704 may include a data field 704C for the first segment added in the graphical user interface 700B. The data field 704C may display information about the first segment, such as its length, direction, and associated attributes like speed limit and road type. This information may be derived from the annotations provided by the user through the editor interface 105A on the user device(s) 105. [0255] In some cases, the updated mission panel 704 may include a validate button 704D. The validate button 704D may be configured to initiate a validation process when clicked or tapped by the user. This validation process may be performed by a validation module, which may be part of the editor interface 105A or a separate module within the user device(s) 105 or the AV server 120.

[0256] In some aspects, the validation module may use heuristics derived from simulations of different vehicle types on various road or offroad conditions. These simulations may include parameters such as vehicle weight, suspension characteristics, tire or track traction, and environmental conditions to define an operational envelope for vehicle type and road type. The operational envelope may define the heuristics used by the validation module to validate the annotations and the resulting sparse map and trajectory.

[0257] In some cases, the validation module may select specific heuristics for validation based on the type of vehicle that will execute the trajectory and/or based on the type of road or terrain indicated by the editor interface 105A or depicted in the at least one satellite image. For instance, if the AV(s) 115 is a heavy-duty truck and the road type is a highway, the validation module may select heuristics that are specific to heavy-duty trucks operating on highways. These heuristics may take into account the truck's maximum speed, turning radius, braking distance, and other relevant factors.

[0258] In some aspects, the validation module may provide real-time feedback to the user during the map annotation process. For instance, if a user attempts to annotate a turn that is too sharp for the vehicle's actuation limits at the designated speed, the interface will flag this in realtime, prompting the user to adjust the annotation accordingly. This immediate feedback loop ensures that the user can make corrections on the fly, optimizing the map for the vehicle's navigation system without the delay of back-and-forth validation checks.

[0259] In some cases, the validation module may also provide recommendations for altering the trajectory or the sparse map if they do not satisfy the validation criteria. These recommendations may be displayed on the graphical user interface 700C, allowing the user to make adjustments as suggested by the validation module. This interactive validation process may help to ensure that the sparse map and the trajectory are not just geometrically and topologically accurate, but also practical and safe for the AV(s) 115 to navigate.

[0260] Referring to FIG. 7D, in some aspects, the system may display a graphical user interface 700D (e.g., as an updated version of on the graphical user interface 700B) with a segment definition panel 708 to add additional segments. The segment definition panel 708 may be part of the editor interface 105A on the user device(s) 105. The segment definition panel 708 may facilitate the creation of corridor boundaries 712 based on a plurality of annotations 710, including a first annotation 710A, a second annotation 71 OB, and a third annotation 71 OC. These annotations 710 may include designating a set of segment points on the at least one satellite image. The user may input these annotations 710 by clicking or tapping on specific locations on the satellite image, or by drawing lines or shapes on the image.

[0261] In some cases, the system may automatically generate a vehicle corridor based on the set of segment points. The vehicle corridor may be a drivable area within the environment 100 that is suitable for the AV(s) 115 to navigate. The vehicle corridor may be defined by the user through the editor interface 105A on the user device(s) 105. The user may input annotations to define the boundaries of the vehicle corridor, such as the edges of a road or the boundaries of a lane. The convertor module 105C may then generate the trajectory by following the vehicle corridor from the starting point 504 to the destination point 506.

[0262] In some aspects, the segment definition panel 708 may also include corridor settings 714, which allow the user to adjust various parameters of the vehicle corridor. These corridor settings 714 may include a corridor type setting 714A, a speed limit setting 714B, and a corridor width setting 714C. The user may input these settings based on the characteristics of the road or terrain, the capabilities of the AV(s) 115, and the mission objectives. For instance, the user may set a lower speed limit for a narrow or winding road, or a higher speed limit for a wide and straight highway. The user may also adjust the corridor width to accommodate the size of the AV(s) 115 and the traffic conditions.

[0263] In some cases, the system may validate the corridor settings 714 using a validation module to alert the user to potential issues 716. This validation module may use heuristics derived from simulations of different vehicle types on various road or offroad conditions. For instance, the validation module may check if the corridor width is sufficient for the AV(s) 115 to navigate safely, or if the speed limit is within the operational limits of the AV(s) 115. If the validation module detects any issues with the corridor settings 714, it may provide feedback to the user through the segment definition panel 708, prompting the user to adjust the corridor settings 714 accordingly.

[0264] In some aspects, the validation module may be further configured to recommend modifications to the annotations if the trajectory does not satisfy validation conditions of the validation module. These validation conditions may indicate a trajectory is deemed infeasible or suboptimal for an intended vehicle type or driving conditions. For instance, if a user attempts to annotate a turn that is too sharp for the vehicle's actuation limits at the designated speed, the interface will flag this in real-time, prompting the user to adjust the annotation accordingly. This immediate feedback loop ensures that the user can make corrections on the fly, optimizing the map for the vehicle's navigation system without the delay of back-and-forth validation checks. [0265] In some cases, the validation module provides feedback to a user through the editor interface 105A, and the feedback includes recommendations for altering the trajectory. These recommendations are selected to improve drivability, safety, and/or efficiency based on heuristics. For example, if the validation module detects that a proposed trajectory involves a sharp turn at a high speed, it may recommend that the user adjust the trajectory to include a wider turn or a lower speed limit at the turn. This feedback may be provided in real-time as the user inputs the annotations, allowing the user to make adjustments immediately and ensure that the trajectory is safe and feasible for the AV(s) 115.

[0266] Referring to FIG. 7E, FIG. 7E depicts a graphical user interface 700E displaying a second data field 704E for a second segment.

[0267] Referring to FIG. 7F, in some aspects, the system may display a graphical user interface 700F with a waypoint panel 718 on the user device(s) 105. The waypoint panel 718 may be part of the editor interface 105A and may be configured to facilitate the addition and management of waypoints for the AV(s) 115. The waypoint panel 718 may include a search bar positioned at the top of the interface, allowing users to search by destination name, MGRS, numeric latitude/longitude, or click on the map to add a waypoint.

[0268] In some cases, the waypoint panel 718 may receive user inputs to define a first waypoint 720 in a mission editor interface. The first waypoint 720 may be a specific geographic coordinate that the AV(s) 115 is expected to reach during its mission. The first waypoint 720 may be represented as a specific geographic coordinate, such as a latitude and longitude, or as a pixel coordinate on the satellite image. The user may input the first waypoint 720 by clicking or tapping on a specific location on the satellite image, or by entering the geographic coordinates into a text input field.

[0269] In some aspects, the waypoint panel 718 may include a waypoint type dropdown menu 722, which allows the user to select the type of waypoint, such as position, hold, call for help, or assisted autonomy. The waypoint type dropdown menu 722 may provide a list of predefined waypoint types that the user can select from. Each waypoint type may correspond to a specific behavior or action that the AV(s) 115 is expected to perform when it reaches the waypoint. For instance, a "position" waypoint may indicate a location that the AV(s) 115 is expected to reach, while a "hold" waypoint may indicate a location where the AV(s) 115 is expected to stop and wait for further instructions.

[0270] In some cases, the waypoint panel 718 may also include additional input fields or controls that allow the user to specify other attributes or instructions associated with the waypoint. For instance, the user may input a desired speed at which the AV(s) 115 is expected to travel when approaching or leaving the waypoint, or a specific maneuver that the AV(s) 115 is expected to perform at the waypoint, such as a turn, merge, or stop. The user may also input any other relevant instructions or constraints, such as road type, traffic conditions, or weather conditions, that may affect the AV's navigation at the waypoint.

[0271] In some aspects, the waypoint panel 718 may include a validation function that checks the waypoints for errors or inconsistencies. This validation function may be part of the first validation module 105B on the user device(s) 105. The validation function may use heuristics derived from simulations of different vehicle types on various road or offroad conditions to validate the waypoints. If the validation function detects any issues with the waypoints, it may provide feedback to the user through the waypoint panel 718, prompting the user to adjust the waypoints accordingly.

[0272] Referring to FIG. 7G, in some aspects, the system may display an updated waypoint panel 718 on the graphical user interface 700G to add waypoints, such as a start or a destination waypoint. The updated waypoint panel 718 may be part of the editor interface 105A on the user device(s) 105. The updated waypoint panel 718 may allow the user to add metadata or instructions for the first waypoint 720. The metadata or instructions may include information such as the desired speed at which the AV(s) 115 is expected to travel when approaching or leaving the waypoint, the type of maneuver that is expected (e.g., turn, merge, stop), and any other relevant instructions or constraints. This could include information about road type, such as whether the waypoint is on a paved road or an off-road trail, which would affect the vehicle's speed and handling characteristics.

[0273] In some cases, the user may input the metadata or instructions through various input fields or controls on the waypoint panel 718. For instance, the waypoint panel 718 may include a waypoint type dropdown menu 722, which allows the user to select the type of waypoint, such as position, hold, call for help, or assisted autonomy. The waypoint type dropdown menu 722 may provide a list of predefined waypoint types that the user can select from. Each waypoint type may correspond to a specific behavior or action that the AV(s) 115 is expected to perform when it reaches the waypoint.

[0274] In some aspects, the waypoint panel 718 may also include additional input fields or controls that allow the user to specify other attributes or instructions associated with the waypoint. For instance, the user may input a desired speed at which the AV(s) 115 is expected to travel when approaching or leaving the waypoint, or a specific maneuver that the AV(s) 115 is expected to perform at the waypoint, such as a turn, merge, or stop. The user may also input any other relevant instructions or constraints, such as road type, traffic conditions, or weather conditions, that may affect the AV's navigation at the waypoint.

[0275] In some cases, the waypoint panel 718 may include a validation function that checks the waypoints for errors or inconsistencies. This validation function may be part of the first validation module 105B on the user device(s) 105. The validation function may use heuristics derived from simulations of different vehicle types on various road or offroad conditions to validate the waypoints. If the validation function detects any issues with the waypoints, it may provide feedback to the user through the waypoint panel 718, prompting the user to adjust the waypoints accordingly.

[0276] Referring to FIG. 7H, in some aspects, the system may display an updated mission panel 704 on the graphical user interface 700H to display a third data field 704F for a first waypoint. The updated mission panel 704 may be part of the editor interface 105A on the user device(s) 105. The updated mission panel 704 may include a third data field 704F for the first waypoint. The third data field 704F may display information about the first waypoint, such as its geographic coordinates, desired speed, and associated instructions. This information may be derived from the annotations provided by the user through the editor interface 105A on the user device(s) 105.

[0277] In some cases, the updated mission panel 704 may include a validate button 704D. The validate button 704D may be configured to initiate a validation process when clicked or tapped by the user. This validation process may be performed by a validation module, which may be part of the editor interface 105A or a separate module within the user device(s) 105 or the AV server 120. The validation module may use heuristics derived from simulations of different vehicle types on various road or offroad conditions to validate the waypoints and the resulting sparse map and trajectory.

[0278] In some aspects, the validation module may provide real-time feedback to the user during the map annotation process. For instance, if a user attempts to annotate a turn that is too sharp for the vehicle's actuation limits at the designated speed, the interface will flag this in realtime, prompting the user to adjust the annotation accordingly. This immediate feedback loop ensures that the user can make corrections on the fly, optimizing the map for the vehicle's navigation system without the delay of back-and-forth validation checks.

[0279] In some cases, the updated mission panel 704 may also include options for adding segments to the mission route. These segments may be defined by the user through the editor interface 105A on the user device(s) 105. The user may input annotations to define the boundaries of the segments, such as the edges of a road or the boundaries of a lane. The convertor module 105C may then generate the trajectory by following the segments from the starting point 504 to the destination point 506.

[0280] Referring to FIG. 7I, in some aspects, the system may display a graphical user interface 700I for adding additional waypoints, such as a destination waypoint.

[0281] Referring to FIG. 7J, in some aspects, the system may display a graphical user interface 700J for managing missions. The graphical user interface 700J may be part of the editor interface 105A on the user device(s) 105. The graphical user interface 700J may include a mission assignment user panel 726, which may display a plurality of defined missions 724. Each of the plurality of defined missions 724 may include details such as the mission name, number of waypoints, total dwell time, source, destination, creation date, and creator information. These details may provide users with a clear overview of all active and past missions, allowing them to easily manage and track the progress of each mission.

[0282] In some cases, the mission assignment user panel 726 may allow users to select a mission from the plurality of defined missions 724 to start. The user may select a mission by clicking or tapping on the mission in the mission assignment user panel 726. Upon selection of a mission, the system may initiate the mission, causing the AV(s) 115 to execute the trajectory associated with the selected mission.

[0283] In some aspects, the mission assignment user panel 726 may also allow users to assign missions to specific AV(s) 115. For instance, the user may select an AV(s) 115 from a list of available vehicles, and then assign a mission to the selected AV(s) 115 from the plurality of defined missions 724. This assignment process may be facilitated by drag-and-drop functionality, dropdown menus, or other user interface elements.

[0284] In some cases, the mission assignment user panel 726 may also provide a route preview for the selected mission. The route preview may display a visual representation of the mission route on a map, allowing the user to review the route before assigning the mission to the AV(s) 115. The route preview may include annotations indicating the starting point, destination point, and any intermediate waypoints or segments along the route.

[0285] In some aspects, the graphical user interface 700J may also include a new mission button, which allows users to create new missions. Upon clicking or tapping the new mission button, the system may display a mission creation interface, such as the mission editor interface 702, which allows the user to define a new mission route and waypoints. The new mission may then be added to the plurality of defined missions 724 and made available for assignment to the AV(s) 115.

[0286] In some cases, the system may update the graphical user interface 700J in real-time to reflect changes in the status of the missions or the AV(s) 115. For instance, if a mission is completed or in progress, the system may display their respective statuses.

[0287] Referring to FIG. 7K, in some aspects, the system may a graphical user interface 700K with a mission assignment user panel 726 on the user device(s) 105. The mission assignment user panel 726 may be part of the editor interface 105A and may be configured to facilitate the assignment and management of missions for the AV(s) 115. The mission assignment user panel 726 may display the status of various trucks, allowing users to monitor the progress of each truck and make informed decisions about mission assignments.

[0288] In some cases, the mission assignment user panel 726 may allow users to assign missions to specific trucks. For instance, the user may select a truck from a list of available trucks, and then assign a mission to the selected truck from a plurality of defined missions 724. This assignment process may be facilitated by drag-and-drop functionality, dropdown menus, or other user interface elements. The user may also have the option to assign the same mission to multiple trucks, or to assign different missions to different trucks based on their current status and capabilities.

[0289] In some aspects, the mission assignment user panel 726 may also provide a route preview for the selected mission. The route preview may display a visual representation of the mission route on a map, allowing the user to review the route before assigning the mission to the truck. The route preview may include annotations indicating the starting point, destination point, and any intermediate waypoints or segments along the route. This route preview feature may help the user to understand the complexity and requirements of the mission, and to ensure that the selected truck is capable of executing the mission successfully.

[0290] In some cases, the mission assignment user panel 726 may also include a validation function that checks the mission assignments for errors or inconsistencies. This validation function may be part of the first validation module 105B on the user device(s) 105. The validation function may use heuristics derived from simulations of different vehicle types on various road or offroad conditions to validate the mission assignments. If the validation function detects any issues with the mission assignments, it may provide feedback to the user through the mission assignment user panel 726, prompting the user to adjust the mission assignments accordingly.

[0291] In some aspects, the system may update the mission assignment user panel 726 in realtime to reflect changes in the status of the trucks or the missions. For instance, if a truck completes a mission or encounters an issue during the mission, the system may update the status of the truck on the mission assignment user panel 726. This real-time update feature may allow the user to monitor the progress of the missions and make timely adjustments as needed. [0292] In some aspects, the system may include a pre-validation service that checks if the sparse map would be accepted by the AV(s) 115 before it is sent directly to the AV(s) 115. This pre-validation service may be part of the first validation module 105B on the user device(s) 105. The pre-validation service may use heuristics derived from simulations of different vehicle types on various road or offroad conditions to validate the sparse map. If the pre-validation service detects any issues with the sparse map, it may provide feedback to the user through the editor interface 105A, prompting the user to adjust the annotations accordingly. This pre-validation process may help to ensure that the sparse map is not just geometrically and topologically accurate, but also practical and safe for the AV(s) 115 to navigate.

[0293] In some cases, the pre-validation service may also check if the trajectory would be feasible for the AV(s) 115 to execute. This feasibility check may involve assessing the trajectory against a set of predefined rules or constraints, such as the maximum speed, turning radius, and sensor range of the AV(s) 115, as well as the characteristics of the road or terrain. If the trajectory does not meet these validation criteria, the pre-validation service may provide feedback to the user, prompting them to adjust the trajectory accordingly.

[0294] In some aspects, the system may also include a second validation module 120A on the AV server 120. The second validation module 120A may provide further validations and/or feedback on the sparse map and the trajectory. For instance, the second validation module 120A may use a larger set of validations than the first validation module 105B, taking into account additional factors such as the current traffic conditions, weather conditions, and other dynamic factors that may affect the AV's navigation. The second validation module 120A may also use more sophisticated validation techniques, such as machine learning algorithms or advanced simulation models, to provide more accurate and comprehensive validations. [0295] In some cases, the system may utilize the technology to automatically generate sparse maps, potentially speeding up the map creation process by two orders of magnitude. This automatic map generation may be facilitated by the convertor module 105C, which may be configured to process satellite images and generate sparse maps without the need for manual annotations. The convertor module 105C may use machine learning models that have been trained to recognize and label features relevant to autonomous vehicle navigation, such as road boundaries, lane markings, traffic signs, and other navigational elements. This automatic map generation may greatly reduce the time and effort involved in creating sparse maps, enabling rapid deployment of AV(s) 115 in dynamic and time-sensitive environments.

Computer System

[0296] FIG. 8 depicts an example system that may execute techniques presented herein. FIG. 8 is a simplified functional block diagram of a computer that may be configured to execute techniques described herein, according to exemplary cases of the present disclosure. Specifically, the computer (or “platform” as it may not be a single physical computer infrastructure) may include a data communication interface 860 for packet data communication. The platform may also include a central processing unit (“CPU”) 820, in the form of one or more processors, for executing program instructions. The platform may include an internal communication bus 810, and the platform may also include a program storage and/or a data storage for various data files to be processed and/or communicated by the platform such as ROM 830 and RAM 840, although the system 800 may receive programming and data via network communications. The system 800 also may include input and output ports 850 to connect with input and output devices such as keyboards, mice, touchscreens, monitors, displays, etc. Of course, the various system functions may be implemented in a distributed fashion on a number of similar platforms, to distribute the processing load. Alternatively, the systems may be implemented by appropriate programming of one computer hardware platform. [0297] The general discussion of this disclosure provides a brief, general description of a suitable computing environment in which the present disclosure may be implemented. In some cases, any of the disclosed systems, methods, and/or graphical user interfaces may be executed by or implemented by a computing system consistent with or similar to that depicted and/or explained in this disclosure. Although not required, aspects of the present disclosure are described in the context of computer-executable instructions, such as routines executed by a data processing device, e.g., a server computer, wireless device, and/or personal computer. Those skilled in the relevant art will appreciate that aspects of the present disclosure can be practiced with other communications, data processing, or computer system configurations, including: Internet appliances, hand-held devices (including personal digital assistants (“PDAs”)), wearable computers, all manner of cellular or mobile phones (including Voice over IP (“VoIP”) phones), dumb terminals, media players, gaming devices, virtual reality devices, multiprocessor systems, microprocessor-based or programmable consumer electronics, set-top boxes, network PCs, mini-computers, mainframe computers, and the like. Indeed, the terms “computer,” “server,” and the like, are generally used interchangeably herein, and refer to any of the above devices and systems, as well as any data processor.

[0298] Aspects of the present disclosure may be embodied in a special purpose computer and/or data processor that is specifically programmed, configured, and/or constructed to perform one or more of the computer-executable instructions explained in detail herein. While aspects of the present disclosure, such as certain functions, are described as being performed exclusively on a single device, the present disclosure may also be practiced in distributed environments where functions or modules are shared among disparate processing devices, which are linked through a communications network, such as a Local Area Network (“LAN”), Wide Area Network (“WAN”), and/or the Internet. Similarly, techniques presented herein as involving multiple devices may be implemented in a single device. In a distributed computing environment, program modules may be located in both local and/or remote memory storage devices.

[0299] Aspects of the present disclosure may be stored and/or distributed on non-transitory computer-readable media, including magnetically or optically readable computer discs, hardwired or preprogrammed chips (e.g., EEPROM semiconductor chips), nanotechnology memory, biological memory, or other data storage media. Alternatively, computer implemented instructions, data structures, screen displays, and other data under aspects of the present disclosure may be distributed over the Internet and/or over other networks (including wireless networks), on a propagated signal on a propagation medium (e.g., an electromagnetic wave(s), a sound wave, etc.) over a period of time, and/or they may be provided on any analog or digital network (packet switched, circuit switched, or other scheme).

[0300] Program aspects of the technology may be thought of as “products” or “articles of manufacture” typically in the form of executable code and/or associated data that is carried on or embodied in a type of machine-readable medium. “Storage” type media include any or all of the tangible memory of the computers, processors or the like, or associated modules thereof, such as various semiconductor memories, tape drives, disk drives and the like, which may provide non-transitory storage at any time for the software programming. All or portions of the software may at times be communicated through the Internet or various other telecommunication networks. Such communications, for example, may enable loading of the software from one computer or processor into another, for example, from a management server or host computer of the mobile communication network into the computer platform of a server and/or from a server to the mobile device. Thus, another type of media that may bear the software elements includes optical, electrical and electromagnetic waves, such as used across physical interfaces between local devices, through wired and optical landline networks and over various air-links. The physical elements that carry such waves, such as wired or wireless links, optical links, or the like, also may be considered as media bearing the software. As used herein, unless restricted to non-transitory, tangible “storage” media, terms such as computer or machine “readable medium” refer to any medium that participates in providing instructions to a processor for execution.

Terminology

[0301] The terminology used above may be interpreted in its broadest reasonable manner, even though it is being used in conjunction with a detailed description of certain specific examples of the present disclosure. Indeed, certain terms may even be emphasized above; however, any terminology intended to be interpreted in any restricted manner will be overtly and specifically defined as such in this Detailed Description section. Both the foregoing general description and the detailed description are exemplary and explanatory only and are not restrictive of the features, as claimed.

[0302] As used herein, the terms “comprises,” “comprising,” “having,” including,” or other variations thereof, are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements, but may include other elements not expressly listed or inherent to such a process, method, article, or apparatus.

[0303] In this disclosure, relative terms, such as, for example, “about,” “substantially,” “generally,” and “approximately” are used to indicate a possible variation of ±10% in a stated value.

[0304] The term “exemplary” is used in the sense of “example” rather than “ideal.” As used herein, the singular forms “a,” “an,” and “the” include plural reference unless the context dictates otherwise.

Examples

[0305] Exemplary embodiments of the systems and methods disclosed herein are described in the numbered paragraphs below.

A1. A system for AV maps based on satellite images, the system comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, wherein the operations include: obtain at least one satellite image; obtain annotations for the at least one satellite image to obtain at least one annotated satellite image; convert the at least one annotated satellite image into a trajectory and a sparse map; sense an environment and update the sparse map and the trajectory; and execute the trajectory through the environment.

A2. The system of A1 , wherein the sparse map is populated with features generated based on the at least one annotated satellite image.

A3. The system of A2, wherein the features are output from machine learning models that detect and label portions of the at least one annotated satellite image.

A4. The system of any of A1-A3, wherein the operations further include receiving user inputs, on a user device, to obtain the annotations for the at least one satellite image.

A5. The system of A4, wherein the user inputs are received by an editor interface to facilitate the annotations for the at least one satellite image.

A6. The system of A4, wherein the annotations include designating a starting point and a destination point on the at least one satellite image.

A7. The system of A6, wherein the operations further include automatically generating the trajectory between the starting point and the destination point based on the annotations.

A8. The system of A6, wherein the annotations include designating a set of segment points on the at least one satellite image.

A9. The system of A8, wherein the operations further include automatically generating a vehicle corridor based on the set of segment points.

A10. The system of A9, wherein the operations further include automatically generating the trajectory based on (a) the vehicle corridor and (b) the starting point and the destination point.

A11. The system of any of A1-A10, wherein the operations further include determining drivable area properties from the at least one satellite image, including a number of lanes and expected width of lanes.

A12. The system of A11 , wherein the operations further include populating default values for unknown drivable area properties until updated by real-time sensor data from the environment.

A13. The system of any of A1-A12, wherein the operations further include updating the sparse map with environmental data collected by sensors onboard an autonomous vehicle executing the trajectory.

A14. The system of A13, wherein the operations further include caching the environmental data collected by the autonomous vehicle and/or new sparse map features based on the environmental data for subsequent transmission to a server. A15. The system of any of A1-A14, wherein converting the at least one annotated satellite image into the sparse map includes identifying and delineating drivable and non- drivable areas within the at least one satellite image.

A16. The system of A15, wherein the operations further include utilizing a convertor to process the at least one annotated satellite image and generate the sparse map.

A17. The system of A16, wherein the convertor is configured to apply image processing techniques and pattern recognition algorithms.

A18. The system of A16, wherein the convertor is configured to employ a combination of edge detection, object recognition, and semantic segmentation to distinguish between different types of terrain and infrastructure elements depicted in the at least one annotated satellite image.

A19. The system of A16, wherein the convertor is configured to generate a topological representation and geometric representation of the environment based on the at least one annotated satellite image for any segments that have not previously been mapped.

A20. The system of A19, wherein the topological representation includes nodes and edges connecting the nodes, and the nodes correspond to segments.

A21. The system of any of A1-A20, wherein the operations further include stitching the sparse map to an existing sparse map to create an extended sparse map of the environment.

A22. The system of A21, wherein stitching involves aligning and merging overlapping features from the sparse map and the existing sparse map to ensure continuity and accuracy of the extended sparse map.

A23. The system of A22, wherein the operations further include employing a stitching module that utilizes geo-referencing techniques to accurately overlay the sparse map onto the existing sparse map based on geographic coordinates.

A24. The system of A21 , wherein stitching the sparse map to the existing sparse map includes performing topological stitching of the sparse map to the existing sparse map, and the topological stitching includes integration of topological information from the sparse map with topological information from the existing sparse map to maintain navigational continuity and logical connectivity within the extended sparse map.

A25. The system of A24, wherein the topological stitching is configured to account for a hierarchical structure of road and/or offroad vehicle pathways, thereby ensuring that topological relationships, including intersections, merges, and splits, are accurately represented in the extended sparse map.

A26. The system of A25, wherein the hierarchical structure includes a set of nodes and edges connecting individual nodes of the set of nodes, and the set of nodes correspond to segments of vehicle corridors. A27. The system of A26, wherein the segments of the vehicle corridors are defined by geometric boundaries and features, including a number of lanes, direction of travel, and road type.

A28. The system of A26, wherein stitching the sparse map to the existing sparse map includes selecting a first node of the sparse map closet to the existing sparse map and selecting a second node of the existing sparse map, and connecting the first node to the second node with a new edge.

A29. The system of A21 , wherein stitching the sparse map to the existing sparse map includes performing geometric stitching of the sparse map to the existing sparse map, and the geometric stitching includes alignment and adjustment of geometric features from the sparse map with corresponding geometric features from the existing sparse map to create a continuous map representation.

A30. The system of A29, wherein the geometric stitching utilizes geometric transformation algorithms to correct for discrepancies in scale, rotation, and translation between the sparse map and the existing sparse map.

A31. The system of any of A1-A30, wherein annotating the at least one satellite image is performed by an editor interface, and the editor interface includes a validation module configured to validate the annotations on the at least one satellite image.

A32. The system of A31, wherein the validation module uses heuristics derived from simulations of different vehicle types on various road or offroad conditions.

A33. The system of A32, wherein the validation module selects specific heuristics for validation (a) based on a type of vehicle that will execute the trajectory and/or (b) based on a type of road or terrain indicated by the editor interface or depicted in the at least one satellite image.

A34. The system of A32, wherein the simulations include parameters such as vehicle weight, suspension characteristics, tire or track traction, and environmental conditions to define an operational envelope for vehicle type and road type, and the operational envelope defining the heuristics.

A35. The system of A31 , wherein the validation module is further configured to recommend modifications to the annotations if the trajectory does not satisfy validation conditions of the validation module.

A36. The system of A35, wherein the validation conditions indicate a trajectory is deemed infeasible or suboptimal for an intended vehicle type or driving conditions.

A37. The system of A31 , wherein the validation module provides feedback to a user through the editor interface, and the feedback includes recommendations for altering the trajectory.

A38. The system of A37, wherein the recommendations are selected to improve drivability, safety, and/or efficiency based on heuristics. A39. The system of A31 , wherein the editor interface is on a user device connected to an autonomous vehicle or a network connected to the autonomous vehicle in an operations center.

A40. The system of A39, wherein the user device is configured to transmit the at least one annotated satellite image directly to the autonomous vehicle, and the autonomous vehicle hosts the validation module configured to validate the annotations and, if the annotations are successful passed by the validation module, to proceed with the trajectory for execution.

A41. The system of A39, wherein the validation module is a first validation module, the user device is configured to transmit the at least one annotated satellite image to a cloud-based server, and the cloud-based server comprising a second validation module configured to provide further validations and/or feedback on the annotations.

A42. The system of A41 , wherein the cloud-based server is configured to transmit, if the annotations are successful passed by the second validation module, the trajectory and the sparse map to the autonomous vehicle for execution of the trajectory through the environment.

A43. The system of A41 , wherein the second validation module utilizes a larger set of validations than the first validation module.

A44. The system of any of A1-A43, wherein the operations further include prompting a user to input the annotations on the at least one satellite image upon determining that an autonomous vehicle may not continue on a current trajectory.

A45. The system of A44, wherein prompting is based on a prediction that the trajectory will lead to an area without pre-existing map data or where the pre-existing map data is determined to be unreliable.

A46. The system of A44, wherein prompting is triggered by the autonomous vehicle encountering an unexpected obstacle or condition that indicates a trajectory or route change.

A47. The system of A44, wherein a user is prompted through an editor interface, and the editor interface is configured to display the at least one satellite image and receive the annotations.

A48. The system of A44, wherein the operations further include generating a request for user intervention when sensors of the autonomous vehicle detect a discrepancy between a sensed version of the environment and the sparse map.

A49. The system of A44, wherein the user inputs the annotations to provide adjustments to the trajectory to avoid an area causing the autonomous vehicle to be unable to continue on the current trajectory.

A50. The system of A44, wherein the operations further include providing the user with options for alternative trajectories based on the at least one satellite image and real-time data from the autonomous vehicle. A51. The system of A44, wherein the operations further include updating the sparse map with the annotations and recalculating a new trajectory for the autonomous vehicle to execute.

A52. The system of any of A1-A51 , wherein the operations further include stitching the sparse map to an existing sparse map to form an extended sparse map; and updating the extended sparse map with environmental data collected by sensors onboard an autonomous vehicle executing the trajectory to generate an updated extended sparse map, the environmental data including changes in road conditions, new construction, or detected obstacles.

A53. The system of A52, wherein the operations further include transmitting the updated extended sparse map to a server, and the server is configured to receive and integrate the updated extended sparse map with existing sparse maps.

A54. The system of A53, wherein the server is configured to update tiles within a regional or global map database based on the updated extended sparse map to form updated tiles, the tiles being portions of the regional or global map database that correspond to specific geographic areas.

A55. The system of A54, wherein the updated tiles are made available to other autonomous vehicles for use in future navigation, and the updated tiles reflect latest sensed environmental changes and the annotations.

A56. The system of A52, wherein the operations further include the autonomous vehicle executing the trajectory based on the updated extended sparse map, and executing the trajectory includes dynamically adjusting the trajectory in response to updated environmental features and conditions.

A57. A method for creating maps for autonomous vehicle (AV) navigation based on satellite images, the method comprising: obtaining, by at least one processor, at least one satellite image; obtaining, by the at least one processor, annotations for the at least one satellite image to generate at least one annotated satellite image; converting, by the at least one processor, the at least one annotated satellite image into a trajectory and a sparse map; sensing, by the at least one processor, an environment and updating the sparse map and the trajectory based on sensed environmental data; and executing, by the at least one processor, the trajectory through the environment based on the updated sparse map and trajectory.

A58. A non-transitory computer-readable medium (CRM) comprising instructions that, when executed by at least one processor of an autonomous vehicle (AV) system, cause the at least one processor to perform operations comprising: obtaining at least one satellite image; obtaining annotations for the at least one satellite image to obtain at least one annotated satellite image; converting the at least one annotated satellite image into a trajectory and a sparse map; sensing an environment and updating the sparse map and the trajectory; and executing the trajectory through the environment.

[0306] Other aspects of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the invention disclosed herein. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the invention being indicated by the following claims.

Claims

Claims What is claimed is:

1. A system for AV maps based on satellite images, the system comprising: at least one processor; and at least one memory storing instructions that, when executed by the at least one processor, cause the at least one processor to perform operations, wherein the operations include: obtain at least one satellite image; obtain annotations for the at least one satellite image to obtain at least one annotated satellite image; convert the at least one annotated satellite image into a trajectory and a sparse map; sense an environment and update the sparse map and the trajectory; and execute the trajectory through the environment.

2. The system of claim 1, wherein the sparse map is populated with features generated based on the at least one annotated satellite image.

3. The system of claim 2, wherein the features are output from machine learning models that detect and label portions of the at least one annotated satellite image.

4. The system of claim 1, wherein the operations further include receiving user inputs, on a user device, to obtain the annotations for the at least one satellite image.

5. The system of claim 4, wherein the user inputs are received by an editor interface to facilitate the annotations for the at least one satellite image.

6. The system of claim 4, wherein the annotations include designating a starting point and a destination point on the at least one satellite image.

7. The system of claim 6, wherein the operations further include automatically generating the trajectory between the starting point and the destination point based on the annotations.

8. The system of claim 6, wherein the annotations include designating a set of segment points on the at least one satellite image.

9. The system of claim 8, wherein the operations further include automatically generating a vehicle corridor based on the set of segment points.

10. The system of claim 9, wherein the operations further include automatically generating the trajectory based on (a) the vehicle corridor and (b) the starting point and the destination point.

11. The system of claim 1 , wherein the operations further include determining drivable area properties from the at least one satellite image, including a number of lanes and expected width of lanes.

12. The system of claim 11 , wherein the operations further include populating default values for unknown drivable area properties until updated by real-time sensor data from the environment.

13. The system of claim 1, wherein the operations further include updating the sparse map with environmental data collected by sensors onboard an autonomous vehicle executing the trajectory.

14. A method for creating maps for autonomous vehicle (AV) navigation based on satellite images, the method comprising: obtaining, by at least one processor, at least one satellite image; obtaining, by the at least one processor, annotations for the at least one satellite image to generate at least one annotated satellite image; converting, by the at least one processor, the at least one annotated satellite image into a trajectory and a sparse map; sensing, by the at least one processor, an environment and updating the sparse map and the trajectory based on sensed environmental data; and executing, by the at least one processor, the trajectory through the environment based on the updated sparse map and trajectory.

15. A non-transitory computer-readable medium (CRM) comprising instructions that, when executed by at least one processor of an autonomous vehicle (AV) system, cause the at least one processor to perform operations comprising: obtaining at least one satellite image; obtaining annotations for the at least one satellite image to obtain at least one annotated satellite image; converting the at least one annotated satellite image into a trajectory and a sparse map; sensing an environment and updating the sparse map and the trajectory; and executing the trajectory through the environment.