CN115600481A

CN115600481A - Autonomous path generation with path optimization

Info

Publication number: CN115600481A
Application number: CN202210685861.2A
Authority: CN
Inventors: 何唐从睿
Original assignee: Waymo LLC
Current assignee: Waymo LLC
Priority date: 2021-06-16
Filing date: 2022-06-16
Publication date: 2023-01-13
Also published as: EP4105606A1; US20220402521A1

Abstract

A system includes a storage device and a processing device operatively coupled to the storage device to receive a set of input data including a road map. The road map includes an autonomous vehicle driving path. The processing apparatus is also for determining that an autonomous vehicle driving path is affected by one or more obstacles, identifying a set of candidate paths that avoid the one or more obstacles, each candidate path in the set of candidate paths associated with a cost value, selecting a candidate path from the set of candidate paths having an optimal cost value to obtain a selected candidate path, generating a synthetic scenario based on the selected candidate path, and training a machine learning model to navigate the autonomous vehicle based on the synthetic scenario.

Description

Autonomous path generation with path optimization

Technical Field

This description relates generally to autonomous vehicles. More particularly, the present description relates to enabling autonomous path generation with path optimization.

Background

Autonomous (fully and partially Autonomous) Vehicles (AV) operate by sensing the external environment with a variety of electromagnetic (e.g., radar and optical) and non-electromagnetic (e.g., audio and humidity) sensors. Some autonomous vehicles map a driving path through the environment based on sensed data. The driving path may be determined based on Global Positioning System (GPS) data and road map data. While GPS and road map data can provide information about static aspects of the environment (buildings, street layout, road closures, etc.), dynamic information (such as other vehicles, pedestrians, street lights, etc.) is obtained from sensed data collected at the same time. The accuracy and safety of the driving path and speed range selected by the autonomous vehicle depends on the timely and accurate identification of the various objects present in the driving environment, as well as the ability of the driving algorithm to process information about the environment and provide the correct instructions to the vehicle control and drive system.

Disclosure of Invention

In one implementation, a system is disclosed that includes a memory device and a processing device coupled to the memory device. The processing device will receive a set of input data including a road map. The road map includes an autonomous vehicle driving path. The processing apparatus is also for determining that the autonomous vehicle driving path is affected by one or more obstacles, identifying a set of candidate paths that avoid the one or more obstacles, each candidate path in the set of candidate paths associated with a cost value, selecting a candidate path from the set of candidate paths having an optimal cost value to obtain a selected candidate path, generating a synthetic scenario based on the selected candidate path, and training a machine learning model to navigate the autonomous vehicle based on the synthetic scenario.

In another implementation, a method is disclosed that includes receiving, by a processing device, a first set of input data that includes a road map. The road map includes an autonomous vehicle driving path. The method also includes determining, by the processing device, that the autonomous vehicle driving path is affected by one or more obstacles, identifying, by the processing device, a set of candidate paths that avoid the one or more obstacles, each candidate path in the set of candidate paths associated with a cost value, selecting, by the processing device, a candidate path from the set of candidate paths having an optimal cost value to obtain a selected candidate path, generating, by the processing device, a synthetic scenario based on the selected candidate path, and training, by the processing device, a machine learning model to navigate the autonomous vehicle based on the synthetic scenario.

In yet another implementation, a non-transitory computer-readable storage medium having instructions stored thereon that, when executed by a processing device, cause the processing device to obtain a machine learning model trained using synthetic data for navigating an autonomous vehicle is disclosed. The synthetic data includes synthetic scenes generated based on candidate paths having optimal cost values to avoid one or more obstacles. The non-transitory computer-readable storage medium has stored thereon further instructions that, when executed by the processing device, cause the processing device to identify a set of artifacts within the scene using the machine learning model as the autonomous vehicle travels along the driving path, and modify the driving path according to the artifacts within the scene.

Drawings

The present disclosure is illustrated by way of example, and not by way of limitation, and may be more completely understood by reference to the following detailed description when considered in connection with the accompanying drawings in which:

fig. 1 is a schematic diagram illustrating components of an example autonomous vehicle capable of implementing a composite construction zone (construction zone) according to some implementations of the present disclosure.

Fig. 2 is a schematic diagram illustrating an example system for generating and utilizing composite scenes in accordance with some implementations of the present disclosure.

FIG. 3 is a schematic diagram illustrating conversion of an original road map to a modified road map including composite objects according to some implementations of the present disclosure.

Fig. 4 is a schematic diagram illustrating a framework for generating a composite scene in accordance with some implementations of the present disclosure.

Fig. 5A is a schematic diagram illustrating an example scene configuration, according to some implementations of the present disclosure.

Fig. 5B illustrates a sample dependency graph based on the scenario configuration of fig. 6A in accordance with some implementations of the present disclosure.

6A-6D are schematic diagrams illustrating an example application of a road map solver, according to some implementations of the present disclosure.

FIG. 7 is a schematic diagram illustrating an example system for implementing a road map solver, according to some implementations of the present disclosure.

Fig. 8 is a schematic diagram illustrating an example of discrete path optimization performed to obtain at least one coarse optimization path according to some implementations of the present disclosure.

FIG. 9 is a schematic diagram illustrating a coarse optimization path and a fine optimization path in accordance with some implementations of the present disclosure.

10A-10C are schematic diagrams illustrating examples of continuous path optimization performed to obtain at least one fine optimization path according to some implementations of the present disclosure.

Fig. 11 is a flow diagram of an example method of training a machine learning model of an Autonomous Vehicle (AV) using synthetic scenarios in accordance with some implementations of the present disclosure.

Fig. 12 is a flow diagram of an example method of using a trained machine learning model to enable control of an Autonomous Vehicle (AV), according to some implementations of the present disclosure.

Fig. 13 depicts a block diagram of an example computer device within which a set of instructions for causing a machine to perform any one of the one or more methods discussed herein may be executed, according to some implementations of the disclosure.

Detailed Description

The vehicle travels along a route from a starting location to a destination location. A route includes segments of different grades (e.g., elevation, grade, uphill, downhill) having different lengths. The route also includes segments having different radii of curvature (e.g., winding roads of different lengths and grades). Some route segments are associated with historical data, such as historically windy segments, historically heavily trafficked segments, historically recommended lanes in segments, and so forth.

An Autonomous Vehicle (AV) performs vehicle actions, such as braking, steering, and throttling, to move the AV from a starting location to a destination location along a route. The AV has a planning module that receives route data (e.g., from a server) that includes particular roads that travel from a start location to a destination location. A planning module (also referred to herein as a "route module") receives sensor data from a perception system (e.g., a vehicle sensor) indicative of the location of other objects. The route module generates short time range route data using the sensor data and the route data. The short time range route data includes instructions on how to control the AV within a short time interval (e.g., the next 10 seconds). Short-time-range route data may be generated (e.g., regenerated, refreshed) very frequently (e.g., every 100 milliseconds (ms)). Short-time-horizon route data, by being generated very frequently, may reflect changes in the vehicle or world (e.g., engine degradation, other objects changing course or speed, or a sudden appearance). The route module provides the short time range route data to the motion control module. The motion control module controls the vehicle system for a next time interval (e.g., a next 10 seconds, a next 100 milliseconds) based on the short time horizon planning data (e.g., and the refreshed or regenerated short time horizon planning). The route module continues to generate (e.g., refresh) new short time range route data for subsequent time intervals based on the route data from the sensing system and the current sensor data. The motion control module continues to control the vehicle based on the new short time range planning data.

Construction areas are a type of scenario that AV is currently trying to address. Machine learning models for construction zone understanding with AV may require a large amount of construction zone data with truth labels on how to navigate within the construction zone. Traditionally, construction area data is collected from real-world scenarios ("real construction area data"), and some real construction area data may be tagged by a person for paired construction area connectivity. While such real construction area data may have high fidelity, it may also suffer from limited data size and diversity. Regardless of driving distance, the natural scarcity of real construction area data relative to total driving distance limits the amount of real world data available. Further, manual labeling of the construction area may be important and/or expensive. Therefore, it is difficult to effectively train a machine learning model for AV construction area understanding using real-world construction area data.

Aspects of the present disclosure address the above and other challenges by implementing autonomous path generation with path optimization for synthetic scene data to train a machine learning model for controlling AV (e.g., to predict drivable lanes from loading observations). The synthetic scene data may be used to train a machine learning model for scene understanding without the need for "real" annotated (e.g., tagged) data, and may help to enhance such "real" annotated data. For example, if the composite scene is a composite construction area, composite construction area data may be generated to include object configurations (e.g., composite cones, construction vehicles, construction markers, direction markers, speed limit markers, road blocks, etc.) and line graphs representing "road maps" within the composite construction area. For example, a line graph representing a "road map" may be generated with information including the layout of the construction area, and the object configuration may be generated with information including the true-ground-truth (ground-drive) cone boundary and drivable lanes in the construction area. The layout of the construction area may include the location of construction cones, carriers, constructors, etc.

As described above, the composite scene may be generated by automatically generating a true value annotation (e.g., label) of the composite scene using a road map solver. The road map solver may modify an original road map representing an original layout of the driving path without the configuration of the objects to obtain a modified road map (based on the original layout) representing a changed layout of the driving path. For example, the object configuration may reflect a construction area that blocks at least one path of the original layout, and the changed layout may include the optimal path(s) or detour that should be taken due to construction traffic.

The road map solver may identify the optimal path according to the object configuration. The optimal path may have an optimal cost value. In some implementations, a variety of techniques may be employed to identify the optimal path. For example, a path may be selected using a coarse optimization technique to obtain a coarse optimized path, and a fine optimization technique may be used to modify the coarse optimized path to obtain a fine optimized path to generate a composite scene. The coarse optimization technique may be a discrete path optimization technique employing dynamic programming. The fine optimization technique may be a continuous path optimization technique. For example, a fine optimization technique may be employed using an Iterative Linear Quadratic Regulator (iLQR).

The aspects and implementations disclosed herein provide a number of advantages over the prior art. For example, generating synthetic scenario data that can be used to efficiently train machine learning models for autonomous vehicle operation may increase scale and diversity. In addition, composite construction area data may be generated to configure for a variety of scenario test cases. Examples of synthesizing scene data include, but are not limited to, boosting machine learning models, generating fully controllable test cases, training machine learning models jointly with artificially labeled data, and performing targeted enhancements to long tail cases.

Fig. 1 is a schematic diagram illustrating components of an example Autonomous Vehicle (AV) 100 capable of object classification and tracking using motion patterns in accordance with some implementations of the present disclosure. Fig. 1 illustrates operation of an example autonomous vehicle. Autonomous vehicles may include motorized vehicles (automobiles, trucks, buses, motorcycles, all terrain vehicles, recreational vehicles, any dedicated agricultural or construction vehicle, etc.), aircraft (airplanes, helicopters, drones, etc.), naval vehicles (ships, boats, yachts, submarines, etc.), or any other self-propelled vehicle (e.g., a sidewalk-delivering robotic vehicle) capable of operating in an autonomous driving mode (without or with reduced human input).

Driving environment 110 may include any object (moving or non-moving) located outside of the AV, such as roads, buildings, trees, bushes, sidewalks, bridges, mountains, other vehicles, pedestrians, and so forth. The driving environment 110 may be a city, suburban area, rural area, etc. In some implementations, the driving environment 110 may be an off-road environment (e.g., a farming or agricultural land). In some implementations, the driving environment may be an indoor environment, such as a factory, a transportation warehouse, a hazardous area of a building, or the like. In some implementations, the driving environment 110 may be substantially flat with various objects moving parallel to the surface (e.g., parallel to the earth's surface). In other implementations, the driving environment may be three-dimensional and may include objects (e.g., balloons, leaves, etc.) that are capable of moving in all three directions. In the following, the term "driving environment" should be understood to include all environments in which autonomous movement of the self-propelled vehicle can take place. For example, a "piloting environment" may include any possible flight environment of an aircraft or marine environment of a warship. Objects of the driving environment 110 may be located at any distance from the AV, a short distance from a few feet (or less) to a few miles (or more).

The example AV 100 can include a sensing system 120. Sensing system 120 may include a variety of electromagnetic (e.g., optical) and non-electromagnetic (e.g., acoustic) sensing subsystems and/or devices. References throughout this disclosure to the terms "optical" and "light" should be understood to encompass any electromagnetic radiation (waves) that may be used for object sensing to facilitate autonomous driving, such as distance sensing, speed sensing, acceleration sensing, rotational motion sensing, and the like. For example, "optical" sensing may utilize the light range visible to the human eye (e.g., 380 to 700nm wavelength range), the ultraviolet range (below 380 nm), the infrared range (above 700 nm), the radio frequency range (above 1 m), and the like. In implementations, "optical" and "light" may include any other suitable range of the electromagnetic spectrum.

The sensing system 120 may include a radar unit 126, which may be any system that utilizes radio or microwave frequency signals to sense objects within the driving environment 110 of the AV 100. The radar units may be configured to sense the spatial position of objects (including their spatial dimensions) and their velocity (e.g., using doppler shift techniques). Hereinafter, "speed" refers to both the speed at which the object moves (the velocity of the object) and the direction in which the object moves.

Sensing system 120 may include one or more lidar sensors 122 (e.g., lidar rangefinders), which may be laser-based units capable of determining (e.g., using ToF techniques) a range to an object in driving environment 110. The lidar sensor(s) may utilize shorter electromagnetic wave wavelengths than radio wave wavelengths and may therefore provide higher spatial resolution and sensitivity than radar units. The lidar sensor(s) may include a coherent lidar sensor, such as a Frequency Modulated Continuous Wave (FMCW) lidar sensor. The lidar sensor(s) may use optical heterodyne detection for velocity determination. In some implementations, the functions of the ToF and coherent lidar sensor(s) are combined into a single (e.g., hybrid) unit that is capable of determining both the distance to the reflecting object and the radial velocity of the reflecting object. Such a hybrid unit may be configured to operate in a non-coherent sensing mode (ToF mode) and/or a coherent sensing mode (e.g. a mode using heterodyne detection) or in both modes simultaneously. In some implementations, multiple lidar sensor 122 units may be mounted on the AV, e.g., at different locations spatially separated, to provide additional information about the lateral component of the velocity of the reflecting object (as described in more detail below).

Lidar sensor(s) 122 may include one or more laser sources that generate and emit signals and one or more detectors of signals reflected back from objects. Lidar sensor(s) 122 may include a spectral filter to filter out stray electromagnetic waves having a wavelength (frequency) different from the wavelength (frequency) of the transmitted signal. In some implementations, lidar sensor(s) 122 may include a directional filter (e.g., an aperture, a diffraction grating, etc.) to filter out electromagnetic waves that may reach the detector along a direction different from the retro-reflection direction of the transmitted signal. Lidar sensor(s) 122 may use a variety of other optical components (lenses, mirrors, gratings, optical films, interferometers, spectrometers, local oscillators, etc.) to improve the sensing capabilities of the sensor.

In some implementations, lidar sensor(s) 122 may scan 360 degrees in the horizontal direction. In some implementations, lidar sensor(s) 122 are capable of spatial scanning along both horizontal and vertical directions. In some implementations, the field of view may be up to 90 degrees in the vertical direction (e.g., at least a portion of the area above the horizon is scanned by the lidar signal). In some implementations, the field of view may be a complete sphere (consisting of two hemispheres). For the sake of brevity and clarity, when reference is made generally to "lidar technology", "lidar sensing", "lidar data" and "lidar" in this disclosure, such reference should be understood to also encompass other sensing technologies that typically operate at near infrared wavelengths, but may include sensing technologies that operate at other wavelengths.

The sensing system 120 may also include one or more cameras 129 to capture images of the driving environment 110. The image may be a two-dimensional projection of the driving environment 110 (or a portion of the driving environment 110) onto a projection plane (planar or non-planar, e.g., fisheye) of the camera. Some of the cameras 129 of the sensing system 120 may be video cameras configured to capture a continuous (or quasi-continuous) stream of images of the driving environment 110. In some implementations, the sensing system 120 can also include one or more sonars 128, which can be ultrasonic sonars.

The sensed data obtained by the sensing system 120 may be processed by the data processing system 130 of the AV 100. For example, the data processing system 130 may include a perception system 132. The perception system 132 may be configured to detect and/or track an object in the driving environment 110 and identify the object. For example, the perception system 132 may analyze images captured by the camera 129 and may be capable of detecting traffic light signals, road signs, road layouts (e.g., boundaries of traffic lanes, topology of intersections, designation of parking spaces, etc.), the presence of obstacles, and the like. Perception system 132 may also receive lidar sensing data (coherent doppler data and incoherent ToF data) to determine distances to and velocities (radial, and in some implementations, transverse) of objects in environment 110. In some implementations, the perception system 132 may use lidar data in conjunction with data captured by the camera(s) 129. In one example, the camera(s) 129 may detect images of a scene, such as a construction area scene. Using data from camera(s) 129, lidar data, and the like, perception system 132 is able to determine the presence of objects (e.g., cones) within the scene. For example, the perception system 132 may include a scene recognition component 133. The scene recognition component 133 may receive data from the sensing system 120 and may identify a scene (e.g., a construction area scene) based on the data.

The perception system 132 may also receive information from a GPS transceiver (not shown) configured to obtain information about the location of the AV relative to the earth. The GPS data processing module 134 may use the GPS data in conjunction with the sensed data to help accurately determine the location of the AV with respect to fixed objects of the driving environment 110, such as roads, lane boundaries, intersections, sidewalks, crosswalks, road signs, surrounding buildings, etc., the location of which may be provided by the map information 135. In some implementations, the data processing system 130 may receive non-electromagnetic data, such as sonar data (e.g., ultrasonic sensor data), temperature sensor data, pressure sensor data, meteorological data (e.g., wind speed and direction, precipitation data), and so forth.

The data processing system 130 may also include an environment monitoring and prediction component 136 that can monitor how the driving environment 110 evolves over time, for example, by tracking the position and velocity (relative to the earth) of a moving object. In some implementations, the environment monitoring and prediction component 136 can track changes in the appearance of the environment due to movement of the AV relative to the environment. In some implementations, the environment monitoring and prediction component 136 can make predictions as to how various active objects of the driving environment 110 will be located within a predicted time frame. The prediction may be based on the current position and velocity of the active object, as well as the tracked dynamics of the active object during a certain (e.g., predetermined) time period. For example, based on stored data of object 1 indicating an accelerated motion of object 1 during a previous 3 second time period, environment monitoring and prediction component 136 may infer that object 1 is resuming its motion from a stop sign or red traffic light signal. Thus, given the layout of the roads and the presence of other vehicles, the environmental monitoring and prediction component 136 can predict where object 1 is likely to move within the next 3 or 5 seconds. As another example, based on stored data of object 2 indicating a deceleration motion of object 2 during a previous 2 second time period, environment monitoring and prediction component 136 may infer that object 2 is stopping at a stop sign or a red traffic light signal. Thus, the environment monitoring and prediction component 136 can predict where the object 2 is likely to be within the next 1 or 3 seconds. The environmental monitoring and prediction component 136 can perform periodic checks on the accuracy of its predictions and modify the predictions based on new data obtained from the sensing system 120.

The data generated by the perception system 132, the GPS data processing module 134 and the environmental monitoring and prediction component 136, as well as the synthetic scene data training model 142, may be received by an autopilot system, such as an AV control system (AVCS) 140. The AVCS 140 may include one or more algorithms that control how the AV behaves under a variety of driving conditions and circumstances. Synthetic scenario data training model 142 is a model trained using synthetic data. The composite data may include a composite scene (e.g., a composite construction area scene) generated by a composite data generator using a road map solver, as will be described in further detail herein. For example, the synthetic data generator may be implemented on a non-load system. As another example, the synthetic data generator may be implemented as part of the perception system 132.

For example, the AVCS 140 may include a navigation system for determining a global driving route to a destination point. The AVCS 140 may also include a driving path selection system for selecting a particular path through the direct driving environment, which may include selecting lanes, negotiating traffic congestion, selecting a location for a U-turn, selecting a trajectory for a parking maneuver, and the like. The AVCS 140 may also include an obstacle avoidance system for safely avoiding various obstacles (cones, rocks, flameout vehicles, pedestrians crossing roads, etc.) within the AV driving environment. The obstacle avoidance system may be configured to assess the size of the obstacle and the trajectory of the obstacle (if the obstacle is active), and select an optimal driving strategy (e.g., braking, steering, accelerating, etc.) to avoid the obstacle.

The algorithms and modules of AVCS 140 may generate instructions for various systems and components of the vehicle, such as power system and steering 150, vehicle electronics 160, signals 170, and other systems and components not explicitly shown in fig. 1. The powertrain system and steering 150 may include an engine (internal combustion, electric, etc.), transmission, differential, axles, wheels, steering mechanism, and other systems. Vehicle electronics 160 may include on-board computers, engine management, ignition, communication systems, automotive computers, telematics, in-vehicle entertainment systems, and other systems and components. The signals 170 may include high and low headlights, brake lights, turn signals and backup lights, speakers and alarms, interior lighting systems, dashboard notification systems, passenger notification systems, radio and wireless network transmission systems, and the like. Some of the instructions output by AVCS 140 may be delivered directly to power system and steering 150 (or signal 170), while other instructions output by AVCS 140 are first delivered to vehicle electronics 160, which generates commands to power system and steering 150 and/or signal 170.

In one example, the AVCS 140 may determine to avoid the obstacle identified by the data processing system 130 by decelerating the vehicle until a safe rate is reached, and then steering the vehicle around the obstacle. The AVCS 140 may output commands to the powertrain system and steering 150 (directly or via vehicle electronics 160) to 1) reduce fuel flow to the engine by modifying throttle settings to reduce engine speed, 2) downshift the driveline to a lower gear via an automatic transmission, 3) engage a brake unit to reduce (simultaneously in cooperation with the engine and transmission) vehicle speed until a safe speed is reached, and 4) perform a steering maneuver using the power steering mechanism until an obstacle is safely circumvented. AVCS 140 may then output a command to power system and steering 150 to restore the vehicle's previous rate setting.

FIG. 2 is a schematic diagram illustrating a system 200 for generating and utilizing composite scenes in accordance with some implementations of the present disclosure. In some implementations, system 200 may be included within a non-loader aware system (e.g., a non-loader AV server) that is physically separate from an Autonomous Vehicle (AV). In some implementations, the system 200 can be included in a load-aware system of the AV. As shown, input data 210 is received by a scene compositor 220. The input data 210 may include one or more messages for a real travel segment without a scene. A real travel segment refers to a segment that is actually driven and imaged (e.g., by a camera and/or lidar). For example, the one or more messages may include one or more communication messages (e.g., based on images captured by a camera and/or lidar).

Scene compositor 220 analyzes input data 210 to automatically generate composite scenes. In some implementations, the composite scene includes a composite construction area. As will be discussed in more detail below, depending on the object configuration, a road map solver may be used to generate the composite scene.

In some implementations, the scene compositor 220 includes a data extractor 222 and a compositor 224. The data extractor 222 may extract data of interest from the input data 210 to obtain extracted data. For example, the extracted data may include an original road map including a set of paths, AV tracks, and the like. Extracting the data of interest may include receiving a set of messages for the travel segment, selecting one or more messages of the set of messages to obtain one or more messages of interest with respect to scene composition, and organizing the one or more messages of interest into a set of synchronization frames.

For example, the set of messages may be received as a time ordered list (e.g., by timestamps), and selecting one or more messages may include analyzing the set of messages in chronological order. Each message of interest may have a corresponding type (e.g., pose, local pose, perceptual object, sensor field of view, marker detection result), and each synchronization frame may include each type of message of interest, one message of interest for each type. The timestamps of the messages of interest within a synchronization frame may be close enough that it is reasonable to consider those messages of interest as occurring simultaneously.

The compositor 224 may then use the extracted data to generate a composite scene 230. For example, the sync frames may be received by the compositor 224 to generate a composite scene 230. The use case comprises (1) extracting the autonomous vehicle track to restrict the position of the synthetic construction area; (2) Determining a segment of the original road map on which the composite scene 230 was generated; and (3) provide useful information for synthetic scene generation (e.g., moving/parked vehicles, sensor field of view).

To generate the composite scene 230, the synthesizer 224 may autonomously generate truth annotations (e.g., lane annotations and boundary annotations) for the composite scene 230 based on the original road map and composite scene configuration, and the truth annotations should have a geometry that is sufficiently smooth and reasonable so as not to encounter scene artifacts or objects. For example, where the composite scene 230 is a composite construction area, the truth annotation may indicate that the driver is likely to be traversing the construction area scene and should have a geometry that is sufficiently smooth and reasonable so that it does not hit construction area objects (e.g., cones, construction vehicles, construction logos).

To generate a true-value annotation, a modified road map may be obtained by modifying the original road map in a manner that reflects a possible real scene (e.g., a real construction area scenario). For example, scene semantics and composite object configurations may be defined in the original road map, and depending on the scene semantics and object configurations, the original road map may be modified by shifting the paths and/or merging the paths into adjacent paths. That is, the original road map represents the original layout of the driving route without any indication of the construction area, and the modified road map represents a changed layout (based on the original layout) that reflects the driving route of the construction area to be defined within the composite scene 230 (e.g., when traffic needs to be directed to a different route due to construction). Accordingly, modifying the road map includes synthesizing the true lanes of the scene 230.

In some implementations, the composite object configuration may include placing one or more composite objects into the original road map, and modifying the road map includes synthesizing the true lanes of the scene 230. For example, if the composite object configuration includes a set of cones defining a construction area boundary, a modified road map may be obtained by shifting and/or merging one or more lanes around the construction area boundary. According to implementations described herein, the synthetic scene 230 may reside in any suitable coordinate system. For example, composite scene 230 may be located in a latitude-longitude-altitude (lat-lng-alt) coordinate system. A high-level overview of the process of converting a road map into a modified road map that includes composite objects using the compositor 224 will be described in more detail below with reference to fig. 3.

In some implementations, composite scene 230 may be provided to composite scene viewer 240. The composite scene viewer 240 may view the composite scene 230 by taking a series of "screenshots" of the composite scene 230 from the viewpoint or point of view of the AV to generate a set of data frames 250 comprising one or more object frames. That is, synthetic scene observer 240 may simulate the perceptual processing of a scene by an AV load-aware system (e.g., perceptual system 132 of FIG. 1). For example, the observation frame may be generated by converting the synthetic scene 230 into a local perceptual coordinate frame (e.g., a smooth coordinate frame) of the AV for model training. The visibility test of each synthetic artifact may then be performed according to, for example, the field of view of the sensor or a circle of a predefined radius within which the object is considered visible. Visible objects may be added to the observation frame while invisible objects are not included in the observation frame. Optionally, a marker observation for the colored marker may also be included in the observation frame. Such marker observation may be obtained from a loading module for colored marker detection, or may be synthesized by switching lane markers in a road map. The marked observations may be stored as polylines in the observation frame. The observation frames may be generated from a plurality of viewpoints (including a top view, a perspective view, etc.).

To generate the set of data frames 250, the synthetic scene viewer 240 may receive additional input data. The additional input data may include a stream AV pose and a stream aware field of view. The composite scene viewer 240 can handle a variety of aspects including pose divergence, composite object visibility, and composite data format.

Gesture refers to the definition of AV location. For example, the pose may include one or more of coordinates, roll, pitch, yaw, latitude, longitude, altitude, and the like. With respect to pose divergence (e.g., due to positional divergence for navigating a composite scene that is not present in the real log), a composite scene (e.g., a composite construction area) can be divided into two categories: composite scenes that affect AV progression and composite scenes that do not affect AV progression. Being synthetic, synthetic scenes do not really exist in real logs. Thus, the pose of the AV may need to be modified, which introduces a divergence of poses. Generally, a limited amount of gesture divergence is acceptable (e.g., within about 5 meters). Too much gesture divergence may make the perception unrealistic.

With respect to synthetic object visibility, to simulate what may be observed from a load-aware system (e.g., the aware system 132 of fig. 1), the pose and the perceptual field of view of the AV may be used at a particular timestamp to filter out synthetic objects that are not visible to the AV (e.g., are occluded and/or too far from the AV).

With respect to the composite data format, at least two forms of data may be generated. For example, one form of data may be used for simulated loading usage and another form of data may be used for training and testing machine learning models. For stowage use, the composite cone may be packed in the same format as a stowage real cone and published at a similar frequency (e.g., from about 10Hz to about 15 Hz) as an alps _ main. The load usage data may be stored in a suitable format (e.g.,. Clf log format).

The set of data frames 250 may be used to generate a set of target output data for model training. For example, the set of target output data generated based on the set of data frames 250 may include a message (e.g., a communication message) with an injection marker and/or a perception object, a tensoflow example, and/or the like.

The set of data frames 250 and the set of target output data may then be provided to a training engine 260 to train a machine learning model for navigating the AV, such as the synthetic scene data training model 142. For example, a machine learning model may be trained to learn how to react to particular scenarios (e.g., construction areas) encountered while operating the AV. The synthetic scene data training model 142 may then be used by the AVCS 140 to control how the AV behaves in a variety of driving situations and environments.

FIG. 3 depicts a schematic diagram 300 illustrating conversion of an original road map to a modified road map including a composite object in accordance with some implementations of the present disclosure. For example, the schematic diagram 300 may reflect a composite construction area scene. However, such implementations should not be considered limiting.

As shown, the schematic diagram 300 depicts an original road map 310 having a first road map lane 312-1 and a second road map lane 312-2. A first road map path 314-1 associated with an AV driving path within the first road map lane 312-1 and a second road map path 314-2 associated with an AV driving path within the second road map lane 312-2 are shown. For purposes of this illustrative example, road map paths 314-1 and 314-2 travel in the same direction to simulate that traffic should move in the same direction within each of road map lanes 312-1 and 312-2. However, in other implementations, one of the road map paths 314-1 or 314-2 may travel in the opposite direction to simulate that traffic should move in the opposite direction.

The diagram 300 further depicts an original road map 310, denoted 320, with defined composite scene semantics and object configuration. A number of synthetic artifacts (artifacts) or objects 322 have been placed to define regions within the synthetic scene. For example, the composite artifact 322 may represent a plurality of cones placed along the boundary of the composite construction area.

The diagram 300 further depicts a modified road map 330 obtained by modifying the original road map according to scene semantics and object configuration (e.g., composite object 322). In this illustrative example, to simulate how path changes occur to simulate a composite construction area, corresponding portions of road map path 314-2 from original road map 310 are shifted and merged into road map path 314-1 to generate modified second path 332. That is, the modified second path 332 is generated after the object configuration is defined.

The original road map itself may not be designed to be modifiable. To modify the original road map, the original road map may be represented by a variable version of the original road map or a variable road map. A variable road map is a data structure representing a path map at a higher level. New paths may be attached to points on an existing graph, existing paths may be disabled, etc. Building blocks of the variable road map are referred to as abstract paths. An abstract path is a data structure that defines a one-dimensional (1D) space, and stores attributes of a composite construction area at multiple locations of a road map (e.g., using offsets from any suitable reference location). Examples of such attributes include, but are not limited to, path center location, path heading, distance to left/right boundary, speed limits, drivability, and the like. The abstract path data structure may have many derived classes. One derived class is called a "road map path" and represents the unchanged road map path in the original road map. The path attributes may be derived from the original road map. Another derived class is called a "composition path" and represents a path modified during scene composition. The composite path attributes may be specified during path creation.

During scene generation, the scene compositor 220 may implement random sampling and a road map solver. Random sampling generates scene configuration and semantics without lane labels, and the road map solver autonomously generates true-value lane annotations (e.g., labels). In some implementations, a probabilistic programming language is used to enable random sampling. Using a probabilistic programming language, a programmatic synthetic scenario generation process for any scenario type can be supported. After the scene compositor 220 has generated one or more composite scenes, the road map solver may autonomously generate lane annotations. In some implementations, in the context of a construction area, the road map solver may also come from a main deformed lane depending on the construction area placed within the scene. Further details regarding a probabilistic programming language and roadmap solver will now be described below with reference to FIG. 4.

Fig. 4 is a schematic diagram illustrating a framework 400 for generating composite scenes in accordance with some implementations of the present disclosure. As shown, framework 400 may include a scene configuration generator 410 configured to generate a scene configuration 420. To generate real and diverse scene data (e.g., construction area data), samples may be obtained from a library of scene types (e.g., construction areas) that simulate a "real scene". Such scene generation can be very difficult to model. For example, on one hand, data scarcity may limit the use of modern deep generative models, while on the other hand, large changes in the real world may not be captured with a single rule-based system.

To address this issue, the scenario configuration generator 410 may generate a scenario configuration 420 based on a scenario type library. For example, a scene type library may include a number of scene types (or script types), each corresponding to a scene compositor, and a weighted combination of each scene type may approximate the distribution of all scenes to obtain a scene configuration.

The distribution of scene types may be generated by a plurality of scene synthesizers. The scene compositor may include at least some of the following features: (1) Each scene compositor models a corresponding distribution of a particular subset of its scene types (e.g., "lane shift due to construction area along road edges," "small construction area within intersection," etc.); (2) Each scene compositor shares a common interface so that they can be substituted for each other, or freely combined with weights; (3) Each scene synthesizer is independent of each other, so that many entities can contribute to the scene synthesizer simultaneously; and (4) sufficient functionality to enable the addition of new scene types to the scene type library.

In some implementations, the scenario configuration generator 410 may implement a Probabilistic Programming Language (PPL). The PPL is a lightweight framework that may be nested in any suitable general-purpose programming language (e.g., C + +). The PPL may include two parts: (1) Definition of scene distributions, and (2) a generic sampling engine that samples from a library of scene types according to the appropriate scene distribution. The scene distribution is defined as a function, where an a priori distribution ("a priori") and a set of conditions or constraints may be specified (e.g., by a user). The prior distribution is a spatial relationship map with randomness that can be constructed with a library in a code library (e.g., a math/geometry/road map library). As will be described in further detail herein, the PPL module 410 may employ random spatial reference and conditional sampling.

The set of constraints may include one or more hard constraints and/or one or more soft constraints. The hard constraints may be user-defined boolean expressions. Each hard constraint will hold for all sampling scenarios. Soft constraints are used to ensure that a certain variable obeys a user-defined distribution. Soft constraints associate variables in the scene generation process with probability density functions (continuous or discrete), and for all sampled scenes, the distribution of the variables will obey the probability density functions.

Instead of specifying the scene configuration 420 directly (e.g., by setting the coordinates of each object in the scene), the scene may be described in a spatial relationship (e.g., a random spatial relationship). The spatial relationship may define a generation flow of the scene. One benefit of expressing a scene in this manner is that once defined, the scene may be generated at any suitable location. This feature improves the generalization ability of machine learning models trained on synthetic scene data. For a target scenario, the model may learn to process the scenario in a location-agnostic manner (in any city, town, etc.). Example spatial relationships will now be described below with reference to fig. 5A-5B.

Fig. 5A is a schematic diagram illustrating an example scene configuration 500A in accordance with some implementations of the present disclosure. Scene configuration 500A is illustratively depicted as a construction area scene. However, any suitable scene configuration may be obtained in accordance with implementations described herein.

As shown, the scene configuration 500A includes a boundary curve 510 and a construction area 520 at a location relative to the boundary curve 510. For example, the boundary curve 510 may be a curve corresponding to a curb (curb). In this example, the construction area 520 is rectangular. However, according to implementations described herein, the construction region 520 may be implemented in any suitable shape. Reference points 530 along the boundary curve 510 are sampled along the curve. Then, a normal vector 540-1 and a tangent vector 540-2 corresponding to the reference point 530 may be queried. Based on vectors 540-1 and 540-2, a set of parameters for construction region 520 may be sampled. The parameters of the construction area may include, for example, the center of the construction area 520, represented as center point 525, the orientation of the construction area 520, the width of the construction area 520, and the length of the construction area 520. As shown by normal vector 540-1 and tangential vector 540-2, center 525 may be offset along the normal direction, oriented in the tangential direction. Many objects may be placed along the construction area 520. In this example, the first cone: cone 1 550-1 is placed at a first corner of the construction area 520 and the second cone: cone 2550-2 is placed at a second corner of the work area 520. The location of cone 1 550-1 and cone 2550-2 may be determined from a set of parameters (e.g., dimensions) of the construction area 520. Once defined, the scene configuration 500A may be placed in any suitable location. For example, the scene configuration 500A may be placed anywhere in relation to boundary curves (roads, lanes, curbs, etc.).

In the "real world," the two cones 550-1 and 550-2 may represent the construction area 520 as the area where the construction or work area vehicles are present. The parameters of the construction area 520 may correspond to physical parameters (e.g., dimensions) of the construction or work area vehicle (e.g., length, width, and orientation of the construction or work area vehicle). Further, the right edge of the construction area 520 may be defined by other vehicles (e.g., parallel parked vehicles) located near the boundary curve 510.

Fig. 5B is a dependency graph 500B of a scenario configuration 500A according to some implementations of the present disclosure. Dependency graph 500B includes boundary curve node 560 corresponding to boundary curve 510, reference point node 570 corresponding to reference point 540, construction region node 580 corresponding to construction region 530, cone 1 node 590-1 corresponding to cone 1530-1, and cone 2 node 590-2 corresponding to cone 2-2.

Real scenes (e.g., construction area scenes) may have large internal differences in their appearance. Even if a single type of scene is created twice in the same place, the results may differ. For example, in a construction area, noise in object placement (e.g., cones), size of construction vehicles, preference of construction workers, and the like. Such different results may result. Such intra-class differences (e.g., within a single scene type) can be captured by the composite data to generalize the machine learning model. For example, intra-class differences may be resolved by adding randomness (e.g., random shape, size, and orientation) to the spatial relationship.

Referring back to FIG. 4, after the scene configuration 420 is obtained, the road map solver component 430 implements a road map solver. The road map solver may be used to autonomously generate truth annotations ("annotations") 440 according to the scene configuration 420. For example, the annotations 460 may include lane annotations (e.g., lane labels). The road map solver component 430 can receive information including polygons, road edges, etc., that can be used to obtain a modified road map. That is, depending on the scene semantics and object configuration within the scene configuration 420, the road map solver component 430 may solve the modified road map by morphing or modifying the original road map. According to implementations described herein, the road map solver component 430 can implement any suitable method for generating the annotations 440 from the master. As described above, the annotation 440 may include an identification of the driving path associated with the modified road map.

6A-6D are schematic diagrams 600A-600D illustrating generating annotations including identification of driving paths associated with a modified road map according to some implementations of the present disclosure. For example, annotations including an identification of a driving path associated with the modified road map can be generated by a road map solver, such as the road map solver component 430 of fig. 4. In FIG. 6A, a schematic diagram 600A is shown that includes paths 610-1 through 610-4. Additional paths 620 (e.g., a near road, a turning lane, a ramp, etc.) are shown connecting path 610-1 and path 610-4. That is, the diagram 600A corresponds to the original road map. In fig. 6B, the schematic 600B is shown to include an area 630 and a path 640 (e.g., right turn to a parallel road, a detour road, etc.). Region 630 may be an obstacle affecting the path of the original road map (e.g., a construction area). In fig. 6C, an optimization process is initiated to identify a set of candidate paths that avoid barrier region 630, where each candidate path is associated with a cost value. For example, the paths of the original road map are modified according to the area 630 to generate a set of candidate paths that avoid the obstacle area 630. From the set of candidate paths, the candidate path 650-1 with the best cost value is selected to replace the affected path 610-3. In FIG. 6D, in addition to path 650-1, new paths 650-2 through 650-4 are generated (using an optimization process) to replace affected paths 610-2 and 610-4 (e.g., via deformed paths 610-2 and 610-4). Path 650-4 merges into path 610-1. Accordingly, an optimization procedure is performed to solve for a path that can circumvent the blockage caused by the region 630.

FIG. 7 is a schematic diagram illustrating an example system 700 for implementing a road map solver, according to some implementations of the present disclosure. The system 700 may be implemented within a road map solver component, such as the road map solver component 430 of FIG. 4.

As shown, the affected path identification component 730 receives a variable road map ("road map") 710 and a set of regions 720 (e.g., construction regions). The road map 710 and the set of regions 720 may be included in a scene configuration, such as the scene configuration 420 of fig. 4. The set of regions 720 may include polygons.

The affected path identification component 730 may identify the affected regions from the set of regions 720 and identify at least one affected path ("affected path") 740 of the road map 710 from the set of regions 720. Affected path 740 (e.g., path 650-x of fig. 6D) may be identified based on a minimum distance to the affected area.

A two-stage optimization process may be performed based on the affected path 740 to find a path to an avoidance area (e.g., the construction area 630 of fig. 6B-6D). The two-stage optimization process may implement reinforcement learning to find an optimal path that will avoid obstacles (e.g., areas, road edges), attempt to approach the affected path 740, and be smooth.

For example, as shown, the affected path 740 may be received by the discrete path optimization component 750. The discrete path optimization component 750 may perform a coarse optimization to generate at least one coarse optimization path ("coarse optimization path") 760 from the affected paths 740. The goal of the coarse optimization is to provide a suitable initialization for the fine optimization, which will be described in further detail below. The discrete path optimization component 750 may receive additional data 745. The additional data 745 may include additional road map modification information. Examples of data that may be included in the additional data 745 include, but are not limited to, data related to where to place a path closure, data related to which direction to shift the path, data related to where to place a multi-lane shift, and the like. For example, the discrete path optimization component 750 can perform a coarse optimization using a dynamic planning method. Further details regarding the operation of the discrete path optimization component 750 will now be described below with reference to FIG. 8.

Fig. 8 is a schematic diagram 800 illustrating an example of discrete path optimization performed to obtain at least one coarse optimization path according to some implementations of the present disclosure. For example, discrete path optimization may be performed by the discrete path optimization component 750 of FIG. 7.

Diagram 800 shows an original path 810 affected by an area 820 (e.g., a construction area). Accordingly, discrete path optimization will be performed to identify a coarsely optimized path that can replace the original path 810. Therefore, the dynamic planning method can realize that: (1) searching a space; (2) a cost function; and (3) an optimization method.

With respect to the search space, the search space may include paths defined on a discrete grid around the candidate path. Such a grid may have two dimensions: step-a position along the candidate path; point (slot) -for each step, the position at that step along the path vertical direction. Each path in the search space occupies one point at each step, sequentially from the first step to the last step. The path geometry is a polyline connecting the point locations at each step. For example, as shown, a number of steps are defined including step 830 and a number of points are defined including point 840.

With respect to a set of cost functions, the goal of discrete path optimization is to find short and smooth candidate paths in the search space, avoid non-drivable areas (e.g., curbs, construction areas), stay close to the original path, and have the same starting and ending points as the original path. Thus, the cost function may be based on the sum of the lengths of each polyline segment in the path and the sum of the costs for each point location. If a point location falls within a non-drivable region, the cost associated with that point location is infinite. Any point other than the point corresponding to the original path is associated with infinite cost for the start and end points. At each step, the cost increases as the distance between the point location and the candidate path increases.

With respect to the optimization method, given the search space and the cost function, the optimization method is used to find the lowest cost candidate path. For example, the candidate path may be the cheapest path that passes through one point for each waypoint (waypoint) and connects at the start and end points.

The optimization method can be implemented with dynamic programming. For example, a dynamic programming approach may be employed by populating a state value matrix based on the following equation:

state _value (i,j)＝state _cost (i,j)+min _k {action _cost (i,j,k)+state _value (i+1,k)}

where i corresponds to the current step, j corresponds to the point location of the current step i, k corresponds to the point location of the subsequent step i +1, state _value (i, j) corresponds to the minimum cost, state, of the path starting from point j of step i _cost (i, j) cost, action, corresponding to point j in step i _cost (i, j, k) corresponds to the cost, state, of moving from point j to point k _value (i +1,k) corresponds to the minimum cost of the path from point k at step i +1, and min _k () Is minimized in k. Since the value at step i depends on step i +1, the matrix of state values can be filled back starting from the last step. And recording the next optimal point position by the state value matrix. The dynamic planning method can be used to select the cheapest path by taking the best movement of each step from the start (i.e. by obtaining the best point of the recording). In this example, the least expensive path is identified as coarse optimization path 850.

Referring back to fig. 7, the continuous path optimization component 770 receives the coarse optimization path 760 to "smooth" the coarse optimization path 760 and obtain at least one fine optimization path ("fine optimization path") 780. The fine optimization path 780 is found by simulating how the vehicle's path is driven in a real-world environment. Each refined path is stitched together along with the unaffected paths of the road map 710 to form a graphical representation of lane labels ("lane labels") 790 from which the machine-learned path prediction model can retrieve truth data. Further details regarding the coarse optimization path 760 and the fine optimization path 780 are described in further detail below with reference to FIG. 9.

The continuous path optimization component 770 may compute an optimal path by optimizing one or more cost functions. In some implementations, the continuous path optimization component 770 implements a Linear Quadratic Regulator (LQR). For example, the LQR may be Iterative LQR (iLQR). Cost terms that may be included in the cost function include, but are not limited to, strict obstacle rejections (e.g., zones and/or edges), attractions that come to rest near the path 710 and reach the target, and constraints on physical state (e.g., velocity, acceleration). The parameters and weights of the cost term can be found by inverse reinforcement learning from the real vehicle trajectory. For example, inverse reinforcement learning may search for an optimal set of parameters such that when the iLQR is constrained with a cost function, the resulting optimized path most closely resembles a real vehicle path. Further details regarding the operation of the continuous path optimization component 770 will be described in further detail below with reference to fig. 10.

One example of a cost function that can be optimized is an "reach target" cost function. A corresponding cost penalty optimizes the distance between the last point of the trajectory to the target location. The cost may be proportional to the square of the distance.

Another example of a cost function that can be optimized is a "follow candidate path" cost function. A corresponding cost penalty optimizes the deviation of the path from the candidate path. The cost may be proportional to the sum of the least squares distances from each point on the optimized path to the candidate path.

Another example of a cost function is an "obstacle avoidance" cost function. When the optimal path encounters an undrivable area (e.g., curb, construction area), the corresponding cost penalizes the optimal path strictly. The cost may be proportional to the sum of the cost terms for each point on the optimized path. For example, if a point is outside the non-drivable range by a constant interval (margin) (e.g., 2.0 meters), the corresponding cost term may be 0. Otherwise, the cost term increases as a function of how deep the point is within the non-drivable range. For example, the cost term may increase as the square of the signed distance between the point and the polygon defining the undrivable area (i.e., the cost term may increase twice).

Another example of a cost function is a "smooth path" cost function, which constrains the physical state in the path and is therefore reasonable for the AV to drive along. For example, the curvature of the path may be constrained to be small enough so that the AV can handle turns, the acceleration will be flat enough so that there is no use of handbrakes and/or impossible accelerations, etc.

Fig. 9 is a schematic diagram 900 illustrating a coarse optimization path and a fine optimization path in accordance with some implementations of the present disclosure. The diagram 900 includes a coarse optimization path example 910 that illustrates an obstacle 912 (e.g., an area) and a coarse optimization path 920 formed from a number of discrete path segments that traverse the obstacle 912. The outline of the original path 915 through the obstacle 912 is also shown. The diagram 900 also includes a fine optimization path example 920 showing a fine optimization path 922 formed by a continuous path segment through the obstacle 912.

10A-10C are schematic diagrams 1000A-1000C illustrating an example of continuous path optimization performed to obtain at least one fine optimization path according to some implementations of the present disclosure. For example, the graphs 1000A-1000C may represent respective iterations of the iLQR method performed by the continuous path optimization component 770 of fig. 7. As will be described, the continuous path optimization may be performed in a rolling manner to enable a fixed time frame regardless of the length of the target path, thereby improving path stability. Each subsequent iteration may be performed to improve a cost function associated with the path.

In FIG. 10A, a schematic diagram 1000A shows a discrete path 1010 having a start point 1012 and an end point or target 1014. A first iteration of the iLQR method is performed to obtain a first intermediate path segment 1020 having a start point 1012 and an end point 1022 corresponding to a first intermediate path segment target according to a cost function.

In fig. 10B, diagram 1000B shows a second iteration of the iLQR method, which is performed to obtain a second intermediate path segment 1030 having a start point at some progression along the first intermediate path segment 1020 and an end point 1032 corresponding to the second intermediate path segment target according to a cost function. The second intermediate path segment 1030 may be generated from a given distance along the first intermediate path segment 1020. For example, the given distance may be expressed as a percentage progression along the first intermediate path segment 1020.

In FIG. 10C, a diagram 1000C illustrates a final iteration of the iLQR method, which is performed to obtain a fine optimization path 1040, according to a cost function. The fine optimization path 1040 begins at a start point 1012 and ends at an end point or target 1022. Any suitable number of additional iterations (not shown) of the iLQR method may be performed between the second iteration and the last iteration to complete the fine optimization path 1040.

Fig. 11 is a flow diagram of an example method 1100 of training a machine learning model of an Autonomous Vehicle (AV) using synthetic scenarios in accordance with some implementations of the present disclosure. Method 1100 may be performed by processing logic that may comprise hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuits, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. For example, the processing logic may be included in a non-load system. Although shown in a particular order or sequence, the order of the processes may be modified unless otherwise indicated. Thus, the illustrated implementations should be understood to be examples only, and the illustrated processes may be performed in a different order, and some processes may be performed in parallel. Additionally, in various implementations, one or more processes may be omitted. Thus, not all processes are required for every implementation. Other process flows are also possible.

At operation 1102, processing logic receives a set of input data including a road map with an autonomous vehicle driving path. The road map may correspond to a data structure representing a one-dimensional space having a set of attributes to be queried. For example, the set of attributes may include at least one of: path center position, path run, distance to left/right boundary, speed limit, and drivability. The set of input data may also include messages for actual travel segments without scenes.

At operation 1104, processing logic determines that the autonomous vehicle driving path is affected by one or more obstacles. For example, the obstacle may be an area (e.g., a construction area), an edge (e.g., a road edge), and so forth. In some implementations, determining that the autonomous vehicle driving path is affected by the one or more obstacles includes generating a scene configuration of a road map using a Probabilistic Programming Language (PPL), and identifying the one or more obstacles from the scene configuration.

At operation 1106, processing logic identifies a set of candidate paths that avoid the one or more obstacles, each candidate path in the set of candidate paths associated with a cost value. For example, each candidate path in the set of candidate paths may have a respective set of inputs for a cost function that generates a respective cost value.

At operation 1108, processing logic selects a candidate path from the set of candidate paths having the optimal cost value to obtain a selected candidate path (e.g., candidate path 650-1, 650-2, 650-3, or 650-4 of FIG. 6D). In some implementations, the candidate paths are selected using discrete path optimization to obtain a coarse optimized path. For example, obtaining the coarse optimization path may include employing a dynamic planning method.

At operation 1110, processing logic generates a composite scene based on the selected candidate paths. In some implementations, the composite scene includes a composite construction area. In some implementations, generating the composite scene includes modifying the selected candidate paths using continuous path optimization to obtain fine optimized paths (e.g., paths 1040 of fig. 10C), and generating the composite scene based on the fine optimized paths. For example, modifying the coarse optimization path may include employing iLQR or other suitable continuous path optimization method. Obtaining the composite scene may include modifying an autonomous vehicle driving path of the road map to obtain a modified composite path of the modified road map having a true-value lane label. Modifying the composite path may include shifting to and/or merging into the second composite path of the modified road map.

At operation 1112, processing logic trains the machine learning model to navigate the autonomous vehicle based on the synthetic scene. The machine learning model may generate an output that may be used by the autonomous vehicle to identify a scene (e.g., a construction area), and thus enable the autonomous vehicle to modify its heading along the path according to the scene. For example, if the scene is a construction area, the autonomous vehicle may modify its route to follow the detour route (e.g., lane segmentation and/or merging) by identifying construction area objects (e.g., cones) that delineate the detour route. For example, training a machine learning model may include generating a set of training input data including a set of data frames from a composite scene, obtaining a set of target output data (e.g., truth annotations or labels) for the set of training input data, and training the machine learning model based on the set of training input data and the set of target output data. The set of target output data may include at least one of a message or a tensorflow instance with an injection marker and/or a perception object. Further details regarding operations 1102-1112 are described above with reference to FIGS. 1-10.

Fig. 12 is a flow diagram of an example method 1200 of using a trained machine learning model to enable control of an Autonomous Vehicle (AV) in accordance with some implementations of the present disclosure. Method 1200 may be performed by processing logic that may comprise hardware (e.g., processing device, circuitry, dedicated logic, programmable logic, microcode, hardware of a device, integrated circuits, etc.), software (e.g., instructions run or executed on a processing device), or a combination thereof. For example, the processing logic may be included within a control system of an AV (e.g., AVCS 140). Although shown in a particular order or sequence, the order of the processes may be modified unless otherwise indicated. Thus, the illustrated embodiments should be understood to be merely examples, and the illustrated processes may be performed in a different order, and some processes may be performed in parallel. Additionally, in various implementations, one or more processes may be omitted. Thus, not all processes are required for every implementation. Other process flows are also possible.

At operation 1202, processing logic obtains a machine learning model trained using synthetic data for navigating an Autonomous Vehicle (AV). The machine learning model may be a machine learning model trained in the manner described above with reference to fig. 1-11.

At operation 1204, processing logic receives detection results including a set of artifacts within the scene while the AV is traveling along the driving path. For example, the detection results may be received from an upstream module of the AV. In some implementations, the set of artifacts may specify lane closures and/or lane modifications that require AV detours. For example, if the scene is a construction area scene, the set of artifacts may include construction area artifacts (e.g., cones) for guiding the vehicle around the construction area.

At operation 1206, processing logic causes the driving path to be modified according to a set of artifacts within the scene. For example, processing logic may determine a detour (e.g., lane path and/or offset) relative to the driving path according to the objects identified within the scene, and may cause the AV to adjust its route according to the detour.

Fig. 13 depicts a block diagram of an example computer device 1300 in which a set of instructions for causing a machine to perform any one of the one or more methods discussed herein may be executed, according to some implementations of the disclosure. The example computer device 1300 may be connected to other computer devices in a LAN, an intranet, an extranet, and/or the internet. Computer device 1300 may operate in the client-server network environment in the identity of a server. Computer device 1300 may be a Personal Computer (PC), a set-top box (STB), a server, a network router, switch or bridge, or any device capable of executing a set of instructions (sequential or otherwise) that specify actions to be taken by that device. Moreover, while only a single example computer device is illustrated, the term "computer" includes any collection of computers that individually or jointly execute a set (or multiple sets) of instructions to perform any one or more of the methodologies discussed herein. In some implementations, the computer device 1300 is an AV server 150. In some implementations, AV 101 includes computer device 1300 (e.g., AVCS 140 is computer device 1300).

Example computer device 1300 may include a processing device 1302 (also referred to as a processor or CPU) that may include processing logic 1303, a main memory 1304 (e.g., read Only Memory (ROM), flash memory, dynamic Random Access Memory (DRAM) such as Synchronous DRAM (SDRAM), etc.), a static memory 1306 (e.g., flash memory, static Random Access Memory (SRAM), etc.), and a secondary memory (e.g., data storage device 1318) that may communicate with each other via a bus 1330.

Processing device 1302 represents one or more general-purpose processing devices such as a microprocessor, central processing unit, or the like. More specifically, the processing device 1302 may be a Complex Instruction Set Computing (CISC) microprocessor, reduced Instruction Set Computing (RISC) microprocessor, very Long Instruction Word (VLIW) microprocessor, processor implementing other instruction sets, or processors implementing a combination of instruction sets. The processing device 1302 may also be one or more special-purpose processing devices such as an Application Specific Integrated Circuit (ASIC), a Field Programmable Gate Array (FPGA), a Digital Signal Processor (DSP), network processor, or the like.

Example computer device 1300 may also include a network interface device 1308, which may be communicatively coupled to a network 1320. Example computer device 1300 may also include a video display 1310 (e.g., a Liquid Crystal Display (LCD), touch screen, or Cathode Ray Tube (CRT)), an alphanumeric input device 1312 (e.g., a keyboard), a cursor control device 1314 (e.g., a mouse), and a sound signal generation device 1316 (e.g., a speaker).

The data storage device 1318 may include a computer-readable storage medium (or, more particularly, a non-transitory computer-readable storage medium) 1328 on which is stored one or more sets of executable instructions 1322.

The executable instructions 1322 may also reside, completely or at least partially, within the main memory 1304 and/or within the processing device 1302 during execution thereof by the example computer device 1300, the main memory 1304 and the processing device 1302 also constituting computer-readable storage media. The executable instructions 1322 may also be transmitted or received over a network via the network interface device 1308.

While the computer-readable storage medium 1328 is illustrated in fig. 13 as a single medium, the term "computer-readable storage medium" should be taken to include a single medium or multiple media (e.g., a centralized or distributed database, and/or associated caches and servers) that store the one or more sets of VM operating instructions. The term "computer-readable storage medium" includes any medium that is capable of storing or encoding a set of instructions for execution by the machine and that cause the machine to perform any one or more of the methodologies described herein. The term "computer readable storage medium" includes, but is not limited to, solid-state memories, and optical and magnetic media.

Some portions of the preceding detailed descriptions have been presented in terms of algorithms and symbolic representations of operations on data bits within a computer memory. These algorithmic descriptions and representations are the means used by those skilled in the data processing arts to most effectively convey the substance of their work to others skilled in the art. An algorithm is here, and generally, considered to be a self-consistent sequence of operations leading to a desired result. The operations are those requiring physical manipulations of physical quantities. Usually, though not necessarily, these quantities take the form of electrical or magnetic signals capable of being stored, combined, compared, and otherwise manipulated. It has proven convenient at times, principally for reasons of common usage, to refer to these signals as bits, values, elements, symbols, characters, terms, numbers, or the like.

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. The present disclosure may relate to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage systems.

The present disclosure also relates to an apparatus for performing the operations herein. This apparatus may be specially constructed for the intended purposes, or it may comprise a general purpose computer selectively activated or reconfigured by a computer program stored in the computer. Such a computer program may be stored in a computer readable storage medium, such as, but is not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random Access Memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions, and each coupled to a computer system bus.

The algorithms and displays presented herein are not inherently related to any particular computer or other apparatus. Various general-purpose systems may be used with programs in accordance with the teachings herein, or it may prove convenient to construct a more specialized apparatus to perform the method. The structure for a variety of these systems will be set forth in the description below. In addition, this disclosure is not described with reference to any particular programming language. It will be appreciated that a variety of programming languages may be used to implement the teachings of the disclosure as described herein.

The present disclosure may be provided as a computer program product or software which may include a machine-readable medium having stored thereon instructions which may be used to program a computer system (or other electronic devices) to perform a process according to the present disclosure. A machine-readable medium includes any mechanism for storing information in a form readable by a machine (e.g., a computer). In some implementations, a machine-readable (e.g., computer-readable) medium includes a machine (e.g., computer) -readable storage medium, such as read only memory ("ROM"), random access memory ("RAM"), magnetic disk storage media, optical storage media, flash memory devices, and so forth. The word "example" or "exemplary" is used herein to mean serving as an example, instance, or illustration. Any aspect or design described herein as "exemplary" or "exemplary" is not necessarily to be construed as preferred or advantageous over other aspects or designs. Rather, use of the word "example" or "exemplary" is intended to present concepts in a concrete fashion. As used in this application, the term "or" is intended to mean an inclusive "or" rather than an exclusive "or". That is, unless specified otherwise or clear from context, "X comprises a or B" is intended to mean any of the natural inclusive permutations. That is, if X comprises A; x comprises B; or X includes A and B, then "X includes A or B" is satisfied under any of the foregoing circumstances. In addition, the articles "a" and "an" as used in this application and the appended claims may generally be construed to mean "one or more" unless specified otherwise or clear from context to be directed to a singular form. Furthermore, the terms "embodiment," "one embodiment," "some embodiments," "implementation," "one implementation," "some implementations," and the like, as used throughout may or may not represent the same embodiment or implementation. One or more embodiments or implementations described herein may be combined in a particular embodiment or implementation. The terms "first," "second," "third," "fourth," and the like, as used herein, are intended as labels to distinguish between different elements and not necessarily have an ordinal meaning according to their numerical designation.

In the foregoing specification, implementations of the present disclosure have been described with reference to specific example implementations thereof. It will be evident that various modifications may be made thereto without departing from the broader spirit and scope of implementations of the disclosure as set forth in the following claims. The specification and drawings are, accordingly, to be regarded in an illustrative rather than a restrictive sense.

Claims

1. A system, comprising:

a storage device; and

a processing device, operatively coupled to the storage device, for:

receiving a set of input data comprising a road map comprising an autonomous vehicle driving path;

determining that an autonomous vehicle driving path is affected by one or more obstacles;

identifying a set of candidate paths that avoid the one or more obstacles, each candidate path in the set of candidate paths associated with a cost value;

selecting a candidate path having an optimal cost value from the set of candidate paths to obtain a selected candidate path;

generating a synthetic scene based on the selected candidate path; and

the machine learning model is trained to navigate the autonomous vehicle based on the synthetic scene.

2. The system of claim 1, wherein the composite scene is a composite construction area.

3. The system of claim 1, wherein the set of input data further comprises a message for a true travel segment without a scene.

4. The system of claim 1, wherein the selected candidate path is a modified autonomous vehicle driving path comprising at least one of: a path of a second autonomous vehicle driving path shifted to or merged into the road map.

5. The system of claim 1, wherein the selected candidate path is a coarse optimization path, wherein the processing device further modifies the selected candidate path using continuous path optimization to obtain a fine optimization path, and wherein the synthetic scenario is generated based on the fine optimization path.

6. The system of claim 5, wherein to select a candidate path, the processing device is to obtain a coarse optimization path by employing a dynamic planning method.

7. The system of claim 5, wherein to modify the coarse optimization path, the processing device employs an Iterative Linear Quadratic Regulator (iLQR).

8. The system of claim 1, wherein the processing device is further to:

generating a set of training input data comprising a set of data frames from the combined scene; and

a set of target output data for the set of training input data is obtained, wherein the machine learning model is trained using the set of training input data and the set of target output data.

9. The system of claim 8, wherein the set of target output data includes at least one of: a message with an injected marker or perceptual object, or a tensorflow example.

10. A method, comprising:

receiving, by a processing device, a first set of input data comprising a road map, the road map comprising an autonomous vehicle driving path;

determining, by a processing device, that an autonomous vehicle driving path is affected by one or more obstacles;

identifying, by a processing device, a set of candidate paths that avoid the one or more obstacles, each candidate path in the set of candidate paths associated with a cost value;

selecting, by the processing device, a candidate path having an optimal cost value from the set of candidate paths to obtain a selected candidate path;

generating, by a processing device, a composite scene based on the selected candidate path; and

training, by a processing device, a machine learning model to navigate an autonomous vehicle based on a synthetic scene.

11. The method of claim 10, wherein the composite scene is a composite construction area.

12. The method of claim 10, wherein the set of input data further comprises a message for a real travel segment without a scene.

13. The method of claim 10, wherein the selected candidate path is a modified autonomous vehicle driving path comprising at least one of: a path shifted to or merged with a second autonomous vehicle driving path of the road map.

14. The method of claim 10, further comprising modifying, by a processing device, the selected candidate path using continuous path optimization to obtain a fine optimized path, wherein the selected candidate path is a coarse optimized path, and wherein a synthetic scenario is generated based on the fine optimized path.

15. The method of claim 14, wherein selecting a candidate path comprises obtaining a coarse optimization path by employing a dynamic planning method.

16. The method of claim 14, wherein modifying the coarse optimization path comprises employing an Iterative Linear Quadratic Regulator (iLQR).

17. The method of claim 10, further comprising:

generating, by a processing device, a set of training input data comprising a set of data frames from the combined scene;

obtaining, by a processing device, a set of target output data for the set of training input data, wherein the machine learning model is trained using the set of training input data and the set of target output data.

18. The method of claim 17, wherein the set of target output data includes at least one of: message with injected markup or perceptual object, or tensorflow example.

19. A non-transitory computer readable storage medium having stored thereon instructions that, when executed by a processing device, cause the processing device to:

obtaining a machine learning model trained using synthetic data for navigating an autonomous vehicle, wherein the synthetic data comprises synthetic scenarios generated based on candidate paths having optimal cost values for avoiding one or more obstacles;

identifying a set of artifacts within a scene using a trained machine learning model while an autonomous vehicle is traveling along a driving path; and

modifying the driving path according to the set of artifacts within the scene.

20. The non-transitory computer-readable storage medium of claim 15, wherein the scene is a construction area.