Classifications

• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
  • B60W30/18—Propelling the vehicle
  • B60W30/18009—Propelling the vehicle related to particular drive situations
  • B60W30/18145—Cornering
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W60/00—Drive control systems specially adapted for autonomous road vehicles
  • B60W60/001—Planning or execution of driving tasks
  • B60W60/0027—Planning or execution of driving tasks using trajectory prediction for other traffic participants
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W30/00—Purposes of road vehicle drive control systems not related to the control of a particular sub-unit, e.g. of systems using conjoint control of vehicle sub-units
  • B60W30/18—Propelling the vehicle
  • B60W30/18009—Propelling the vehicle related to particular drive situations
  • B60W30/18159—Traversing an intersection
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N20/00—Machine learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/08—Learning methods
  • G06N3/09—Supervised learning
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V20/00—Scenes; Scene-specific elements
  • G06V20/50—Context or environment of the image
  • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
  • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W2554/00—Input parameters relating to objects
  • B60W2554/40—Dynamic objects, e.g. animals, windblown objects
  • B60W2554/402—Type
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W2554/00—Input parameters relating to objects
  • B60W2554/40—Dynamic objects, e.g. animals, windblown objects
  • B60W2554/404—Characteristics
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W2554/00—Input parameters relating to objects
  • B60W2554/80—Spatial relation or speed relative to objects
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W2556/00—Input parameters relating to data
  • B60W2556/10—Historical data
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W2556/00—Input parameters relating to data
  • B60W2556/25—Data precision
• • B—PERFORMING OPERATIONS; TRANSPORTING
  • B60—VEHICLES IN GENERAL
  • B60W—CONJOINT CONTROL OF VEHICLE SUB-UNITS OF DIFFERENT TYPE OR DIFFERENT FUNCTION; CONTROL SYSTEMS SPECIALLY ADAPTED FOR HYBRID VEHICLES; ROAD VEHICLE DRIVE CONTROL SYSTEMS FOR PURPOSES NOT RELATED TO THE CONTROL OF A PARTICULAR SUB-UNIT
  • B60W2556/00—Input parameters relating to data
  • B60W2556/45—External transmission of data to or from the vehicle
  • B60W2556/50—External transmission of data to or from the vehicle of positioning data, e.g. GPS [Global Positioning System] data
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/044—Recurrent networks, e.g. Hopfield networks
  • G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/045—Combinations of networks
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
  • G06N3/00—Computing arrangements based on biological models
  • G06N3/02—Neural networks
  • G06N3/04—Architecture, e.g. interconnection topology
  • G06N3/0464—Convolutional networks [CNN, ConvNet]

Definitions

• FIG. 3 illustrates example detail of a detection component and a tracking component of a machine-learned model according to some of the technologies disclosed herein.
• a bounding box may indicate, among other things, a location of an object in the environment at to, an orientation of the object at to, a size of the object, or the like.
• the object data may indicate a classification and/or type of the object, such as whether the object is a dynamic object that is capable of movement (e.g., a vehicle, motorcycle, bicycle, pedestrian, animal, etc.) and/or a static object (e.g., abuilding, road surface, tree, sign, barrier, curb, parked vehicle, etc.).
• the object data may indicate other information associated with the object, such as a velocity of the object at to, a confidence associated with the object, or the like.
• the machine-learned model may include an obj ect tracking portion.
• the object tracking portion may be configured to perform multiple subprocesses to determine trajectories traversed by the detected objects during a period of time from to-N-to, where “N” represents any integer greater than or equal to one.
• the object tracking portion may include, among other things, a location estimation subprocess, a track-location association subprocess, a trajectory estimation subprocess, and/or a track storage subprocess.
• the vehicle may be controlled based at least in part on the tracked object data. For instance, at least a portion of the tracked object data may be used as in input to another system associated with the vehicle, such as a prediction system, planning system, or the like.
• this disclosure is also directed to techniques for end-to-end training of a machine-learned model that includes multiple portions (e.g., stages, subprocesses, components, etc.).
• a machine-learned model is trained end-to-end, outputs of the machine-learned model are used to train individual portions of the machine-learned model, which may otherwise exist as independent machine-learned models.
• the intermediary outputs of one portion of the machine-learned model may be specifically tailored for use as inputs by other portions of the machine-learned model.
• a method associated with end-to-end training of a machine-learned model may include techniques of receiving sensor data representing a vehicle traversing an environment.
• the sensor data may comprise stored log data associated with the vehicle.
• the sensor data/log data may be image data, lidar data, radar data, time of flight data, or the like.
• the sensor data may be a time-ordered collection of image frames representing the sensor data associated with the environment, such that a first frame represents the environment at a first time, a second frame represents the environment at a second time, and so forth.
• the techniques may include receiving an output from the machine-learned model.
• the output may include predicted tracked object data that includes, among other things, the predicted bounding box and the predicted trajectory.
• the ground truth data and the predicted tracked object data may be compared to determine whether differences exist between the ground truth data and predicted tracked object data. In some instances, if a difference is determined to meet or exceed a threshold difference, a parameter of the machine-learned model may be altered to minimize the difference. In some instances, a parameter of one or more portions of the machine-learned model may be altered.
• the objects 108(1) and 108(2) may be referred to herein collectively as “objects 108.”
• the sensor component s) may include lidar sensors, radar sensors, ultrasonic transducers, sonar sensors, location sensors (e.g., global positioning component (GPS), compass, etc.), inertial sensors (e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.), cameras (e.g., RGB, IR, intensity, depth, etc.), wheel encoders, microphones, environment sensors (e.g., temperature sensors, humidity sensors, light sensors, pressure sensors, etc.), time of flight (ToF) sensors, etc.
• GPS global positioning component
• inertial sensors e.g., inertial measurement units, accelerometers, magnetometers, gyroscopes, etc.
• cameras e.g., RGB, IR, intensity, depth, etc.
• wheel encoders e.g., microphones, environment sensors (e.g
• the obj ect data 116 may, in some instances, be used in whole or in part as an input to a tracking component 122 of the machine-learned model 112.
• the tracking component 122 may also receive, as an input, stored tracking data associated with the objects 108(1) and 108(2), which is discussed in further detail below.
• the tracking component 122 may determine, and the machine-learned model 112 may output, tracked object data 124.
• the tracked object data 124 may include the top-down data 118 representing the view of the environment 104 from the top-down perspective, with movement data indicative of object movement.
• the tracked object data 124 may also include trajectories traversed by the objects in the environment 104, such as the trajectory 126(1) traversed by the object 108(1) and the trajectory 126(2) traversed by the object 108(2).
• the tracked object data 124 may, in some examples, be forwarded to a prediction component 128 executing on the computing device(s) 110.
• the prediction component 128 includes one or more machine-learned models that are trained to make predictions about the objects in the environment 104.
• the prediction component 128 may determine prediction data 130 associated with the objects based at least in part on some or all of the tracked object data 124.
• the prediction data 130 may include the top-down data 118 representing the view of the environment 104 from the top-down perspective. Additionally, or alternatively, the prediction data 130 may indicate one or more prediction(s) associated with the objects. For instance, the prediction(s) 132(1) and 132(2) associated with the objects 108(1) and 108(2), respectively, include trajectories that the objects 108(1) and 108(2) are predicted to traverse in the environment 104 during a future period of time.
• the prediction data 130 may be forwarded as an input to a planner component 134 of the vehicle 102 that is executing on the computing device(s) 110.
• the planner component 134 includes one or more machine-learned models and/or other algorithms that are configured to determine a planned trajectory for the vehicle 102 to follow through the environment 104. As such, the planner component 134 may determine the planned trajectory of the vehicle 102 based at least in part on the tracked object data 124 and/or the prediction data 130.
• any one of the sensor data 106, the object data 116, the tracked object data 124, or the prediction data 130 may be formatted as a multi-channel image, where individual channels of the multi-channel image may represent a selection of information.
• a first channel of a multi-channel image may include bounding box(es) associated with object(s)
• a second channel of the multi-channel image may include velocity information associated with the object(s)
• a third channel of the multi-channel image may include environmental data (e.g., surface or lane markings, traffic light information, etc.), and so forth.
• multiple different channels of the multichannel image may include a portion of the same information.
• a first channel and a second channel of the multi-channel image may each include bounding boxes associated with the same objects in an environment.
• a channel can be represented as a color and/or a layer in a three-dimensional image stack, for example.
• the LSTM component 316 may receive data and other information associated with the objects/features, such as a feature tensor, finite differences, and an output from the MLP component 312, and determine one or more trajectory -lev el representation(s) 318.
• the MLP refinement component 320 may utilize the trajectory level representation(s) 318 to determine score(s), position offset(s), and other data 328.
• the methods 400 and 500 are illustrated as collections of blocks in logical flow graphs, which represent sequences of operations that can be implemented in hardware, software, or a combination thereof.
• the blocks represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations.
• computer-executable instructions include routines, programs, objects, components, data structures, and the like that perform particular functions or implement particular abstract data types.
• the order in which the operations are described is not intended to be construed as a limitation, and any number of the described blocks can be combined in any order and/or in parallel to implement the processes. In some embodiments, one or more blocks of the process may be omitted entirely.
• the methods 400 and 500 may be combined in whole or in part with each other or with other methods.
• the method 400 includes performing a first action.
• the first action can include determining that the machine-learned model is a trained machine-learned model. Additionally, or alternatively, the first action can include sending the machine-learned model to the vehicle for use by the vehicle to traverse an environment. In some examples, the first action can include validating the performance of the machine-learned model in a simulation or with additional sensor data.
• the method 400 includes altering a parameter of the first portion of the machine-learned model. For instance, a parameter of the detection component 114 or the tracking component 122 may be altered, as well as, or in the alternative, subprocesses performed by those components.
• altering the parameter may comprise altering a software component of the first portion of the model or retraining the first portion of the model based at least in part on the difference.
• the first data determined by the first portion of the model may be adjusted such that, in future predictions, the first data is tailored for use as an input for the second portion of the machine- learned model.
• the vehicle computing device(s) 604 can include processor(s) 616 and memory 618 communicatively coupled with the processor(s) 616.
• the memory 618 of the vehicle computing device(s) 604 stores a localization component 620, a perception component 622, a prediction component 130, a planner component 134, and one or more system controller(s) 624.
• the perception component 622 can include the machine-learned model 112, as well as the tracking data 206.
• the machine-learned model 112, or at least a tracking component of the machine-learned model 112, may utilize the tracking data 206 to estimate locations of objects in the environment, as described herein.
• the tracking data 206 is stored in a memory, such as a long short-term memory.
• the vehicle 602 can also include communication connect! on(s) 610 that enable communication between the vehicle 602 and other local or remote computing device(s), such as the computing device(s) 628, as well as other remote or local computing device(s).
• the communication connection(s) 610 can facilitate communication with other local computing device(s) on the vehicle 602 and/or the drive system(s) 614.
• the communication connection(s) 610 can allow the vehicle to communicate with other nearby computing device(s) (e.g., other nearby vehicles, traffic signals, etc.).
• the communications connect! on(s) 610 also enable the vehicle 602 to communicate with a remote teleoperations computing device or other remote services.
• the communications connect! on(s) 610 can include physical and/or logical interfaces for connecting the vehicle computing device(s) 604 to another computing device or a network, such as network(s) 626.
• the communications connect! on(s) 610 can enable Wi-Fi-based communication such as via frequencies defined by the IEEE 602.11 standards, short range wireless frequencies such as BLUETOOTH®, or any suitable wired or wireless communications protocol that enables the respective computing device to interface with the other computing device(s).
• the training data 638 may also include ground truth data 642.
• ground truth data 642 for every instance of log data 640 used to train the machine-learned model(s) 636, a corresponding instance of ground truth data 642 may be stored.
• the ground truth data 642 may include tracked object data associated with an object.
• the ground truth data 642 may include a bounding box associated with an object, a trajectory traversed by the object, a classification of the object, or the like.
• the ground truth data 642 may be determined by a human labeler, a trained machine- learned model, or the like.
• the processor(s) 616 of the vehicle 602 and the processor(s) 630 of the computing device(s) 628 can be any suitable processor capable of executing instructions to process data and perform operations as described herein.
• the processor(s) 616 and 632 can comprise one or more Central Processing Units (CPUs), Graphics Processing Units (GPUs), or any other device or portion of a device that processes electronic data to transform that electronic data into other electronic data that can be stored in registers and/or memory.
• integrated circuits e.g., ASICs, etc.
• gate arrays e.g., FPGAs, etc.
• other hardware devices can also be considered processors in so far as they are configured to implement encoded instructions.
• components of the vehicle 602 can be associated with the computing device(s) 628 and/or the components of the computing device(s) 628 can be associated with the vehicle 602. That is, the vehicle 602 can perform one or more of the functions associated with the computing device(s) 628, and vice versa.
• vehicle computing device(s) 604 and the computing device(s) 628 are shown to include multiple components, in some examples, such components can be associated with more or fewer individual components.
• the localization component 620, the perception component 622, the prediction component 130, and/or the planner component 134 can be combined into a single component. That is, while depicted as separate components, any one or more of the components can be combined.
• a method comprising: receiving, at a first time, sensor data associated with a vehicle operating in an environment; inputting the sensor data into a machine-learned model that is configured to: determine a location of an object in the environment based at least in part on the sensor data; determine an estimated location of a tracked object in the environment based at least in part on a trajectory traversed by the tracked object, the trajectory determined based at least in part on additional sensor data received by the machine-learned model prior to the first time; and associate the object with the tracked object based at least in part on the location and the estimated location; receiving, as an output from the machine-learned model, an indication of the location and the trajectory; and controlling the vehicle based at least in part on the output.
• N The method as recited in any one of paragraphs F-M, wherein the output from the machine- learned model further includes a bounding box associated with the object, the bounding box indicative of at least one of: the location of the object in the environment relative to the vehicle; a size associated with the object; or an orientation associated with the object.
• the sensor data comprises at least one of: image data; lidar data; radar data; or time of flight data.
• the machine-learned model comprises: a first portion that is configured to determine the predicted bounding box, the predicted bounding box indicating a location of the object in the environment; and a second portion that is configured to: determine, based at least in part on tracking data associated with the object, an estimated location of the object in the environment; associate the predicted bounding box with the tracking data based at least in part on the location and the estimated location; and determine a predicted trajectory traversed by the object based at least in part on the location and the tracking data.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Software Systems (AREA)
• Evolutionary Computation (AREA)
• Artificial Intelligence (AREA)
• Computing Systems (AREA)
• Multimedia (AREA)
• Transportation (AREA)
• Automation & Control Theory (AREA)
• Mechanical Engineering (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Medical Informatics (AREA)
• General Health & Medical Sciences (AREA)
• Health & Medical Sciences (AREA)
• Data Mining & Analysis (AREA)
• Mathematical Physics (AREA)
• General Engineering & Computer Science (AREA)
• Databases & Information Systems (AREA)
• Human Computer Interaction (AREA)
• Life Sciences & Earth Sciences (AREA)
• Biomedical Technology (AREA)
• Biophysics (AREA)
• Computational Linguistics (AREA)
• Molecular Biology (AREA)
• Image Analysis (AREA)
• Traffic Control Systems (AREA)
• Control Of Driving Devices And Active Controlling Of Vehicle (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Publications (1)

Family

ID=86608987

Family Applications (1)

Country Status (5)

Families Citing this family (2)

Citations (5)

Family Cites Families (33)

• 2021
  • 2021-12-03 US US17/542,352 patent/US12030528B2/en active Active
• 2022
  • 2022-12-01 CN CN202280079871.XA patent/CN118339067A/zh active Pending
  • 2022-12-01 WO PCT/US2022/051550 patent/WO2023102126A1/en not_active Ceased
  • 2022-12-01 JP JP2024532785A patent/JP2024544196A/ja active Pending
  • 2022-12-01 EP EP22902186.0A patent/EP4440897A4/en active Pending

Patent Citations (5)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events