KR20230140517A - Predicting and controlling object crossings on vehicle routes - Google Patents

Abstract

Methods, systems and computer program products are provided for predicting vehicle crossing and yielding, which may include receiving sensor information indicative of objects around the vehicle. Some methods may also determine a future position of the vehicle based on at least a first trajectory of the vehicle, determine a future position of the object based on a second trajectory of the object, and determine a future position of the object based on the future position of the vehicle and the future position of the object. Includes determining vehicle control. The method also includes training a model using vehicle control, a first trajectory of the vehicle, and a second trajectory of the object.

Description

PREDICTING AND CONTROLLING OBJECT CROSSINGS ON VEHICLE ROUTES}

An autonomous vehicle is a vehicle that can sense its environment and drive without human input. Self-driving vehicles are equipped with several types of sensors that recognize their surroundings and provide data representing the surrounding environment to the self-driving vehicle. Autonomous vehicles use machine learning models to process or compute data, and the results of those computations may include predictions about whether other actors in the autonomous vehicle's environment will cross the autonomous vehicle's route. Make movement decisions based on

1 is an example environment in which a vehicle including one or more components of an autonomous system may be implemented.
2 is a diagram of one or more systems of a vehicle including an autonomous driving system.
Figure 3 is a diagram of components of one or more devices and/or one or more systems of Figures 1 and 2;
4 is a diagram of specific components of an autonomous driving system.
5 is a diagram of an example of a process for predicting and controlling object crossing of a vehicle route.
6 is a flow diagram of an example of a process for predicting and controlling object crossing of a vehicle route.

In the following description, for purposes of explanation, numerous specific details are set forth to provide a thorough understanding of the disclosure. However, it will be clear that the embodiments described by this disclosure may be practiced without these specific details. In some cases, well-known structures and devices are illustrated in block diagram form to avoid unnecessarily obscuring aspects of the disclosure.

Specific arrangements or orders of schematic elements, such as those representing systems, devices, modules, instruction blocks, data elements, etc., are illustrated in the drawings for ease of explanation. However, one of ordinary skill in the art will recognize that a specific order or arrangement of elements schematically in the drawings requires a specific processing order or sequence of processes, or separation of processes, unless explicitly stated as such. It will be understood that it is not meant to be implied. Moreover, the inclusion of a schematic element in the drawings indicates that in some embodiments, unless explicitly stated as such, such element is required in all embodiments or that the features represented by such element are different from other elements. It is not meant to imply that it may not be included in or may not be combined with other elements.

Moreover, where connecting elements such as solid or broken lines or arrows are used in the drawings to illustrate a connection, relationship or association between two or more other schematic elements, the absence of any such connecting elements indicates a connection, relationship or association between two or more other schematic elements. It is not meant to imply that a connection cannot exist. In other words, some connections, relationships or associations between elements are not illustrated in the drawings in order not to obscure the present disclosure. Additionally, for ease of illustration, a single connected element may be used to indicate multiple connections, relationships, or associations between elements. For example, if a connecting element represents the communication of signals, data, or instructions (e.g., “software instructions”), one of ordinary skill in the art would recognize that such element is necessary to effectuate the communication. It will be appreciated that it may represent one or multiple signal paths (e.g., buses) that may be routed.

Although terms such as first, second, third, etc. are used to describe various components, these elements should not be limited by these terms. Terms such as first, second, third, etc. are used only to distinguish one element from another. For example, without departing from the scope of the described embodiments, a first contact may be referred to as a second contact, and similarly the second contact may be referred to as a first contact. Although both the first contact and the second contact are contacts, they are not the same contact.

Terminology used in the description of the various described embodiments herein is included only to describe the specific embodiments and is not intended to be limiting. As used in the description of the various described embodiments and the appended claims, the singular forms “a”, “an”, and “the” are intended to include the plural forms as well, and the context may vary. Unless explicitly stated, it may be used interchangeably with “one or more” or “at least one.” It will also be understood that the term “and/or”, as used herein, refers to and includes any and all possible combinations of one or more of the associated listed items. When the terms “includes,” including, “comprises,” and/or “comprising” are used in this description, they refer to the features, integers, or steps referred to. , it is further understood that it specifies the presence of operations, elements, and/or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and/or groups thereof. It will be.

As used herein, the terms “communication” and “communicate” refer to receiving, receiving, or receiving information (or information, e.g., expressed by data, signals, messages, instructions, commands, etc.). Refers to at least one of transmission, delivery, provision, etc. For one unit (e.g., a device, system, component of a device or system, combinations thereof, etc.) to communicate with another unit means that one unit directly or indirectly receives information from the other unit and/or communicates with the other unit. This means that information can be transmitted (e.g., transmitted). This may refer to a direct or indirect connection that is wired and/or wireless in nature. Additionally, the two units may be in communication with each other although information being transmitted may be modified, processed, relayed and/or routed between the first unit and the second unit. For example, a first unit may be in communication with a second unit even though the first unit is passively receiving information and not actively transmitting information to the second unit. As another example, at least one intermediate unit (e.g., a third unit located between the first unit and the second unit) processes information received from the first unit and transmits the processed information to the second unit. In this case, the first unit may be communicating with the second unit. In some embodiments, a message may refer to a network packet containing data (eg, a data packet, etc.).

As used herein, the term “if” means, optionally, “when” or “when” or “in response to determining that” or “in response to detecting that,” depending on the context. It is interpreted to mean “in response.” Similarly, the phrases “if it is determined that” or “if [the stated condition or event] is detected” can, optionally, depending on the context, mean “upon determining that”, “in response to determining that ", "upon detecting the [mentioned condition or event]", "in response to detecting the [mentioned condition or event]", etc. Additionally, as used herein, the terms “has,” “have,” and the like are intended to be open-ended terms. Moreover, the phrase “based on” is intended to mean “based at least in part on,” unless explicitly stated otherwise.

Embodiments, examples of which are illustrated in the accompanying drawings, will now be referred to in detail. In the following detailed description, numerous specific details are set forth in order to provide a thorough understanding of the various described embodiments. However, it will be apparent to one skilled in the art that the various described embodiments may be practiced without these specific details. In other instances, well-known methods, procedures, components, circuits, and networks have not been described in detail so as not to unnecessarily obscure aspects of the embodiments.

general overview

In some aspects and/or embodiments, the systems, methods, and computer program products described herein include and/or implement predicting and controlling object crossing of a vehicle route. In some embodiments, a vehicle (e.g., an autonomous vehicle) can predict or determine whether an object will traverse the vehicle's route or yield to the vehicle, and maintain or adjust the movement of the vehicle based on the trajectory of the object and the vehicle. Determine vehicle control for, and train a model (e.g., machine learning model) based on the trajectory and vehicle control. For example, information is provided to the vehicle about whether an object detected in the vehicle's surrounding environment will cross in front of the vehicle or yield to the vehicle. This information is based on determining trajectory and associated timestamp information for the vehicle and for the object. The decision to traverse or yield involves detecting false positives, filtering out these results and improving the accuracy of object movement detection and training the model accordingly.

For example, a vehicle's sensors detect the presence of an object, as well as movement data of the object that the vehicle can use to predict the object's trajectory for a given time period or duration. The vehicle also determines its own trajectory for the same duration and predicts whether the object will cross the vehicle's route or yield to the vehicle. When the future positions of the object and the vehicle intersect and the object passes the intersection before the vehicle, the object traverses the vehicle's route. When the future positions of an object and a vehicle intersect and the vehicle passes the intersection before the object, the object is said to give way to the vehicle.

In some embodiments, the vehicle determines actual or “ground truth” trajectories of objects and vehicles and compares these trajectories to their respective predicted trajectories to identify any false positive or false negative object crossings. do. A false positive object crossing refers to a scenario where the predicted trajectories of the object and vehicle indicate that the object traverses the route of the vehicle, but the ground truth trajectories of the object and vehicle indicate the opposite, and a false negative object crossing refers to a scenario where the predicted trajectory of the object and vehicle indicates the opposite. refers to a scenario where indicates that the object does not traverse the vehicle's route, while the ground truth trajectory indicates that the object does traverse the vehicle's route. Predicted trajectories with identified false positive crossings and false negative crossings are filtered and predicted and/or used to make improved predictions regarding object and/or vehicle trajectories and whether the object crosses or yields to vehicles. A model (e.g., a machine learning model) is trained using the ground truth trajectories.

In some embodiments, the vehicle determines vehicle controls to control movement of the vehicle based on predicted trajectories of the vehicle and the object. Vehicle control includes, but is not limited to, changing or maintaining the speed of the vehicle (eg, to avoid a collision, i.e., to avoid reaching an intersection location at or substantially the same time as the object). A model is then trained with the predicted trajectories of the object and/or vehicle and vehicle control to make improved predictions about the object and/or vehicle trajectory and whether the object crosses or yields to the vehicle.

By implementing the systems, methods and computer program products described herein, techniques for predicting and controlling an object's crossing of a vehicle's route are provided to enable more accurate determination of vehicle responsiveness to the likelihood that an object will cross the vehicle's route. Uses trajectory and timestamp information of vehicles and objects. Additional consideration of timestamp information is helpful in determining whether an object traverses the vehicle's route, as well as in determining false positive object crossings. These decisions improve vehicle and object safety by allowing the vehicle to adjust its direction (eg, steering angle, lane, etc.), speed, etc.

Referring now to FIG. 1 , an example environment 100 is illustrated in which vehicles including autonomous driving systems as well as vehicles without autonomous driving systems operate. As illustrated, environment 100 includes vehicles 102a through 102n, objects 104a through 104n, routes 106a through 106n, area 108, and vehicle-to-infrastructure. V2I) device 110, network 112, remote autonomous vehicle (AV) system 114, fleet management system 116, and V2I system 118. Vehicles 102a through 102n, vehicle-to-infrastructure (V2I) device 110, network 112, autonomous vehicle (AV) system 114, fleet management system 116, and V2I system 118 Interconnect (e.g., establish a connection for communication, etc.) through wired connections, wireless connections, or a combination of wired or wireless connections. In some embodiments, objects 104a - 104n are connected to vehicles 102a - 102n, vehicle-to-infrastructure (V2I) device 110, via wired connections, wireless connections, or a combination of wired or wireless connections. It interconnects with at least one of a network 112, an autonomous vehicle (AV) system 114, a fleet management system 116, and a V2I system 118.

Vehicles 102a - 102n (individually referred to as vehicle 102 and collectively referred to as vehicles 102 ) include at least one device configured to transport goods and/or people. In some embodiments, vehicles 102 are configured to communicate with V2I device 110, remote AV system 114, fleet management system 116, and/or V2I system 118 via network 112. do. In some embodiments, vehicles 102 include cars, buses, trucks, trains, etc. In some embodiments, vehicles 102 are the same or similar to vehicles 200 (see FIG. 2) described herein. In some embodiments, vehicle 200 of the group of vehicles 200 is associated with an autonomous fleet manager. In some embodiments, vehicles 102 travel on respective routes 106a through 106n (individually referred to as route 106 and collectively referred to as routes 106 ), as described herein. ) drive along. In some embodiments, one or more vehicles 102 include an autonomous driving system (e.g., the same or similar autonomous driving system as autonomous driving system 202).

Objects 104a - 104n (individually referred to as object 104 and collectively referred to as objects 104 ) may be, for example, at least one vehicle, at least one pedestrian, at least one cyclist. Includes people, at least one structure (e.g., building, sign, fire hydrant, etc.). Each object 104 may be stationary (e.g., located at a fixed position for a period of time) or moving (e.g., has a velocity and is associated with at least one trajectory). In some embodiments, objects 104 are associated with corresponding locations within region 108.

Routes 106a through 106n (individually referred to as route 106 and collectively as routes 106) each represent a sequence of actions (also referred to as a trajectory) connecting the states in which the AV can travel. Associated with (e.g., defining). Each route 106 has an initial state (e.g., a state corresponding to a first spatiotemporal position, speed, etc.) and a final target state (e.g., a state corresponding to a second spatiotemporal position different from the first spatiotemporal position) or It starts from a target region (e.g., a subspace of allowable states (e.g., terminal states)). In some embodiments, the first state includes a location where the individual or individuals must be picked up by the AV and the second state or area includes a location where the individual or individuals picked up by the AV drop-off. Includes the location or locations that need to be done. In some embodiments, routes 106 include a plurality of allowable state sequences (e.g., a plurality of spatiotemporal position sequences), and the plurality of state sequences are associated with a plurality of trajectories (e.g., For example, define this). In one example, routes 106 include only high-level actions or imprecise state locations, such as a series of connected roads indicating turns at road intersections. Additionally or alternatively, routes 106 may include more precise actions or states, such as, for example, precise locations within specific target lanes or lane areas and target speeds at those locations. can do. In one example, routes 106 include a plurality of precise state sequences along at least one high-level action sequence with a bounded lookahead horizon to reach intermediate goals, wherein the bounded horizon state sequence The combination of successive iterations of these cumulatively corresponds to a plurality of trajectories, which collectively form a higher-level route that ends in the final target state or region.

Area 108 includes a physical area (eg, geographic area) in which vehicles 102 may operate. In one example, area 108 includes at least one state (e.g., a nation, a province, an individual state of a plurality of states included in a nation, etc.), at least one portion of a state, at least one city, Includes at least one part of a city, etc. In some embodiments, area 108 includes at least one named thoroughfare (referred to herein as a “road”), such as a thoroughfare, interstate thoroughfare, parkway, city street, etc. Additionally or alternatively, in some examples, area 108 includes at least one unnamed road, such as a driveway, a section of a parking lot, a section of a vacant lot and/or undeveloped lot, a dirt path, etc. In some embodiments, a road includes at least one lane (e.g., a portion of the road that may be traversed by vehicle 102). In one example, a road includes at least one lane associated with (e.g., identified based on) at least one lane marking.

Vehicle-to-Infrastructure (V2I) device 110 (sometimes referred to as a Vehicle-to-Infrastructure (V2X) device) includes at least one device configured to communicate with vehicles 102 and/or V2I infrastructure system 118. Includes. In some embodiments, V2I device 110 is configured to communicate with vehicles 102, remote AV system 114, fleet management system 116, and/or V2I system 118 over network 112. do. In some embodiments, V2I device 110 may include a radio frequency identification (RFID) device, signage, a camera (e.g., a two-dimensional (2D) and/or three-dimensional (3D) camera), and a lane marker. , street lights, parking meters, etc. In some embodiments, V2I device 110 is configured to communicate directly with vehicles 102. Additionally or alternatively, in some embodiments, V2I device 110 is configured to communicate with vehicles 102, remote AV system 114, and/or fleet management system 116 via V2I system 118. It is composed. In some embodiments, V2I device 110 is configured to communicate with V2I system 118 over network 112.

Network 112 includes one or more wired and/or wireless networks. In one example, network 112 may be a cellular network (e.g., a long term evolution (LTE) network, a third generation (3G) network, a fourth generation (4G) network, a fifth generation (5G) network, or a code division multiple (CDMA) network. access network, etc.), public land mobile network (PLMN), local area network (LAN), wide area network (WAN), metropolitan area network (MAN), telephone network (e.g., public switched telephone network (PSTN)) , private networks, ad hoc networks, intranets, the Internet, fiber-optic networks, cloud computing networks, etc., and combinations of some or all of these networks.

Remote AV system 114 may be connected to vehicles 102, V2I device 110, network 112, remote AV system 114, fleet management system 116, and/or V2I system ( 118) and includes at least one device configured to communicate with. In one example, remote AV system 114 includes a server, a group of servers, and/or other similar devices. In some embodiments, remote AV system 114 is co-located with fleet management system 116. In some embodiments, remote AV system 114 is involved in the installation of some or all of the components of the vehicle, including the autonomous driving system, the autonomous vehicle computer, software implemented by the autonomous vehicle computer, and the like. In some embodiments, remote AV system 114 maintains (e.g., updates and/or replaces) such components and/or software over the life of the vehicle.

Fleet management system 116 includes at least one device configured to communicate with vehicles 102, a V2I device 110, a remote AV system 114, and/or a V2I infrastructure system 118. In one example, fleet management system 116 includes servers, groups of servers, and/or other similar devices. In some embodiments, the fleet management system 116 may be used by a ridesharing company (e.g., a fleet of multiple vehicles (e.g., vehicles that include autonomous driving systems and/or self-driving systems). It is associated with organizations that control the operation of vehicles that do not operate, etc.

In some embodiments, V2I system 118 is configured to communicate with vehicles 102, V2I device 110, remote AV system 114, and/or fleet management system 116 over network 112. Contains at least one device. In some examples, V2I system 118 is configured to communicate with V2I device 110 through a different connection than network 112. In some embodiments, V2I system 118 includes a server, a group of servers, and/or other similar devices. In some embodiments, V2I system 118 is associated with a local government or private organization (eg, a private organization that maintains V2I device 110, etc.).

The number and arrangement of elements illustrated in Figure 1 are provided by way of example. There may be additional elements, fewer elements, different elements, and/or differently arranged elements than illustrated in FIG. 1 . Additionally or alternatively, at least one element of environment 100 may perform one or more functions described as being performed by at least one different element of FIG. 1 . Additionally or alternatively, at least one set of elements of environment 100 may perform one or more functions described as being performed by at least one different set of elements of environment 100.

Referring now to FIG. 2 , vehicle 200 includes an autonomous driving system 202, a powertrain control system 204, a steering control system 206, and a braking system 208. In some embodiments, vehicle 200 is the same or similar to vehicle 102 (see FIG. 1). In some embodiments, vehicle 102 has autonomous driving capabilities (e.g., fully autonomous vehicles (e.g., vehicles that do not rely on human intervention), highly autonomous vehicles (e.g., implements at least one function, feature, device, etc. that allows vehicle 200 to be operated partially or completely without human intervention, including, without limitation, vehicles that do not rely on human intervention in certain situations), etc. do). For a detailed description of fully autonomous vehicles and highly autonomous vehicles, see SAE International's standard J3016: Taxonomy and Definitions for Terms Related to On-Road Motor Vehicle Automated Driving Systems, which is incorporated by reference in its entirety. there is. In some embodiments, vehicle 200 is associated with an autonomous fleet manager and/or ride sharing company.

Autonomous driving system 202 includes a sensor suite that includes one or more devices such as cameras 202a, LiDAR sensors 202b, radar sensors 202c, and microphones 202d. . In some embodiments, autonomous driving system 202 may include more or fewer devices and/or different devices (e.g., ultrasonic sensors, inertial sensors, GPS receivers (discussed below), vehicle (200) may include odometry sensors, etc., which generate data associated with an indication of the distance traveled. In some embodiments, autonomous driving system 202 uses one or more devices included in autonomous driving system 202 to generate data associated with environment 100 described herein. Data generated by one or more devices of autonomous driving system 202 may be used by one or more systems described herein to observe the environment in which vehicle 200 is located (e.g., environment 100). . In some embodiments, autonomous driving system 202 includes a communication device 202e, an autonomous vehicle computer 202f, and a drive-by-wire (DBW) system 202h.

Cameras 202a communicate with communication device 202e, autonomous vehicle computer 202f, and/or safety controller 202g via a bus (e.g., the same or similar bus as bus 302 in FIG. 3). Includes at least one device configured to Cameras 202a include at least one camera (e.g., charge-coupled device (CCD)) for capturing images containing physical objects (e.g., cars, buses, curbs, people, etc.) Includes digital cameras, thermal cameras, infrared (IR) cameras, event cameras, etc. that use optical sensors such as In some embodiments, camera 202a produces camera data as output. In some examples, camera 202a generates camera data that includes image data associated with an image. In this example, the image data may specify at least one parameter corresponding to the image (eg, image characteristics such as exposure, brightness, etc., image timestamp, etc.). In such examples, the image may be in one format (eg, RAW, JPEG, PNG, etc.). In some embodiments, camera 202a is a plurality of independent cameras configured on the vehicle (e.g., positioned on the vehicle) to capture images for stereopsis (stereo vision). includes them. In some examples, camera 202a includes a plurality of cameras that generate image data and transmit the image data to autonomous vehicle computer 202f and/or a fleet management system (e.g., of FIG. 1 ). It is transmitted to a fleet management system (same or similar to the fleet management system 116). In such an example, autonomous vehicle computer 202f determines the depth to one or more objects within the field of view of at least two of the plurality of cameras based on image data from the at least two cameras. In some embodiments, cameras 202a are configured to capture images of objects within a distance (eg, up to 100 meters, up to 1 kilometer, etc.) from cameras 202a. Accordingly, cameras 202a include features such as sensors and lenses that are optimized to recognize objects at one or more distances from cameras 202a.

In one embodiment, camera 202a includes at least one camera configured to capture one or more images associated with one or more traffic lights, street signs, and/or other physical objects that provide visual navigation information. In some examples, camera 202a generates TLD data associated with one or more images containing a format (eg, RAW, JPEG, PNG, etc.). In some embodiments, camera 202a generating TLD data may include one or more cameras (e.g., wide angle lenses, It differs from other systems described herein that include cameras in that it may include a fisheye lens, a lens with a viewing angle of approximately 120 degrees or greater, etc.).

Laser Detection and Ranging (LiDAR) sensors 202b are connected to the communication device 202e, the autonomous vehicle computer 202f, and/or via a bus (e.g., the same or similar bus as bus 302 in FIG. 3). or at least one device configured to communicate with safety controller 202g. LiDAR sensors 202b include a system configured to transmit light from a light emitter (eg, a laser transmitter). Light emitted by LiDAR sensors 202b includes light outside the visible spectrum (eg, infrared light, etc.). In some embodiments, during operation, light emitted by LiDAR sensors 202b encounters a physical object (e.g., a vehicle) and is reflected back to LiDAR sensors 202b. In some embodiments, the light emitted by LiDAR sensors 202b does not penetrate the physical objects the light encounters. LiDAR sensors 202b also include at least one light detector that detects light emitted from the light emitter after it encounters a physical object. In some embodiments, at least one data processing system associated with the LiDAR sensors 202b processes images (e.g., a point cloud, a combined point cloud) representing objects included in the field of view of the LiDAR sensors 202b. point cloud, etc.) is created. In some examples, at least one data processing system associated with LiDAR sensor 202b generates an image representative of boundaries of a physical object, surfaces of the physical object (e.g., topology of surfaces), etc. In such an example, the image is used to determine the boundaries of physical objects within the field of view of LiDAR sensors 202b.

Radar (Radio Detection and Ranging) sensors 202c are connected to communication device 202e, autonomous vehicle computer 202f, and and/or at least one device configured to communicate with safety controller 202g. Radar sensors 202c include a system configured to transmit radio waves (either pulsed or continuously). Radio waves transmitted by radar sensors 202c include radio waves that are within a predetermined spectrum. In some embodiments, during operation, radio waves transmitted by radar sensors 202c encounter a physical object and are reflected back to radar sensors 202c. In some embodiments, radio waves transmitted by radar sensors 202c are not reflected by some objects. In some embodiments, at least one data processing system associated with radar sensors 202c generates signals representative of objects included in the field of view of radar sensors 202c. For example, at least one data processing system associated with radar sensor 202c generates an image representative of boundaries of a physical object, surfaces of the physical object (e.g., topology of surfaces), etc. In some examples, the image is used to determine boundaries of physical objects within the field of view of radar sensors 202c.

Microphones 202d communicate with communication device 202e, autonomous vehicle computer 202f, and/or safety controller 202g via a bus (e.g., the same or similar bus as bus 302 in FIG. 3). Includes at least one device configured to Microphones 202d include one or more microphones (e.g., array microphone, external microphone, etc.) that capture audio signals and generate data associated with (e.g., representative of) the audio signals. In some examples, microphones 202d include transducer devices and/or similar devices. In some embodiments, one or more systems described herein may receive data generated by microphones 202d and determine the location of an object relative to vehicle 200 based on audio signals associated with the data (e.g., distance, etc.) can be determined.

Communication device 202e may include cameras 202a, LiDAR sensors 202b, radar sensors 202c, microphones 202d, autonomous vehicle computer 202f, safety controller 202g, and/or DBW. and at least one device configured to communicate with system 202h. For example, communication device 202e may include the same or similar device as communication interface 314 of FIG. 3 . In some embodiments, communication device 202e includes a vehicle-to-vehicle (V2V) communication device (e.g., a device that enables wireless communication of data between vehicles).

Autonomous vehicle computer 202f may include cameras 202a, LiDAR sensors 202b, radar sensors 202c, microphones 202d, communication device 202e, safety controller 202g, and/or DBW. and at least one device configured to communicate with system 202h. In some examples, autonomous vehicle computer 202f may be a client device, a mobile device (e.g., a cellular phone, tablet, etc.), a server (e.g., a computing device that includes one or more central processing units, graphics processing units, etc. ), etc. In some embodiments, autonomous vehicle computer 202f is the same or similar to autonomous vehicle computer 400 described herein. Additionally or alternatively, in some embodiments, autonomous vehicle computer 202f may be configured to operate on an autonomous vehicle system (e.g., an autonomous vehicle system the same or similar to remote AV system 114 of FIG. 1), fleet management, etc. A system (e.g., a fleet management system that is the same as or similar to the fleet management system 116 of FIG. 1), a V2I device (e.g., a V2I device that is the same or similar to the V2I device 110 of FIG. 1), and/or It is configured to communicate with a V2I system (e.g., a V2I system that is the same or similar to V2I system 118 of FIG. 1).

Safety controller 202g includes cameras 202a, LiDAR sensors 202b, radar sensors 202c, microphones 202d, communication device 202e, autonomous vehicle computer 202f, and/or DBW. and at least one device configured to communicate with system 202h. In some examples, safety controller 202g provides controls to operate one or more devices of vehicle 200 (e.g., powertrain control system 204, steering control system 206, brake system 208, etc.) It includes one or more controllers (electrical controller, electromechanical controller, etc.) configured to generate and/or transmit signals. In some embodiments, safety controller 202g is configured to generate control signals that override (eg, override) control signals generated and/or transmitted by autonomous vehicle computer 202f.

DBW system 202h includes at least one device configured to communicate with communication device 202e and/or autonomous vehicle computer 202f. In some examples, DBW system 202h provides controls to operate one or more devices of vehicle 200 (e.g., powertrain control system 204, steering control system 206, brake system 208, etc.) and one or more controllers (eg, electrical controllers, electromechanical controllers, etc.) configured to generate and/or transmit signals. Additionally or alternatively, one or more controllers of DBW system 202h may provide control signals to operate at least one different device of vehicle 200 (e.g., turn signals, headlights, door locks, window shield wipers, etc.). configured to generate and/or transmit them.

Powertrain control system 204 includes at least one device configured to communicate with DBW system 202h. In some examples, powertrain control system 204 includes at least one controller, actuator, etc. In some embodiments, powertrain control system 204 receives control signals from DBW system 202h, and powertrain control system 204 causes vehicle 200 to start moving forward and stop moving forward. start moving backwards, stop moving backwards, accelerate in one direction, decelerate in one direction, make a left turn, make a right turn, etc. In one example, the powertrain control system 204 causes the energy (e.g., fuel, electricity, etc.) provided to the vehicle's motor to increase, remain the same, or decrease, thereby causing the vehicle 200 ) at least one wheel rotates or does not rotate.

Steering control system 206 includes at least one device configured to rotate one or more wheels of vehicle 200 . In some examples, steering control system 206 includes at least one controller, actuator, etc. In some embodiments, the steering control system 206 can cause the front two wheels and/or the rear two wheels of the vehicle 200 to turn left or right to cause the vehicle 200 to turn left or right. do.

Brake system 208 includes at least one device configured to actuate one or more brakes to cause vehicle 200 to reduce speed and/or remain stationary. In some examples, the brake system 208 includes at least one controller configured to cause one or more calipers associated with one or more wheels of the vehicle 200 to close at a corresponding rotor of the vehicle 200, and/ Or it includes an actuator. Additionally or alternatively, in some examples, braking system 208 includes an automatic emergency braking (AEB) system, a regenerative braking system, etc.

In some embodiments, vehicle 200 includes at least one platform sensor (not explicitly illustrated) that measures or infers attributes of a state or condition of vehicle 200. In some examples, vehicle 200 includes platform sensors such as a global positioning system (GPS) receiver, inertial measurement unit (IMU), wheel speed sensor, wheel brake pressure sensor, wheel torque sensor, engine torque sensor, steering angle sensor, etc. .

Referring now to Figure 3, a schematic diagram of device 300 is illustrated. As illustrated, device 300 includes a processor 304, memory 306, storage component 308, input interface 310, output interface 312, communication interface 314, and bus 302. Includes. In some embodiments, device 300 includes at least one device of vehicles 102 (e.g., a system of vehicles 102, such as, but not limited to, autonomous driving system 202). at least one device), and/or one or more devices of network 112 (e.g., one or more devices of a system of network 112). For example, in some embodiments, one or more devices of vehicles 102 (e.g., one or more of a system of vehicles 102, such as, but not limited to, autonomous driving system 202) device), and/or one or more devices of network 112 (e.g., one or more devices of a system of network 112) may include at least one device 300 and/or at least one component of device 300 Includes.

Bus 302 includes components that enable communication between components of device 300. In some embodiments, processor 304 is implemented in hardware, software, or a combination of hardware and software. In some examples, processor 304 may be a processor (e.g., a central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), etc.), which may be programmed to perform at least one function; It includes a microphone, a digital signal processor (DSP), and/or any processing components (e.g., field-programmable gate array (FPGA), application specific integrated circuit (ASIC), etc.). Memory 306 may include random access memory (RAM), read-only memory (ROM), and/or other types of dynamic and/or static storage devices (e.g., Examples include flash memory, magnetic memory, optical memory, etc.).

Storage component 308 stores data and/or software related to the operation and use of device 300. In some examples, storage component 308 may be a hard disk (e.g., magnetic disk, optical disk, magneto-optical disk, solid state disk, etc.), compact disc (CD), digital versatile disc (DVD), floppy disk, cartridge. , magnetic tape, CD-ROM, RAM, PROM, EPROM, FLASH-EPROM, NV-RAM, and/or other types of computer-readable media, together with corresponding drives.

Input interface 310 allows device 300 to receive information, e.g., through user input (e.g., touchscreen display, keyboard, keypad, mouse, buttons, switches, microphone, camera, etc.). Contains components. Additionally or alternatively, in some embodiments, input interface 310 includes a sensor (e.g., global positioning system (GPS) receiver, accelerometer, gyroscope, actuator, etc.) that senses information. Output interface 312 includes components that provide output information from device 300 (eg, a display, a speaker, one or more light emitting diodes (LEDs), etc.).

In some embodiments, communication interface 314 is a transceiver-like component that allows device 300 to communicate with other devices via a wired connection, a wireless connection, or a combination of wired and wireless connections (e.g., transceivers, individual receivers and transmitters, etc.). In some examples, communication interface 314 allows device 300 to receive information from and/or provide information to another device. In some examples, communication interface 314 includes an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, etc. .

In some embodiments, device 300 performs one or more processes described herein. Device 300 performs these processes based on processor 304 executing software instructions stored by a computer-readable medium, such as memory 306 and/or storage component 308. Computer-readable media (e.g., non-transitory computer-readable media) are defined herein as non-transitory memory devices. Non-transitory memory devices include memory space located within a single physical storage device or memory space distributed across multiple physical storage devices.

In some embodiments, software instructions are read into memory 306 and/or storage component 308 from another device or from another computer-readable medium via communications interface 314. When executed, software instructions stored in memory 306 and/or storage component 308 cause processor 304 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry may be used in place of or in combination with software instructions to perform one or more processes described herein. Accordingly, the embodiments described herein are not limited to any particular combination of hardware circuitry and software, unless explicitly stated otherwise.

Memory 306 and/or storage component 308 includes data storage or at least one data structure (eg, database, etc.). Device 300 is capable of receiving information from, storing information therein, communicating information to, or storing information therein at least one data structure within data storage or memory 306 or storage component 308. You can search for information. In some examples, the information includes network data, input data, output data, or any combination thereof.

In some embodiments, device 300 is configured to execute software instructions stored in memory 306 and/or in the memory of another device (e.g., another device that is the same or similar to device 300). As used herein, the term “module” refers to a device ( Refers to at least one instruction stored in memory 306 and/or in the memory of another device that causes 300) (e.g., at least one component of device 300) to perform one or more processes described herein. do. In some embodiments, a module is implemented in software, firmware, hardware, etc.

The number and arrangement of components illustrated in Figure 3 are provided as examples. In some embodiments, device 300 may include additional components, fewer components, different components, or differently arranged components than those illustrated in FIG. 3 . Additionally or alternatively, a set of components (e.g., one or more components) of device 300 may perform one or more functions described as being performed by another component or set of components of device 300.

Referring now to Figure 4, an example block diagram of autonomous vehicle computer 400 (sometimes referred to as the “AV stack”) is illustrated. As illustrated, autonomous vehicle computer 400 includes a cognitive system 402 (sometimes referred to as a cognitive module), a planning system 404 (sometimes referred to as a planning module), and a localization system 406 (sometimes referred to as a localization module). (sometimes referred to as a control module), a control system 408 (sometimes referred to as a control module), and a database 410. In some embodiments, the cognitive system 402, the planning system 404, the localization system 406, the control system 408, and the database 410 may support the vehicle's autonomous navigation system (e.g., vehicle 200 ) is included and implemented in the autonomous vehicle computer 202f). Additionally or alternatively, in some embodiments, cognitive system 402, planning system 404, localization system 406, control system 408, and database 410 can be combined with one or more standalone systems (e.g. For example, it is included in one or more systems that are the same or similar to the autonomous vehicle computer 400, etc. In some examples, cognitive system 402, planning system 404, localization system 406, control system 408, and database 410 can operate on a vehicle and/or at least one remote system as described herein. Included in one or more standalone systems located in In some embodiments, some and/or all of the systems included in autonomous vehicle computer 400 may include software (e.g., software instructions stored in memory), computer hardware (e.g., microprocessor, microprocessor, It is implemented using a controller, application-specific integrated circuit (ASIC), field programmable gate array (FPGA), etc.), or a combination of computer software and computer hardware. In some embodiments, autonomous vehicle computer 400 may be configured to support a remote system (e.g., an autonomous vehicle system the same or similar to remote AV system 114, a fleet management system the same or similar to fleet management system 116, It will also be understood that the V2I system is configured to communicate with a V2I system that is the same or similar to V2I system 118, etc.

In some embodiments, cognitive system 402 receives data associated with at least one physical object in the environment (e.g., data used by cognitive system 402 to detect the at least one physical object) and classifies at least one physical object. In some examples, perception system 402 receives image data captured by at least one camera (e.g., cameras 202a), wherein the image is associated with one or more physical objects within the field of view of the at least one camera. It is done (for example, it expresses this). In such an example, cognitive system 402 classifies at least one physical object based on one or more groupings of physical objects (eg, bicycles, vehicles, traffic signs, pedestrians, etc.). In some embodiments, based on the classification of physical objects by cognitive system 402, cognitive system 402 transmits data associated with the classification of physical objects to planning system 404.

In some embodiments, planning system 404 receives data associated with a destination and determines at least one route (e.g., routes) along which a vehicle (e.g., vehicles 102) can travel toward the destination. (106)) and generate data associated with it. In some embodiments, planning system 404 periodically or continuously receives data from cognitive system 402 (e.g., data associated with classification of physical objects, as described above), and planning system 404 ) updates at least one trajectory or creates at least one different trajectory based on data generated by the cognitive system 402. In some embodiments, planning system 404 receives data associated with an updated location of a vehicle (e.g., vehicles 102) from localization system 406, and planning system 404 provides localization Update at least one trajectory or generate at least one different trajectory based on data generated by system 406.

In some embodiments, localization system 406 receives data associated with (e.g., indicative of) a location of a vehicle (e.g., vehicles 102) in an area. In some examples, localization system 406 receives LiDAR data associated with at least one point cloud generated by at least one LiDAR sensor (e.g., LiDAR sensors 202b). In certain examples, localization system 406 receives data associated with at least one point cloud from multiple LiDAR sensors and localization system 406 generates a combined point cloud based on each of the point clouds. . In these examples, localization system 406 compares at least one point cloud or combined point cloud to a two-dimensional (2D) and/or three-dimensional (3D) map of the area stored in database 410. . Based on localization system 406 comparing the at least one point cloud or combined point cloud to the map, localization system 406 then determines the location of the vehicle in that area. In some embodiments, the map includes a combined point cloud of the area that is generated prior to operation of the vehicle. In some embodiments, the map may be, without limitation, a high-precision map of roadway geometric characteristics, a map describing roadway network connectivity characteristics, roadway physical characteristics (e.g., traffic speed, traffic volume, vehicular traffic lanes and cyclist traffic lanes). a map describing the number, lane width, lane traffic direction, or lane marker type and location, or combinations thereof), and the spatial locations of roadway features, such as crosswalks, traffic signs, or various types of other travel signals. Includes a map that In some embodiments, the map is created in real time based on data received by the cognitive system.

In another example, localization system 406 receives Global Navigation Satellite System (GNSS) data generated by a global positioning system (GPS) receiver. In some examples, localization system 406 receives GNSS data associated with the vehicle's location within the area and localization system 406 determines the latitude and longitude of the vehicle within the area. In such an example, localization system 406 determines the vehicle's location in the area based on the vehicle's latitude and longitude. In some embodiments, localization system 406 generates data associated with the location of the vehicle. In some examples, based on localization system 406 determining the location of the vehicle, localization system 406 generates data associated with the location of the vehicle. In such examples, data associated with the location of the vehicle includes data associated with one or more semantic features corresponding to the location of the vehicle.

In some embodiments, control system 408 receives data associated with at least one trajectory from planning system 404 and control system 408 controls the operation of the vehicle. In some examples, control system 408 receives data associated with at least one trajectory from planning system 404, and control system 408 supports a powertrain control system (e.g., DBW system 202h, powertrain control system 204, etc.), a steering control system (e.g., steering control system 206), and/or a brake system (e.g., brake system 208) to generate and transmit control signals that cause the operation. Controls the operation of the vehicle by In an example where the trajectory includes a left turn, control system 408 transmits a control signal that causes steering control system 206 to adjust the steering angle of vehicle 200, thereby causing vehicle 200 to turn left. Additionally or alternatively, control system 408 may generate control signals that cause other devices of vehicle 200 (e.g., headlights, turn signals, door locks, window shield wipers, etc.) to change states. Send.

In some embodiments, cognitive system 402, planning system 404, localization system 406, and/or control system 408 may use at least one machine learning model (e.g., at least one multilayer perceptron (MLP), at least one convolutional neural network (CNN), at least one recurrent neural network (RNN), at least one autoencoder, at least one transformer, etc.). In some examples, the cognitive system 402, the planning system 404, the localization system 406, and/or the control system 408, alone or in combination with one or more of the systems mentioned above, can be used to control at least one machine. Implement the learning model. In some examples, the perception system 402, the planning system 404, the localization system 406, and/or the control system 408 may use a pipeline (e.g., a pipe for identifying one or more objects located in the environment). Implement at least one machine learning model as part of a line, etc.

Database 410 stores data transmitted to, received from, and/or updated by cognitive system 402, planning system 404, localization system 406, and/or control system 408. . In some examples, database 410 may be a storage component that stores data and/or software related to the operation and use of at least one system of autonomous vehicle computer 400 (e.g., storage component 308 of FIG. 3 ). includes the same or similar storage component). In some embodiments, database 410 stores data associated with a 2D and/or 3D map of at least one area. In some examples, database 410 may be a 2D and /Or save data related to the 3D map. In such examples, a vehicle (e.g., a vehicle identical or similar to vehicles 102 and/or vehicle 200) may be located in one or more drivable areas (e.g., a single lane road, a multi-lane road, a main road, driving along a back road, off-road trail, etc.), and having at least one LiDAR sensor (e.g., the same or similar LiDAR sensor as LiDAR sensors 202b) Data associated with images representing objects included in can be generated.

In some embodiments, database 410 may be implemented across multiple devices. In some examples, database 410 may be configured to include vehicles (e.g., vehicles identical or similar to vehicles 102 and/or vehicles 200), autonomous vehicle systems (e.g., remote AV systems 114 and the same or similar autonomous vehicle system), a fleet management system (e.g., the same or similar fleet management system as the fleet management system 116 of FIG. 1), a V2I system (e.g., the V2I system 118 of FIG. 1) may be included in the same or similar V2I system), etc.

Referring now to Figure 5, a diagram of an implementation 500 of a process for predicting and controlling object crossings on a vehicle route is illustrated. In some implementations, implementation 500 includes vehicle 502 and vehicle 506 including autonomous vehicle computer 504 . In some embodiments, vehicle 502 is the same as or similar to vehicle 102a-102n or vehicle 200, autonomous vehicle computer 504 is the same as or similar to autonomous vehicle computer 202f, 400, and The object 506 is the same as or similar to the objects 104a-104n.

In some embodiments, vehicle 502 has a sensor suite that allows the vehicle to collect sensor information about the environment surrounding vehicle 502. For example, vehicle 502 may include devices such as imaging devices (e.g., cameras), LiDAR sensors, radar sensors, microphones, GPS, etc. that collect sensor information about objects 506 surrounding vehicle 502. It may be possible, but it is not limited to this. The sensor information may include captured images and/or point cloud images of the object 506 and the surrounding environment, respectively captured by an imaging device and/or LiDAR sensor on the vehicle 502. Another example of sensor information includes a radar signal representing an object 506 that is generated by a radar sensor in response to radio waves transmitted by the radar sensor toward the object 506. The object may be a vehicle, a pedestrian, a cyclist, etc.

In some embodiments, AV computer 504 of vehicle 502 determines a trajectory (e.g., alternatively referred to as a route) 512 of vehicle 502 based on sensor information received from a sensor suite of vehicle 502. and predict the trajectory 514 of the object 506 (e.g., using an internal ML model). In some cases, AV computer 504 predicts trajectory 512 and trajectory 514 over a predetermined time horizon, such as from about 2 seconds to about 16 seconds, from about 4 seconds to about 12 seconds. , may range from about 6 seconds to about 10 seconds, about 8 seconds (including values and subranges therebetween), and the AV computer 504 may continue to do so periodically with a period of time at least substantially equal to the predetermined time period. do. For example, if the predetermined time period is about 8 seconds, AV computer 504 predicts trajectory 512 and trajectory 514 for 8 seconds, and continues to predict trajectory 512 and trajectory 514 every 8 seconds. Update or predict.

In some embodiments, trajectories 512 and 514 include velocity vectors of vehicle 502 and object 506, where the velocity vectors are from starting space-time location P1a (508a) and starting space-time location P2a (510a). It represents the speed and direction (eg, heading angle) of movement of the vehicle 502 and the object 506, respectively. The spatiotemporal location includes the position of the vehicle 502 or object 506 and, in some cases, its timestamp at this position. Based at least in part on trajectory 512 and trajectory 514 , AV computer 504 may determine future spatiotemporal positions of vehicle 502 and object 506 , respectively. For example, AV computer 504 may determine the future positions of vehicle 502 and object 506 and their timestamps when their respective trajectories or routes intersect based at least in part on trajectory 512 and trajectory 514. can be calculated. Referring to FIG. 5, the AV computer 504 determines the future spatiotemporal positions P1b (508b) and P2b (510b) of the vehicle 502 and the object 506 when the trajectories 512 and 514 intersect, respectively. You can.

Not only the positions of the space-time positions of the vehicle 502 and the object 506 along the trajectories 512 and 514, respectively, but also the positions of the intersection points of the trajectories 512 and 514 are also determined by the vehicle 502 and the object ( 506) may depend on the physical dimensions (e.g., length, width, etc.). In some cases, for example, when the time difference between vehicle 502 and object 506 passing an intersection is large, AV computer 504 (e.g., a machine learning model running therein) may interact with vehicle 502 and Object 506 may be modeled as a point object, such that the position of the point may represent the spatiotemporal positions of vehicle 502 and object 506 along trajectory 512 and trajectory 514, respectively. As another example, particularly when vehicle 502 and object 506 pass through an intersection within a short period of time, AV computer 504 may represent vehicle 502 and object 506 in two dimensions (e.g., square, rectangular, etc.). Alternatively, it can be modeled as a three-dimensional object. AV computer 504 can then use any point (e.g., front, middle, back, etc.) of the 2D or 3D object as the position of the space-time location of vehicle 502 and object 506. For example, the AV computer 504 uses the leading point of the 2D or 3D object representing the vehicle 502 as the position of the spatiotemporal location of the vehicle 502 and the front point of the 2D or 3D object representing the object 506. The last point can be used as the position of the spatiotemporal location of the object 506. In this case, AV computer 504 determines that object 506 has crossed vehicle 502 when the leading point passes the intersection of trajectory 512 and trajectory 514 before the rearmost point reaches the intersection. A decision can be made (e.g., without conflict).

In some embodiments, AV computer 504 determines and compares the spatiotemporal positions of vehicle 502 and object 506 when trajectories 512 and trajectories 514 intersect, such that object 506 is connected to vehicle 502. Decide whether to traverse. As noted above, AV computer 504 determines the spatiotemporal position P1b 508b of vehicle 502 based on trajectory 512 and the spatiotemporal position P2b 510b of object 506 based on trajectory 514. Determines , and the space-time positions P1b (508b) and P2b (510b) are the predicted space-time positions of the vehicle 502 and the object 506 when the trajectory 512 and the trajectory 514 intersect. Then, the AV computer 504 determines if the object 506 passes the same intersection before the arrival of the vehicle 502 at the intersection of the trajectories 512 and 514, and the spatiotemporal positions P1b (508b) and P2b If the difference between the position and timestamp associated with 510b meets the respective threshold, it is determined that object 506 has crossed vehicle 502.

For example, the AV computer 504 determines that the object 506 passed the intersection P1b 508b prior to the arrival of the vehicle 502 at the same intersection. AV computer 504 also determines that the distance difference between the positions associated with space-time positions P1b 508b and P2b 510b meets a distance threshold (e.g., is less than a distance threshold). Additionally, AV computer 504 also determines that the time difference between the timestamps associated with space-time locations P1b 508b and P2b 510b meets a time threshold (e.g., is less than a time threshold). The AV computer 504 determines that the object 506 may be detected by the vehicle 502 if both the distance difference and the time difference meet their respective thresholds (e.g., the distance difference and the time difference are less than the distance threshold and the time threshold, respectively). ), and determine that the object 506 has passed the same intersection before the vehicle 502 arrives at the intersection of the trajectory 512 and the trajectory 514. The condition that the distance difference and the time difference are less than their respective thresholds means that the AV computer 504 can simulate a scenario in which the vehicle 502 passes the intersection after a long time (e.g., longer than the threshold time) after the object 506 has passed. Allows filtering.

In some embodiments, the distance difference and time difference may not have met their respective thresholds (e.g., the distance difference and time difference may be greater than the distance threshold and time threshold, respectively), and AV computer 504 Object 506 may still be considered to have crossed vehicle 502 if the intersection is part of a crosswalk or at least substantially close to a crosswalk. For example, a vehicle 502 stopped at a crosswalk may begin moving again after a period of time greater than a time threshold has elapsed since the object 506 crossed the crosswalk (e.g., and vehicle 502 ). As another example, object 506 may have traveled a distance greater than the distance threshold since object 506 crossed a crosswalk (e.g., and vehicle 502). In this case, AV computer 504 may consider this crossing to be a valid crossing even if the distance difference and/or time difference is greater than the respective threshold (e.g., AV computer 504 may determine the distance associated with the crosswalk The threshold and/or time threshold can be set higher, such that most or all objects that cross vehicles stopped at the crosswalk are considered to have made a valid crossing).

In some embodiments, AV computer 504 first detects object 506 before vehicle 502 arrives at the intersection of trajectories 512 and 514 or when trajectories 512 and 514 do not intersect. ) did not pass the same intersection, and/or if one or both of the distance difference and the time difference do not meet their respective thresholds, it is determined that the object 506 did not cross the vehicle 502.

In some embodiments, AV computer 504 compares predicted trajectories 512 and 514 to the respective “ground truth” trajectories of vehicle 502 and object 506, thereby determining whether object 506 is It can be established whether the decision to traverse or not to traverse is a false positive or false negative, respectively. A false positive object crossing is, for example, as described above, where the predicted trajectories 512 and 514 indicate that the object 506 crosses the vehicle 502, but the intersection of the vehicle 502 and the object 506 Ground truth trajectory refers to when the object 506 does not cross the vehicle 502 . A false negative object crossing is where the predicted trajectories 512 and 514 indicate that the object 506 does not cross the vehicle 502, but the ground truth trajectories of the vehicle 502 and the object 506 do. This refers to when crossing the vehicle 502.

In some embodiments, as described above, AV computer 504 identifies trajectories 512 and predictions of trajectories 514 that are associated with false positive and/or false negative object crossings, These “false object crossing” predictions are filtered out from the set of trajectory predictions for object 506. AV computer 504 may perform “false object crossing” predictions and The remainder of the associated set of filtered trajectories and/or predicted trajectories may be used. For example, AV computer 504 may train a machine learning model to predict the probability that object 506 will cross vehicle 502 . As another example, AV computer 504 may train a machine learning model to calculate the probability that predicted vehicle and object trajectories are associated with false positive or false negative object crossings.

In some embodiments, AV computer 504 determines vehicle controls to control movement or movement of vehicle 502 based on predicted trajectory 512 and/or predicted trajectory 514 . Examples of vehicle control include changing the speed of the vehicle 502 (e.g., increasing or decreasing the speed), maintaining the speed of the vehicle 502, changing the direction of movement of the vehicle 502, maintaining the direction of the vehicle 502, etc. , but is not limited to this. For example, as described above, AV computer 504 predicts the trajectory 512 of vehicle 502 and the trajectory 514 of object 506 and determines whether object 506 traverses vehicle 502. You can decide to do it. In this case, AV computer 504 may determine vehicle 502 based on predicted trajectories 512, 514 (e.g., and, based on the predictions, determine, for example, that object 506 will cross vehicle 502). Vehicle control to control the movement of can be determined. For example, vehicle control may include changing the speed and direction of movement of the vehicle 502 such that the vehicle 502 has a trajectory 516 that crosses the trajectory 518 of the object 506 . That is, the AV computer 504 determines that the vehicle 502 is at the space-time location P1c (508c) after the vehicle 502 passes the intersection of the trajectories 516 and 518 (e.g., corresponding to the space-time location P2c (510c) of the object 506). A vehicle configured to control the movement of the vehicle 502 such that the trajectory 516 of the vehicle 502 intersects the trajectory 518 of the object 506 at the spatiotemporal location P2c 510c of the object 506 when present. Determine control.

In some embodiments, vehicle computer 504 may perform vehicle control, predicted trajectory 512 of vehicle 502, predicted trajectory 514 of object 506, and actual data of vehicle 502 as a result of vehicle control. Train an internally implemented machine learning model based on the trajectory 516 and/or the actual trajectory 518 of the object 506. For example, a machine learning model can be trained to make predictions or improved predictions about the trajectory of vehicle 502 and/or object 506. Additionally, as another example, the machine learning model determines the likelihood that the object 506 will cross the vehicle 502, i.e., before the vehicle 502 reaches the intersection of the trajectories of the vehicle 502 and the object 506 ( 506) trajectories can be trained to predict the likelihood of crossing an intersection or to improve the prediction thereof.

Referring now to Figure 6, a flow diagram of a process 600 for predicting vehicle crossing and yielding is illustrated. In some embodiments, one or more steps described with respect to process 600 are performed (e.g., fully, partially, etc.) by planning system 404 of autonomous vehicle computer 400. Additionally or alternatively, in some embodiments, one or more steps described with respect to process 600 may be performed by another step that is separate from or includes planning system 404, such as control system 408 of autonomous vehicle computer 400. Performed (e.g. fully, partially, etc.) by a device or group of devices.

At block 602, sensor information representative of at least one object around the vehicle is received. For example, sensor information is captured from at least one of a radar sensor, an imaging device, a Global Positioning System (GPS), and a LiDAR sensor.

At block 604, a future position of the vehicle is determined based on at least the first trajectory of the vehicle.

At block 606, a future position of at least one object is determined based on the second trajectory of the at least one object.

At block 608, vehicle control is determined based on the future position of the vehicle and the future position of at least one object.

At block 610, at least one model is trained using the vehicle control, the first trajectory of the vehicle, and the second trajectory of the at least one object.

In some embodiments of process 600, a first timestamp associated with a future position of the vehicle is determined. Additionally, a second timestamp associated with the future position of the at least one object is determined. Additionally, a first difference between the future position of the vehicle and the future position of the at least one object is determined. Additionally, a second difference between the first timestamp and the second timestamp is determined. Additionally, vehicle control is determined based at least on whether the first difference and the second difference meet their respective thresholds.

In some embodiments of process 600, the future position of the vehicle and the future position of the at least one object are additionally based on one or more dimensions of the vehicle and/or the at least one object.

In some embodiments of process 600, control signals related to vehicle control are generated based on a model trained to operate the vehicle.

In some embodiments of process 600, controlling the vehicle is at least one of changing speed, changing steering angle, maintaining the current speed of the vehicle, and maintaining the current direction of the vehicle.

In some embodiments of process 600, the future position of the vehicle and the future position of the at least one object are determined by determining whether the future position of the vehicle and the future position of the at least one object intersect. Additionally, in response to determining that the future position of the vehicle and the future position of the object intersect, vehicle controls are selected, such as changing speed or changing steering angle, and control signals are generated to operate the vehicle based on the selected vehicle controls. Additionally, in response to determining that the future position of the vehicle and the future position of the object do not intersect, vehicle controls are selected to maintain the current vehicle state, and control signals are generated to operate the vehicle based on the selected vehicle controls.

In some embodiments of process 600, at least one model is trained by successively determining a first trajectory of a vehicle and a second trajectory of at least one object. For example, the first trajectory of the vehicle and the second trajectory of the at least one object are continuously determined approximately every 8 seconds.

In the foregoing description, aspects and embodiments of the disclosure have been described with reference to numerous specific details that may vary from implementation to implementation. Accordingly, the description and drawings are to be regarded in an illustrative rather than a restrictive sense. The only exclusive indicator of the scope of the invention, and what the applicant intends to be the scope of the invention, is the literal equivalent range of the series of claims appearing in the particular form in this application, including any subsequent amendments. Any definitions expressly recited herein for terms contained in such claims determine the meaning of such terms when used in the claims. Additionally, when the term "further comprising" is used in the foregoing description and the claims below, what follows this phrase may be an additional step or entity, or a sub-step/sub-entity of a previously mentioned step or entity. You can.

Claims (20)

In the method,
Using at least one processor, receiving sensor information representing at least one object around the vehicle;
determining, using the at least one processor, a future position of the vehicle based on at least a first trajectory of the vehicle;
determining, using the at least one processor, a future position of the at least one object based on a second trajectory of the at least one object;
determining vehicle control based on a future position of the vehicle and a future position of the at least one object, using the at least one processor; and
Using the at least one processor, training at least one model using the vehicle control, a first trajectory of the vehicle, and a second trajectory of the at least one object.
Method, including. In claim 1,
determining, using the at least one processor, a first timestamp associated with a future position of the vehicle;
determining, using the at least one processor, a second timestamp associated with a future position of the at least one object;
determining, using the at least one processor, a first difference between a future position of the vehicle and a future position of the at least one object;
determining, using the at least one processor, a second difference between the first timestamp and the second timestamp; and
determining the vehicle control based at least on whether the first difference and the second difference meet their respective thresholds.
A method further comprising: In claim 1 or claim 2,
The method wherein the future position of the vehicle and the future position of the at least one object are additionally based on one or more dimensions of at least one of the vehicle and the at least one object. In claim 1 or claim 2,
Using the at least one processor, generating control signals related to controlling the vehicle based on the trained model to operate the vehicle.
A method further comprising: In claim 1 or claim 2,
The method wherein the vehicle control is at least one of changing speed, changing steering angle, maintaining the current speed of the vehicle, and maintaining the current direction of the vehicle. In claim 1 or claim 2,
Determining the future position of the vehicle and the future position of the at least one object includes:
The method further comprising determining, using the at least one processor, whether the future position of the vehicle and the future position of the at least one object intersect. In claim 6,
In response to determining that the future position of the vehicle intersects the future position of the object, using the at least one processor, selecting control of the vehicle as a speed change or a steering angle change; and
Using the at least one processor, generating the control signal based on the selected vehicle control to operate the vehicle.
A method further comprising: In claim 6,
In response to determining that the future position of the vehicle and the future position of the object do not intersect, using the at least one processor, selecting control of the vehicle as maintaining a current vehicle state; and
Using the at least one processor, generating the control signal based on the selected vehicle control to operate the vehicle.
A method further comprising: In claim 1 or claim 2,
The steps of training the at least one model are:
and training the at least one model by successively determining, using the at least one processor, a first trajectory of the vehicle and a second trajectory of the at least one object. In claim 9,
The method of claim 1, wherein the first trajectory of the vehicle and the second trajectory of the at least one object are continuously determined approximately every 8 seconds. In claim 1 or claim 2,
The method of claim 1, wherein the sensor information is captured from at least one of a radar sensor, an imaging device, a global positioning system (GPS), and a LiDAR sensor. In the system,
at least one non-transitory storage medium storing instructions; and
At least one processor coupled to the at least one non-transitory storage medium
Including,
The at least one processor causes the system to:
Receiving sensor information indicating at least one object around the vehicle, using at least one processor;
determining, using the at least one processor, a future position of the vehicle based on at least a first trajectory of the vehicle;
determining a future position of the at least one object based on a second trajectory of the at least one object, using the at least one processor;
determining vehicle control based on a future position of the vehicle and a future position of the at least one object, using the at least one processor; and
Using the at least one processor, training at least one model using the vehicle control, a first trajectory of the vehicle, and a second trajectory of the at least one object.
The system is configured to read the instructions from the at least one non-transitory storage medium to perform operations including. In claim 12,
The above operations are:
determining, using the at least one processor, a first timestamp associated with a future position of the vehicle;
determining, using the at least one processor, a second timestamp associated with a future position of the at least one object;
determining, using the at least one processor, a first difference between a future position of the vehicle and a future position of the at least one object;
determining, using the at least one processor, a second difference between the first timestamp and the second timestamp; and
determining the vehicle control based at least on whether the first difference and the second difference meet their respective thresholds.
A system further comprising: In claim 12 or claim 13,
The system, wherein the future position of the vehicle and the future position of the at least one object are additionally based on one or more dimensions of at least one of the vehicle and the at least one object. In claim 12 or claim 13,
The above operations are:
The system further comprising generating, using the at least one processor, a control signal related to controlling the vehicle based on the trained model to operate the vehicle. In claim 12 or claim 13,
The system, wherein the vehicle control is at least one of changing speed, changing steering angle, maintaining the current speed of the vehicle, and maintaining the current direction of the vehicle. In claim 12 or claim 13,
The operation of determining the future position of the vehicle and the future position of the at least one object is:
Using the at least one processor, determining whether a future position of the vehicle intersects a future position of the at least one object.
A system further comprising: In claim 12 or claim 13,
The operation of training the at least one model is:
and training the at least one model by successively determining, using the at least one processor, a first trajectory of the vehicle and a second trajectory of the at least one object. A non-transitory machine-readable medium storing machine-readable instructions, comprising:
The machine readable instructions are:
Receiving sensor information indicating at least one object around the vehicle, using at least one processor;
determining, using the at least one processor, a future position of the vehicle based on at least a first trajectory of the vehicle;
determining a future position of the at least one object based on a second trajectory of the at least one object, using the at least one processor;
determining vehicle control based on a future position of the vehicle and a future position of the at least one object, using the at least one processor; and
Using the at least one processor, training at least one model using the vehicle control, a first trajectory of the vehicle, and a second trajectory of the at least one object.
A non-transitory machine-readable medium executable to perform operations including. In claim 19,
wherein the sensor information is captured from at least one of a radar sensor, an imaging device, a GPS, and a LiDAR sensor.

Images