EP3516467A1 - Véhicule autonome : combinaison de niveau objet - Google Patents

Véhicule autonome : combinaison de niveau objet

Info

Publication number
EP3516467A1
EP3516467A1 EP16790768.2A EP16790768A EP3516467A1 EP 3516467 A1 EP3516467 A1 EP 3516467A1 EP 16790768 A EP16790768 A EP 16790768A EP 3516467 A1 EP3516467 A1 EP 3516467A1
Authority
EP
European Patent Office
Prior art keywords
information
tracks
object tracks
kinematic
detected objects
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP16790768.2A
Other languages
German (de)
English (en)
Inventor
Matthew Graham
Troy Jones
Kyra Horne
Scott Lennox
Jon D. Demerly
Stephen Sheldon Strong
Afrah Faiz Naik
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Charles Stark Draper Laboratory Inc
Veoneer US LLC
Original Assignee
Charles Stark Draper Laboratory Inc
Veoneer US LLC
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Charles Stark Draper Laboratory Inc, Veoneer US LLC filed Critical Charles Stark Draper Laboratory Inc
Publication of EP3516467A1 publication Critical patent/EP3516467A1/fr
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G05—CONTROLLING; REGULATING
    • G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
    • G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
    • G05D1/0088—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G01S13/865—Combination of radar systems with lidar systems
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/86—Combinations of radar systems with non-radar systems, e.g. sonar, direction finder
    • G01S13/867—Combination of radar systems with cameras
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/88—Radar or analogous systems specially adapted for specific applications
    • G01S13/93—Radar or analogous systems specially adapted for specific applications for anti-collision purposes
    • G01S13/931—Radar or analogous systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
    • G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
    • G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
    • G01S7/4802—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S13/00—Systems using the reflection or reradiation of radio waves, e.g. radar systems; Analogous systems using reflection or reradiation of waves whose nature or wavelength is irrelevant or unspecified
    • G01S13/66—Radar-tracking systems; Analogous systems
    • G01S13/72—Radar-tracking systems; Analogous systems for two-dimensional tracking, e.g. combination of angle and range tracking, track-while-scan radar
    • G01S13/723—Radar-tracking systems; Analogous systems for two-dimensional tracking, e.g. combination of angle and range tracking, track-while-scan radar by using numerical data

Definitions

  • vehicles can employ automated systems such as lane assist, pre- collision breaking, and rear cross-track detection. These systems can assist a driver of the vehicle from making human error and to avoid crashes with other vehicles, moving objects, or pedestrians. However, these systems only automate certain vehicle functions, and still rely on the driver of the vehicle for other operations.
  • a method includes converting sensor data of detected objects from a plurality of heterogeneous sensors to a common coordinate frame.
  • the method includes predicting position, velocity, orientation and bounding boxes of existing object tracks at a current measurement time.
  • the method further includes associating detected objects to existing object tracks by determining a similarity of at least two of kinematic information, geometric information, and object classification information based on the converted sensor data.
  • the method further includes updating the kinematic, geometric and object classification information for object tracks that are associated to detected objects.
  • the method further includes reporting a fused object list having a resulting set of updated object tracks.
  • the method, system, and computer readable medium are for providing situational awareness in an autonomous vehicle.
  • the method includes initiating new object tracks for detected objects that are unassociated to existing object tracks.
  • the method includes deleting object tracks that are outside of a field-of-view of the one or more heterogeneous sensors view of the autonomous vehicle.
  • the method includes deleting the object tracks further includes exempting object tracks from deletion if the object tracks are within a blind spot of the one or more heterogeneous sensors.
  • associating detected objects to existing object tracks further determines a similarity of at least three of kinematic information, geometric information, and object classification information.
  • the method includes associating detected feature tracks to existing object tracks by determining a similarity of at least two of kinematic information, geometric information, and object classification information.
  • kinematic information includes position information, velocity, information, and orientation information
  • geometric information includes bounding boxes and object contours
  • object classification information includes an object type.
  • a system for analyzing detected objects to provide situational awareness for an autonomous vehicle includes a preprocessing module configured to convert sensor data of detected objects from a plurality of heterogeneous sensors to a common coordinate frame.
  • the system further includes a track prediction module configured to predict position, velocity, orientation and bounding boxes of existing object tracks at a current measurement time.
  • the system further includes a data association module configured to associate detected objects to existing object tracks by determining a similarity of at least two of kinematic information, geometric information, and object classification information using the converted sensor data.
  • the system further includes a track update module configured to update the kinematic, geometric and object classification information for object tracks that are associated to detected objects.
  • the system further includes a reporting module configured to report a fused object list having a resulting set of updated object tracks.
  • a non-transitory computer-readable medium is configured to store instructions for operating an autonomous vehicle.
  • the instructions when loaded and executed by a processor, causes the processor to convert sensor data of detected objects from a plurality of heterogeneous sensors to a common coordinate frame.
  • the instructions further cause the processor to predict position, velocity, orientation and bounding boxes of existing object tracks at a current measurement time.
  • the instructions further cause the processor to associate detected objects to existing object tracks by determining a similarity of at least two of kinematic information, geometric information, and object classification information based on the converted sensor data.
  • the instructions further cause the processor to update the kinematic, geometric and object classification information for object tracks that are associated to detected objects.
  • the instructions further cause the processor to report a fused object list having a resulting set of updated object tracks.
  • Fig. 1 is a diagram illustrating steps in an embodiment of an automated control system of the Observe, Orient, Decide, and Act (OODA) model.
  • Fig. 2 is a block diagram of an embodiment of an autonomous vehicle high-level architecture.
  • Fig. 3 is a block diagram illustrating an embodiment of the sensor interaction controller (SIC), perception controller (PC), and localization controller (LC).
  • SIC sensor interaction controller
  • PC perception controller
  • LC localization controller
  • Fig. 4 is a block diagram illustrating an example embodiment of the automatic driving controller (ADC), vehicle controller (VC) and actuator controller.
  • ADC automatic driving controller
  • VC vehicle controller
  • actuator controller actuator controller
  • Fig. 5 is a diagram illustrating decision time scales of the ADC and VC.
  • Fig. 6 is a block diagram illustrating an example embodiment of the system controller, human interface controller (HC) and machine interface controller (MC).
  • HC human interface controller
  • MC machine interface controller
  • Fig. 7 is a block diagram illustrating an embodiment of an object fusion filter.
  • Fig. 8 A is a block diagram illustrating an embodiment of an object fusion server design.
  • Fig. 8B is a block diagram illustrating an embodiment of an object fusion module.
  • Fig. 9 is a block diagram illustrating an embodiment of track prediction.
  • Figs. 10A-D are diagrams illustrating elements of data association according to one embodiment.
  • Fig. 11 is a diagram illustrating an example embodiment of updating tracks.
  • Fig. 12 illustrates a computer network or similar digital processing environment in which embodiments of the present invention may be implemented.
  • Fig. 13 is a diagram of an example internal structure of a computer (e.g., client processor/device or server computers) in the computer system of Fig. 11.
  • a computer e.g., client processor/device or server computers
  • Fig. 1 is a diagram illustrating steps in an embodiment of an automated control system of the Observe, Orient, Decide, and Act (OODA) model.
  • Automated systems such as highly-automated driving systems, or, self-driving cars, or autonomous vehicles, employ an OODA model.
  • the observe virtual layer 102 involves sensing features from the world using machine sensors, such as laser ranging, radar, infra-red, vision systems, or other systems.
  • the orientation virtual layer 104 involves perceiving situational awareness based on the sensed information. Examples of orientation virtual layer activities are Kalman filtering, model based matching, machine or deep learning, and Bayesian predictions.
  • the decide virtual layer 106 selects an action from multiple objects to a final decision.
  • the act virtual layer 108 provides guidance and control for executing the decision.
  • Fig. 2 is a block diagram 200 of an embodiment of an autonomous vehicle high-level architecture 206.
  • the architecture 206 is built using a top-down approach to enable fully automated driving.
  • the architecture 206 is preferably modular such that it can be adaptable with hardware from different vehicle manufacturers.
  • the architecture 206 therefore, has several modular elements functionally divided to maximize these properties.
  • the modular architecture 206 described herein can interface with sensor systems 202 of any vehicle 204. Further, the modular architecture 206 can receive vehicle information from and communicate with any vehicle 204.
  • Elements of the modular architecture 206 include sensors 202, Sensor Interface Controller (SIC) 208, localization controller (LC) 210, perception controller (PC) 212, automated driving controller 214 (ADC), vehicle controller 216 (VC), system controller 218 (SC), human interaction controller 220 (HC) and machine interaction controller 222 (MC).
  • the observation layer of the model includes gathering sensor readings, for example, from vision sensors, Radar (Radio Detection And Ranging), LIDAR (Light Detection And
  • the sensors 202 shown in Fig. 2 shows such an observation layer.
  • Examples of the orientation layer of the model can include determining where a car is relative to the world, relative to the road it is driving on, and relative to lane markings on the road, shown by Perception Controller (PC) 212 and
  • LC Localization Controller
  • ADC Automatic Driving Controller
  • VC Vehicle Controller
  • Act layer include converting that corridor into commands to the vehicle's driving systems (e.g., steering sub-system, acceleration sub-system, and breaking sub-system) that direct the car along the corridor, such as actuator control 410 of Fig. 4.
  • a person of ordinary skill in the art can recognize that the layers of the system are not strictly sequential, and as observations change, so do the results of the other layers.
  • changing conditions on the road such as detection of another object, may direct the car to modify its corridor, or enact emergency procedures to prevent a collision.
  • the commands of the vehicle controller may need to be adjusted dynamically to compensate for drift, skidding, or other changes to expected vehicle behavior.
  • the module architecture 206 receives measurements from sensors 202. While different sensors may output different sets of information in different formats, the modular architecture 206 includes Sensor Interface Controller (SIC) 208, sometimes also referred to as a Sensor Interface Server (SIS), configured to translate the sensor data into data having a vendor-neutral format that can be read by the modular architecture 206. Therefore, the modular architecture 206 learns about the environment around the vehicle 204 from the vehicle's sensors, no matter the vendor, manufacturer, or configuration of the sensors. The SIS 208 can further tag each sensor's data with a metadata tag having its location and orientation in the car, which can be used by the perception controller to determine the unique angle, perspective, and blind spot of each sensor.
  • SIC Sensor Interface Controller
  • SIS Sensor Interface Server
  • the modular architecture 206 includes vehicle controller 216 (VC).
  • the VC 216 is configured to send commands to the vehicle and receive status messages from the vehicle.
  • the vehicle controller 216 receives status messages from the vehicle 204 indicating the vehicle's status, such as information regarding the vehicle's speed, attitude, steering position, braking status, and fuel level, or any other information about the vehicle's subsystems that is relevant for autonomous driving.
  • the modular architecture 206 based on the information from the vehicle 204 and the sensors 202, therefore can calculate commands to send from the VC 216 to the vehicle 204 to implement self-driving.
  • the functions of the various modules within the modular architecture 206 are described in further detail below.
  • the modular architecture 206 when viewing the modular architecture 206 at a high level, it receives (a) sensor information from the sensors 202 and (b) vehicle status information from the vehicle 204, and in turn, provides the vehicle instructions to the vehicle 204.
  • Such an architecture allows the modular architecture to be employed for any vehicle with any sensor configuration.
  • any vehicle platform that includes a sensor subsystem (e.g., sensors 202) and an actuation subsystem having the ability to provide vehicle status and accept driving commands (e.g., actuator control 410 of Fig. 4) can integrate with the modular architecture 206.
  • a sensor subsystem e.g., sensors 202
  • an actuation subsystem having the ability to provide vehicle status and accept driving commands (e.g., actuator control 410 of Fig. 4) can integrate with the modular architecture 206.
  • the modular architecture 206 various modules work together to implement automated driving according to the OODA model.
  • the sensors 202 and SIC 208 reside in the "observe" virtual layer.
  • the SIC 208 receives measurements (e.g., sensor data) having various formats.
  • the SIC 208 is configured to convert vendor-specific data directly from the sensors to vendor-neutral data.
  • the set of sensors 202 can include any brand of Radar, LIDAR, image sensor, or other sensors, and the modular architecture 206 can use their perceptions of the environment effectively.
  • the measurements output by the sensor interface server are then processed by perception controller (PC) 212 and localization controller (LC) 210.
  • the PC 212 and LC 210 both reside in the "orient" virtual layer of the OODA model.
  • the LC 210 determines a robust world-location of the vehicle that can be more precise than a GPS signal, and still determines the world-location of the vehicle when there is no available or an inaccurate GPS signal.
  • the LC 210 determines the location based on GPS data and sensor data.
  • the PC 212 on the other hand, generates prediction models representing a state of the environment around the car, including objects around the car and state of the road.
  • Fig. 3 provides further details regarding the SIC 208, LC 210 and PC 212.
  • Automated driving controller 214 and vehicle controller 216 (VC) receive the outputs of the perception controller and localization controller.
  • the ADC 214 and VC 216 reside in the "decide" virtual layer of the OODA model.
  • the ADC 214 is responsible for destination selection, route and lane guidance, and high-level traffic surveillance.
  • the ADC 214 further is responsible for lane selection within the route, and identification of safe harbor areas to diver the vehicle in case of an emergency.
  • the ADC 214 selects a route to reach the destination, and a corridor within the route to direct the vehicle.
  • the ADC 214 passes this corridor onto the VC 216. Given the corridor, the VC 216 provides lower level driving functions to direct the vehicle through the corridor safely.
  • the VC 216 first determines the best way to maneuver through the corridor while providing comfort to the driver, an ability to reach safe harbor, emergency maneuverability, and ability to follow the vehicle's trajectory. In emergency situation, the VC 216 overrides the corridor provided by the ADC 214 and immediately guides the car into a safe harbor corridor, returning to the corridor provided by the ADC 214 when it is safe to do so. The VC 216, after determining how to maneuver the vehicle, including safety maneuvers, then provides actuation commands to the vehicle 204, which executes the commands in its steering, throttle, and braking subsystems. This element of the VC 216 is therefore in the "act" virtual layer of the OODA model. Fig. 4 describes the ADC 214 and VC 216 in further detail.
  • the modular architecture 206 further coordinates communication with various modules through system controller 218 (SC).
  • SC system controller 218
  • the SC 218 enables operation of human interaction controller 220 (HC) and machine interaction controller 222 (MC).
  • HC human interaction controller 220
  • MC machine interaction controller 222
  • the HC 220 provides information about the autonomous vehicle's operation in a human readable format based on status messages coordinated by the system controller.
  • the HC 220 further allows for human input to be factored into the car's decisions.
  • the HC 220 enables the operator of the vehicle to enter or modify the destination or route of the vehicle, as one example.
  • the SC 218 interprets the operator's input and relays the information to the VC 216 or ADC 214 as necessary.
  • the MC 222 can coordinate messages with other machines or vehicles.
  • other vehicles can electronically and wirelessly transmit turn signals to autonomous vehicles, and the MC 222 can receive such information, and relay it to the VC 216 and ADC 214 via the SC 218.
  • the MC 222 can send information to other vehicles wirelessly.
  • the MC 222 can receive a notification that the vehicle intends to turn.
  • the MC 222 receives this information via the VC 216 sending a status message to the SC 218, which relays the status to the MC 222.
  • other examples of machine communication can also be implemented.
  • Fig. 3 is a block diagram 300 illustrating an embodiment of the sensor interaction controller 304 (SIC), perception controller (PC) 306, and localization controller (LC) 308.
  • a sensor array 302 of the vehicle can include various types of sensors, such as a camera 302a, radar 302b, LIDAR 302c, GPS 302d, IMU 302e, or vehicle-to-everything (V2X) 302f. Each sensor sends individual vendor defined data types to the SIC 304.
  • the camera 302a sends object lists and images
  • the radar 302b sends object lists, and in-phase/quadrature (IQ) data
  • the LIDAR 302c sends object lists and scan points
  • the GPS 302d sends position and velocity
  • the IMU 302e sends acceleration data
  • the V2X 302f controller sends tracks of other vehicles, turn signals, other sensor data, or traffic light data.
  • the SIC 304 monitors and diagnoses faults at each of the sensors 302a-f.
  • the SIC 304 isolates the data from each sensor from its vendor specific package and sends vendor neutral data types to the perception controller (PC) 306 and localization controller 308 (LC).
  • the SIC 304 forwards localization feature measurements and position and attitude measurements to the LC 308, and forwards tracked object measurements, driving surface measurements, and position & attitude measurements to the PC 306.
  • the SIC 304 can further be updated with firmware so that new sensors having different formats can be used with the same modular architecture.
  • the LC 308 fuses GPS and FMU data with Radar, Lidar, and Vision data to determine a location in GPS location, and to increase the precision of the GPS location.
  • the LC 308 reports that robust determined location, velocity, and attitude to the PC 306.
  • the LC 308 further monitors measurements representing position, velocity, and attitude data for accuracy relative to each other, such that if one sensor measurement fails or becomes degraded, such as a GPS signal in a city, the LC 308 can correct for it.
  • the PC 306 identifies and locates objects around the vehicle based on the sensed information.
  • the PC 306 further estimates drivable surface regions surrounding the vehicle, and further estimates other surfaces such as road shoulders or drivable terrain in the case of an emergency.
  • the PC 306 further provides a stochastic prediction of future locations of objects.
  • the PC 306 further stores a history of objects and drivable surfaces.
  • the PC 306 outputs two predictions, a strategic prediction, and a tactical prediction.
  • the tactical prediction represents the world around 2-4 seconds into the future, which only predicts the nearest traffic and road to the vehicle. This prediction includes a free space harbor on shoulder of the road or other location.
  • the strategic prediction is a long term prediction that predicts areas of the car's visible environment beyond the visible range. This prediction is for greater than four seconds into the future, but has a higher uncertainty than the tactical prediction because objects (e.g., cars and people) may change their currently observed behavior in an unanticipated manner.
  • the strategic prediction has higher uncertainty because it assumes that currently observed behavior continues for the entire prediction period.
  • Such a prediction can also be based on sensor measurements from external sources including other autonomous vehicles, manual vehicles with a sensor system and sensor communication network, sensors positioned near or on the roadway or received over a network from transponders on the objects, and traffic lights, signs, or other signals configured to communicate wirelessly with the autonomous vehicle.
  • Fig. 4 is a block diagram 400 illustrating an example embodiment of the automatic driving controller (ADC) 402, vehicle controller (VC) 404 and actuator controller 410.
  • ADC automatic driving controller
  • VC vehicle controller
  • actuator controller 410 The ADC 402 and VC 404 execute the "decide" virtual layer of the OODA model.
  • the ADC 402 based on destination input by the operator and current position, first creates an overall route from the current position to the destination including a list of roads and junctions between roads in order to reach the destination.
  • This strategic route plan may be based on traffic conditions, and can change based on updating traffic conditions, however such changes are generally enforced for large changes in estimated time of arrival (ETA).
  • ETA estimated time of arrival
  • the ADC 402 plans a safe, collision-free, corridor for the autonomous vehicle to drive through based on the surrounding objects and permissible drivable surface - both supplied by the PC.
  • This corridor is continuously sent as a request to the VC 404 and is updated as traffic and other conditions change.
  • the VC 404 receives the updates to the corridor in real time.
  • the ADC 402 receives back from the VC 404 the current actual trajectory of the vehicle, which is also used to modify the next planned update to the driving corridor request.
  • the ADC 402 generates a strategic corridor for the vehicle to navigate.
  • the ADC 402 generates the corridor based on predictions of the free space on the road in the strategic/tactical prediction.
  • the ADC 402 further receives the vehicle position information and vehicle attitude information from the perception controller of Fig. 3.
  • the VC 404 further provides the ADC 402 with an actual trajectory of the vehicle from the vehicle's actuator control 410. Based on this information, the ADC 402 calculates feasible corridors to drive the road. In the example of being on an empty road, the corridor may follow the lane ahead of the car.
  • the ADC 402 can determine whether there is free space in a passing lane and in front of the car to safely execute the pass.
  • the ADC 402 can automatically calculate based on (a) the current distance to the car to be passed, (b) amount of drivable road space available in the passing lane, (c) amount of free space in front of the car to be passed, (d) speed of the vehicle to be passed, (e) current speed of the autonomous vehicle, and (f) known acceleration of the autonomous vehicle, a corridor for the vehicle to travel through to execute the pass maneuver.
  • the ADC 402 can determine a corridor to switch lanes when approaching a highway exit. In addition to all of the above factors, the ADC 402 monitors the planned route to the destination and, upon approaching a junction, calculates the best corridor to safely and legally continue on the planned route.
  • the ADC 402 further calculates a tactical trajectory within the corridor, which allows the vehicle to maintain a safe separation between objects.
  • the tactical trajectory also includes a backup safe harbor trajectory in the case of an emergency, such as a vehicle unexpectedly decelerating or stopping, or another vehicle swerving in front of the
  • the ADC 402 the provides the requested trajectory corridor 406 to the VC 404, which works in tandem with the ADC 402 to allow the vehicle to navigate the corridor.
  • the requested trajectory corridor 406 places geometric and velocity constraints on trajectory for a number of seconds into the future.
  • the VC 404 determines a route to maneuver within the corridor 406.
  • the VC 404 bases its maneuvering decisions from the tactical / maneuvering prediction received from the perception controller and the position of the vehicle and the attitude of the vehicle. As described previously, the tactical / maneuvering prediction is for a shorter time period, but has less uncertainty. Therefore, for lower-level maneuvering and safety calculations, the VC 404 effectively uses the tactical / maneuvering prediction to plan collision-free trajectories within requested corridor 406. As needed in emergency situations, the VC 404 plans trajectories outside the corridor 406 to avoid collisions with other objects.
  • the VC 404 determines, based on the requested corridor 406, the current velocity and acceleration of the car, and the nearest objects, how to drive the car through that corridor 406 while avoiding collisions with objects and remain on the drivable surface. As necessary to avoid collisions, the VC 404 may be required to command a maneuver suddenly outside of the requested corridor from the ADC 402. This emergency maneuver can be initiated entirely by the VC 404 as it has faster response times than the ADC 402 to imminent collision threats. This capability isolates the safety critical collision avoidance responsibility within the VC 404. The VC 404 sends maneuvering commands to the actuators that control steering, throttling, and braking of the vehicle platform.
  • the VC 404 executes its maneuvering strategy by sending a current vehicle trajectory 408 having driving commands (e.g., steering, throttle, braking) to the vehicle's actuator controls 410.
  • the vehicle's actuator controls 410 apply the commands to the car's respective steering, throttle, and braking systems.
  • the VC 404 sending the trajectory 408 to the actuator controls represent the "Act" virtual layer of the OODA model.
  • the VC is the only component needing configuration to control a specific model of car (e.g., format of each command, acceleration performance, turning performance, and braking performance), whereas the ADC remaining highly agnostic to the specific vehicle capacities.
  • the VC 404 can be updated with firmware configured to allow interfacing with particular vehicle's actuator control systems, or a fleet-wide firmware update for all vehicles.
  • Fig. 5 is a diagram 500 illustrating decision time scales of the ADC 402 and VC 404.
  • the ADC 402 implements higher-level, strategic 502 and tactical 504 decisions by generating the corridor.
  • the ADC 402 therefore implements the decisions having a longer range/ or time scale.
  • the estimate of world state used by the ADC 402 for planning strategic routes and tactical driving corridors for behaviors such as passing or making turns has higher uncertainty, but predicts longer into the future, which is necessary for planning these autonomous actions.
  • the strategic predictions have high uncertainty because they predict beyond the car's visible range, relying solely on non-vision technologies, such as Radar, for predictions of objects far away from the car.
  • Many tactical decisions, such as passing a car at highway speed require perception Beyond the Visible Range (BVR) of an autonomous vehicle (e.g., 100m or greater), whereas all maneuverability 506 decisions are made based on locally perceived objects to avoid collisions.
  • BVR Visible Range
  • the VC 404 generates maneuverability decisions 506 using maneuverability predictions that are short time frame/range predictions of object behaviors and the driving surface. These maneuverability predictions have a lower uncertainty because of the shorter time scale of the predictions, however, they rely solely on measurements taken within visible range of the sensors on the autonomous vehicle. Therefore, the VC 404 uses these maneuverability predictions (or estimates) of the state of the environment immediately around the car for fast response planning of collision-free trajectories for the autonomous vehicle.
  • the VC 402 issues actuation commands, on the lowest end of the time scale, representing the execution of the already planned corridor and maneuvering through the corridor.
  • Fig. 6 is a block diagram 600 illustrating an example embodiment of the system controller 602, human interface controller 604 (HC) and machine interface controller 606 (MC).
  • the human interaction controller 604 (HC) receives input command requests from the operator and provides outputs to the operator, passengers of the vehicle, and humans external to the autonomous vehicle.
  • the HC 604 provides the operator and passengers (via visual, audio, haptic, or other interfaces) a human-understandable representation of the system status and rationale of the decision making of the autonomous vehicle.
  • the HC 604 can display the vehicle's long-term route, or planned corridor and safe harbor areas.
  • the HC 604 reads sensor measurements about the state of the driver, allowing the HC 604 to monitor the availability of the driver to assist with operations of the car at any time.
  • a sensor system within the vehicle could sense whether the operator has hands on the steering wheel. If so, the HC 604 can signal that a transition to operator steering can be allowed, but otherwise, the HC 604 can prevent a turnover of steering controls to the operator.
  • the HC 604 can synthesize and summarize decision making rationale to the operator, such as reasons why it selected a particular route.
  • the machine interaction controller 606 interacts with other autonomous vehicles or automated system to coordinate activities such as formation driving or traffic management.
  • the MC 606 reads the internal system status and generates an output data type that can be read by collaborating machine systems, such as the V2X data type. This status can be broadcast over a network by collaborating systems.
  • the MC 606 can translate any command requests from external machine systems (e.g., slow down, change route, merge request, traffic signal status) into commands requests routed to the SC for arbitration against the other command requests from the HC 604.
  • the MC 606 can further authenticate (e.g., using signed messages from other trusted manufacturers) messages from other systems to ensure that they are valid and represent the environment around the car. Such an
  • the system controller 602 serves as an overall manager of the elements within the architecture.
  • the SC 602 aggregates the status data from all of the system elements to determine total operational status, and sends commands to the elements to execute system functions. If elements of the system report failures, the SC 602 initiates diagnostic and recovery behaviors to ensure autonomous operation such that the vehicle remains safe. Any transitions of the vehicle to/from an automated state of driving are approved or denied by the SC 602 pending the internal evaluation of operational readiness for automated driving and the availability of the human driver.
  • a system and method fuses objects in a perception layer/module.
  • Fig. 7 is a block diagram 700 illustrating an example embodiment of the object level fusion server 702 receiving information from all sensors, such as vision sensors 702, LIDAR 706, and RADAR 704.
  • the sensors include two stereo vision sensors, RADAR, and LIDAR.
  • Stereo vision702 uses imaging technology to output bounded boxes of objects (e.g., cars, bikes, people, lanes, signs, and objects) found in the field of view of the cameras.
  • the RADAR systems 704 track locations and ranges of any objects that reflect RADAR - but cannot classify or identify those objects.
  • Fig. 8 A is a block diagram 800 illustrating an example embodiment of an object fusion server 814.
  • Each sensor 802, 804, and 806 communicates with a server module that may include a dedicated processor configured to run a tracking loop, process the raw data of each sensor individually, and output object data, instead of raw data of the sensor.
  • each object output by each sensor server 802, 804, and 806 is in a coordinate system with axes X, Y, and Z, where X is the axis from the back of the car to the front of the car, Y is the axes from the left of the car to the right of the car, and Z is the axis from the ground going upward.
  • X is the axis from the back of the car to the front of the car
  • Y is the axes from the left of the car to the right of the car
  • Z is the axis from the ground going upward.
  • each sensor server converts the object data to a message format agnostic of the sensor or sensor's format.
  • the Perception Object Fusion Server 814 then combines the messages into a fusion tracked object list message.
  • the object fusion module 814 then analyzes all of the data - vision, LIDAR, and RADAR - to determine a complete picture of objects surrounding the self-driving car.
  • Previous object-level fusion systems only used RADAR systems, or only used vision systems, but did not use both.
  • One unique challenge overcome in combining data from all systems is that while LIDAR and RADAR output objects as points within a coordinate- system as well as a range to that point, vision systems output a bounded box that the object is likely within, but having a less accurate range. Therefore, the data of the different systems needs to be associated in an accurate way.
  • RADAR objects that are within bounded boxes can be determined to be the same objects.
  • a RADAR ping just outside a bounded box could be a separate object.
  • the object fusion module reconciles the data from all sensors to determine a most likely scenario of all objects near the self-driving car.
  • the object fusion module employs Kalman filtering and probabilistic data association.
  • multi-hypothesis tracking, probability hypothesis density filtering or other techniques can also be employed in other embodiments.
  • Fig. 8B is a more detailed block diagram 850 of the object fusion module.
  • Each sensor's object data 858, 860, and 862 is pre-processed by one or more preprocessors 864.
  • a data association block associates the objects from the three different domains (e.g., LIDAR, RADAR, and Vision System) to determine an object track.
  • a radar tracker 874 additionally processes the radar object list 870 before forwarding the data to the data association module 872.
  • the data association module 872 applies probabilistic data association using a global nearest neighbor method to associate sensor object tracks to existing fused object tracks.
  • the global nearest neighbor method selects the association that maximizes the sum of the similarities for each associated sensor object track and fused object track.
  • the data association module 872 measures the similarity of object tracks using at least two of kinematic information, geometric information, and object classification information based on the converted sensor data. In some embodiments, the data association module 872 measures the similarity of object tracks using all three of kinematic information, geometric information, and object classification information based on the converted sensor data.
  • the data association module 872 sends unassociated object tracks 878 to a track management module 882, which initializes new tracks 888, which are sent to the track prediction module 884.
  • the data association module 872 associates the remaining objects to existing fused object tracks 876, which are passed to a track update module 880.
  • the track update module 880 sends updated tracks 892 to track prediction module 884.
  • the track update module 880 uses a Bayesian filter to update the object tracks and also to predict next time step tracks.
  • the track update module employs a Kalman filter.
  • extended Kalman filters, unscented Kalman filters or particle filters could also be employed in other embodiments.
  • the track prediction module 884 issues a prediction 886 of existing object tracks that are sent to the data association module 872. That prediction is used for data association in future time steps. This loop continues as long as the car is active.
  • the track update module 880 further removes old tracks when the sensors detect that the object is no longer in view. However, the track update module 880 can create an exception for objects in known blind spots of the sensors.
  • Fig. 9 is a diagram 900 illustrating track prediction.
  • the goal of track prediction is to track location and velocity of an object at the current measurement time given a most recent tack of estimates.
  • estimation uncertainty grows after prediction the further out in time the prediction is.
  • a bounding box corresponding with the object should increase, and a degree of certainty corresponding to the object decreases as the prediction is further distant in time from the present.
  • Figs. 10A-D are diagrams 1000, 1020, 1040, and 1060 illustrating data
  • the data association module associates a most recent set of measurements to existing tracks (Fig. 10A). First, the system calculates association scores for each
  • the scores can be distance from the objects to each other. Then, the system disallows or removes associations that are not plausible, as in Fig. IOC. Then, the data association module chooses the measurement-to- track assignment that minimizes the sum of the association scores. Then, based on the associated measurement, the tracks are updated. Updating the track considers the predicted track and the measured values to come to a new track.
  • Fig. 11 is a diagram 1100 illustrating an example embodiment of updating tracks. Tracks can be updated based on how objects are associated. For example, if two objects are associated, the system can update the tracks to combine the tracks of the two objects, having the knowledge that the objects are associated and likely have the same or similar tracks.
  • Fig. 12 illustrates a computer network or similar digital processing environment in which embodiments of the present invention may be implemented.
  • Client computer(s)/devices 50 and server computer(s) 60 provide processing, storage, and input/output devices executing application programs and the like.
  • the client computer(s)/devices 50 can also be linked through communications network 70 to other computing devices, including other client devices/processes 50 and server computer(s) 60.
  • the communications network 70 can be part of a remote access network, a global network (e.g., the Internet), a worldwide collection of computers, local area or wide area networks, and gateways that currently use respective protocols (TCP/IP, Bluetooth®, etc.) to communicate with one another.
  • Other electronic device/computer network architectures are suitable.
  • Fig. 13 is a diagram of an example internal structure of a computer (e.g., client processor/device 50 or server computers 60) in the computer system of Fig. 12.
  • Each computer 50, 60 contains a system bus 79, where a bus is a set of hardware lines used for data transfer among the components of a computer or processing system.
  • the system bus 79 is essentially a shared conduit that connects different elements of a computer system (e.g., processor, disk storage, memory, input/output ports, network ports, etc.) that enables the transfer of information between the elements.
  • Attached to the system bus 79 is an I/O device interface 82 for connecting various input and output devices (e.g., keyboard, mouse, displays, printers, speakers, etc.) to the computer 50, 60.
  • a network interface 86 allows the computer to connect to various other devices attached to a network (e.g., network 70 of Fig. 12).
  • Memory 90 provides volatile storage for computer software instructions 92 and data 94 used to implement an embodiment of the present invention (e.g., sensor interface controller, perception controller, localization controller, automated driving controller, vehicle controller, system controller, human interaction controller, machine interaction controller, object fusion server, preprocessors, data association module, track updates module, track management module, track prediction module, and radar tracker detailed above).
  • Disk storage 95 provides non-volatile storage for computer software instructions 92 and data 94 used to implement an embodiment of the present invention.
  • a central processor unit 84 is also attached to the system bus 79 and provides for the execution of computer instructions.
  • the processor routines 92 and data 94 are a computer program product (generally referenced 92), including a non-transitory computer-readable medium (e.g., a removable storage medium such as one or more DVD-ROM's, CD-ROM's, diskettes, tapes, etc.) that provides at least a portion of the software instructions for the invention system.
  • the computer program product 92 can be installed by any suitable software installation procedure, as is well known in the art.
  • at least a portion of the software instructions may also be downloaded over a cable communication and/or wireless connection.
  • the invention programs are a computer program propagated signal product embodied on a propagated signal on a propagation medium (e.g., a radio wave, an infrared wave, a laser wave, a sound wave, or an electrical wave propagated over a global network such as the Internet, or other network(s)).
  • a propagation medium e.g., a radio wave, an infrared wave, a laser wave, a sound wave, or an electrical wave propagated over a global network such as the Internet, or other network(s)
  • Such carrier medium or signals may be employed to provide at least a portion of the software instructions for the present invention routines/program 92.

Landscapes

  • Engineering & Computer Science (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Electromagnetism (AREA)
  • Health & Medical Sciences (AREA)
  • Business, Economics & Management (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Game Theory and Decision Science (AREA)
  • Medical Informatics (AREA)
  • Aviation & Aerospace Engineering (AREA)
  • Automation & Control Theory (AREA)
  • Traffic Control Systems (AREA)
  • Control Of Driving Devices And Active Controlling Of Vehicle (AREA)

Abstract

Les systèmes de voiture sans conducteur précédents peuvent détecter des objets séparément avec des systèmes de vision, des systèmes RADAR ou des systèmes LIDAR. Dans un mode de réalisation de la présente invention, un module de combinaison d'objets normalise une sortie de capteur provenant de systèmes de vision, RADAR et LIDAR en un format commun. Ensuite, le système combine les données de capteur de niveau objet dans l'ensemble des systèmes en associant tous les objets détectés et en prédisant des trajectoires pour tous les objets. Le présent système améliore les systèmes précédents en utilisant les données provenant de tous les capteurs réunies pour développer un ensemble unique de connaissances concernant les objets autour de la voiture sans conducteur, au lieu d'un fonctionnement individuel de chaque capteur.
EP16790768.2A 2016-09-29 2016-09-29 Véhicule autonome : combinaison de niveau objet Withdrawn EP3516467A1 (fr)

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/US2016/054423 WO2018063241A1 (fr) 2016-09-29 2016-09-29 Véhicule autonome : combinaison de niveau objet

Publications (1)

Publication Number Publication Date
EP3516467A1 true EP3516467A1 (fr) 2019-07-31

Family

ID=57227078

Family Applications (1)

Application Number Title Priority Date Filing Date
EP16790768.2A Withdrawn EP3516467A1 (fr) 2016-09-29 2016-09-29 Véhicule autonome : combinaison de niveau objet

Country Status (3)

Country Link
EP (1) EP3516467A1 (fr)
JP (1) JP2019530608A (fr)
WO (1) WO2018063241A1 (fr)

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102019102920A1 (de) * 2019-02-06 2020-08-06 Bayerische Motoren Werke Aktiengesellschaft Verfahren und eine Vorrichtung zur Sensordatenfusion für ein Fahrzeug
US10984543B1 (en) 2019-05-09 2021-04-20 Zoox, Inc. Image-based depth data and relative depth data
US10937178B1 (en) * 2019-05-09 2021-03-02 Zoox, Inc. Image-based depth data and bounding boxes
US11087494B1 (en) 2019-05-09 2021-08-10 Zoox, Inc. Image-based depth data and localization
CN113112643A (zh) * 2020-01-13 2021-07-13 北京地平线机器人技术研发有限公司 预测轨迹的评价方法、装置以及电子设备和存储介质
CN111288984B (zh) * 2020-03-04 2021-12-14 武汉大学 一种基于车联网的多车联合绝对定位的方法
CN112572471B (zh) * 2020-12-08 2022-11-04 西人马帝言(北京)科技有限公司 自动驾驶方法、装置、电子设备及计算机存储介质
CN113342057B (zh) * 2021-08-05 2021-12-03 特金智能科技(上海)有限公司 轨迹的融合方法、装置、无人机探测系统、设备和介质
WO2023139706A1 (fr) * 2022-01-19 2023-07-27 日本電気株式会社 Dispositif de traitement

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9472097B2 (en) * 2010-11-15 2016-10-18 Image Sensing Systems, Inc. Roadway sensing systems
US9381916B1 (en) * 2012-02-06 2016-07-05 Google Inc. System and method for predicting behaviors of detected objects through environment representation
US9547989B2 (en) * 2014-03-04 2017-01-17 Google Inc. Reporting road event data and sharing with other vehicles

Also Published As

Publication number Publication date
WO2018063241A1 (fr) 2018-04-05
JP2019530608A (ja) 2019-10-24

Similar Documents

Publication Publication Date Title
US10599150B2 (en) Autonomous vehicle: object-level fusion
US10377375B2 (en) Autonomous vehicle: modular architecture
US10963462B2 (en) Enhancing autonomous vehicle perception with off-vehicle collected data
US20180087907A1 (en) Autonomous vehicle: vehicle localization
EP3516467A1 (fr) Véhicule autonome : combinaison de niveau objet
US11314249B2 (en) Teleoperation system and method for trajectory modification of autonomous vehicles
EP3516422A1 (fr) Localisation de véhicule autonome
US11249474B2 (en) Safety of autonomous vehicles using a virtual augmented support environment
US10860019B2 (en) Planning autonomous motion
US9507346B1 (en) Teleoperation system and method for trajectory modification of autonomous vehicles
US11513518B2 (en) Avoidance of obscured roadway obstacles
WO2018063250A1 (fr) Véhicule autonome à architecture modulaire
US20180061237A1 (en) Modifying Behavior of Autonomous Vehicle Based on Advanced Predicted Behavior Analysis of Nearby Drivers
JP2018152056A (ja) 視界に制限のある交差点への接近のためのリスクベースの運転者支援
KR101439019B1 (ko) 차량 제어 장치 및 그 자율 주행 방법
KR20220054534A (ko) 거동 규칙 검사를 사용한 차량 작동
EP4057252A1 (fr) Dispositif, procédé et programme de traitement d'informations
US20230127379A1 (en) Method and system for remote assistance of an autonomous agent
WO2018199941A1 (fr) Amélioration de la perception de véhicule autonome à l'aide de données collectées hors véhicule
EP4285083A1 (fr) Procédés et système pour générer une carte de niveau voie pour une zone d'intérêt pour la navigation d'un véhicule autonome
CN117813227A (zh) 用于自主交通工具的互补控制系统
EP3454269A1 (fr) Planification de mouvements autonomes
Karnati et al. Artificial Intelligence in Self Driving Cars: Applications, Implications and Challenges
US20230399026A1 (en) State Identification For Road Actors With Uncertain Measurements Based On Compliant Priors
Anderson et al. Constraint-based semi-autonomy for unmanned ground vehicles using local sensing

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20190425

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20200603