JP2019530608A - Autonomous vehicle with object level fusion - Google Patents

Abstract

The present invention provides an autonomous vehicle method improved over the prior art. Sensor data about a detected object, a process of converting sensor data from a plurality of different types of sensors into a common coordinate frame, and a position, speed, and direction at the current measurement time of an existing object trajectory And a process of predicting a bounding box and a process of associating a detected object with an existing object trajectory by obtaining at least two similarities among motion information, geometric information, and object type information based on the converted sensor data And, for the object trajectory associated with the detected object, update the motion information, geometric information, and object type information, and report a fused object list that includes the updated set of object trajectories. A process. [Selection] Figure 2

Description

Related applications

  The present application is “Autonomous Vehicle: Vehicle Localization” by Paul DeBitetto, Matthew Graham, Troy Jones and Peter Lommel, filed at the same time on September 29, 2016 (agent management number 5000.1006- 000 (CSDL-2489)), and “Autonomous Vehicle: Modular Architecture” by Troy Jones, Scott Lennox, John Sgueglia and Jon Demerly, filed simultaneously on September 29, 2016 ( Related to agent management number 5000.1007-000 (CSDL-2490)).

  The entire teachings of the above application are incorporated herein by reference.

  Some vehicles today employ automated systems such as lane assist, pre-collision braking, and rear cross track detection. These systems may help the vehicle driver avoid human error and avoid collisions with other vehicles, moving objects or pedestrians.

  However, these systems only automate some vehicle functions and still rely on the vehicle driver for other operations.

  In one embodiment, the method comprises converting sensor data from a plurality of disparate sensors into a common coordinate frame for sensor objects. The method comprises the step of predicting the position, velocity, orientation and bounding box at the current measurement time of an existing object trajectory. The method further includes associating a detected object with an existing object trajectory by determining similarity between at least two of motion information, geometric information, and object type information based on the converted sensor data. . The method further includes the step of updating the motion information, the geometric information, and the object type information for the object trajectory associated with the detected object. The method further comprises reporting an object list that is a fused object list that includes the resulting set of updated object trajectories. In one embodiment, the method, system, and computer readable medium provide situational awareness to autonomous vehicles.

  In one embodiment, the method comprises initiating a new object trajectory for a detected object that is not associated with an existing object trajectory.

  In one embodiment, the method comprises deleting an object trajectory that is outside the field of view of at least one of the heterogeneous sensors of an autonomous vehicle.

  In one embodiment, the method comprises a step of deleting the object trajectory, which further excludes the object trajectory from deletion if the object trajectory is within a blind spot of the at least one heterogeneous sensor. Including sub-processes.

  In one embodiment, the process of associating the detected object with an existing object trajectory further determines at least three similarities of motion information, geometric information, and object type information.

  In one embodiment, the method comprises associating a detected feature trajectory with an existing object trajectory by determining similarity between at least two of motion information, geometric information, and object type information.

  In one embodiment, the motion information includes position information, velocity information, and orientation information, the geometric information includes a bounding box and an object outline, and the object type information includes an object type.

  In one embodiment, a system for analyzing a detected object to provide situational awareness of an autonomous vehicle is sensor data about the detected object, wherein the sensor data from multiple dissimilar sensors is converted into a common coordinate frame Pre-processing means configured to do. The system further comprises trajectory prediction means configured to predict position, velocity, orientation and bounding box at the current measurement time of an existing object trajectory. The system is further configured to associate a detected object with an existing object trajectory by determining at least two similarities among motion information, geometric information, and object type information using the converted sensor data. Provided with data association means. The system further includes trajectory updating means configured to update the motion information, the geometric information, and the object type information for the object trajectory associated with the detected object. The system further comprises reporting means configured to report an object list that is a fused object list and includes the obtained set of updated object trajectories.

  In one embodiment, a non-transient computer readable medium is configured to store instructions for operating an autonomous vehicle. The instructions, when loaded and executed by a processor, cause the processor to execute a procedure for converting sensor data about a detection object, which is sensor data from a plurality of different types of sensors, into a common coordinate frame. The instructions further cause the processor to execute a procedure for predicting position, velocity, orientation and bounding box at the current measurement time of an existing object trajectory. The instruction further determines the detected object as an existing object trajectory by obtaining at least two similarities among motion information, geometric information, and object type information based on the converted sensor data. The procedure to associate is executed. The instructions further cause the processor to execute a procedure for updating the motion information, the geometric information, and the object type information for an object trajectory associated with a detected object. The instructions further cause the processor to execute a procedure for reporting a merged object list that includes an obtained set of updated object trajectories.

  The foregoing will become apparent from the following more detailed description of the exemplary embodiments of the invention illustrated in the accompanying drawings. Throughout the different figures, the same reference signs refer to the same elements / components. The drawings are not necessarily to scale, but rather emphasis is placed on illustrating embodiments of the invention.

It is a figure which shows each step in one Embodiment of the automatic control system of a model of observation, situation judgment, decision making, and action (OODA). 1 is a block diagram of one embodiment of a higher-order architecture for an autonomous vehicle. FIG. It is a block diagram which shows one Embodiment of a sensor interaction control part (SIC), a recognition control part (PC), and a position specific control part (LC). It is a block diagram which shows one illustrative embodiment of an automatic driving | operation control part (ADC), a vehicle control part (VC), and a drive part (actuator) control part. It is a figure which shows the decision time scale of ADC and VC. FIG. 3 is a block diagram illustrating an exemplary embodiment of a system controller, a human interface controller (HC), and a machine interface controller (MC). It is a block diagram which shows one Embodiment of an object fusion filter. FIG. 6 is a block diagram illustrating one embodiment of an object fusion server design. FIG. 6 is a block diagram illustrating one embodiment of an object fusion module. It is a block diagram which shows one Embodiment of a locus | trajectory (track) prediction. It is a figure which shows one element of the data association (association) in one Embodiment. It is a figure which shows the other element of the data relation in one Embodiment. FIG. 6 is a diagram illustrating yet another element of data association in one embodiment. FIG. 6 is a diagram illustrating yet another element of data association in one embodiment. FIG. 6 illustrates an exemplary embodiment of trajectory update. FIG. 2 illustrates a computer network or similar digital processing environment in which embodiments of the invention can be implemented. FIG. 13 is a diagram of an exemplary internal structure of a computer (eg, client processor / device, server computer, etc.) in the computer system of FIG.

  In the following, exemplary embodiments of the invention will be described.

  FIG. 1 is a diagram illustrating each step in an embodiment of an automatic control system for observation, situation determination, decision making, and behavior (OODA) models. Automated systems such as highly automated driving systems and autonomous vehicles (ie autonomous vehicles) employ the OODA model. The observation virtual layer 102 involves detecting features from the real world (Earth) using machine sensors such as laser rangefinders, radar, infrared, or vision systems. The situation determination virtual layer 104 involves performing situation recognition based on detected information. Examples of activities of the situation determination virtual layer include Kalman filtering, model-based matching, machine learning or deep learning, and Bayesian prediction. The decision making virtual layer 106 selects an action from a plurality of objects and makes a final decision. The behavioral virtual layer 108 provides guidance and control to make that decision. FIG. 2 is a block diagram 200 of one embodiment of an autonomous vehicle higher-order architecture 206. Architecture 206 is built using a top-down approach to allow fully automated operation. The architecture 206 is also preferably a modular architecture so that it can be adapted to different vehicle manufacturer hardware. Accordingly, the architecture 206 comprises a plurality of modular elements that are functionally divided to maximize the hardware characteristics of different vehicle manufacturers. In one embodiment, the modular architecture 206 described herein can interface to the sensor system 202 of any vehicle 204. The modular architecture 206 can further receive vehicle information from any vehicle 204 and can communicate with any vehicle 204.

  The components of the modular architecture 206 include a sensor 202, a sensor interface control unit (SIC) 208, a position specifying control unit (LC) 210, a recognition control unit (PC) 212, an automatic driving control unit 214 (ADC), and vehicle control. A unit 216 (VC), a system control unit 218 (SC), a human interaction control unit 220 (HC), and a machine interaction control unit 222 (MC).

  Referring once again to the OODA model of FIG. From the viewpoint of an autonomous vehicle, the observation layer of the model includes, for example, a vision sensor, a radar (Radar: “radio wave detection and ranging”), a rider (LIDAR: “light detection and ranging”), and global positioning. Collecting sensor readings from a system (GPS) or the like. The sensor 202 shown in FIG. 2 represents such an observation layer. An example of the situation determination layer of the model may include determining where in the real world a vehicle is relative to a running road and to a lane marking on the road, as shown in FIG. It is represented by a recognition control unit (PC) 212 and a position identification control unit (LC) 210. Examples of the decision-making layer of the model include determining a route (corridor) for automatically driving a vehicle, and includes an automatic driving control unit (ADC) 214 and a vehicle control unit ( VC) 216 and the like. In the example of the action layer, the route is connected to the drive system of the vehicle (for example, the steering sub system, the acceleration sub system, the brake sub system, etc.) such as the drive unit control unit 410 of FIG. Including conversion to commands that guide the car along. One skilled in the art will understand that these layers of the system do not necessarily proceed in this order, and that the results of other layers will change as observations change. For example, even after the system selects a route to travel, it may change the route of a car or execute an emergency procedure to prevent a collision due to changes in road conditions such as detection of other objects. Good. Also, the command of the vehicle control unit may need to be dynamically adjusted to compensate for anticipated changes in vehicle behavior such as drift and skid.

  At a higher level, module architecture 206 receives measurements from sensor 202. Different sensors may output different sets of information and different formats, but the modular architecture 206 is a vendor-neutral (independent vendor-specific) format of data that can be interpreted by the modular architecture 206. It includes a sensor interface controller (SIC) 208 (sometimes referred to as a sensor interface server (SIS)) configured to convert. Therefore, the modular architecture 206 can know the environment around the vehicle 204 from the sensor of the vehicle, regardless of the vendor, manufacturer or configuration of the sensor. The SIC 208 can also attach a metadata tag to each sensor data that includes the position and orientation of the sensor in the car. The metadata tag can be used by the recognition control unit to determine the unique angle, field of view, and blind spot of each sensor.

  The modular architecture 206 further includes a vehicle controller 216 (VC). The VC 216 is configured to send commands to the vehicle and receive status messages from the vehicle. The vehicle control unit 216 receives from the vehicle 204 information on sub-systems related to autonomous traveling of the vehicle, such as information on the vehicle state (for example, information on the vehicle speed, posture, steering position, braking state, and fuel level). A status message indicating any information). Based on information from the vehicle 204 and the sensor 202, the modular architecture 206 can calculate a command to be sent from the VC 216 to the vehicle 204 to achieve automatic driving. The functions of the various modules within the modular architecture 206 will be described in detail later. However, looking at the modular architecture 206 at a higher level, the modular architecture 206 receives (a) sensor information from the sensor 202 and (b) vehicle status information from the vehicle 204 and sends vehicle instructions to the vehicle. 204. With this architecture, the modular architecture can be adopted for any vehicle having an arbitrary sensor configuration. Therefore, a vehicle equipped with a sensor sub-system (for example, the sensor 202 etc.) and further a drive sub-system (for example, the drive unit control unit 410 in FIG. 4) capable of outputting a vehicle state and receiving a drive command. Any vehicle platform can be integrated with the modular architecture 206.

  In the modular architecture 206, various modules cooperate to realize automatic driving according to the OODA model. Sensor 202 and SIC 208 reside in the “observation” virtual layer. As described above, the SIC 208 receives measurements in various formats (eg, sensor data, etc.). The SIC 208 is configured to convert vendor specific data directly from the sensor into vendor neutral data. This allows the set of sensors 202 to include sensors such as any brand of radar, lidar or image sensor, and the modular architecture 206 can effectively exploit the environmental perception of the sensors 202.

  Next, the measurement value output from the sensor interface server is processed by the recognition control unit (PC) 212 and the position specification control unit (LC) 210. Both the PC 212 and the LC 210 exist in the “situation determination” virtual layer of the OODA model. The LC 210 determines a robust real world position of the vehicle that may be more accurate than the GPS signal, and also determines the real world position of the vehicle when the GPS signal is not available or not accurate. The LC 210 determines the position based on GPS data and sensor data. On the other hand, the PC 212 generates a prediction model that represents the state of the environment around the vehicle, including the conditions around the vehicle and the road. FIG. 3 shows the SIC 208, LC 210 and PC 212 in more detail.

  The automatic driving control unit 214 (ADC) and the vehicle control unit 216 (VC) receive the outputs of the recognition control unit and the position specifying control unit. The ADC 214 and VC 216 reside in the “decision” virtual layer of the OODA model. The ADC 214 is responsible for destination selection, route (lane) and lane guidance, and higher traffic monitoring. The ADC 214 is further responsible for selecting a lane within the route and specifying a safe evacuation area for changing the course of the vehicle in an emergency. That is, the ADC 214 selects a route for reaching the destination and a route for guiding the vehicle in the route. The ADC 214 passes this path to the VC 216. The VC 216 given the route provides a low-order drive function for guiding the vehicle to travel safely on the route. The VC 216 first follows the best way to maneuver along the route while providing comfort to the driver, the ability to reach a safe retreat, emergency maneuverability, and the vehicle's trajectory. Determine the ability to do. In an emergency, the VC 216 invalidates the route supplied by the ADC 214 and immediately guides the vehicle to a safe retreat route, and returns to the route supplied by the ADC 214 when safe. The VC 216 supplies a drive command to the vehicle 204 after determining the vehicle operation method (including safe operation). The vehicle 204 executes the command in the steering sub system, the throttle sub system, and the braking sub system of the vehicle 204. That is, this element of VC 216 exists in the “behavior” virtual layer of the OODA model. In FIG. 4, further details about ADC 214 and VC 216 are described.

  The modular architecture 206 further coordinates communication with various modules through the system controller 218 (SC). The SC 218 enables the operations of the human interaction control unit 220 (HC) and the machine interaction control unit 222 (MC) by exchanging messages with the ADC 214 and the VC 216. Based on the status message coordinated by the system control unit, the HC 220 provides information about the operation of the autonomous vehicle in a human-readable format. The HC 220 further allows human input to be considered in vehicle decision making. For example, by way of example, the HC 220 allows the vehicle operator to enter or change the destination or route of the vehicle. The SC 218 interprets the operator input and passes this information to the VC 216 or ADC 214 as necessary.

  In addition, the MC 222 can coordinate messages with other machines or vehicles. For example, another vehicle can electronically and wirelessly transmit a route change signal to an autonomous vehicle, and the MC 222 receives such information and delivers the information to the VC 216 and the ADC 214 via the SC 218. it can. In addition, the MC 222 can transmit information to other vehicles wirelessly. As an example of a course change signal, MC 222 may receive a notification that this car is about to change course. MC 222 receives this information by VC 216 sending a status message to SC 218 and SC 218 passing the status to MC 222. However, other examples of machine communication can also be realized. For example, sensor information of other vehicles or fixed sensors can wirelessly transmit to the autonomously traveling vehicle data that gives the vehicle a more reliable view of the environment. For example, it may be possible for another machine to transmit information about the object in the blind spot of the vehicle. In a further example, other vehicles can transmit the trajectory of the other vehicle. In yet another example, the traffic signal light can transmit a digital signal about the state of the traffic signal light for assistance when the traffic light is not visible from the vehicle. A person skilled in the art will understand that any information used by the autonomous vehicle can be transmitted to and received from other vehicles, which is useful for autonomous driving. . In FIG. 6, further details about HC 220, MC 222 and SC 218 are described.

  FIG. 3 is a block diagram 300 illustrating one embodiment of a sensor interaction control unit 304 (SIC), a recognition control unit (PC) 306, and a location specification control unit (LC) 308. The vehicle sensor array 302 may include various types of sensors such as a camera 302a, radar 302b, rider 302c, GPS 302d, IMU 302e, and vehicle-to-everything (V2X) 302f. Each sensor sends a vendor-defined individual data type to the SIC 304. For example, camera 302a sends object lists and images, radar 302b sends object lists and in-phase / quadrature (IQ) data, lidar 302c sends object lists and scan points, GPS 302d sends position and velocity, and IMU 302e The acceleration data is sent, and the V2X302f control unit sends sensor data such as a trajectory and a route change signal of other vehicles and traffic signal light data. However, those skilled in the art will appreciate that the sensor array 302 may use other types of sensors. The SIC 304 monitors and diagnoses abnormalities in the sensors 302a to 302f. The SIC 304 further separates the data for each sensor from the vendor specific package of the data and sends the vendor neutral data type to the recognition control unit (PC) 306 and the location control unit 308 (LC). The SIC 304 transfers the position-specific feature measurement value and the position and orientation measurement value to the LC 308, and transfers the tracking target object measurement value, the running surface measurement value, and the position and orientation measurement value to the PC 306. The SIC 304 can also be updated in firmware so that new sensors of different formats can be used with the same modular architecture.

  The LC 308 fuses GPS data and IMU data with radar data, rider data, and vision data, thereby determining the position within the GPS position and increasing the accuracy of the GPS position. The LC 308 then reports the determined position, velocity, and attitude to the PC 306. In order to further improve the accuracy, the LC 308 uses the measurement values representing the position data, the velocity data, and the posture data when a certain sensor measurement value (for example, a GPS signal in a city) becomes unavailable or deteriorates. The relative monitoring is performed so that the LC 308 can correct the measured value. The PC 306 identifies and locates objects around the vehicle based on the detected information. Further, the PC 306 estimates a travelable surface area in the vicinity of the vehicle, and also evaluates a surface such as a road shoulder or an emergency travelable terrain. The PC 306 further provides a probability prediction of the future position of the object. The PC 306 further stores the history of the object and the travelable surface.

  The PC 306 outputs two types of predictions, strategic predictions and tactical predictions. Tactical prediction represents the real world of the future in about 2-4 seconds and only predicts traffic and roads that are closest to the vehicle. This prediction includes empty evacuation space in places such as shoulders.

  Strategic prediction is long-term prediction that predicts an area beyond the visible range of the vehicle's visible environment. This prediction is for the future beyond 4 seconds, but has greater uncertainty than the tactical prediction. This is because the behavior of an object (eg, car, person, etc.) may change unpredictably from the currently observed behavior. The reason for the large uncertainty of strategic prediction is that the prediction makes the assumption that the currently observed behavior will continue throughout the prediction period. Also, such predictions may include sensor measurements from external sources (including other autonomous vehicles, non-autonomous vehicles with sensor systems and sensor communication networks and sensors located near or on the roadway) It can be based on sensor measurements received from a transponder on an object via a network, or signals such as traffic lights or signs configured to communicate wirelessly with the autonomous vehicle.

  FIG. 4 is a block diagram 400 illustrating an exemplary embodiment of an automatic operation controller (ADC) 402, a vehicle controller (VC) 404, and a drive controller 410. The ADC 402 and VC 404 execute the “decision-making” virtual layer of the OODA model.

  The ADC 402 first generates an entire route from the current position to the destination (including a list of roads and branch points between the roads to reach the destination) based on the destination input by the operator and the current position. . This strategic route plan may be based on traffic conditions and can change based on the latest traffic conditions, but such changes generally have a large estimated arrival time (ETA). Implemented when changing. The ADC 402 then plans a safe and collision-free route for the autonomous vehicle to travel based on the nearby objects and the allowable travelable surfaces provided by the PC. This route is always sent as a request to the VC 404 and is updated whenever a situation such as a traffic situation changes. The VC 404 receives the route update in real time. Conversely, the ADC 402 receives the latest actual track of the vehicle from the VC 404. The latest actual trajectory is also used to change the next planned update for the request for the driving route.

  The ADC 402 generates a strategic route for the vehicle to travel. The ADC 402 generates the route based on predictions about free space on the road in the strategic / tactical prediction. The ADC 402 further receives vehicle position information and vehicle attitude information from the recognition control unit in FIG. The VC 404 further outputs the actual track of the vehicle from the drive unit control unit 410 of the vehicle to the ADC 402. Based on this information, the ADC 402 calculates an appropriate route for traveling on the road. In an example where the road is free, the route may follow a lane ahead of the vehicle.

  In another example where a car needs to overtake another car, the ADC 402 may determine whether there is a vacant space in the overtaking lane and ahead of the other car for safe overtaking. it can. The ADC 402 follows a route on which the vehicle travels to perform the overtaking operation: (a) the current distance to the vehicle to be overtaken, (b) the amount of road space available in the overtaking lane, (c ) Based on the amount of empty space ahead of the vehicle to be overtaken, (d) the speed of the vehicle to be overtaken, (e) the current speed of the autonomous vehicle, and (f) the known acceleration of the autonomous vehicle. Can be calculated automatically.

  In another example, the ADC 402 can determine a route to change lanes when heading for a highway exit. In addition to all the factors described above, the ADC 402 monitors the planned route to the destination, and when approaching the branch point, the ADC 402 calculates the best route for continuing to operate the planned route safely and legally.

  The ADC 402 further calculates a tactical trajectory in the path. Thereby, the said vehicle can maintain the safe space | interval between objects. The tactical trajectory also includes a safe back-up retreat trajectory in an emergency such as when one vehicle unexpectedly decelerates or stops or another vehicle interrupts in front of the autonomous vehicle.

  The ADC 402 supplies the required trajectory path 406 to the VC 404. This supply is performed in cooperation with the ADC 402 so that the vehicle can travel along the route. The requested trajectory path 406 imposes geometric and velocity constraints on future trajectories after a few seconds. The VC 404 determines a route for driving in the route 406. The VC 404 makes a steering decision of the VC 404 based on the tactical / steering prediction received from the recognition control unit, the position of the vehicle, and the attitude of the vehicle. As mentioned above, the tactical / maneuvering prediction is for a short period of time, but with less uncertainty. Thus, for low order maneuvering and safety operations, VC 404 effectively uses the tactical / maneuvering prediction to plan a trajectory that does not cause a collision in request path 406. When needed in an emergency, the VC 404 plans a trajectory outside the path 406 so as not to collide with other objects.

  Next, based on the requested route 406, the current speed and acceleration of the car, and the nearest object, the VC 404 avoids collision with the object and stays on the travelable surface while traveling on the route 406. To decide. When the VC 404 is forced to avoid a collision, it may have to command a maneuver that suddenly departs from the requested path of the ADC 402. This emergency maneuver can be initiated not by the ADC 402 but by the VC 404 alone, which has a faster response time than the ADC 402 against the impending collision threat. With this function, the safety serious collision avoidance responsibility is only in the VC 404. The VC 404 sends a steering command to the drive unit that controls the steering operation, throttle operation, and braking operation of the vehicle platform.

  The VC 404 executes the steering strategy of the VC 404 by sending a current vehicle trajectory 408 including a drive command (for example, steering, throttle, braking, etc.) to the drive controller 410 of the vehicle. The drive control unit 410 of the vehicle applies corresponding commands to the vehicle steering system, throttle system, and braking system. The VC 404 that sends the trajectory 408 to the drive controller represents the “behavior” virtual layer of the OODA model. By conceptualizing the architecture of the autonomous vehicle in this way, a component that requires settings for controlling a vehicle of a specific model (for example, the format of each command, acceleration performance, turning performance, braking performance, etc.) Is the VC only, and the ADC can be largely independent of specific vehicle performance. In one example, the VC 404 can be updated with firmware configured to interface with a particular drive control system of the vehicle or with a fleet-scale firmware update for the entire vehicle.

  FIG. 5 is a diagram 500 illustrating a decision time scale for the ADC 402 and VC 404. The ADC 402 implements higher order strategic decisions 502 and tactical decisions 504 by generating the path. Thus, the ADC 402 achieves long range and / or time scale decisions. The estimation of the real world state that the ADC 402 uses to plan strategic routes and tactical driving paths for behavior such as overtaking and turns predicts a long-time future, albeit with high uncertainty. This is necessary to plan such autonomous operation. When the strategic prediction is predicted beyond the visible range of the vehicle, the uncertainty of the object that is far from the vehicle depends on a non-vision technique such as radar for prediction of an object far away from the vehicle. While many of the tactical decisions, such as overtaking a car at the speed of a highway, require recognition beyond the visible range of autonomous vehicles (eg, BVR) (eg, 100 meters or more), All are based on locally recognized objects to avoid collisions.

  On the other hand, the VC 404 generates a maneuverability decision 506 using maneuverability prediction, which is a short-term frame / range prediction for object behavior and running surface. Since these maneuverability predictions are predictions on a short-time scale, uncertainty is reduced. However, the maneuverability prediction uses only measured values obtained within the visible range of the sensor in the autonomous vehicle. That is, the VC 404 uses these maneuverability predictions (or evaluation values) for the state of the surrounding environment in the immediate vicinity of the vehicle in order to plan the path of the autonomously traveling vehicle that does not cause a collision with a high-speed response. The VC 404 issues a drive command at the lowest end of the time scale, shifts the planned route to execution, and steers on the route.

  FIG. 6 is a block diagram 600 illustrating an exemplary embodiment of a system controller 602, human interface controller 604 (HC), and machine interface controller 606 (MC). The human interaction control unit 604 (HC) receives an input command request from an operator and provides an output to the operator, the vehicle occupant, and a person outside the autonomous vehicle. The HC 604 provides the operator and the occupant (via an interface such as a visual interface, an auditory interface, or a tactile interface) with a human understandable understanding of the system state and the basis for decision making of the autonomous vehicle. For example, the HC 604 can display a long-term route (or planned route) or a safe evacuation area of the vehicle. The HC 604 can also read sensor measurements about the driver's condition and monitor the driver's readiness to assist in driving the vehicle at any moment. As an example, the sensor system in the vehicle can detect whether the operator is placing his hand on the steering wheel. When the hand is placed, the HC 604 can notify that the switching to the steering of the operator can be permitted, and when the hand is not placed, the HC 604 can prevent the steering control from being delivered to the operator. . In another example, the HC 604 may summarize the decision base (eg, the reason for selecting a particular route, etc.) to the operator.

  The machine interaction control unit 606 (MC) interacts with other autonomous vehicles and automation systems so as to coordinate activities such as formation drive and traffic management. The MC 606 reads the internal system state and generates an output data type (for example, a V2X data type) that can be understood by the cooperative machine system. This state can be broadcast over the network by a collaborative system. MC 606 forwards any command request from an external machine system (eg, deceleration, route change, merge request, traffic signal status, etc.) to the SC for arbitration with other command requests from HC 604. Can be converted into a command request. Also, the MC 606 authenticates messages from other systems (for example, using a signed message from a reliable third-party manufacturer) so that these messages are valid and represent the environment around the vehicle. Can be guaranteed. Such authentication can prevent tampering by a malicious person.

  The system control unit 602 (SC) functions as an overall management unit for each component in the architecture. The SC 602 aggregates the state data from all system components to determine the overall operating state, and sends a command to execute the system function for each component. When a system component reports a malfunction, the SC 602 initiates diagnosis / recovery behavior to ensure autonomous operation and maintain the vehicle in a safe state. Any switching of the vehicle to / from the automatic driving state is permitted by the SC 602 during the internal diagnosis of whether or not the operation is ready for automatic driving and the readiness of the human driver. Rejected.

  In one embodiment of the present invention, the system and method fuse objects within a cognitive layer / module. FIG. 7 is a block diagram 700 illustrating an exemplary embodiment of an object level fusion server 702 that receives information from all sensors such as vision sensor 702, rider 706, radar 704, and the like. In an exemplary embodiment, the sensor includes two stereo vision sensors, a radar and a lidar. The stereo vision 702 outputs a bounding box (boundary box) of an object (for example, a car, a two-wheeled vehicle, a person, a lane, a sign, and an object) existing in the field of view of the camera by using an imaging technique. The radar system 704 tracks the position and distance (range) of any object that reflects the radar, but cannot classify or identify these objects. The rider system 706 can identify objects such as cars and motorcycles, and outputs the types of these objects.

  FIG. 8A is a block diagram 800 illustrating an exemplary embodiment of an object fusion server 814. Each sensor 802, 804, 806 communicates with a server module that can include a dedicated processor. The dedicated processor is configured to execute a tracking loop, process raw data for each sensor separately, and output object data instead of raw sensor data. In one embodiment, all object outputs by the sensor servers 802, 804, 806 are the X axis (axial center from the rear of the vehicle to the front of the vehicle), the Y axis (axial center from the left side of the vehicle to the right side of the vehicle), and the Z axis. This is a coordinate system including (an axis extending upward from the ground). However, those skilled in the art will be able to envisage coordinate systems with different axis configurations (ie, coordinate systems such as spherical coordinate systems). In other words, each sensor server converts object data into a message format independent of the sensor or sensor format. Next, the cognitive object fusion server 814 combines these messages into a fusion tracking target object list message.

  The object fusion module 814 then analyzes all data (vision, lidar and radar) to determine the overall image of the object near the autonomous vehicle. Conventional object level fusion systems have used only radar systems, only vision systems, or none. One of the unique challenges that must be overcome when combining data from the entire system is that the lidar or radar outputs the object as a point in the coordinate system and the distance to that point, whereas the vision system Will output a bounding box that is likely to contain, but the bounding box is less accurate. That is, data of different systems needs to be accurately associated with each other. For example, radar objects inside the bounding box can be determined to be the same object. The radar ping outside the bounding box may be another object. The object fusion module accepts data from all sensors to determine the most likely scenario for all objects near the autonomous vehicle. In an exemplary embodiment, the object fusion module uses Kalman filtering and stochastic data combining. However, in other embodiments, techniques such as multiple hypothesis tracking or probability hypothesis density filtering may be used.

  FIG. 8B is a more detailed block diagram 850 of the object fusion module. The object data 858, 860, 862 of each sensor is preprocessed by at least one preprocessor 864. Matching the object data formats in the object lists 866, 868, 870, the data association block (data association block) associates objects from these three different domains (eg, lidar, radar, vision system, etc.). Thus, the object trajectory is determined. One skilled in the art will recognize that the radar object list 870 is further processed by the radar tracker 874 and then forwarded to the data association module (association module) 872.

  Data association module 872 applies probabilistic data association using a global nearest neighbor method to associate sensor object trajectories with existing fused object trajectories. The global nearest neighbor method selects an association that maximizes the similarity between each associated sensor object trajectory and the fused object trajectory. The data association module 872 measures the similarity of the object trajectory based on the converted sensor data using at least two of motion information, geometric information, and object type information. In some embodiments, the data association module 872 measures the similarity of the object trajectory using all three of motion information, geometric information, and object type information based on the converted sensor data. The data association module 872 sends the unrelated object trajectory 878 to the trajectory management module 882, and the trajectory management module 882 starts a new trajectory 888. The new trajectory 888 is sent to the trajectory prediction module 884. Data association module 872 on the other hand associates the remaining objects with the existing fused object trajectory 876. These are passed to the trajectory update module 880. The trajectory update module 880 sends the updated trajectory 892 to the trajectory prediction module 884. The trajectory update module 880 uses a Bayesian filter to update the object trajectory and predict the trajectory at the next time step. In an exemplary embodiment, the trajectory update module uses a Kalman filter. However, in other embodiments, an extended Kalman filter, an unscented Kalman filter, or a particle filter may be used.

  The trajectory prediction module 884 issues a prediction 886 for an existing object trajectory that is sent to the data association module 872. This prediction is used for data association in future time steps. This loop continues as long as the car is moving.

  Trajectory update module 880 further removes old trajectories when the sensor detects that the object is no longer in view. However, the trajectory update module 880 can make an exception for an object within the blind spot known to the sensor.

  FIG. 9 is a diagram 900 illustrating trajectory prediction. The goal of trajectory prediction is to track the position and velocity of the object at the time of the current measurement, given the latest course of evaluation values. For example, assuming that a trajectory 902 at time t = k is given, the trajectory prediction module calculates the position and trajectory 904 of the object at time t = k + 1 at the previous time (for example, the most recent time step). Estimation can be based on data from the collected data. However, the estimation uncertainty increases as the distance from the prediction increases in time after the prediction time. As the prediction becomes farther away from the present time, the bounding box that matches the object naturally increases, and the degree of certainty corresponding to the object decreases.

  10A to 10D are diagrams 1000, 1020, 1040, and 1060 showing data association. The data association module associates the latest set of measurements with an existing trajectory (FIG. 10A). First, as shown in FIG. 10B, the system calculates a relevance score for each pair of trajectory and measurement value. In one embodiment, the score may be a distance between objects. Next, the system disallows or excludes associations that do not seem valid, as shown in FIG. 10C. Then, the data association module selects the measurement value-trajectory allocation that minimizes the sum of the relevance scores. The trajectory is then updated based on the associated measurement value. The update of the trajectory becomes a new trajectory in consideration of the predicted trajectory and the measured value.

  FIG. 11 is a drawing 1100 illustrating an exemplary embodiment of trajectory update. The trajectory can be updated based on how the objects are related. For example, when two objects are related to each other, the system is based on the knowledge that the objects are related and the trajectories of the objects are likely to be the same or similar. The trajectory can be updated to combine the trajectories of two objects.

  While the invention has been particularly shown and described with reference to illustrative embodiments, those skilled in the art will recognize the invention in form and detail without departing from the scope of the invention as encompassed by the appended claims. It will be understood that various changes may be made.

  FIG. 12 illustrates a computer network or similar digital processing environment in which embodiments of the present invention may be implemented.

  At least one client computer / device 50 and at least one server computer 60 provide processing devices, storage devices, and input / output devices for executing application programs and the like. The at least one client computer / device 50 may further be connected to other computing devices (including other client devices / processes 50 and one or more other server computers 60) via the communication network 70. it can. The communication network 70 may be a remote access network, a global network (eg, the Internet, etc.), a collection of computers around the world, a local or wide area network, and currently compatible protocols (TCP / IP, Bluetooth®). Etc.) may be part of a gateway that communicates with each other. Other electronic device / computer network architectures are also suitable.

  FIG. 13 is a diagram of an exemplary internal structure of a computer (eg, client processor / device 50, server computer 60, etc.) in the computer system of FIG. Computers 50 and 60 include a system bus 79 which is a series of hardware lines used for data transmission between components of the computer or processing system. The system bus 79 essentially connects different components of a computer system (eg, processor, disk storage, memory, input / output port, network port, etc.) and allows information to be transmitted between the components. Is a shared conduit. An input / output device interface 82 for connecting various input / output devices (for example, a keyboard, a mouse, a display, a printer, a speaker, etc.) to the computers 50 and 60 is attached to the system bus 79. The network interface 86 allows the computer to connect to various other devices attached to the network (eg, the network 70 of FIG. 12). The memory 90 is an embodiment of the present invention (for example, the sensor interface control unit, the recognition control unit, the position specifying control unit, the automatic driving control unit, the vehicle control unit, the system control unit, the human interaction control unit, the machine described in detail above) Volatile storage of computer software instructions 92 and data 94 used to implement an interaction controller, object fusion server, preprocessor, data association module, trajectory update module, trajectory management module, trajectory prediction module, radar tracker, etc. It is a storage unit. The disk storage 95 is a non-volatile storage that stores computer software instructions 92 and data 94 used to implement one embodiment of the present invention. Further, a central processing unit 84 for executing computer instructions is attached to the system bus 79.

  In one embodiment, the processor routines 92 and data 94 are computer program products (generally designated 92). The computer program product is a non-transitory computer readable medium (eg, removable at least one DVD-ROM, CD-ROM, diskette, tape, etc.) that provides at least some of the software instructions for the system of the present invention. Storage medium etc.). The computer program product 92 may be capable of being installed by any suitable software installation method well known in the art. In other embodiments, at least some of the software instructions may be downloaded via cable communication and / or wireless connection. In another embodiment, the program of the present invention is a propagation signal in a propagation medium (for example, a radio wave, an infrared wave, a laser wave, a sound wave, an electric wave propagated by a global network such as the Internet or at least one other network). Is a computer program propagation signal product incorporated in Such a carrier medium or signal may be used to provide at least a portion of the software instructions for the routine / program 92 of the present invention.

In one embodiment, the processor routines 92 and data 94 are computer program products (generally designated 92). The computer program product is a non-transitory computer readable medium (eg, removable at least one DVD-ROM, CD-ROM, diskette, tape, etc.) that provides at least some of the software instructions for the system of the present invention. Storage medium etc.). The computer program product 92 may be capable of being installed by any suitable software installation method well known in the art. In other embodiments, at least some of the software instructions may be downloaded via cable communication and / or wireless connection. In another embodiment, the program of the present invention is a propagation signal in a propagation medium (for example, a radio wave, an infrared wave, a laser wave, a sound wave, an electric wave propagated by a global network such as the Internet or at least one other network). Is a computer program propagation signal product incorporated in Such a carrier medium or signal may be used to provide at least a portion of the software instructions for the routine / program 92 of the present invention.
In addition, this invention contains the following content as an aspect.
[Aspect 1]
A process of converting sensor data from a plurality of different sensors into a common coordinate frame, which is sensor data about a detected object;
Predicting the position, velocity, orientation and bounding box of an existing object trajectory at the current measurement time;
Associating a detected object with an existing object trajectory by determining at least two similarities among motion information, geometric information and object type information based on the converted sensor data;
Updating the motion information, the geometric information, and the object type information for the object trajectory associated with the detected object;
Reporting a merged object list, including an obtained set of updated object trajectories;
A method comprising:
[Aspect 2]
The method according to aspect 1, further comprising:
Starting a new object trajectory for a detected object not associated with an existing object trajectory;
A method comprising:
[Aspect 3]
The method according to aspect 1, further comprising:
Deleting an object trajectory outside the field of view of the at least one heterogeneous sensor of the autonomous vehicle;
A method comprising:
[Aspect 4]
In the method according to aspect 3, the step of deleting the object trajectory further includes a sub-process of excluding the object trajectory from the deletion target when the object trajectory is within the blind spot of the at least one different sensor. ,Method.
[Aspect 5]
The method of claim 1, wherein the step of associating a detected object with an existing object trajectory further determines at least three similarities of motion information, geometric information and object type information.
[Aspect 6]
The method according to aspect 1, further comprising:
Associating a trajectory of a detected feature with an existing object trajectory by obtaining at least two similarities among motion information, geometric information and object type information;
A method comprising:
[Aspect 7]
The method according to aspect 1, wherein the motion information includes position information, velocity information, and orientation information, the geometric information includes a bounding box and an object outline, and the object type information includes an object type.
[Aspect 8]
Pre-processing means configured to convert sensor data from a plurality of different types of sensors into a common coordinate frame, which is sensor data about a detected object;
Trajectory predicting means configured to predict position, velocity, orientation and bounding box at the current measurement time of an existing object trajectory;
Data association means configured to associate a detected object with an existing object trajectory by obtaining at least two similarities among motion information, geometric information and object type information using the converted sensor data; ,
Trajectory update means configured to update the motion information, the geometric information, and the object type information for the object trajectory associated with the detected object;
A reporting means configured to report a fused object list, the object list including the resulting set of updated object trajectories;
A system comprising:
[Aspect 9]
The system according to aspect 8, further comprising:
Trajectory management means configured to start a new object trajectory for a detected object that is not associated with an existing object trajectory,
A system comprising:
[Aspect 10]
9. The system according to aspect 8, wherein the trajectory updating means is further configured to delete an object trajectory that is outside the field of view of the at least one heterogeneous sensor of the autonomous vehicle.
[Aspect 11]
In the system according to aspect 10, when the object trajectory is further within the blind spot of the at least one heterogeneous sensor, the trajectory updating unit further excludes the object trajectory from the deletion target, thereby removing the object trajectory. A system that is configured to be removed.
[Aspect 12]
9. The system according to aspect 8, wherein the data association means is further configured to obtain at least three similarities among motion information, geometric information, and object type information.
[Aspect 13]
In the system according to aspect 8, the data association means further obtains a trajectory of the detected feature object by obtaining at least two similarities among the motion information, the geometric information, and the object type information. A system that is configured to associate with.
[Aspect 14]
The system according to aspect 8, wherein the motion information includes position information, velocity information, and orientation information, the geometric information includes a bounding box and an object outline, and the object type information includes an object type.
[Aspect 15]
A non-transient computer readable medium configured to store instructions for operating an autonomous vehicle, wherein the instructions, when loaded and executed by a processor,
A procedure for converting sensor data from a plurality of different sensors into a common coordinate frame, which is sensor data about a detected object,
A procedure for predicting the position, velocity, orientation and bounding box at the current measurement time of an existing object trajectory;
A step of associating a detected object with an existing object trajectory by obtaining at least two similarities among motion information, geometric information and object type information based on the converted sensor data;
A procedure for updating the motion information, the geometric information, and the object type information for the object trajectory associated with the detected object;
A procedure for reporting a merged object list that includes an obtained set of updated object trajectories;
A non-transient computer readable medium that causes
[Aspect 16]
The non-transient computer readable medium of aspect 15, wherein the instructions are further to the processor,
Procedure for starting a new object trajectory for a detected object that is not associated with an existing object trajectory,
A non-transitory computer readable medium configured to cause execution.
[Aspect 17]
The non-transient computer readable medium of aspect 15, wherein the instructions are further to the processor,
A procedure for deleting an object locus outside the field of view of the at least one heterogeneous sensor of the autonomous vehicle;
A non-transitory computer readable medium configured to cause execution.
[Aspect 18]
The non-transient computer-readable medium according to aspect 17, wherein the step of deleting an object trajectory is further subject to deletion when the object trajectory is within a blind spot of the at least one heterogeneous sensor. A non-transient computer readable medium that includes excluding from.
[Aspect 19]
The non-transient computer readable medium according to aspect 15, wherein the procedure of associating a detected object with an existing object trajectory further determines the similarity of at least three of motion information, geometric information, and object type information. Non-transient computer readable medium.
[Aspect 20]
The non-transient computer readable medium of aspect 15, wherein the instructions are further directed to the processor:
A procedure for associating a trajectory of a detected feature with an existing object trajectory by obtaining at least two similarities among motion information, geometric information, and object type information;
A non-transitory computer readable medium configured to cause execution.

Claims (20)

A process of converting sensor data from a plurality of different sensors into a common coordinate frame, which is sensor data about a detected object;
Predicting the position, velocity, orientation and bounding box of an existing object trajectory at the current measurement time;
Associating a detected object with an existing object trajectory by determining at least two similarities among motion information, geometric information and object type information based on the converted sensor data;
Updating the motion information, the geometric information, and the object type information for the object trajectory associated with the detected object;
Reporting a merged object list, including an obtained set of updated object trajectories;
A method comprising: The method of claim 1, further comprising:
Starting a new object trajectory for a detected object not associated with an existing object trajectory;
A method comprising: The method of claim 1, further comprising:
Deleting an object trajectory outside the field of view of the at least one heterogeneous sensor of the autonomous vehicle;
A method comprising:   4. The method of claim 3, wherein the step of deleting an object trajectory further includes a sub-process of excluding the object trajectory from the deletion target when the object trajectory is within a blind spot of the at least one different sensor. Including.   The method of claim 1, wherein the step of associating a detected object with an existing object trajectory further determines at least three similarities of motion information, geometric information, and object type information. The method of claim 1, further comprising:
Associating a trajectory of a detected feature with an existing object trajectory by obtaining at least two similarities among motion information, geometric information and object type information;
A method comprising:   The method according to claim 1, wherein the motion information includes position information, velocity information, and orientation information, the geometric information includes a bounding box and an object outline, and the object type information includes an object type. Pre-processing means configured to convert sensor data from a plurality of different types of sensors into a common coordinate frame, which is sensor data about a detected object;
Trajectory predicting means configured to predict position, velocity, orientation and bounding box at the current measurement time of an existing object trajectory;
Data association means configured to associate a detected object with an existing object trajectory by obtaining at least two similarities among motion information, geometric information and object type information using the converted sensor data; ,
Trajectory update means configured to update the motion information, the geometric information, and the object type information for the object trajectory associated with the detected object;
A reporting means configured to report a fused object list, the object list including the resulting set of updated object trajectories;
A system comprising: 9. The system according to claim 8, further comprising:
Trajectory management means configured to start a new object trajectory for a detected object that is not associated with an existing object trajectory,
A system comprising:   9. The system according to claim 8, wherein the trajectory updating means is further configured to delete an object trajectory outside the field of view of the at least one heterogeneous sensor of the autonomous vehicle.   The system according to claim 10, wherein the trajectory updating means further excludes the object trajectory from a deletion target when the object trajectory is within a blind spot of the at least one heterogeneous sensor. The system is configured to remove.   9. The system according to claim 8, wherein the data association means is further configured to obtain similarity of at least three of motion information, geometric information, and object type information.   9. The system according to claim 8, wherein the data association means further obtains a trajectory of the detected feature object by obtaining at least two similarities among motion information, geometric information, and object type information. A system that is configured to be associated with a trajectory.   9. The system according to claim 8, wherein the motion information includes position information, velocity information, and orientation information, the geometric information includes a bounding box and an object outline, and the object type information includes an object type. A non-transient computer readable medium configured to store instructions for operating an autonomous vehicle, wherein the instructions, when loaded and executed by a processor,
A procedure for converting sensor data from a plurality of different sensors into a common coordinate frame, which is sensor data about a detected object,
A procedure for predicting the position, velocity, orientation and bounding box at the current measurement time of an existing object trajectory;
A step of associating a detected object with an existing object trajectory by obtaining at least two similarities among motion information, geometric information and object type information based on the converted sensor data;
A procedure for updating the motion information, the geometric information, and the object type information for the object trajectory associated with the detected object;
A procedure for reporting a merged object list that includes an obtained set of updated object trajectories;
A non-transient computer readable medium that causes 16. The non-transient computer readable medium of claim 15, wherein the instructions are further to the processor.
Procedure for starting a new object trajectory for a detected object that is not associated with an existing object trajectory,
A non-transitory computer readable medium configured to cause execution. 16. The non-transient computer readable medium of claim 15, wherein the instructions are further to the processor.
A procedure for deleting an object locus outside the field of view of the at least one heterogeneous sensor of the autonomous vehicle;
A non-transitory computer readable medium configured to cause execution.   18. The non-transient computer readable medium of claim 17, wherein the procedure of deleting an object trajectory further deletes the object trajectory if the object trajectory is within a blind spot of the at least one disparate sensor. A non-transient computer-readable medium that includes exclusion from the subject.   16. The non-transient computer readable medium of claim 15, wherein the procedure of associating a detected object with an existing object trajectory further determines a similarity of at least three of motion information, geometric information and object type information. Non-transient computer readable medium. 16. The non-transient computer readable medium of claim 15, wherein the instructions are further to the processor:
A procedure for associating a trajectory of a detected feature with an existing object trajectory by obtaining at least two similarities among motion information, geometric information, and object type information;
A non-transitory computer readable medium configured to cause execution.

Images