US20170075356A1 - Classifying objects detected by 3d sensors for autonomous vehicle operation - Google Patents
Classifying objects detected by 3d sensors for autonomous vehicle operation Download PDFInfo
- Publication number
- US20170075356A1 US20170075356A1 US14/852,806 US201514852806A US2017075356A1 US 20170075356 A1 US20170075356 A1 US 20170075356A1 US 201514852806 A US201514852806 A US 201514852806A US 2017075356 A1 US2017075356 A1 US 2017075356A1
- Authority
- US
- United States
- Prior art keywords
- track
- cluster
- clusters
- points
- autonomous driving
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 238000000034 method Methods 0.000 claims abstract description 29
- 230000002123 temporal effect Effects 0.000 claims abstract description 5
- 230000001133 acceleration Effects 0.000 claims description 6
- 230000033001 locomotion Effects 0.000 claims description 3
- 230000015654 memory Effects 0.000 description 13
- 238000001514 detection method Methods 0.000 description 10
- 230000008447 perception Effects 0.000 description 10
- 238000013439 planning Methods 0.000 description 8
- 238000004891 communication Methods 0.000 description 5
- 230000008569 process Effects 0.000 description 5
- 238000010295 mobile communication Methods 0.000 description 3
- 238000005303 weighing Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 2
- 230000003993 interaction Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 238000001228 spectrum Methods 0.000 description 2
- 230000001413 cellular effect Effects 0.000 description 1
- 230000008859 change Effects 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000010606 normalization Methods 0.000 description 1
- 239000002245 particle Substances 0.000 description 1
- 238000012913 prioritisation Methods 0.000 description 1
- 238000002310 reflectometry Methods 0.000 description 1
- 238000011160 research Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
- G05D1/0242—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using non-visible light signals, e.g. IR or UV signals
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
- G05D1/0238—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using obstacle or wall sensors
- G05D1/024—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using obstacle or wall sensors in combination with a laser
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course, altitude or attitude of land, water, air or space vehicles, e.g. using automatic pilots
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
- G05D1/0246—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means
- G05D1/0251—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using a video camera in combination with image processing means extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision
-
- G06K9/00805—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
Definitions
- the embodiments disclosed herein generally relate to autonomous operation systems for vehicles and, more particularly, to the classification of objects detected by 3D sensors in autonomous operation systems.
- Some vehicles include an autonomous operation system with an operational mode in which the vehicle is driven along a travel route with minimal or no input from a human driver.
- the autonomous operation system is configured to detect information about the environment surrounding the vehicle, including the presence of objects, and process the detected information in order to plan how to drive the vehicle along a travel route while avoiding the objects.
- the autonomous operation system In real world traffic situations, as a part of this detection and planning, it is desirable for the autonomous operation system to classify the objects in the environment surrounding the vehicle in order to account not only for other vehicles, but also for pedestrians and bicycles, among other objects. Improving the autonomous operation system's classification of the objects in the environment surrounding the vehicle is the subject of ongoing research.
- a method of autonomous driving includes generating, with a 3D sensor, 3D points representing objects in the environment surrounding a vehicle.
- the method further includes, with a computing device, identifying, from the 3D points, a temporal series of clusters of 3D points representing the same object in the environment surrounding the vehicle as a track, identifying cluster-based classifiers for the object based on identified local features for the clusters in the track, identifying track-based classifiers for the object based on identified global features for the track, combining the cluster-based classifiers and the track-based classifiers to classify the object, with the cluster-based classifiers being weighted based on an amount of information on the clusters from which they are identified, and with the weight increasing with increasing amounts of information, and driving the vehicle along a route based on the object's classification.
- FIG. 1 is a schematic representation of a vehicle including an autonomous operation system whose operation is supported by a 3D sensor;
- FIG. 2 is a schematic representation of the system architecture of a detection module for the autonomous operation system
- FIG. 3 is a flowchart showing the operations of a process for a classifier thread in the detection module
- FIGS. 4A and 4B show example estimations of the principle direction of a cluster of 3D points representing a vehicle
- FIG. 5 shows example spin images from clusters of 3D points representing different objects
- FIGS. 6A-C show example virtual orthographic images of a cluster of 3D points representing a vehicle
- FIG. 7 is an example graphical model encoding the probabilistic independencies between local features for a track's clusters of 3D points and global features for the track;
- FIG. 8 shows aspects of an example weighing factor for a cluster-based classifier
- FIG. 9 shows aspects of a classifier confidence over time for track.
- This disclosure teaches a vehicle that includes an autonomous operation system that, in operation, classifies objects in the environment surrounding the vehicle represented by temporal series of clusters of 3D points, and drives the vehicle along a route based on the objects' classifications.
- the autonomous operation system classifies the objects based on both a track-based classifier and cluster-based classifiers, but weighs the cluster-based classifiers based on the amount of information on the clusters from which they are identified.
- FIG. 1 shows a vehicle 10 including an autonomous operation system 20 whose operation is supported by a LIDAR sensor 22 and one or more optional auxiliary sensors 24 .
- the LIDAR sensor 22 and the auxiliary sensors 24 are mounted on the vehicle 10 and positioned to have fields of view in the environment surrounding the vehicle 10 .
- the vehicle 10 is provided as a non-limiting example of a mobile platform, it will be understood that the autonomous operation system 20 could be implemented in other mobile platforms.
- the LIDAR sensor 22 is provided as a non-limiting example of a 3D sensor, it will be understood that this description is applicable in principle to other 3D sensors.
- the LIDAR sensor 22 is configured to scan the environment surrounding the vehicle 10 and generate signals, including but not limited to 3D points, representing the objects in the environment surrounding the vehicle 10 .
- the LIDAR sensor 22 can include a transmitter and a receiver.
- the transmitter can be a component or group of components operable to transmit laser signals (e.g., laser light energy).
- the transmitter may be a laser, laser rangefinder, LIDAR, and/or laser scanner.
- the laser signals may have any suitable characteristics.
- the laser signals may be from any suitable portion of the electromagnetic spectrum, such as from the ultraviolet, visible, or near infrared portions of the electromagnetic spectrum.
- the laser signals may be eye safe.
- the laser signals may be transmitted into the environment surrounding the vehicle 10 , where they impinge upon objects therein that are located in the path of the laser signals.
- the laser signals may be transmitted in series of 360 degree spins around a vertical Z axis of the vehicle 10 , for example.
- a portion of the laser signals is returned (e.g., by reflection) to the LIDAR sensor 22 .
- the returned portion of the laser signals can be captured at the LIDAR sensor 22 by its receiver, which may be, or include, one or more photodetectors, solid state photodetectors, photodiodes or photomultipliers, or any combination of these.
- the LIDAR sensor 22 may be configured to output signals representing objects, or the lack thereof, in the environment surrounding the vehicle 10 .
- the LIDAR sensor 22 may include a global positioning system (GPS) or other positioning system for identifying its position, and an inertial measurement unit (IMU) for identifying its pose.
- the signals may include 3D points representing the location in space of the points from which the returned laser signals are received, and therefore, the location in space of points of objects on which the laser signals impinged.
- the LIDAR sensor 22 may determine the location in space of points of objects based on the distance from the LIDAR sensor 22 to the points, as well as the position and pose of the LIDAR sensor 22 associated with the returned laser signals.
- the distance to the points may be determined from the returned laser signals using the time of flight (TOF) method, for instance.
- TOF time of flight
- the signals may also represent the locations in space from which no returned laser signals are received, and therefore, the lack of points of objects in those locations in space on which the laser signals would otherwise have impinged.
- the signals may further represent other aspects of the returned laser signals, which, in turn, may represent other properties of points of objects on which the incident laser signals impinged.
- These aspects of the returned laser signals can include their intensity or reflectivity, for instance, or any combination of these.
- the auxiliary sensors 24 may also be configured to scan the environment surrounding the vehicle 10 and generate signals representing objects, or the lack thereof, in the environment surrounding the vehicle 10 .
- the auxiliary sensors 24 may have fields of view individually, or collectively, common to the field of view of the LIDAR sensor 22 in the environment surrounding the vehicle 10 .
- the auxiliary sensors 24 can be, or include, one or more image sensors configured for capturing light or other electromagnetic energy from the environment surrounding the vehicle 10 .
- These image sensors may be, or include, one or more photodetectors, solid state photodetectors, photodiodes or photomultipliers, or any combination of these.
- the environment can be illuminated by the transmitter of the LIDAR sensor 22 .
- the auxiliary sensors 24 Responsive to capturing light or other electromagnetic energy, the auxiliary sensors 24 may be configured to output signals representing objects, or the lack thereof, in the environment surrounding the vehicle 10 .
- the vehicle 10 includes a computing device 30 to which the LIDAR sensor 22 and the auxiliary sensors 24 are communicatively connected through one or more communication links 32 .
- the computing device 30 and either or both of the LIDAR sensor 22 and the auxiliary sensors 24 may be dedicated to the autonomous operation system 20 , it is contemplated that some or all of these could also support the operation of other systems of the vehicle 10 .
- the computing device 30 may include a processor 40 communicatively coupled with a memory 42 .
- the processor 40 may include any device capable of executing machine-readable instructions, which may be stored on a non-transitory computer-readable medium, for example the memory 42 .
- the processor 40 may include a controller, an integrated circuit, a microchip, a computer, and/or any other computing device.
- the memory 42 may include any type of computer readable medium suitable for storing data and algorithms.
- the memory 42 may include RAM, ROM, a flash memory, a hard drive, and/or any device capable of storing machine-readable instructions.
- the computing device 30 may also include an input/output interface 44 for facilitating communication between the processor 40 and the LIDAR sensor 22 and the auxiliary sensors 24 .
- the computing device 30 is schematically illustrated as including a single processor 40 and a single memory 42 , in practice the computing device 30 may include a plurality of components, each having one or more memories 42 and/or processors 40 that may be communicatively coupled with one or more of the other components.
- the computing device 30 may be a separate standalone unit or may be configured as a part of a central control system for the vehicle 10 .
- the various algorithms and data for the autonomous operation system 20 and the other systems of the vehicle 10 may reside in whole or in part in the memory 42 of the computing device 30 .
- the signals output by the LIDAR sensor 22 and the auxiliary sensors 24 are stored in the memory 42 .
- the algorithms and data for the autonomous operation system 20 include a detection module 50 and a planning module 52 .
- the vehicle 10 may also include an integrated mobile communication system 60 with variously configured communication hardware for wirelessly transmitting data between the computing device 30 and a mobile network, such as a cellular network.
- the mobile communication system 60 and the mobile network together enable the computing device 30 to wirelessly communicate with other devices connected to the mobile network, such as a remote server that may similarly be, or include, a computing device including one or more processors and one or more memories, or another vehicle that may similarly include an object detection system with a computing device including one or more processors and one or more memories.
- the mobile communication system 60 of the vehicle 10 may include an integrated mobile network transceiver 62 configured to transmit and receive data over the mobile network.
- the mobile network transceiver 62 may be communicatively connected to the computing device 30 though a mobile network transceiver communication link 64 , with the input/output interface 44 facilitating communication between the processor 40 and the memory 42 and the mobile network transceiver 62 .
- the mobile network transceiver 62 includes a transmitter for wirelessly transferring data from the computing device 30 to the mobile network and a receiver for wirelessly transferring data from the mobile network to the computing device 30 .
- the overall operations of performing object detection in the detection module 50 of the autonomous operation system 20 of the vehicle 10 are introduced with reference to FIG. 2 .
- the detection module 50 of the autonomous operation system 20 has a perception and tracking thread and a classifier thread.
- the perception and tracking thread as the LIDAR sensor 22 scans the environment surrounding the vehicle 10 , 3D points and other signals output by the LIDAR sensor 22 and representing the objects in the environment surrounding the vehicle 10 are received.
- a digital map containing a 3D road network with positions for each lane and associated traffic rules e.g., speed limits, the priority of each road at intersections and roundabouts, and stop line positions
- traffic rules e.g., speed limits, the priority of each road at intersections and roundabouts, and stop line positions
- the 3D points representing the objects in the environment surrounding the vehicle 10 are received over multiple timesteps each corresponding, for instance, to a 360 degree spin around the vertical Z axis of the vehicle 10 .
- the 3D points may be evaluated to discriminate between 3D points representing obstacles and 3D points representing other objects, such as the ground, and collected into clusters of 3D points representing respective objects in the environment surrounding the vehicle 10 .
- the clustering may implement Markov random field-based clustering, for example.
- a given cluster of 3D points at one timestep representing an object in the environment surrounding the vehicle 10 is associated to clusters of 3D points at previous timesteps representing the same object in the environment surrounding the vehicle 10 .
- a so-called track is generated, which is a temporal series of clusters of 3D points representing the same object in the environment surrounding the vehicle 10 .
- the generation of a track may be implemented by, or from, both particle and Kalman filtering, for example.
- the tracks, once generated, are updated with new clusters of 3D points representing the object in the environment surrounding the vehicle 10 in subsequent iterations.
- the perception and tracking thread and the classifier thread run in parallel and communicate via concurrent result and request queues.
- the result queue is used to pass classification results from the classifier thread to the perception and tracking thread, while the request queue is filled by the perception and tracking thread with tracks for which classification results are needed.
- the detection module 50 may implement an anytime system in which the results of the classifier thread may be prioritized and integrated into the perception and tracking thread to ensure output for real-time decision-making to the planning module 52 despite the analytical costs of the classifier thread.
- each track may be given a score for prioritization prior to being inserted into the request queue.
- Tracks having clusters of 3D points in front of and in closest proximity to the vehicle 10 may be given the highest priority, for instance.
- the score given to each track may simply be the distance from the vehicle 10 to a track's clusters of 3D points plus a penalty (e.g., 50 meters) for being behind the vehicle 10 , with the tracks with lower scores being prioritized over those with higher scores.
- tracks having clusters of 3D points in certain areas of interest in the environment surrounding the vehicle 10 or tracks whose clusters of 3D points represent an object with an uncertain classification, for instance, could be prioritized over other tracks.
- the classifier thread pulls the highest priority track from the request queue, identifies a number of features of the track's clusters of 3D points, identifies classifiers for the track's clusters of 3D points and classifies the object represented by the track's clusters of 3D points based on its most likely object class, as explained in additional detail below, and puts the classifiers and an object class label reflecting the object's resulting classification into the result queue.
- the perception and tracking thread After the perception and tracking thread generates a track, it updates its clusters of 3D points with results of the classifier thread from the result queue. Then, all unprocessed tracks are cleared from the request queue, and the next track's clusters of 3D points are pushed onto it. Then, the perception and tracking thread sends each of the tracks to the planning module 52 . Each of the tracks has an associated history, so past results of the classifier thread may be used for tracks not addressed in a given iteration.
- a classifier may be expressed as the one-vs-all class log-odds
- log-odds can be converted to a probability, with the vector containing probabilities for each object class being
- This vector along with the most likely object class for the object represented by the track's clusters of 3D points, as well as typical tracking information such as position, velocity, and size of the object, are passed to the to the planning module 52 .
- the classification results from the classifier thread advantageously allow the planning module 52 to address the range of real world traffic situations otherwise faced by human drivers, such as interactions between pedestrians, bicycles and other vehicles. Addressing interactions between the vehicle 10 and pedestrians and bicycles is particularly important given the desire to ensure safety for occupants of the vehicle 10 , pedestrians and operators of bicycles, the potential for large speed differentials and sudden relative lateral motions between the vehicle 10 and pedestrians and bicycles, and the relative vulnerability of pedestrians and operators of bicycles.
- proper classification of these and other objects in the environment surrounding the vehicle 10 may, for example, provide information to the planning module 52 used in the determination of how much leeway to give the objects while they are being passed, or, in the determination of whether to pass those objects in the first place.
- FIG. 3 The operations of a process 100 for the classifier thread in the detection module 50 of the autonomous operation system 20 of the vehicle 10 are shown in FIG. 3 .
- the process 100 culminates in the combination of cluster-based classifiers identified based on local features for a track's clusters of 3D points, or cluster features, and a track-based classifier based on the global features for the track itself, or holistic features.
- the cluster features are based on the track's clusters of 3D points, which change from one timestep t to the next with changing distances, viewpoints and orientation of the LIDAR sensor 22 , among other things.
- the cluster features may, for example, correspond in whole or in part to the appearance of the track's clusters of 3D points.
- For the track there is a local, or cluster, feature set z 1:T for timesteps 1 through T.
- the holistic features are higher level summary statistics of the object represented by the track's clusters of 3D points.
- the holistic features may, for example, correspond in whole or in part to the motion of the object represented by the track's clusters of 3D points.
- each of the resulting feature sets corresponds to a classifier, so there will be T cluster-based classifiers and one track-based classifier incorporated into the object's ultimate classification.
- the local features for a track's clusters of 3D points, or cluster features may be identified, for instance, from spin images and histogram of oriented gradients (HOG) features derived from virtual orthographic images of the track's clusters of 3D points.
- HOG histogram of oriented gradients
- this identification requires the track's clusters of 3D points to be oriented consistently, which can be accomplished by estimating the principle direction of each of the track's clusters of 3D points.
- the Z axis can be assumed for up, and the principle direction can be searched for in the XY plane.
- the 3D points may be projected onto the XY plane, and a random sample consensus (RANSAC) may be ran on all of the 3D points to find the direction that the most 3D points align to (e.g., within 10 cm).
- RANSAC random sample consensus
- a threshold of 50% of the 3D points may be used, for example.
- FIG. 4A An example estimation of the principle direction of a cluster of 3D points representing a vehicle when viewed from the side is shown in FIG. 4A , where the principle direction PD is that to which the most 3D points align.
- this estimation of the principle direction of a cluster of 3D points sometimes fails for clusters of 3D points representing a vehicle when viewed from the back, since the principle direction to which the most 3D points align is perpendicular to the actual orientation of the vehicle.
- the learner will be able to consider local features for a track's clusters of 3D points from both views in classification.
- Example spin images for a tree, bicycle, sedan and station wagon are shown in FIG. 5 .
- a virtual image plane may be spun around the Z axis around the closest 3D points to the center top, front center and side of a given of the track's clusters of 3D points representing an object, accumulating all the 3D points hit along the way into bins.
- Example virtual orthographic images of a cluster of 3D points representing a vehicle when viewed from the front, side and top of the cluster of 3D points are shown in FIGS. 6A-C , respectively.
- Each virtual orthographic image is an orthographic projection of the cluster of 3D points oriented along the principal direction of the cluster of 3D points and centered on its bounding box.
- Other local features for a track's clusters of 3D points may be identified for a track's clusters of 3D points.
- the estimated principle direction of a cluster of 3D points may be used to orient a bounding box of the cluster of 3D points, and the local features may include, for example, the height, width and length of the bounding box of the cluster of 3D points, as well as the volume of the bounding box of the cluster of 3D points.
- the centroid of the bounding box of the cluster of 3D points may be identified, and the local features may include a distance to the centroid of the bounding box from the LIDAR sensor 22 , for instance, or otherwise from the vehicle 10 .
- the global features for a track may be, or include, a velocity of the track's clusters of 3D points that represents a velocity of the object represented by the track's clusters of 3D points. Accordingly, the global features for a track may include a maximum velocity of the track's clusters of 3D points or a maximum velocity of the track's clusters of 3D points, or both, for instance. Alternatively, or additionally, the global features for a track may be, or include, an acceleration of the track's clusters of 3D points that represents an acceleration of the object represented by the track's clusters of 3D points.
- the global features for a track may include a maximum acceleration of the track's clusters of 3D points or a maximum acceleration of the track's clusters of 3D points, or both, for instance.
- These and other global features for a track may be identified, for example, using a Kalman filter over the centroids of the track's clusters of 3D points.
- FIG. 7 is an example graphical model encoding the probabilistic independencies between the local features for a track's clusters of 3D points, or cluster features, and the global features for the track, or holistic features.
- the learning in operation 106 may implement a decision-tree-based Gentle ADABoost.
- cluster-based classifiers are identified based on the local features for a track's clusters of 3D points, or cluster features, and in operation 110 , a track-based classifier is identified based on the global features for the track itself, or holistic features.
- the cluster-based classifier and the track-based classifier may generally be predictions of which of the pedestrian (c p ), bicycle (c b ) vehicle (c v ) and background (c bg ) object classes that the object represented by the track's clusters of 3D points belongs to.
- the cluster-based classifier and the track-based classifier may be expressed as a one-vs-all log-odds that the object belongs to one of the pedestrian (c p ), bicycle (c b ) vehicle (c v ) and background (C bg ) object classes.
- a strong classifier H may be given by the sum of K weak classifiers h
- the weak classifiers h are regression trees of limited depth, using the local features and the global features for splits. This outputs real values, and the sum of the weak classifiers h may be directly used as opposed to a discrete output sign(H(x, c)). In the limit, the sum of the weak classifiers h converges to the log-odds L(x, c).
- these log-odds may be identified for both the cluster-based classifiers and the track-based classifier for the pedestrian (c p ), bicycle (c b ) and vehicle (c v ) object classes.
- the cluster-based classifiers and the track-based classifier are combined to classify the object represented by the track's clusters of 3D point based on its most likely object class apparent from the combined cluster-based classifiers and track-based classifier.
- T+1 classifier results from the T cluster-based classifiers and the single track-based classifier. As described below, these may be combined, for instance, using a variation of the normalized Discrete Bayes Filter (DBF) that weighs the cluster-based classifiers based on the amount of information on the cluster of 3D points from which they are identified.
- DBF normalized Discrete Bayes Filter
- the log-odds ratio is L(x).
- L 0 is the log prior odds, and equals log
- H is an empirical estimate of log prior odds for the cluster-based classifier
- L 0 H is an empirical estimate of log prior odds for the track-based classifier
- H c is the cluster-based classifier that returns the log-odds as identified for the local features for a track's clusters of 3D points, or cluster features
- H H is the track-based classifier that returns the log-odds as identified for the global features for a track, or holistic features.
- a weighing factor ⁇ t may be applied to the cluster-based classifiers to down-weight the cluster-based classifiers with decreasing amounts of information on the cluster of 3D points from which they are identified, or up-weight the cluster-based classifiers with increasing amounts of information on the cluster of 3D points from which they are identified, as the case may be:
- the weighing factor ⁇ t for a given cluster-based classifier may increase with increasing amounts of information on the cluster of 3D points from which it is identified according to
- n t is the number of 3D points in the cluster at time t and n ⁇ is a parameter controlling how quickly ⁇ grows with the number of 3D points.
- thresholds can be defined and enforced for the amounts of information on the cluster of 3D points from which a given cluster-based classifier is identified. These thresholds may be defined and enforced, for instance, on the amount of 3D points in the cluster (e.g., 25 3D points), the proximity between the vehicle 10 and the object represented by the cluster of 3D points (e.g., 30 meters), or both, and if the thresholds are not satisfied, the cluster-based classifier can be weighted to zero, for example, by setting the log-odds associated to the cluster-based classifier to zero.
- the above classification framework may be applied to the pedestrian (c p ), bicycle (c b ) and vehicle (c v ) object classes, giving the one-vs-all log odds
- the object represented by the track's clusters of 3D points may be classified as belonging to its most likely object class.
- the planning module 52 may plan how to drive the vehicle 10 along a travel route based on the object's classification while avoiding the object, and the autonomous operation system 20 of the vehicle 10 can drive the vehicle 10 along the route according to the plan.
- FIG. 9 shows the classifier confidence over time for an example track in a cases where the object represented by the track's clusters of 3D points is a bicycle.
- the solid lines show the confidence for the combined results from the classifier, while the dashed lines show the confidence for the cluster-based classifiers and the dashed-dot lines show the confidence for the track-based classifiers, for each of the pedestrian (c p ), bicycle (c b ) vehicle (c v ) and background (c bg ) object classes.
- the track-based classifier contributes to the classification of the object because there are too few 3D points in the clusters (i.e., fewer than 25).
- the object is initially classified as bicycle, for the first 40 timesteps, but then is misclassified as car at a distance of 82 meters, for the next 80 timesteps, until there are enough 3D points to use the cluster-based classifiers at a distance of 40 meters, at which point the bicycle (c b ) object class quickly wins out and remains represented in the combined results of the classifier despite several cluster misclassifications later.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Aviation & Aerospace Engineering (AREA)
- Electromagnetism (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Automation & Control Theory (AREA)
- Theoretical Computer Science (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Optics & Photonics (AREA)
- Traffic Control Systems (AREA)
- Control Of Driving Devices And Active Controlling Of Vehicle (AREA)
- Image Analysis (AREA)
Abstract
Description
- The embodiments disclosed herein generally relate to autonomous operation systems for vehicles and, more particularly, to the classification of objects detected by 3D sensors in autonomous operation systems.
- Some vehicles include an autonomous operation system with an operational mode in which the vehicle is driven along a travel route with minimal or no input from a human driver. In these vehicles, the autonomous operation system is configured to detect information about the environment surrounding the vehicle, including the presence of objects, and process the detected information in order to plan how to drive the vehicle along a travel route while avoiding the objects.
- In real world traffic situations, as a part of this detection and planning, it is desirable for the autonomous operation system to classify the objects in the environment surrounding the vehicle in order to account not only for other vehicles, but also for pedestrians and bicycles, among other objects. Improving the autonomous operation system's classification of the objects in the environment surrounding the vehicle is the subject of ongoing research.
- Disclosed herein are systems and methods for autonomous driving. In one aspect, a method of autonomous driving includes generating, with a 3D sensor, 3D points representing objects in the environment surrounding a vehicle. The method further includes, with a computing device, identifying, from the 3D points, a temporal series of clusters of 3D points representing the same object in the environment surrounding the vehicle as a track, identifying cluster-based classifiers for the object based on identified local features for the clusters in the track, identifying track-based classifiers for the object based on identified global features for the track, combining the cluster-based classifiers and the track-based classifiers to classify the object, with the cluster-based classifiers being weighted based on an amount of information on the clusters from which they are identified, and with the weight increasing with increasing amounts of information, and driving the vehicle along a route based on the object's classification.
- These and other aspects will be described in additional detail below.
- The various features, advantages and other uses of the present embodiments will become more apparent by referring to the following detailed description and drawing in which:
-
FIG. 1 is a schematic representation of a vehicle including an autonomous operation system whose operation is supported by a 3D sensor; -
FIG. 2 is a schematic representation of the system architecture of a detection module for the autonomous operation system; -
FIG. 3 is a flowchart showing the operations of a process for a classifier thread in the detection module; -
FIGS. 4A and 4B show example estimations of the principle direction of a cluster of 3D points representing a vehicle; -
FIG. 5 shows example spin images from clusters of 3D points representing different objects; -
FIGS. 6A-C show example virtual orthographic images of a cluster of 3D points representing a vehicle; -
FIG. 7 is an example graphical model encoding the probabilistic independencies between local features for a track's clusters of 3D points and global features for the track; -
FIG. 8 shows aspects of an example weighing factor for a cluster-based classifier; and -
FIG. 9 shows aspects of a classifier confidence over time for track. - This disclosure teaches a vehicle that includes an autonomous operation system that, in operation, classifies objects in the environment surrounding the vehicle represented by temporal series of clusters of 3D points, and drives the vehicle along a route based on the objects' classifications. The autonomous operation system classifies the objects based on both a track-based classifier and cluster-based classifiers, but weighs the cluster-based classifiers based on the amount of information on the clusters from which they are identified.
-
FIG. 1 shows avehicle 10 including an autonomous operation system 20 whose operation is supported by a LIDAR sensor 22 and one or more optionalauxiliary sensors 24. The LIDAR sensor 22 and theauxiliary sensors 24 are mounted on thevehicle 10 and positioned to have fields of view in the environment surrounding thevehicle 10. Although thevehicle 10 is provided as a non-limiting example of a mobile platform, it will be understood that the autonomous operation system 20 could be implemented in other mobile platforms. Additionally, although the LIDAR sensor 22 is provided as a non-limiting example of a 3D sensor, it will be understood that this description is applicable in principle to other 3D sensors. - The LIDAR sensor 22 is configured to scan the environment surrounding the
vehicle 10 and generate signals, including but not limited to 3D points, representing the objects in the environment surrounding thevehicle 10. - Generally, the LIDAR sensor 22 can include a transmitter and a receiver. The transmitter can be a component or group of components operable to transmit laser signals (e.g., laser light energy). As an example, the transmitter may be a laser, laser rangefinder, LIDAR, and/or laser scanner. The laser signals may have any suitable characteristics. In one or more arrangements, the laser signals may be from any suitable portion of the electromagnetic spectrum, such as from the ultraviolet, visible, or near infrared portions of the electromagnetic spectrum. The laser signals may be eye safe.
- The laser signals may be transmitted into the environment surrounding the
vehicle 10, where they impinge upon objects therein that are located in the path of the laser signals. The laser signals may be transmitted in series of 360 degree spins around a vertical Z axis of thevehicle 10, for example. Generally, when the laser signals impinge upon an object, a portion of the laser signals is returned (e.g., by reflection) to the LIDAR sensor 22. The returned portion of the laser signals can be captured at the LIDAR sensor 22 by its receiver, which may be, or include, one or more photodetectors, solid state photodetectors, photodiodes or photomultipliers, or any combination of these. - Responsive to capturing the returned laser signals, the LIDAR sensor 22 may be configured to output signals representing objects, or the lack thereof, in the environment surrounding the
vehicle 10. The LIDAR sensor 22 may include a global positioning system (GPS) or other positioning system for identifying its position, and an inertial measurement unit (IMU) for identifying its pose. According to this configuration, the signals may include 3D points representing the location in space of the points from which the returned laser signals are received, and therefore, the location in space of points of objects on which the laser signals impinged. The LIDAR sensor 22 may determine the location in space of points of objects based on the distance from the LIDAR sensor 22 to the points, as well as the position and pose of the LIDAR sensor 22 associated with the returned laser signals. The distance to the points may be determined from the returned laser signals using the time of flight (TOF) method, for instance. The signals may also represent the locations in space from which no returned laser signals are received, and therefore, the lack of points of objects in those locations in space on which the laser signals would otherwise have impinged. - The signals may further represent other aspects of the returned laser signals, which, in turn, may represent other properties of points of objects on which the incident laser signals impinged. These aspects of the returned laser signals can include their intensity or reflectivity, for instance, or any combination of these.
- The
auxiliary sensors 24 may also be configured to scan the environment surrounding thevehicle 10 and generate signals representing objects, or the lack thereof, in the environment surrounding thevehicle 10. - The
auxiliary sensors 24 may have fields of view individually, or collectively, common to the field of view of the LIDAR sensor 22 in the environment surrounding thevehicle 10. Generally, theauxiliary sensors 24 can be, or include, one or more image sensors configured for capturing light or other electromagnetic energy from the environment surrounding thevehicle 10. These image sensors may be, or include, one or more photodetectors, solid state photodetectors, photodiodes or photomultipliers, or any combination of these. Optionally, the environment can be illuminated by the transmitter of the LIDAR sensor 22. Responsive to capturing light or other electromagnetic energy, theauxiliary sensors 24 may be configured to output signals representing objects, or the lack thereof, in the environment surrounding thevehicle 10. - The
vehicle 10 includes acomputing device 30 to which the LIDAR sensor 22 and theauxiliary sensors 24 are communicatively connected through one ormore communication links 32. Although thecomputing device 30 and either or both of the LIDAR sensor 22 and theauxiliary sensors 24 may be dedicated to the autonomous operation system 20, it is contemplated that some or all of these could also support the operation of other systems of thevehicle 10. - The
computing device 30 may include a processor 40 communicatively coupled with amemory 42. The processor 40 may include any device capable of executing machine-readable instructions, which may be stored on a non-transitory computer-readable medium, for example thememory 42. The processor 40 may include a controller, an integrated circuit, a microchip, a computer, and/or any other computing device. Thememory 42 may include any type of computer readable medium suitable for storing data and algorithms. For example, thememory 42 may include RAM, ROM, a flash memory, a hard drive, and/or any device capable of storing machine-readable instructions. - The
computing device 30 may also include an input/output interface 44 for facilitating communication between the processor 40 and the LIDAR sensor 22 and theauxiliary sensors 24. Although thecomputing device 30 is schematically illustrated as including a single processor 40 and asingle memory 42, in practice thecomputing device 30 may include a plurality of components, each having one ormore memories 42 and/or processors 40 that may be communicatively coupled with one or more of the other components. Thecomputing device 30 may be a separate standalone unit or may be configured as a part of a central control system for thevehicle 10. - The various algorithms and data for the autonomous operation system 20 and the other systems of the
vehicle 10 may reside in whole or in part in thememory 42 of thecomputing device 30. In operation of the autonomous operation system 20, the signals output by the LIDAR sensor 22 and theauxiliary sensors 24 are stored in thememory 42. As described in additional detail below, the algorithms and data for the autonomous operation system 20 include adetection module 50 and aplanning module 52. - Although the various algorithms and data for the autonomous operation system 20 are described with reference to the
computing device 30 onboard thevehicle 10 for simplicity, it will be understood that these may reside in whole or in part in a memory of a computing device separate from thevehicle 10. In these cases, thevehicle 10 may also include an integrated mobile communication system 60 with variously configured communication hardware for wirelessly transmitting data between thecomputing device 30 and a mobile network, such as a cellular network. The mobile communication system 60 and the mobile network together enable thecomputing device 30 to wirelessly communicate with other devices connected to the mobile network, such as a remote server that may similarly be, or include, a computing device including one or more processors and one or more memories, or another vehicle that may similarly include an object detection system with a computing device including one or more processors and one or more memories. - The mobile communication system 60 of the
vehicle 10 may include an integrated mobile network transceiver 62 configured to transmit and receive data over the mobile network. The mobile network transceiver 62 may be communicatively connected to thecomputing device 30 though a mobile networktransceiver communication link 64, with the input/output interface 44 facilitating communication between the processor 40 and thememory 42 and the mobile network transceiver 62. The mobile network transceiver 62 includes a transmitter for wirelessly transferring data from thecomputing device 30 to the mobile network and a receiver for wirelessly transferring data from the mobile network to thecomputing device 30. - The overall operations of performing object detection in the
detection module 50 of the autonomous operation system 20 of thevehicle 10 are introduced with reference toFIG. 2 . - As shown, the
detection module 50 of the autonomous operation system 20 has a perception and tracking thread and a classifier thread. In the perception and tracking thread, as the LIDAR sensor 22 scans the environment surrounding thevehicle vehicle 10 are received. A digital map containing a 3D road network with positions for each lane and associated traffic rules (e.g., speed limits, the priority of each road at intersections and roundabouts, and stop line positions) may also be received. - In the perception and tracking thread, the 3D points representing the objects in the environment surrounding the
vehicle 10 are received over multiple timesteps each corresponding, for instance, to a 360 degree spin around the vertical Z axis of thevehicle 10. For each of the timesteps, the 3D points may be evaluated to discriminate between 3D points representing obstacles and 3D points representing other objects, such as the ground, and collected into clusters of 3D points representing respective objects in the environment surrounding thevehicle 10. The clustering may implement Markov random field-based clustering, for example. - A given cluster of 3D points at one timestep representing an object in the environment surrounding the
vehicle 10 is associated to clusters of 3D points at previous timesteps representing the same object in the environment surrounding thevehicle 10. Over multiple timesteps, a so-called track is generated, which is a temporal series of clusters of 3D points representing the same object in the environment surrounding thevehicle 10. The generation of a track may be implemented by, or from, both particle and Kalman filtering, for example. The tracks, once generated, are updated with new clusters of 3D points representing the object in the environment surrounding thevehicle 10 in subsequent iterations. - The perception and tracking thread and the classifier thread run in parallel and communicate via concurrent result and request queues. The result queue is used to pass classification results from the classifier thread to the perception and tracking thread, while the request queue is filled by the perception and tracking thread with tracks for which classification results are needed.
- With this configuration, the
detection module 50 may implement an anytime system in which the results of the classifier thread may be prioritized and integrated into the perception and tracking thread to ensure output for real-time decision-making to theplanning module 52 despite the analytical costs of the classifier thread. - To implement the anytime system, in the perception and tracking thread, each track may be given a score for prioritization prior to being inserted into the request queue. Tracks having clusters of 3D points in front of and in closest proximity to the
vehicle 10 may be given the highest priority, for instance. In these cases, the score given to each track may simply be the distance from thevehicle 10 to a track's clusters of 3D points plus a penalty (e.g., 50 meters) for being behind thevehicle 10, with the tracks with lower scores being prioritized over those with higher scores. Alternatively, or additionally, tracks having clusters of 3D points in certain areas of interest in the environment surrounding thevehicle 10, or tracks whose clusters of 3D points represent an object with an uncertain classification, for instance, could be prioritized over other tracks. - The classifier thread pulls the highest priority track from the request queue, identifies a number of features of the track's clusters of 3D points, identifies classifiers for the track's clusters of 3D points and classifies the object represented by the track's clusters of 3D points based on its most likely object class, as explained in additional detail below, and puts the classifiers and an object class label reflecting the object's resulting classification into the result queue.
- After the perception and tracking thread generates a track, it updates its clusters of 3D points with results of the classifier thread from the result queue. Then, all unprocessed tracks are cleared from the request queue, and the next track's clusters of 3D points are pushed onto it. Then, the perception and tracking thread sends each of the tracks to the
planning module 52. Each of the tracks has an associated history, so past results of the classifier thread may be used for tracks not addressed in a given iteration. - In general, for a track τ with features xτ, a classifier may be expressed as the one-vs-all class log-odds
-
- where Y is the object class label and c is a discrete object class. As explained in additional detail below, these log-odds can be converted to a probability, with the vector containing probabilities for each object class being
- using the shorthand
- where the given object classes are pedestrian (cp), bicycle (cb), vehicle (cv) and background (cbg) object classes, respectively. This vector, along with the most likely object class for the object represented by the track's clusters of 3D points, as well as typical tracking information such as position, velocity, and size of the object, are passed to the to the
planning module 52. - In the autonomous operation of the
vehicle 10 by its autonomous operation system 20, the classification results from the classifier thread advantageously allow theplanning module 52 to address the range of real world traffic situations otherwise faced by human drivers, such as interactions between pedestrians, bicycles and other vehicles. Addressing interactions between thevehicle 10 and pedestrians and bicycles is particularly important given the desire to ensure safety for occupants of thevehicle 10, pedestrians and operators of bicycles, the potential for large speed differentials and sudden relative lateral motions between thevehicle 10 and pedestrians and bicycles, and the relative vulnerability of pedestrians and operators of bicycles. Among other things, proper classification of these and other objects in the environment surrounding thevehicle 10 may, for example, provide information to theplanning module 52 used in the determination of how much leeway to give the objects while they are being passed, or, in the determination of whether to pass those objects in the first place. - The operations of a
process 100 for the classifier thread in thedetection module 50 of the autonomous operation system 20 of thevehicle 10 are shown inFIG. 3 . - As described below, the
process 100 culminates in the combination of cluster-based classifiers identified based on local features for a track's clusters of 3D points, or cluster features, and a track-based classifier based on the global features for the track itself, or holistic features. - In general, the cluster features are based on the track's clusters of 3D points, which change from one timestep t to the next with changing distances, viewpoints and orientation of the LIDAR sensor 22, among other things. The cluster features may, for example, correspond in whole or in part to the appearance of the track's clusters of 3D points. For the track, there is a local, or cluster, feature set z1:T for
timesteps 1 through T. - In general, the holistic features are higher level summary statistics of the object represented by the track's clusters of 3D points. The holistic features may, for example, correspond in whole or in part to the motion of the object represented by the track's clusters of 3D points. For the track, there is a single global, or holistic, feature set ω. With both the cluster feature set z1:T and the single holistic feature set ω, the feature set for the track at T is xT=z1:T, ω.
- In
operation 102, the local features for a track's clusters of 3D points, or cluster features, are identified, and inoperation 104, the global features for the track itself, or holistic features, are identified. In theprocess 100, for the track, each of the resulting feature sets corresponds to a classifier, so there will be T cluster-based classifiers and one track-based classifier incorporated into the object's ultimate classification. - The local features for a track's clusters of 3D points, or cluster features, may be identified, for instance, from spin images and histogram of oriented gradients (HOG) features derived from virtual orthographic images of the track's clusters of 3D points. In general, this identification requires the track's clusters of 3D points to be oriented consistently, which can be accomplished by estimating the principle direction of each of the track's clusters of 3D points.
- With the
vehicle 10 driving on relatively flat ground as the LIDAR sensor 22 scans the environment surrounding thevehicle 10, the Z axis can be assumed for up, and the principle direction can be searched for in the XY plane. To estimate the principle direction of a given cluster of 3D points, the 3D points may be projected onto the XY plane, and a random sample consensus (RANSAC) may be ran on all of the 3D points to find the direction that the most 3D points align to (e.g., within 10 cm). A threshold of 50% of the 3D points may be used, for example. An example estimation of the principle direction of a cluster of 3D points representing a vehicle when viewed from the side is shown inFIG. 4A , where the principle direction PD is that to which the most 3D points align. - It has been found that this estimation of the principle direction of a cluster of 3D points generally works well for clusters of 3D points representing vehicles and bicycles when viewed from the side, as well as for clusters of 3D points representing pedestrians. As can be seen from the comparison of the estimation of the principle direction PD of a cluster of 3D points representing a vehicle when viewed from the side, shown in
FIG. 4A , with the estimation of the principle direction PD of a cluster of 3D points representing a vehicle when viewed from the back, shown inFIG. 4B , this estimation of the principle direction of a cluster of 3D points sometimes fails for clusters of 3D points representing a vehicle when viewed from the back, since the principle direction to which the most 3D points align is perpendicular to the actual orientation of the vehicle. However, if the same estimation of the principle direction of a cluster of 3D points is used in both learning and in classification, the learner will be able to consider local features for a track's clusters of 3D points from both views in classification. - Example spin images for a tree, bicycle, sedan and station wagon are shown in
FIG. 5 . To generate the example and other spin images, a virtual image plane may be spun around the Z axis around the closest 3D points to the center top, front center and side of a given of the track's clusters of 3D points representing an object, accumulating all the 3D points hit along the way into bins. - Example virtual orthographic images of a cluster of 3D points representing a vehicle when viewed from the front, side and top of the cluster of 3D points are shown in
FIGS. 6A-C , respectively. Each virtual orthographic image is an orthographic projection of the cluster of 3D points oriented along the principal direction of the cluster of 3D points and centered on its bounding box. - Other local features for a track's clusters of 3D points, or cluster features, may be identified for a track's clusters of 3D points. For instance, the estimated principle direction of a cluster of 3D points may be used to orient a bounding box of the cluster of 3D points, and the local features may include, for example, the height, width and length of the bounding box of the cluster of 3D points, as well as the volume of the bounding box of the cluster of 3D points. Additionally, or alternatively, the centroid of the bounding box of the cluster of 3D points may be identified, and the local features may include a distance to the centroid of the bounding box from the LIDAR sensor 22, for instance, or otherwise from the
vehicle 10. - The global features for a track, or holistic features, may be, or include, a velocity of the track's clusters of 3D points that represents a velocity of the object represented by the track's clusters of 3D points. Accordingly, the global features for a track may include a maximum velocity of the track's clusters of 3D points or a maximum velocity of the track's clusters of 3D points, or both, for instance. Alternatively, or additionally, the global features for a track may be, or include, an acceleration of the track's clusters of 3D points that represents an acceleration of the object represented by the track's clusters of 3D points. Accordingly, the global features for a track may include a maximum acceleration of the track's clusters of 3D points or a maximum acceleration of the track's clusters of 3D points, or both, for instance. These and other global features for a track may be identified, for example, using a Kalman filter over the centroids of the track's clusters of 3D points.
- In
operation 106, it is learned which local features, or cluster features, and which global features, or holistic features, are predictive of objects belonging to the pedestrian (cp), bicycle (cb), vehicle (cv) and background (cbg) object classes.FIG. 7 is an example graphical model encoding the probabilistic independencies between the local features for a track's clusters of 3D points, or cluster features, and the global features for the track, or holistic features. The learning inoperation 106 may implement a decision-tree-based Gentle ADABoost. - In
operation 108, cluster-based classifiers are identified based on the local features for a track's clusters of 3D points, or cluster features, and inoperation 110, a track-based classifier is identified based on the global features for the track itself, or holistic features. - The cluster-based classifier and the track-based classifier may generally be predictions of which of the pedestrian (cp), bicycle (cb) vehicle (cv) and background (cbg) object classes that the object represented by the track's clusters of 3D points belongs to.
- In one example, the cluster-based classifier and the track-based classifier may be expressed as a one-vs-all log-odds that the object belongs to one of the pedestrian (cp), bicycle (cb) vehicle (cv) and background (Cbg) object classes. According to this example, in general, a strong classifier H may be given by the sum of K weak classifiers h
-
- where the weak classifiers h are regression trees of limited depth, using the local features and the global features for splits. This outputs real values, and the sum of the weak classifiers h may be directly used as opposed to a discrete output sign(H(x, c)). In the limit, the sum of the weak classifiers h converges to the log-odds L(x, c). Using the Gentle AdaBoost, as implemented in OpenCV, these log-odds may be identified for both the cluster-based classifiers and the track-based classifier for the pedestrian (cp), bicycle (cb) and vehicle (cv) object classes.
- In
operation 112, the cluster-based classifiers and the track-based classifier are combined to classify the object represented by the track's clusters of 3D point based on its most likely object class apparent from the combined cluster-based classifiers and track-based classifier. - To identify the log-odds for a track of length T, for a specific object class c, there are T+1 classifier results from the T cluster-based classifiers and the single track-based classifier. As described below, these may be combined, for instance, using a variation of the normalized Discrete Bayes Filter (DBF) that weighs the cluster-based classifiers based on the amount of information on the cluster of 3D points from which they are identified.
- To simplify notation, the combination is described for one object class c, and the object class notations are omitted below. Accordingly, the log-odds ratio is L(x). As used below, L0 is the log prior odds, and equals log
-
- is an empirical estimate of log prior odds for the cluster-based classifier, L0 H is an empirical estimate of log prior odds for the track-based classifier, Hc is the cluster-based classifier that returns the log-odds as identified for the local features for a track's clusters of 3D points, or cluster features, and HH is the track-based classifier that returns the log-odds as identified for the global features for a track, or holistic features.
- Although the example graphical model in
FIG. 7 assumes conditional independence between the local features for a track's clusters of 3D points, or cluster features, and the global features for a track, or holistic features, a more sophisticated model is described below. To begin with, however, assuming this conditional independence, to identify the log-odds L(ω,z1:T) given all local features for a track's clusters of 3D points and the global features for a track over all timesteps (from 1 to T) for the track, and using Bayes rule: -
- This has the effect of placing unequal weight on the contribution of the track-based classifier, depending on the length of the track. Adding normalization term:
-
- This still has the effect of placing equal weight on every cluster-based classifier. Although this would be correct if the cluster-based classifiers accurately predicted which of the object classes that the object represented by the track's clusters of 3D points belongs to in all cases, it has been found that the predictive accuracy of the cluster-based classifiers significantly increases with increasing amounts of information on the cluster of 3D points from which they are identified. In most instances, the amounts of information on the cluster of 3D points from which a given cluster-based classifier is identified is, or is associated with, the amount of 3D points in the cluster. In this or other instances, increasing amounts of information on these clusters of 3D points may be the product of closer proximity between the
vehicle 10 and the object represented by the cluster of 3D points. - Accordingly, a weighing factor αt may be applied to the cluster-based classifiers to down-weight the cluster-based classifiers with decreasing amounts of information on the cluster of 3D points from which they are identified, or up-weight the cluster-based classifiers with increasing amounts of information on the cluster of 3D points from which they are identified, as the case may be:
-
- As shown with additional reference to
FIG. 8 , the weighing factor αt for a given cluster-based classifier may increase with increasing amounts of information on the cluster of 3D points from which it is identified according to -
- where nt is the number of 3D points in the cluster at time t and nα is a parameter controlling how quickly α grows with the number of 3D points. In
FIG. 8 , nα=250, and it can be seen that 0≦α≦1 and α=0.5 when nt=nα. - Additionally, or alternatively, thresholds can be defined and enforced for the amounts of information on the cluster of 3D points from which a given cluster-based classifier is identified. These thresholds may be defined and enforced, for instance, on the amount of 3D points in the cluster (e.g., 25 3D points), the proximity between the
vehicle 10 and the object represented by the cluster of 3D points (e.g., 30 meters), or both, and if the thresholds are not satisfied, the cluster-based classifier can be weighted to zero, for example, by setting the log-odds associated to the cluster-based classifier to zero. - Returning again to the discussion of the pedestrian (cp), bicycle (cb), vehicle (cv) and background (Cbg) object classes, for each track and timestep, with features ω, Z1:T, the above classification framework may be applied to the pedestrian (cp), bicycle (cb) and vehicle (cv) object classes, giving the one-vs-all log odds
-
L(ω,z1:T;Cp), L(ω, z1:T; cb), l(ω, z1:T; cv) (Eq. 8) - predicting which of the pedestrian (cp), bicycle (cb) and vehicle (cv) object classes that the object represented by the track's clusters of 3D points belongs to. This may be converted to a probability by solving
-
- for the probability of which of the pedestrian (cp), bicycle (cb) and vehicle (cv) object classes that the object represented by the track's clusters of 3D points belongs to. For the background (cbg) object class:
- With these results from the classifier, the object represented by the track's clusters of 3D points may be classified as belonging to its most likely object class. In
operation 114, theplanning module 52 may plan how to drive thevehicle 10 along a travel route based on the object's classification while avoiding the object, and the autonomous operation system 20 of thevehicle 10 can drive thevehicle 10 along the route according to the plan. -
FIG. 9 shows the classifier confidence over time for an example track in a cases where the object represented by the track's clusters of 3D points is a bicycle. The solid lines show the confidence for the combined results from the classifier, while the dashed lines show the confidence for the cluster-based classifiers and the dashed-dot lines show the confidence for the track-based classifiers, for each of the pedestrian (cp), bicycle (cb) vehicle (cv) and background (cbg) object classes. For the first 120 timesteps, only the track-based classifier contributes to the classification of the object because there are too few 3D points in the clusters (i.e., fewer than 25). The object is initially classified as bicycle, for the first 40 timesteps, but then is misclassified as car at a distance of 82 meters, for the next 80 timesteps, until there are enough 3D points to use the cluster-based classifiers at a distance of 40 meters, at which point the bicycle (cb) object class quickly wins out and remains represented in the combined results of the classifier despite several cluster misclassifications later. - While recited characteristics and conditions of the invention have been described in connection with certain embodiments, it is to be understood that the invention is not to be limited to the disclosed embodiments but, on the contrary, is intended to cover various modifications and equivalent arrangements included within the spirit and scope of the appended claims, which scope is to be accorded the broadest interpretation so as to encompass all such modifications and equivalent structures as is permitted under the law.
Claims (18)
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/852,806 US9576185B1 (en) | 2015-09-14 | 2015-09-14 | Classifying objects detected by 3D sensors for autonomous vehicle operation |
DE102016116029.6A DE102016116029B4 (en) | 2015-09-14 | 2016-08-29 | Classification of objects detected by 3D sensors |
JP2016177291A JP6298128B2 (en) | 2015-09-14 | 2016-09-12 | Classification of objects detected by 3D sensors |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/852,806 US9576185B1 (en) | 2015-09-14 | 2015-09-14 | Classifying objects detected by 3D sensors for autonomous vehicle operation |
Publications (2)
Publication Number | Publication Date |
---|---|
US9576185B1 US9576185B1 (en) | 2017-02-21 |
US20170075356A1 true US20170075356A1 (en) | 2017-03-16 |
Family
ID=58017693
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/852,806 Active US9576185B1 (en) | 2015-09-14 | 2015-09-14 | Classifying objects detected by 3D sensors for autonomous vehicle operation |
Country Status (3)
Country | Link |
---|---|
US (1) | US9576185B1 (en) |
JP (1) | JP6298128B2 (en) |
DE (1) | DE102016116029B4 (en) |
Cited By (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170366755A1 (en) * | 2016-06-20 | 2017-12-21 | Gopro, Inc. | Image Alignment Using a Virtual Gyroscope Model |
US10891518B1 (en) | 2018-12-14 | 2021-01-12 | Waymo Llc | Auto labeler |
US20210027102A1 (en) * | 2018-04-05 | 2021-01-28 | Koito Manufacturing Co., Ltd. | Processing device |
US11037305B2 (en) | 2017-12-29 | 2021-06-15 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method and apparatus for processing point cloud data |
US11341604B2 (en) | 2018-04-05 | 2022-05-24 | Koito Manufacturing Co., Ltd. | Processing device for object identification |
US11754714B2 (en) | 2017-08-18 | 2023-09-12 | Koito Manufacturing Co., Ltd. | Recognition sensor and control method thereof, automobile, automotive lamp, object recognition system, and object recognition method |
US11841435B2 (en) | 2018-03-15 | 2023-12-12 | Koito Manufacturing Co., Ltd. | Object identification system |
Families Citing this family (28)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10121082B2 (en) * | 2015-10-07 | 2018-11-06 | Honda Motor Co., Ltd. | System and method for providing laser camera fusion for identifying and tracking a traffic participant |
CA2998956C (en) * | 2015-11-26 | 2023-03-21 | Sportlogiq Inc. | Systems and methods for object tracking and localization in videos with adaptive image representation |
US10740658B2 (en) * | 2016-09-08 | 2020-08-11 | Mentor Graphics Corporation | Object recognition and classification using multiple sensor modalities |
US11390557B2 (en) * | 2017-03-30 | 2022-07-19 | Agc Glass Europe | Glass for autonomous car |
US10710579B2 (en) * | 2017-06-01 | 2020-07-14 | Waymo Llc | Collision prediction system |
US11042155B2 (en) | 2017-06-06 | 2021-06-22 | Plusai Limited | Method and system for closed loop perception in autonomous driving vehicles |
US11392133B2 (en) | 2017-06-06 | 2022-07-19 | Plusai, Inc. | Method and system for object centric stereo in autonomous driving vehicles |
US11573573B2 (en) | 2017-06-06 | 2023-02-07 | Plusai, Inc. | Method and system for distributed learning and adaptation in autonomous driving vehicles |
US10401862B2 (en) | 2017-10-31 | 2019-09-03 | Waymo Llc | Semantic object clustering for autonomous vehicle decision making |
US10713940B2 (en) | 2017-10-31 | 2020-07-14 | Waymo Llc | Detecting and responding to traffic redirection for autonomous vehicles |
CN108230379B (en) * | 2017-12-29 | 2020-12-04 | 百度在线网络技术(北京)有限公司 | Method and device for fusing point cloud data |
CN110555449B (en) * | 2018-05-30 | 2023-03-24 | 百度在线网络技术(北京)有限公司 | Method and apparatus for outputting information |
US10845818B2 (en) * | 2018-07-30 | 2020-11-24 | Toyota Research Institute, Inc. | System and method for 3D scene reconstruction of agent operation sequences using low-level/high-level reasoning and parametric models |
US10754338B2 (en) | 2018-08-14 | 2020-08-25 | GM Global Technology Operations LLC | Controlling an autonomous vehicle based upon labels assigned to objects represented in sensor data |
US10984540B2 (en) * | 2018-10-15 | 2021-04-20 | Tusimple, Inc. | Tracking and modeling processing of image data for LiDAR-based vehicle tracking system and method |
US10878282B2 (en) * | 2018-10-15 | 2020-12-29 | Tusimple, Inc. | Segmentation processing of image data for LiDAR-based vehicle tracking system and method |
US10878580B2 (en) | 2018-10-15 | 2020-12-29 | Tusimple, Inc. | Point cluster refinement processing of image data for LiDAR-based vehicle tracking system and method |
US10983530B2 (en) * | 2018-10-31 | 2021-04-20 | Wipro Limited | Method and system for determining an accurate position of an autonomous vehicle |
WO2020110802A1 (en) | 2018-11-30 | 2020-06-04 | 株式会社小糸製作所 | On-vehicle object identification system, automobile, vehicle lamp, classifier learning method, and arithmetic operation device |
WO2020110801A1 (en) | 2018-11-30 | 2020-06-04 | 株式会社小糸製作所 | Ranging sensor, vehicle lamp, and ranging method |
CN109726692A (en) * | 2018-12-29 | 2019-05-07 | 重庆集诚汽车电子有限责任公司 | High-definition camera 3D object detection system based on deep learning |
WO2020168464A1 (en) * | 2019-02-19 | 2020-08-27 | SZ DJI Technology Co., Ltd. | Local sensing based autonomous navigation, and associated systems and methods |
US11630209B2 (en) | 2019-07-09 | 2023-04-18 | Waymo Llc | Laser waveform embedding |
EP3795940A1 (en) * | 2019-09-19 | 2021-03-24 | sentronics metrology GmbH | Device and method for inspecting flat objects and for detecting boundary layers of said objects |
US11597406B2 (en) * | 2020-02-19 | 2023-03-07 | Uatc, Llc | Systems and methods for detecting actors with respect to an autonomous vehicle |
CN113870347A (en) * | 2020-06-30 | 2021-12-31 | 北京市商汤科技开发有限公司 | Target vehicle control method and device, electronic equipment and storage medium |
JP7001795B1 (en) * | 2020-11-13 | 2022-01-20 | 株式会社ネクスティエレクトロニクス | Data generators, methods and computer programs for simulation |
CN114387305A (en) * | 2021-12-30 | 2022-04-22 | 驭势科技(北京)有限公司 | Trailer queue tracking method and device, computer equipment and storage medium |
Family Cites Families (20)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP3516856B2 (en) * | 1998-01-30 | 2004-04-05 | 富士重工業株式会社 | Outside monitoring device |
JP4239689B2 (en) | 2003-05-30 | 2009-03-18 | スズキ株式会社 | Vehicle alarm system and vehicle alarm generation method |
US7522066B2 (en) | 2006-02-23 | 2009-04-21 | Rockwell Automation Technologies, Inc. | Systems and methods that evaluate distance to potential hazards utilizing overlapping sensing zones |
US7720260B2 (en) * | 2006-09-13 | 2010-05-18 | Ford Motor Company | Object detection system and method |
EP2107503A1 (en) * | 2008-03-31 | 2009-10-07 | Harman Becker Automotive Systems GmbH | Method and device for generating a real time environment model for vehicles |
KR100946723B1 (en) | 2008-04-12 | 2010-03-12 | 재단법인서울대학교산학협력재단 | Steering Method for vehicle and Apparatus thereof |
US8744122B2 (en) * | 2008-10-22 | 2014-06-03 | Sri International | System and method for object detection from a moving platform |
FR2953313B1 (en) * | 2009-11-27 | 2012-09-21 | Thales Sa | OPTRONIC SYSTEM AND METHOD FOR PREPARING THREE-DIMENSIONAL IMAGES FOR IDENTIFICATION |
WO2012117528A1 (en) * | 2011-03-01 | 2012-09-07 | トヨタ自動車株式会社 | State estimation device |
CN102685382B (en) * | 2011-03-18 | 2016-01-20 | 安尼株式会社 | Image processing apparatus and method and moving body collision prevention device |
US8605998B2 (en) * | 2011-05-06 | 2013-12-10 | Toyota Motor Engineering & Manufacturing North America, Inc. | Real-time 3D point cloud obstacle discriminator apparatus and associated methodology for training a classifier via bootstrapping |
DE102011101246A1 (en) | 2011-05-11 | 2012-11-15 | Conti Temic Microelectronic Gmbh | REDUDENT OBJECT DETECTION FOR DRIVER ASSISTANCE SYSTEMS |
FR2981772B1 (en) * | 2011-10-21 | 2017-12-22 | Thales Sa | METHOD FOR 3D RECONSTRUCTION OF AN OBJECT OF A SCENE |
DE102012014300A1 (en) | 2012-07-19 | 2013-01-17 | Daimler Ag | Method for recognizing object e.g. tree in environment of car, involves assigning objects based on object attribute to object class and adjusting operating condition of vehicle based on assignment of object to object class |
US9383753B1 (en) * | 2012-09-26 | 2016-07-05 | Google Inc. | Wide-view LIDAR with areas of special attention |
US8473144B1 (en) * | 2012-10-30 | 2013-06-25 | Google Inc. | Controlling vehicle lateral lane positioning |
US9110163B2 (en) * | 2013-06-14 | 2015-08-18 | Microsoft Technology Licensing, Llc | Lidar-based classification of object movement |
US8989944B1 (en) * | 2013-11-26 | 2015-03-24 | Google Inc. | Methods and devices for determining movements of an object in an environment |
US9091558B2 (en) * | 2013-12-23 | 2015-07-28 | Automotive Research & Testing Center | Autonomous driver assistance system and autonomous driving method thereof |
US9664779B2 (en) * | 2014-07-03 | 2017-05-30 | GM Global Technology Operations LLC | Object classification for vehicle radar systems |
-
2015
- 2015-09-14 US US14/852,806 patent/US9576185B1/en active Active
-
2016
- 2016-08-29 DE DE102016116029.6A patent/DE102016116029B4/en not_active Expired - Fee Related
- 2016-09-12 JP JP2016177291A patent/JP6298128B2/en active Active
Cited By (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20170366755A1 (en) * | 2016-06-20 | 2017-12-21 | Gopro, Inc. | Image Alignment Using a Virtual Gyroscope Model |
US9961261B2 (en) * | 2016-06-20 | 2018-05-01 | Gopro, Inc. | Image alignment using a virtual gyroscope model |
US10382683B2 (en) | 2016-06-20 | 2019-08-13 | Gopro, Inc. | Image alignment using a virtual gyroscope model |
US11503209B2 (en) | 2016-06-20 | 2022-11-15 | Gopro, Inc. | Image alignment using a virtual gyroscope model |
US11754714B2 (en) | 2017-08-18 | 2023-09-12 | Koito Manufacturing Co., Ltd. | Recognition sensor and control method thereof, automobile, automotive lamp, object recognition system, and object recognition method |
US11037305B2 (en) | 2017-12-29 | 2021-06-15 | Baidu Online Network Technology (Beijing) Co., Ltd. | Method and apparatus for processing point cloud data |
US11841435B2 (en) | 2018-03-15 | 2023-12-12 | Koito Manufacturing Co., Ltd. | Object identification system |
US20210027102A1 (en) * | 2018-04-05 | 2021-01-28 | Koito Manufacturing Co., Ltd. | Processing device |
US11341604B2 (en) | 2018-04-05 | 2022-05-24 | Koito Manufacturing Co., Ltd. | Processing device for object identification |
US11676394B2 (en) * | 2018-04-05 | 2023-06-13 | Koito Manufacturing Co., Ltd. | Processing device for conversion of images |
US10891518B1 (en) | 2018-12-14 | 2021-01-12 | Waymo Llc | Auto labeler |
US11556744B1 (en) | 2018-12-14 | 2023-01-17 | Waymo Llc | Auto labeler |
Also Published As
Publication number | Publication date |
---|---|
JP2017056935A (en) | 2017-03-23 |
JP6298128B2 (en) | 2018-03-20 |
DE102016116029B4 (en) | 2019-05-16 |
DE102016116029A1 (en) | 2017-03-23 |
US9576185B1 (en) | 2017-02-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9576185B1 (en) | Classifying objects detected by 3D sensors for autonomous vehicle operation | |
US11175145B2 (en) | System and method for precision localization and mapping | |
US20210382488A1 (en) | Systems and Methods for Prioritizing Object Prediction for Autonomous Vehicles | |
EP3745158B1 (en) | Methods and systems for computer-based determining of presence of dynamic objects | |
US11783568B2 (en) | Object classification using extra-regional context | |
US11769318B2 (en) | Systems and methods for intelligent selection of data for building a machine learning model | |
WO2020264010A1 (en) | Low variance region detection for improved detection | |
US11657591B2 (en) | Autonomous vehicle system for intelligent on-board selection of data for building a remote machine learning model | |
US11361201B2 (en) | Systems and methods for determining an object type and an attribute for an observation based on fused sensor data | |
US20220188695A1 (en) | Autonomous vehicle system for intelligent on-board selection of data for training a remote machine learning model | |
US11361484B1 (en) | Methods and systems for ground segmentation using graph-cuts | |
US12005928B2 (en) | Dangerous road user detection and response | |
EP4145398A1 (en) | Systems and methods for vehicle camera obstruction detection | |
CN116434162A (en) | System and method for reducing missed detection of tracked objects in the surrounding environment of a vehicle | |
US11731662B2 (en) | Autonomous vehicle system for detecting pedestrian presence | |
CN116745188A (en) | Method and system for generating a longitudinal plan for an autonomous vehicle based on the behavior of an uncertain road user | |
Jaiswal et al. | An advanced deep learning model for maneuver prediction in real-time systems using alarming-based hunting optimization. | |
Brehar et al. | Outdoor Traffic Scene Risk Estimation in the Context of Autonomous Driving | |
KR20230102003A (en) | Device and method for estimating the position of pedestrians in the blind spot of an intersection based on lidar sensor | |
CN115690719A (en) | System and method for object proximity monitoring around a vehicle |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: TOYOTA MOTOR ENGINEERING & MANUFACTURING NORTH AME Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:DELP, MICHAEL J.;REEL/FRAME:036642/0473 Effective date: 20150911 |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
AS | Assignment |
Owner name: TOYOTA JIDOSHA KABUSHIKI KAISHA, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:TOYOTA MOTOR ENGINEERING & MANUFACTORING NORTH AMERICA, INC.;REEL/FRAME:041752/0106 Effective date: 20170215 |
|
AS | Assignment |
Owner name: TOYOTA JIDOSHA KABUSHIKI KAISHA, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:TOYOTA MOTOR ENGINEERING & MANUFACTURING NORTH AMERICA, INC.;REEL/FRAME:041912/0861 Effective date: 20170215 |
|
CC | Certificate of correction | ||
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |