US20210223402A1 - Autonomous vehicle controlled based upon a lidar data segmentation system - Google Patents
Autonomous vehicle controlled based upon a lidar data segmentation system Download PDFInfo
- Publication number
- US20210223402A1 US20210223402A1 US17/226,123 US202117226123A US2021223402A1 US 20210223402 A1 US20210223402 A1 US 20210223402A1 US 202117226123 A US202117226123 A US 202117226123A US 2021223402 A1 US2021223402 A1 US 2021223402A1
- Authority
- US
- United States
- Prior art keywords
- point
- points
- lidar
- representative
- segmentation
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 230000011218 segmentation Effects 0.000 title claims abstract description 110
- 238000013528 artificial neural network Methods 0.000 claims abstract description 31
- 238000000034 method Methods 0.000 claims description 31
- 238000004891 communication Methods 0.000 claims description 7
- 238000012805 post-processing Methods 0.000 description 30
- 238000012549 training Methods 0.000 description 12
- 238000010586 diagram Methods 0.000 description 6
- 238000005516 engineering process Methods 0.000 description 5
- 238000007781 pre-processing Methods 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 238000013500 data storage Methods 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 239000013598 vector Substances 0.000 description 3
- 230000004075 alteration Effects 0.000 description 2
- 238000013459 approach Methods 0.000 description 2
- 239000000835 fiber Substances 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000000644 propagated effect Effects 0.000 description 2
- 244000025254 Cannabis sativa Species 0.000 description 1
- 101000822695 Clostridium perfringens (strain 13 / Type A) Small, acid-soluble spore protein C1 Proteins 0.000 description 1
- 101000655262 Clostridium perfringens (strain 13 / Type A) Small, acid-soluble spore protein C2 Proteins 0.000 description 1
- 101000655256 Paraclostridium bifermentans Small, acid-soluble spore protein alpha Proteins 0.000 description 1
- 101000655264 Paraclostridium bifermentans Small, acid-soluble spore protein beta Proteins 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 238000002485 combustion reaction Methods 0.000 description 1
- 238000004590 computer program Methods 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 230000001172 regenerating effect Effects 0.000 description 1
- 238000012360 testing method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/4808—Evaluating distance, position or velocity data
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
- G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
- G05D1/0088—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D1/00—Control of position, course or altitude of land, water, air, or space vehicles, e.g. automatic pilot
- G05D1/02—Control of position or course in two dimensions
- G05D1/021—Control of position or course in two dimensions specially adapted to land vehicles
- G05D1/0231—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means
- G05D1/0238—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using obstacle or wall sensors
- G05D1/024—Control of position or course in two dimensions specially adapted to land vehicles using optical position detecting means using obstacle or wall sensors in combination with a laser
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/217—Validation; Performance evaluation; Active pattern learning techniques
-
- G06K9/00805—
-
- G06K9/6262—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/82—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
- G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
-
- G—PHYSICS
- G05—CONTROLLING; REGULATING
- G05D—SYSTEMS FOR CONTROLLING OR REGULATING NON-ELECTRIC VARIABLES
- G05D2201/00—Application
- G05D2201/02—Control of position of land vehicles
- G05D2201/0213—Road vehicle, e.g. car or truck
Definitions
- An autonomous vehicle is a motorized vehicle that can operate without human conduction.
- An exemplary autonomous vehicle includes a plurality of sensor systems, such as, but not limited to, a lidar sensor system, a camera sensor system, and a radar sensor system, amongst others, wherein the autonomous vehicle operates based upon sensor signals output by the sensor systems.
- autonomous vehicles are configured to identify locations of different objects in a driving environment based upon different sensor signals.
- a radar system can identify a range from the autonomous vehicle to another vehicle in the driving environment.
- an object recognition system may be configured to receive images output by a camera and to identify relative positions of objects captured in the images.
- a lidar point cloud representative of the driving environment it may be difficult to determine by way of conventional algorithms what points are representative of same objects and what points are representative of different objects in the driving environment.
- algorithms based upon human-generated rules have been used to perform segmentation of a lidar point cloud to identify objects and their locations in the driving environment. For example, an algorithm may identify two points in a lidar point cloud as belonging to a same object based upon the points being within 50 centimeters of one another.
- This conventional approach may not be robust to variations in the driving environment, and can result in possible errors with respect to object identification by the autonomous vehicle.
- various conventional algorithms for performing segmentation of a lidar point cloud may perform poorly when certain types of objects are present in a driving environment (e.g., groups of closely-spaced pedestrians, vegetation, non-convex objects, etc.)
- Described herein are various technologies pertaining to controlling operation of an autonomous vehicle. With more specificity, described herein are various technologies pertaining to segmentation of a lidar point cloud, wherein the lidar point cloud is representative of positions of objects in the driving environment of the autonomous vehicle at one or more points in time. With still more specificity, a lidar data segmentation system is described herein, wherein the lidar data segmentation system comprises a neural network that is trained to output data indicative of a probability that a point in a lidar point cloud is representative of at least one of ground (e.g., a road surface, a sidewalk surface, or other ground surface) or vegetation.
- ground e.g., a road surface, a sidewalk surface, or other ground surface
- the lidar data segmentation system can assign a label to the point based upon the probability indicated by the neural network output, the label indicating a classification of a type of object that the point is expected to represent.
- the lidar segmentation system then performs a segmentation of the lidar point cloud based in part upon the label assigned to the point.
- an autonomous vehicle includes a lidar sensor system that comprises at least one lidar sensor, and a lidar segmentation system.
- the lidar sensor system outputs lidar data based on sensor signals received from the at least one lidar sensor, where the lidar data is indicative of positions of objects in the driving environment of the autonomous vehicle at one or more times.
- the lidar data comprises a three-dimensional lidar point cloud, wherein each point in the point cloud indicates a three-dimensional position of an object or surface of an object in the driving environment of the autonomous vehicle.
- the lidar sensor system outputs the lidar data to the lidar segmentation system, whereupon the lidar segmentation system generates a segmentation of the lidar data.
- the segmentation indicates which points in the lidar point cloud belong to same objects in the driving environment. Further, the segmentation can indicate a plurality of groups each representative of a different respective object in the driving environment, wherein each group includes one or more points in the lidar point cloud that are representative of the object.
- the lidar segmentation system in connection with generating the segmentation of the lidar data, receives the lidar data from the lidar sensor system and identifies various input features with respect to each of the points represented in the lidar data.
- input features for a point in a lidar point cloud can include distance to a next-closest point in the point cloud, angle formed by the point and adjacent points in the point cloud, mean and variance of position relative to a local neighborhood of points in the point cloud, etc.
- the lidar segmentation system receives the input features for each of the points and provides the input features as input to a deep neural network (DNN).
- DNN deep neural network
- the DNN is trained to output, based upon the input features for a point, a probability that the point is representative of ground, vegetation, or another type of object.
- the lidar segmentation system assigns labels to points in the point cloud based upon the output of the DNN, the labels indicating a type of object that the point is expected to represent.
- the lidar segmentation system can then compute a segmentation of the lidar point cloud based upon the labels assigned to the points in the lidar point cloud.
- the autonomous vehicle is configured to align one or more sets of labeled data with one or more sensor signals based upon the output of the DNN in real time during operation of the autonomous vehicle in the driving environment.
- the autonomous vehicle comprises a data alignment system, and a data storage system that stores labeled data pertaining to the driving environment (e.g., a semantic map of the driving environment, a height map of the driving environment, etc.).
- the data alignment system is configured to align labeled data stored in the data storage system with the lidar data output by the lidar sensor system.
- the data alignment system receives the output of the DNN and identifies a plurality of points that are indicated in the output of the DNN as having a high probability (e.g., greater than 90%, greater than 95%, or greater than 99%) of being representative of ground cover.
- the data alignment system computes an alignment between the labeled data and the points identified as having a high probability of being representative of ground cover.
- a control system of the autonomous vehicle then controls operation of the autonomous vehicle in the driving environment based upon the alignment.
- the control system of the autonomous vehicle can determine that a direction of travel is unobstructed based jointly upon the lidar data and the labeled data, wherein joint consideration of the lidar data and the labeled data is based upon the alignment computed by the data alignment system.
- FIG. 1 illustrates an exemplary autonomous vehicle.
- FIG. 2 is a functional block diagram of an exemplary object recognition system included within an autonomous vehicle.
- FIG. 3A is a top-down view of an exemplary driving environment of an autonomous vehicle.
- FIG. 3B is a side view of the exemplary driving environment depicted in FIG. 3A .
- FIG. 4 is a functional block diagram of an exemplary system that is configured to learn a neural network component included in the object recognition system depicted in FIG. 2 .
- FIG. 5 is a flow diagram illustrating an exemplary methodology for controlling operation of a mechanical system of an autonomous vehicle based upon output of a lidar segmentation system.
- FIG. 6 is a flow diagram illustrating an exemplary methodology for learning a neural network to be included in an object recognition system of an autonomous vehicle.
- FIG. 7 is a flow diagram illustrating an exemplary methodology for controlling operation of a mechanical system of an autonomous vehicle based upon lidar data and a heightmap of a driving environment of the autonomous vehicle.
- FIG. 8 is an exemplary computing system.
- the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or clear from the context, the phrase “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, the phrase “X employs A or B” is satisfied by any of the following instances: X employs A; X employs B; or X employs both A and B.
- the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from the context to be directed to a singular form.
- the terms “component” and “system” are intended to encompass computer-readable data storage that is configured with computer-executable instructions that cause certain functionality to be performed when executed by a processor.
- the computer-executable instructions may include a routine, a function, or the like. It is also to be understood that a component or system may be localized on a single device or distributed across several devices.
- the term “exemplary” is intended to mean serving as an illustration or example of something and is not intended to indicate a preference.
- the autonomous vehicle 100 can navigate about roadways without human conduction based upon sensor signals output by sensor systems of the autonomous vehicle 100 .
- the autonomous vehicle 100 includes a lidar sensor system 101 and a plurality of additional sensor systems 102 - 104 (a second sensor system 102 through an Nth sensor system 104 ).
- the sensor systems 102 - 104 may be of different types and are arranged about the autonomous vehicle 100 .
- the second sensor system 102 may be a radar sensor system and the Nth sensor system 104 may be a camera (image) system.
- Other exemplary sensor systems include GPS sensor systems, sonar sensor systems, infrared sensor systems, and the like.
- the autonomous vehicle 100 further includes several mechanical systems that are used to effectuate appropriate motion of the autonomous vehicle 100 .
- the mechanical systems can include but are not limited to, an engine 106 , a braking system 108 , and a steering system 110 .
- the engine 106 may be an electric engine or a combustion engine.
- the braking system 108 can include an engine brake, brake pads, actuators, a regenerative braking system, and/or any other suitable componentry that is configured to assist in decelerating the autonomous vehicle 100 .
- the steering system 110 includes suitable componentry that is configured to control the direction of movement of the autonomous vehicle 100 .
- the autonomous vehicle 100 additionally comprises a computing system 112 that is in communication with the sensor systems 101 - 104 and is further in communication with the engine 106 , the braking system 108 , and the steering system 110 .
- the computing system 112 includes a processor 114 and memory 116 that includes computer-executable instructions that are executed by the processor 114 .
- the processor 114 can be or include a graphics processing unit (GPU), a plurality of GPUs, a central processing unit (CPU), a plurality of CPUs, an application-specific integrated circuit (ASIC), a microcontroller, a programmable logic controller (PLC), a field programmable gate array (FPGA), or the like.
- the memory 116 comprises an object recognition system 118 that is configured to identify objects (in proximity to the autonomous vehicle 100 ) captured in sensor signals output by the sensor systems 101 - 104 .
- the object recognition system 118 includes a lidar segmentation system 202 , and may further include a plurality of additional object recognition components 204 - 206 (a second objection recognition component 204 through an Mth objection recognition component 206 ), wherein the lidar segmentation system 202 and the objection recognition components 204 - 206 are configured to output data indicative of positions and/or predefined types of objects in a driving environment of the autonomous vehicle 100 .
- Each of the object recognizer components 202 - 206 independently generates output based upon at least one sensor signal.
- the lidar segmentation system 202 outputs data indicative of positions of objects in the driving environment of the autonomous vehicle 100 based upon lidar data output by the lidar sensor system 101 .
- the second object recognizer component 204 outputs data indicative of types of objects based upon images output by a camera
- the Mth object recognizer component 206 outputs data indicative of ranges to objects based upon radar signals, etc.
- the memory 118 additionally includes a control system 120 that is configured to receive output of the object recognition system 118 , and is further configured to control at least one of the mechanical systems of the autonomous vehicle 100 (the engine 106 , the braking system 108 , and/or the steering system 110 ) based upon the output of the object recognition system 118 .
- FIGS. 3A and 3B depict different views of an exemplary driving environment 300 in which the autonomous vehicle 100 operates.
- the autonomous vehicle 100 is depicted in the driving environment 300 , wherein the driving environment includes a first vehicle 302 , two pedestrians 304 , 306 standing to the right of the vehicle 302 , and a bush 308 that stands to the right of the pedestrians 304 , 306 .
- the autonomous vehicle 100 emits a laser beam 310 into the driving environment 300 (e.g., by way of the lidar sensor system 101 ).
- the laser beam 310 is scanned 360° around the vehicle 100 , and impinges upon the various objects 302 - 308 in the environment 300 .
- the lidar sensor system 101 receives reflections of the beam 310 , and based upon such reflections outputs data indicative of positions of a plurality of points of intersection 312 of the beam 310 with the objects 302 - 308 in the driving environment 300 .
- the lidar sensor system 101 outputs data indicating a three-dimensional position of each of the points 312 (e.g., relative to the autonomous vehicle 100 , relative to a point of reference in the driving environment 300 or outside of the driving environment 300 , etc.).
- the lidar sensor system 101 can emit a plurality of laser beams into a driving environment of the autonomous vehicle 100 .
- a side view of the objects 302 - 308 in the driving environment 300 is depicted.
- the lidar sensor system 101 emits a plurality of laser beams into the driving environment of the vehicle 100 , wherein each of the beams has a different angular alignment with respect to a horizontal reference plane of the vehicle 100 .
- each of a plurality of sensors in the lidar sensor system 101 may be aligned with a different respective lidar beam emitted by the lidar sensor system 101 , such that the lidar sensor system 101 outputs data indicative of a plurality of points of intersection for each of the plurality of beams over a period of time.
- the plurality of laser beams scan across the objects 302 - 308 in the driving environment 300 and intersect the objects along scan lines 316 - 320 .
- Each of the points 312 where a laser beam of the lidar sensor system 101 intersects one of the objects 302 - 308 in the driving environment lies along one of the scan lines 316 - 322 .
- each of the plurality of lidar sensors in the lidar sensor system 101 outputs data indicative of a point of intersection of a respective laser beam with various objects in the driving environment 300 along a respective scan line.
- a lidar point cloud output by the lidar sensor system 101 can therefore be considered a rastered image of objects in the driving environment, comprising as many scan lines as lasers are included in the lidar sensor system 101 .
- the lidar sensor system 101 can output data indicative of positions of a larger number of points than are depicted in FIGS. 3A and 3B (e.g., tens of thousands of points, hundreds of thousands of points, millions of points, etc.).
- points of intersection of laser beams with ground cover may be represented in a lidar point cloud output by the lidar sensor system 101 .
- the scan line 322 lies substantially along a ground surface 324
- the points of intersection 312 lying along the scan line 322 represent intersections of a laser beam of the lidar sensor system 101 with the ground surface 324 .
- the points of intersection of laser beams emitted by the lidar sensor system 101 are relatively regularly arranged along surfaces of the object.
- the points of intersection 312 of the beam 310 with the vehicle 302 are regularly arranged along surfaces of the vehicle (e.g., doors, side panels, etc.).
- the points of intersection with the laser beams are irregularly positioned.
- the points of intersection 312 in the driving environment include points of intersection 314 with the bush 308 .
- the points of intersection 314 may not be regularly spaced around apparent boundaries of the bush 308 , for example due to movement of branches and leaves of the bush 308 in wind, irregular positioning of branches and leaves of the bush 308 , etc. These irregularities associated with vegetation in a driving environment of an autonomous vehicle can interfere with conventional lidar point cloud segmentation systems that identify objects in a lidar point cloud.
- a conventional lidar point cloud segmentation system operating in the driving environment 300 may determine that the points of intersection with the bush 308 represent multiple objects rather than a single object.
- a conventional lidar point cloud segmentation system may determine that points of intersection with the bush 308 represent the same object as points of intersection representative of the pedestrian 306
- the lidar segmentation system 202 is configured to identify points in a lidar point cloud that are likely to be representative of vegetation and/or ground cover, and to perform a segmentation of the lidar point cloud that takes appropriate consideration of such objects.
- the lidar segmentation system 202 can be configured to identify adjacent points that are likely to be representative of vegetation, and to consider these points to be representative of a same object (e.g., a single bush).
- the lidar segmentation system 202 can be configured to identify points that are likely to be representative of ground cover, and to separate these points from adjacent points in the lidar point cloud that are not likely to be representative of ground cover.
- the lidar segmentation system 202 can be prevented from erroneously determining that non-ground objects are part of ground cover in the driving environment of the autonomous vehicle 100 .
- the lidar segmentation system 202 further comprises a lidar preprocessing component 208 , a neural network component 210 , and a lidar postprocessing component 212 .
- the lidar segmentation system 202 receives lidar data from the lidar sensor system 101 , wherein the lidar data is indicative of positions of surfaces of a plurality of objects in a driving environment of the autonomous vehicle 100 .
- the lidar preprocessing component 208 performs various processing operations over the lidar data and outputs a plurality of input features to the neural network component 210 .
- the neural network component 210 is configured to output a probability that a point is representative of at least one of ground cover or vegetation based upon the input features associated with the point.
- the lidar postprocessing component 212 then performs a segmentation of the lidar point cloud based upon the probability output by the neural network component 210 .
- the lidar segmentation system 202 receives a lidar point cloud from the lidar sensor system 101 , wherein the point cloud is indicative of positions of points on surfaces of the objects 302 - 308 (and other objects that may be present) in the driving environment 300 of the vehicle 100 (e.g., as described above with respect to FIGS. 3A and 3B ).
- the lidar preprocessing component 208 performs an analysis over the points in the point cloud to identify various features relative to each of the points in the point cloud.
- the lidar preprocessing component 208 parameterizes the point cloud according to a training scheme used to learn the neural network component 210 .
- the lidar preprocessing component 208 can compute, for each of the points in the point cloud, a distance to a next-closest point in the point cloud, a distance from the point to the autonomous vehicle 100 , a number of points within a threshold distance of the point, etc., where such features are provided as input to a DNN during training of the DNN (e.g., as described below with respect to FIG. 4 ).
- the preprocessor component 208 then provides these features as input features to the neural network component 210 for each of the points in the point cloud.
- the neural network component 210 comprises a DNN that has a plurality of layers including an input layer, a plurality of hidden layers (e.g., greater than 5 layers, greater than 10 layers, greater than 50 layers, etc.), and an output layer. Responsive to receipt of input features for a point in the lidar point cloud at the input layer of the DNN, the input features are propagated through the hidden layers of the DNN.
- the DNN is configured to output, for each of the points in the point cloud, one or more probabilities, each probability indicating a likelihood that the point is representative of an object of a certain type.
- the DNN can be configured such that the output layer is a softmax layer that outputs a first probability that the input point (e.g., the point represented by the input features provided at the input layer of the DNN) represents a first type of object, a second probability that the input point represents a second type of object, etc. through an nth probability that the input points represents an nth type of object, where then probabilities sum to unity.
- the output layer outputs a first probability that the input point is representative of vegetation, a second probability that the input point is representative of ground cover, and a third probability that the input point is representative of any other type of object (e.g., cars, trucks, pedestrians, bicycles, etc.).
- the DNN is learned based upon labeled training data (e.g., labeled lidar point cloud data) such that the output of the DNN indicates a high probability (e.g., greater than 80%, greater than 90%, greater than 95%, etc.) that the input point is ground cover when the input point is actually representative of ground cover, a high probability that the input point is vegetation when the input point is actually representative of vegetation, etc.
- labeled training data e.g., labeled lidar point cloud data
- the lidar post processing component 212 receives the output of the DNN (e.g., the n probabilities indicated by the output layer of the DNN) from the neural network component 210 and outputs, based upon the output of the DNN, a segmentation of the lidar point cloud (e.g., as received from the lidar sensor system 101 ) that indicates which points in the point cloud are representative of same objects in the driving environment of the autonomous vehicle 100 .
- the lidar post processing component 212 performs a segmentation of the lidar point cloud by excluding points from consideration that are indicated by the output of the DNN as being likely to be representative of vegetation and/or ground.
- the lidar post processing component 212 performs a segmentation of only those points in the point cloud that are indicated by the output of the DNN as not being likely to represent ground or vegetation.
- the lidar post processing component 212 can reduce errors associated with poor performance of the segmentation algorithm in identifying ground and vegetation features. For instance, excluding points representative of ground cover from consideration by a segmentation algorithm can reduce errors caused by an inability of the algorithm to distinguish between the ground and objects in contact with the ground (e.g., pedestrians' feet, car tires, or other object in contact with the ground).
- excluding points representative of vegetation from consideration by a segmentation algorithm can reduce errors caused by an inability of the algorithm to identify that scattered points in the point cloud are representative of a same vegetative object, such as a bush, a leafy tree branch, etc.
- the lidar post processing component 212 identifies a first group of points in the point cloud that are indicated by the output of the DNN as being likely to represent ground, and/or a second group of points in the point cloud that are indicated by the output of the DNN as being likely to represent vegetation.
- the lidar post processing component 212 can identify points that are likely to be representative of ground and/or vegetation based upon the output of the DNN and a probability threshold. For instance, the lidar post processing component 212 can label a point as being representative of ground based upon the output of the DNN indicating that the point has a probability of being ground that is above the probability threshold.
- the probability threshold may be 75%, 90%, 95%, or other desirable threshold. Responsive to identifying a point as being likely to be representative of ground or vegetation (e.g., based upon the probability threshold), the lidar post processing component 212 assigns a label to the point that indicates ground or vegetation, accordingly.
- the lidar post processing component 212 Upon determining which points in the lidar point cloud are likely to be representative of ground and/or vegetation, the lidar post processing component 212 generates a segmentation of points in the point cloud that are not labeled as ground or vegetation.
- the segmentation comprises data indicative of which points are representative of which objects in the driving environment of the autonomous vehicle 100 .
- generating the segmentation comprises assigning respective labels to each of the points in the point cloud, wherein each of the labels indicates a group to which its point belongs, each group being representative of a different object in the driving environment.
- the lidar post processing component 212 can generate the segmentation according to any suitable segmentation algorithm.
- the lidar post processing component 212 generates the segmentation based upon output of a neural network that is trained to output data indicative of whether points in a lidar point cloud are representative of same objects.
- the segmentation is generated based upon an algorithm that considers distance between points in the point cloud, relative angles formed by points in the point cloud, convex hulls of subsets of points in the point cloud, etc. Other approaches to generating the segmentation of the points not labeled vegetation or ground are also contemplated.
- the lidar post processing component 212 outputs the segmentation to the object recognition system 118 .
- the object recognition system 118 outputs data indicative of one or more objects in the driving environment to the control system 120 based upon the segmentation generated by the lidar segmentation system 202 .
- the object recognition system 118 outputs data indicative of positions and/or types of objects in the driving environment.
- the object recognition system 118 outputs data indicative of positions and/or types of objects in the driving environment of the autonomous vehicle 100 based upon the segmentation output by the lidar segmentation system 202 and output of one or more of the additional object recognizer components 204 - 206 .
- control system 120 of the vehicle 100 can output control signals to at least one of the engine 106 , the braking system 108 , or the steering system 110 based upon the output of the object recognition system 118 , such that the autonomous vehicle is controlled based at least in part upon the segmentation of the lidar point cloud generated by the lidar segmentation system 202 .
- route information e.g., locations of streets, directions of traffic on streets, road closures, etc.
- locations of interest e.g., restaurants, fueling/charging stations, entertainment venues, etc.
- the computing system 112 of the autonomous vehicle 100 further includes a data store 122 that stores labeled data 124 .
- the labeled data 124 includes data pertaining to a potential operating environment of the autonomous vehicle 100 (e.g., an area that includes the surroundings of the vehicle 100 that make up the driving environment of the vehicle 100 ).
- the labeled data 124 is generated independent of the sensor systems 101 - 104 of the autonomous vehicle 100 and is labeled based upon ground truth data pertaining to the potential operating environment (e.g., based upon human input).
- the labeled data 124 includes a semantic map 126 of a potential operating environment of the autonomous vehicle 100 (e.g., an area that includes the surroundings of the vehicle 100 that make up the driving environment of the vehicle 100 ) and a heightmap 128 of the potential operating environment.
- the heightmap 128 comprises data indicating a height for each of a plurality of points in a two-dimensional space that comprises an overhead view of the potential operating environment of the autonomous vehicle 100 .
- the autonomous vehicle 100 can identify a fastest available route to a destination based upon the semantic map 126 .
- the autonomous vehicle 100 identify that sensor data is representative of a same object in the driving environment that is also represented in the labeled data, and perform one or more functions based upon the sensor data and the labeled data 124 being representative of the same object.
- the sensor data and the labeled data 124 are aligned to ensure that sensor signals are accurately matched to corresponding information in the labeled data 124 .
- the lidar data output by the lidar sensor system 101 is aligned with the height map 128 .
- the lidar data and the height map 128 are aligned such that the computing system 112 can readily determine that first data in the height map 128 is representative of a first location of a first point in the lidar data.
- the computing system 112 can compute a coordinate transform from a coordinate system of the lidar point cloud to a coordinate system of the height map 128 (and vice-versa).
- the computing system 112 can match coordinates of a point in the point cloud to coordinates in the height map 128 to determine a ground height at a position of the point.
- Alignment between sensor data and the labeled data 124 can be initially achieved during calibration of systems of the autonomous vehicle 100 prior to operation of the autonomous vehicle 100 in the driving environment.
- the lidar sensor system 101 in a calibration environment the lidar sensor system 101 generates a lidar point cloud, wherein the lidar point cloud is representative of a known geometry of the calibration environment.
- the computing system 112 of the vehicle 100 can compute an alignment between labeled data 124 and the lidar point cloud based upon the geometry of the calibration environment being known.
- the autonomous vehicle 100 is subject to misalignment of the sensor data and the labeled data 124 over time as the autonomous vehicle 100 operates in a driving environment.
- the autonomous vehicle in order to maintain alignment of sensor data and labeled data, the autonomous vehicle must be periodically recalibrated in the environment having the known geometry, requiring downtime of the autonomous vehicle and removal of the autonomous vehicle from the driving environment.
- the autonomous vehicle 100 is configured to align sensor data from one or more of the sensor systems 101 - 104 with labeled data 124 in real-time during operation of the vehicle 100 in a driving environment.
- the lidar segmentation system 202 further comprises a data alignment system 214 that computes an alignment between lidar data output by the lidar sensor system 101 and the labeled data 124 based upon output of the neural network component 210 .
- the data alignment system 214 receives output of the DNN from the neural network component 210 .
- the data alignment system 214 identifies points in the point cloud for which the output of the DNN indicates a high probability (e.g., greater than 90% probability, greater than 95% probability, greater than 99% probability) that the point is representative of ground.
- the data alignment system 214 then computes an alignment between the coordinate system of the lidar point cloud and the coordinate system of the heightmap 128 based upon the identified points and the heightmap 128 .
- the alignment computed by the data alignment system 214 based upon the identified points is more accurate than an alignment computed based upon a group of points in the lidar point cloud that also includes points that are not representative of ground.
- the lidar post processing component 212 can generate a segmentation of the lidar data based upon the labeled data 124 or update an existing segmentation of the lidar data based upon the labeled data 124 .
- the lidar post processing component 212 receives a coordinate transform from the data alignment system 214 .
- the coordinate transform indicates a transformation from a coordinate system of the lidar data to a coordinate system of at least one dataset in the labeled data 124 .
- the lidar post processing component 212 employs the coordinate transform to update a segmentation based upon characteristics of points in the lidar point cloud that are indicated in the labeled data 124 .
- the lidar post processing component 212 can identify a point in the lidar point cloud for which the output of the DNN indicates a moderate probability (e.g., 40%-80%, 50%-70%, or 55%-65%) of the point being representative of ground cover. For instance, referring again briefly to FIG. 3B , a point 326 near the bottom of a tire of the vehicle 302 in the driving environment 300 can be indicated in the output of the DNN as having a probability of 55% of being representative of ground cover (e.g., due in part to its proximity to the ground surface 324 ). The lidar post processing component 212 can compute, based upon the coordinate transform, coordinates of the identified point in the heightmap 128 .
- a moderate probability e.g. 40%-80%, 50%-70%, or 55%-65%
- the lidar post processing component 212 can determine whether the identified point is or is not representative of ground. For example, if the height of the point as indicated in the lidar point cloud is significantly greater than the height indicated in the heightmap 128 (e.g., 20 centimeters greater, 10 centimeters greater, 5 centimeters greater), the lidar post processing component 212 can assign a label to the point indicating that the point is representative of an object other than ground.
- the lidar post processing component 212 can assign a label to the point indicating that the point is representative of ground cover. It is to be understood that there may be uncertainty associated with heights indicated in the heightmap data 128 .
- the heightmap 128 includes data indicative of an uncertainty threshold (e.g., a range of potential heights) for one or more of the heights indicated in the heightmap 128 .
- the lidar post processing component 212 can assign a label to a point in the lidar point cloud based upon the uncertainty threshold, the label indicating that the point is representative of ground cover. For instance, if a height of a point as indicated by the lidar point cloud is within a range of potential heights indicated in the heightmap 128 , the lidar post processing component 212 assigns a label to the point indicating that the point is representative of ground.
- the lidar post processing component 212 can assign the labels to points in the lidar point cloud based upon the alignment, as described above, prior to performing a segmentation of the lidar point cloud.
- the data alignment system 214 receives the output of the DNN from the neural network component 210 , and computes the alignment as described above.
- the lidar post processing component 212 based upon the alignment and the heightmap 128 , evaluates points in the lidar point cloud that potentially represent ground cover, and assigns labels to those points with heights consistent with the heightmap 128 .
- the lidar post processing component 212 can then generate the segmentation of the lidar point cloud based upon the labeled points being representative of ground.
- the lidar post processing component 212 can evaluate points in the lidar point cloud that are indicated by the output of the DNN as having a probability of representing ground cover that is above a threshold amount (e.g., greater than 30%, greater than 40%, greater than 50%) against heights indicated in the heightmap 128 . To each point indicated by the lidar point cloud as having a height within a range of potential heights indicated in the heightmap 128 , the lidar post processing component 212 assigns a label indicating that the point is representative of ground. The lidar post processing component 212 can then perform a segmentation of the remaining unlabeled points in the lidar point cloud based upon any suitable segmentation algorithm. The control system 120 can subsequently output control signals to at least one of the engine 106 , the braking system 108 , or the steering system 110 to control operation of the autonomous vehicle 100 based upon the segmentation of the lidar point cloud.
- a threshold amount e.g., greater than 30%, greater than 40%, greater than 50%
- the computing system 400 includes a processor 402 and memory 404 , wherein the memory 404 includes instructions that are executed by the processor 402 .
- the computing system 400 additionally includes a data store 406 that comprises labeled training data 408 .
- the memory 404 includes a neural network learner 410 that learns the neural network component 210 .
- the neural network learner 410 receives the labeled training data 408 , wherein the labeled training data 408 comprises lidar point cloud data (e.g., parameterized as described above with respect to input features of the DNN), wherein the lidar point cloud data further includes ground truth data indicating which points are representative of vegetation, and which points are representative of ground in the lidar point cloud data.
- the neural network learner 410 learns the neural network component 210 such that the neural network component 210 outputs a high ground probability for points in the point cloud that are representative of ground, a high vegetation probability for points representative of vegetation, and a low probability of vegetation and ground for points representative of neither vegetation nor ground.
- the neural network component 210 can be validated based upon a test training data set, which is also labeled, to ensure that the neural network component 210 produces outputs with acceptable characteristics (e.g., such that the neural network component 210 outputs a high vegetation probability for points representative of vegetation and a low vegetation probability for points that are not representative of vegetation).
- FIGS. 5-7 illustrate exemplary methodologies relating to controlling an autonomous vehicle based upon data output by a lidar data segmentation system. While the methodologies are shown and described as being a series of acts that are performed in a sequence, it is to be understood and appreciated that the methodologies are not limited by the order of the sequence. For example, some acts can occur in a different order than what is described herein. In addition, an act can occur concurrently with another act. Further, in some instances, not all acts may be required to implement a methodology described herein.
- the acts described herein may be computer-executable instructions that can be implemented by one or more processors and/or stored on a computer-readable medium or media.
- the computer-executable instructions can include a routine, a sub-routine, programs, a thread of execution, and/or the like.
- results of acts of the methodologies can be stored in a computer-readable medium, displayed on a display device, and/or the like.
- the methodology 500 starts at 502 , and at 504 lidar data is received, wherein the lidar data comprises a plurality of points representative of positions of objects in a driving environment of an autonomous vehicle.
- a label is assigned to a first point in the points based upon output of a DNN.
- the DNN responsive to receiving input features pertaining to the first point, the DNN outputs respective probabilities that the first point is representative of ground cover, vegetation, or another type of object.
- the label is assigned to the first point responsive to determining that a probability of the first point being either vegetation or ground as indicated in the output of the DNN is greater than a pre-defined threshold value.
- the label assigned to the first point indicates that the first point is representative of ground cover or vegetation in the driving environment.
- a segmentation of the lidar data is generated based upon the first point being excluded from the segmentation. By generating the segmentation based upon excluding the first point from the segmentation, errors associated with poor performance of a segmentation algorithm relative to ground and vegetation features are avoided.
- at least one of an engine, a braking system, or a steering system of the autonomous vehicle is controlled based upon the segmentation generated at 508 .
- an object recognition system of the autonomous vehicle can generate a prediction of future behavior of one or more objects in the driving environment based upon the segmentation, and can control steering of the autonomous vehicle based upon such prediction.
- the methodology 500 completes at 512 .
- FIG. 6 an exemplary methodology 600 that facilitates learning a DNN is illustrated, wherein the DNN is configured for use in a lidar segmentation system of an autonomous vehicle.
- the methodology 600 starts at 602 , and at 604 labeled training data is received, wherein the training data comprises labeled lidar point cloud data.
- a DNN is learned based upon the training data.
- the DNN is configured such that responsive to receiving a point in a lidar point cloud as input (e.g., when the DNN receives a vector of input features representative of the point at an input layer of the DNN), the DNN outputs a plurality of probabilities, each probability indicative of a likelihood that the point represents a different respective type of object.
- the DNN responsive to receiving a point in a lidar point cloud as input, the DNN outputs a first probability that the point is representative of vegetation, a second probability that the point is representative of ground, and a third probability that the point is representative of some other type of object, where the three probabilities sum to unity.
- the DNN is learned at 606 such that the output of the DNN indicates a high probability that the point is representative of vegetation when the input point is a point representative of vegetation, the output of the DNN indicates a high probability that the point is representative of ground when the input point is a point representative of ground, and the output of the DNN indicates a high probability that the point is representative of some other type of object when the input point is a point representative of neither ground nor vegetation.
- the methodology 600 completes at 608 .
- the methodology 700 begins at 702 and at 704 , lidar data is received, wherein the lidar data comprises a plurality of points representative of positions of objects in a driving environment of an autonomous vehicle.
- an alignment is computed between the lidar data and a heightmap based upon a first point in the points, wherein the first point is selected based upon output of a DNN indicating that the first point is representative of ground.
- At 708 at least one of an engine, a braking system, or a steering system of the autonomous vehicle is controlled based upon the lidar data and the heightmap, wherein the alignment computed at 706 facilitates joint consideration of the lidar data and the heightmap.
- a segmentation of the lidar data can be generated based at least in part upon the heightmap being aligned with the lidar data.
- the methodology 700 ends at 710 .
- the computing device 800 may be or include the computing system 112 or the computing system 400 .
- the computing device 800 includes at least one processor 802 that executes instructions that are stored in a memory 804 .
- the instructions may be, for instance, instructions for implementing functionality described as being carried out by one or more modules and systems discussed above or instructions for implementing one or more of the methods described above.
- the processor 802 may be a GPU, a plurality of GPUs, a CPU, a plurality of CPUs, a multi-core processor, etc.
- the processor 802 may access the memory 804 by way of a system bus 806 .
- the memory 804 may also store point cloud data, vectors of output features for points in a point cloud, segmentation data, training data, etc.
- the computing device 800 additionally includes a data store 808 that is accessible by the processor 802 by way of the system bus 806 .
- the data store 808 may include executable instructions, sensor data, output feature vector data, training data, segmentation data, etc.
- the computing device 800 also includes an input interface 810 that allows external devices to communicate with the computing device 800 .
- the input interface 810 may be used to receive instructions from an external computer device, etc.
- the computing device 800 also includes an output interface 812 that interfaces the computing device 800 with one or more external devices.
- the computing device 800 may transmit control signals to the engine 106 , the braking system 108 , and/or the steering system 110 by way of the output interface 812 .
- the computing device 800 may be a distributed system. Thus, for instance, several devices may be in communication by way of a network connection and may collectively perform tasks described as being performed by the computing device 800 .
- Computer-readable media includes computer-readable storage media.
- a computer-readable storage media can be any available storage media that can be accessed by a computer.
- such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer.
- Disk and disc include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc (BD), where disks usually reproduce data magnetically and discs usually reproduce data optically with lasers. Further, a propagated signal is not included within the scope of computer-readable storage media.
- Computer-readable media also includes communication media including any medium that facilitates transfer of a computer program from one place to another. A connection, for instance, can be a communication medium.
- the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave
- coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave
- the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave
- the functionally described herein can be performed, at least in part, by one or more hardware logic components.
- illustrative types of hardware logic components include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.
Abstract
Description
- This application is a continuation of U.S. patent application Ser. No. 16/054,088, filed on Aug. 3, 2018, and entitled “AUTONOMOUS VEHICLE CONTROLLED BASED UPON A LIDAR DATA SEGMENTATION SYSTEM,” the entirety of which is incorporated herein by reference.
- An autonomous vehicle is a motorized vehicle that can operate without human conduction. An exemplary autonomous vehicle includes a plurality of sensor systems, such as, but not limited to, a lidar sensor system, a camera sensor system, and a radar sensor system, amongst others, wherein the autonomous vehicle operates based upon sensor signals output by the sensor systems.
- Conventionally, autonomous vehicles are configured to identify locations of different objects in a driving environment based upon different sensor signals. For example, a radar system can identify a range from the autonomous vehicle to another vehicle in the driving environment. In another example, an object recognition system may be configured to receive images output by a camera and to identify relative positions of objects captured in the images.
- In some situations, however, it may be difficult to determine what objects are at which locations based upon the sensor signals output by the sensor systems of the autonomous vehicle. For example, in a lidar point cloud representative of the driving environment, it may be difficult to determine by way of conventional algorithms what points are representative of same objects and what points are representative of different objects in the driving environment. Conventionally, algorithms based upon human-generated rules have been used to perform segmentation of a lidar point cloud to identify objects and their locations in the driving environment. For example, an algorithm may identify two points in a lidar point cloud as belonging to a same object based upon the points being within 50 centimeters of one another. This conventional approach may not be robust to variations in the driving environment, and can result in possible errors with respect to object identification by the autonomous vehicle. For instance, various conventional algorithms for performing segmentation of a lidar point cloud may perform poorly when certain types of objects are present in a driving environment (e.g., groups of closely-spaced pedestrians, vegetation, non-convex objects, etc.)
- The following is a brief summary of subject matter that is described in greater detail herein. This summary is not intended to be limiting as to the scope of the claims.
- Described herein are various technologies pertaining to controlling operation of an autonomous vehicle. With more specificity, described herein are various technologies pertaining to segmentation of a lidar point cloud, wherein the lidar point cloud is representative of positions of objects in the driving environment of the autonomous vehicle at one or more points in time. With still more specificity, a lidar data segmentation system is described herein, wherein the lidar data segmentation system comprises a neural network that is trained to output data indicative of a probability that a point in a lidar point cloud is representative of at least one of ground (e.g., a road surface, a sidewalk surface, or other ground surface) or vegetation. The lidar data segmentation system can assign a label to the point based upon the probability indicated by the neural network output, the label indicating a classification of a type of object that the point is expected to represent. The lidar segmentation system then performs a segmentation of the lidar point cloud based in part upon the label assigned to the point.
- In an example, an autonomous vehicle includes a lidar sensor system that comprises at least one lidar sensor, and a lidar segmentation system. The lidar sensor system outputs lidar data based on sensor signals received from the at least one lidar sensor, where the lidar data is indicative of positions of objects in the driving environment of the autonomous vehicle at one or more times. In an exemplary embodiment, the lidar data comprises a three-dimensional lidar point cloud, wherein each point in the point cloud indicates a three-dimensional position of an object or surface of an object in the driving environment of the autonomous vehicle. The lidar sensor system outputs the lidar data to the lidar segmentation system, whereupon the lidar segmentation system generates a segmentation of the lidar data. By way of example, the segmentation indicates which points in the lidar point cloud belong to same objects in the driving environment. Further, the segmentation can indicate a plurality of groups each representative of a different respective object in the driving environment, wherein each group includes one or more points in the lidar point cloud that are representative of the object.
- The lidar segmentation system, in connection with generating the segmentation of the lidar data, receives the lidar data from the lidar sensor system and identifies various input features with respect to each of the points represented in the lidar data. By way of example, and not limitation, input features for a point in a lidar point cloud can include distance to a next-closest point in the point cloud, angle formed by the point and adjacent points in the point cloud, mean and variance of position relative to a local neighborhood of points in the point cloud, etc. The lidar segmentation system receives the input features for each of the points and provides the input features as input to a deep neural network (DNN). The DNN is trained to output, based upon the input features for a point, a probability that the point is representative of ground, vegetation, or another type of object. The lidar segmentation system assigns labels to points in the point cloud based upon the output of the DNN, the labels indicating a type of object that the point is expected to represent. The lidar segmentation system can then compute a segmentation of the lidar point cloud based upon the labels assigned to the points in the lidar point cloud.
- In further embodiments, the autonomous vehicle is configured to align one or more sets of labeled data with one or more sensor signals based upon the output of the DNN in real time during operation of the autonomous vehicle in the driving environment. In an example, the autonomous vehicle comprises a data alignment system, and a data storage system that stores labeled data pertaining to the driving environment (e.g., a semantic map of the driving environment, a height map of the driving environment, etc.). The data alignment system is configured to align labeled data stored in the data storage system with the lidar data output by the lidar sensor system. The data alignment system receives the output of the DNN and identifies a plurality of points that are indicated in the output of the DNN as having a high probability (e.g., greater than 90%, greater than 95%, or greater than 99%) of being representative of ground cover. The data alignment system computes an alignment between the labeled data and the points identified as having a high probability of being representative of ground cover. A control system of the autonomous vehicle then controls operation of the autonomous vehicle in the driving environment based upon the alignment. By way of example, the control system of the autonomous vehicle can determine that a direction of travel is unobstructed based jointly upon the lidar data and the labeled data, wherein joint consideration of the lidar data and the labeled data is based upon the alignment computed by the data alignment system.
- The above summary presents a simplified summary in order to provide a basic understanding of some aspects of the systems and/or methods discussed herein. This summary is not an extensive overview of the systems and/or methods discussed herein. It is not intended to identify key/critical elements or to delineate the scope of such systems and/or methods. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is presented later.
-
FIG. 1 illustrates an exemplary autonomous vehicle. -
FIG. 2 is a functional block diagram of an exemplary object recognition system included within an autonomous vehicle. -
FIG. 3A is a top-down view of an exemplary driving environment of an autonomous vehicle. -
FIG. 3B is a side view of the exemplary driving environment depicted inFIG. 3A . -
FIG. 4 is a functional block diagram of an exemplary system that is configured to learn a neural network component included in the object recognition system depicted inFIG. 2 . -
FIG. 5 is a flow diagram illustrating an exemplary methodology for controlling operation of a mechanical system of an autonomous vehicle based upon output of a lidar segmentation system. -
FIG. 6 is a flow diagram illustrating an exemplary methodology for learning a neural network to be included in an object recognition system of an autonomous vehicle. -
FIG. 7 is a flow diagram illustrating an exemplary methodology for controlling operation of a mechanical system of an autonomous vehicle based upon lidar data and a heightmap of a driving environment of the autonomous vehicle. -
FIG. 8 is an exemplary computing system. - Various technologies pertaining to controlling operation of an autonomous vehicle based upon lidar segmentation data generated through use of a DNN are now described with reference to the drawings, wherein like reference numerals are used to refer to like elements throughout. In the following description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of one or more aspects. It may be evident, however, that such aspect(s) may be practiced without these specific details. In other instances, well-known structures and devices are shown in block diagram form in order to facilitate describing one or more aspects. Further, it is to be understood that functionality that is described as being carried out by certain system components may be performed by multiple components. Similarly, for instance, a component may be configured to perform functionality that is described as being carried out by multiple components.
- Moreover, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or.” That is, unless specified otherwise, or clear from the context, the phrase “X employs A or B” is intended to mean any of the natural inclusive permutations. That is, the phrase “X employs A or B” is satisfied by any of the following instances: X employs A; X employs B; or X employs both A and B. In addition, the articles “a” and “an” as used in this application and the appended claims should generally be construed to mean “one or more” unless specified otherwise or clear from the context to be directed to a singular form.
- Further, as used herein, the terms “component” and “system” are intended to encompass computer-readable data storage that is configured with computer-executable instructions that cause certain functionality to be performed when executed by a processor. The computer-executable instructions may include a routine, a function, or the like. It is also to be understood that a component or system may be localized on a single device or distributed across several devices. Further, as used herein, the term “exemplary” is intended to mean serving as an illustration or example of something and is not intended to indicate a preference.
- With reference now to
FIG. 1 , an exemplaryautonomous vehicle 100 is illustrated. Theautonomous vehicle 100 can navigate about roadways without human conduction based upon sensor signals output by sensor systems of theautonomous vehicle 100. Theautonomous vehicle 100 includes alidar sensor system 101 and a plurality of additional sensor systems 102-104 (asecond sensor system 102 through an Nth sensor system 104). The sensor systems 102-104 may be of different types and are arranged about theautonomous vehicle 100. For example, thesecond sensor system 102 may be a radar sensor system and theNth sensor system 104 may be a camera (image) system. Other exemplary sensor systems include GPS sensor systems, sonar sensor systems, infrared sensor systems, and the like. - The
autonomous vehicle 100 further includes several mechanical systems that are used to effectuate appropriate motion of theautonomous vehicle 100. For instance, the mechanical systems can include but are not limited to, anengine 106, abraking system 108, and asteering system 110. Theengine 106 may be an electric engine or a combustion engine. Thebraking system 108 can include an engine brake, brake pads, actuators, a regenerative braking system, and/or any other suitable componentry that is configured to assist in decelerating theautonomous vehicle 100. Thesteering system 110 includes suitable componentry that is configured to control the direction of movement of theautonomous vehicle 100. - The
autonomous vehicle 100 additionally comprises acomputing system 112 that is in communication with the sensor systems 101-104 and is further in communication with theengine 106, thebraking system 108, and thesteering system 110. Thecomputing system 112 includes aprocessor 114 andmemory 116 that includes computer-executable instructions that are executed by theprocessor 114. In an example, theprocessor 114 can be or include a graphics processing unit (GPU), a plurality of GPUs, a central processing unit (CPU), a plurality of CPUs, an application-specific integrated circuit (ASIC), a microcontroller, a programmable logic controller (PLC), a field programmable gate array (FPGA), or the like. - The
memory 116 comprises anobject recognition system 118 that is configured to identify objects (in proximity to the autonomous vehicle 100) captured in sensor signals output by the sensor systems 101-104. As will be described in greater detail below (and referring briefly toFIG. 2 ), theobject recognition system 118 includes alidar segmentation system 202, and may further include a plurality of additional object recognition components 204-206 (a secondobjection recognition component 204 through an Mth objection recognition component 206), wherein thelidar segmentation system 202 and the objection recognition components 204-206 are configured to output data indicative of positions and/or predefined types of objects in a driving environment of theautonomous vehicle 100. These predefined types of objects can include, but are not limited to, pedestrian, bike, car, truck, bus, ground cover (e.g., paved roadways, dirt, grass) etc. Each of the object recognizer components 202-206 independently generates output based upon at least one sensor signal. For instance, thelidar segmentation system 202 outputs data indicative of positions of objects in the driving environment of theautonomous vehicle 100 based upon lidar data output by thelidar sensor system 101. In further examples, the secondobject recognizer component 204 outputs data indicative of types of objects based upon images output by a camera, the Mthobject recognizer component 206 outputs data indicative of ranges to objects based upon radar signals, etc. - The
memory 118 additionally includes acontrol system 120 that is configured to receive output of theobject recognition system 118, and is further configured to control at least one of the mechanical systems of the autonomous vehicle 100 (theengine 106, thebraking system 108, and/or the steering system 110) based upon the output of theobject recognition system 118. - Exemplary operation of the
autonomous vehicle 100 is now set forth.FIGS. 3A and 3B depict different views of anexemplary driving environment 300 in which theautonomous vehicle 100 operates. With reference now toFIG. 3A , theautonomous vehicle 100 is depicted in the drivingenvironment 300, wherein the driving environment includes afirst vehicle 302, twopedestrians vehicle 302, and abush 308 that stands to the right of thepedestrians autonomous vehicle 100 emits alaser beam 310 into the driving environment 300 (e.g., by way of the lidar sensor system 101). Thelaser beam 310 is scanned 360° around thevehicle 100, and impinges upon the various objects 302-308 in theenvironment 300. Thelidar sensor system 101 receives reflections of thebeam 310, and based upon such reflections outputs data indicative of positions of a plurality of points ofintersection 312 of thebeam 310 with the objects 302-308 in the drivingenvironment 300. By way of example, thelidar sensor system 101 outputs data indicating a three-dimensional position of each of the points 312 (e.g., relative to theautonomous vehicle 100, relative to a point of reference in the drivingenvironment 300 or outside of the drivingenvironment 300, etc.). - It is to be understood that while a
single laser beam 310 and a plurality of points ofintersection 312 of thebeam 310 with objects 302-308 are depicted inFIG. 3A , thelidar sensor system 101 can emit a plurality of laser beams into a driving environment of theautonomous vehicle 100. For instance, and referring now to FIG. 3B, a side view of the objects 302-308 in the drivingenvironment 300 is depicted. In exemplary embodiments, thelidar sensor system 101 emits a plurality of laser beams into the driving environment of thevehicle 100, wherein each of the beams has a different angular alignment with respect to a horizontal reference plane of thevehicle 100. Further, each of a plurality of sensors in thelidar sensor system 101 may be aligned with a different respective lidar beam emitted by thelidar sensor system 101, such that thelidar sensor system 101 outputs data indicative of a plurality of points of intersection for each of the plurality of beams over a period of time. As depicted inFIG. 3B , the plurality of laser beams scan across the objects 302-308 in the drivingenvironment 300 and intersect the objects along scan lines 316-320. Each of thepoints 312 where a laser beam of thelidar sensor system 101 intersects one of the objects 302-308 in the driving environment lies along one of the scan lines 316-322. Hence, each of the plurality of lidar sensors in thelidar sensor system 101 outputs data indicative of a point of intersection of a respective laser beam with various objects in the drivingenvironment 300 along a respective scan line. A lidar point cloud output by thelidar sensor system 101 can therefore be considered a rastered image of objects in the driving environment, comprising as many scan lines as lasers are included in thelidar sensor system 101. It is to be appreciated that over the period of a 360° scan of a driving environment of theautonomous vehicle 100, thelidar sensor system 101 can output data indicative of positions of a larger number of points than are depicted inFIGS. 3A and 3B (e.g., tens of thousands of points, hundreds of thousands of points, millions of points, etc.). Furthermore, it is to be appreciated that points of intersection of laser beams with ground cover may be represented in a lidar point cloud output by thelidar sensor system 101. For example, thescan line 322 lies substantially along aground surface 324, and the points ofintersection 312 lying along thescan line 322 represent intersections of a laser beam of thelidar sensor system 101 with theground surface 324. - For many objects in a driving environment of an autonomous vehicle, the points of intersection of laser beams emitted by the
lidar sensor system 101 are relatively regularly arranged along surfaces of the object. For instance, and referring again toFIG. 3A , the points ofintersection 312 of thebeam 310 with thevehicle 302 are regularly arranged along surfaces of the vehicle (e.g., doors, side panels, etc.). However, for some objects, the points of intersection with the laser beams are irregularly positioned. For instance, the points ofintersection 312 in the driving environment include points ofintersection 314 with thebush 308. The points ofintersection 314 may not be regularly spaced around apparent boundaries of thebush 308, for example due to movement of branches and leaves of thebush 308 in wind, irregular positioning of branches and leaves of thebush 308, etc. These irregularities associated with vegetation in a driving environment of an autonomous vehicle can interfere with conventional lidar point cloud segmentation systems that identify objects in a lidar point cloud. By way of example, a conventional lidar point cloud segmentation system operating in the drivingenvironment 300 may determine that the points of intersection with thebush 308 represent multiple objects rather than a single object. In another example, a conventional lidar point cloud segmentation system may determine that points of intersection with thebush 308 represent the same object as points of intersection representative of thepedestrian 306 - Referring again to
FIG. 2 , various details pertaining to thelidar segmentation system 202 are now described. Thelidar segmentation system 202 is configured to identify points in a lidar point cloud that are likely to be representative of vegetation and/or ground cover, and to perform a segmentation of the lidar point cloud that takes appropriate consideration of such objects. For example, thelidar segmentation system 202 can be configured to identify adjacent points that are likely to be representative of vegetation, and to consider these points to be representative of a same object (e.g., a single bush). In another example, thelidar segmentation system 202 can be configured to identify points that are likely to be representative of ground cover, and to separate these points from adjacent points in the lidar point cloud that are not likely to be representative of ground cover. Hence, thelidar segmentation system 202 can be prevented from erroneously determining that non-ground objects are part of ground cover in the driving environment of theautonomous vehicle 100. - The
lidar segmentation system 202 further comprises alidar preprocessing component 208, aneural network component 210, and alidar postprocessing component 212. Thelidar segmentation system 202 receives lidar data from thelidar sensor system 101, wherein the lidar data is indicative of positions of surfaces of a plurality of objects in a driving environment of theautonomous vehicle 100. Thelidar preprocessing component 208 performs various processing operations over the lidar data and outputs a plurality of input features to theneural network component 210. Theneural network component 210 is configured to output a probability that a point is representative of at least one of ground cover or vegetation based upon the input features associated with the point. Thelidar postprocessing component 212 then performs a segmentation of the lidar point cloud based upon the probability output by theneural network component 210. - Exemplary operations of the
lidar segmentation system 202 are now set forth. Thelidar segmentation system 202 receives a lidar point cloud from thelidar sensor system 101, wherein the point cloud is indicative of positions of points on surfaces of the objects 302-308 (and other objects that may be present) in the drivingenvironment 300 of the vehicle 100 (e.g., as described above with respect toFIGS. 3A and 3B ). Thelidar preprocessing component 208 performs an analysis over the points in the point cloud to identify various features relative to each of the points in the point cloud. By way of example, thelidar preprocessing component 208 parameterizes the point cloud according to a training scheme used to learn theneural network component 210. For instance, thelidar preprocessing component 208 can compute, for each of the points in the point cloud, a distance to a next-closest point in the point cloud, a distance from the point to theautonomous vehicle 100, a number of points within a threshold distance of the point, etc., where such features are provided as input to a DNN during training of the DNN (e.g., as described below with respect toFIG. 4 ). Thepreprocessor component 208 then provides these features as input features to theneural network component 210 for each of the points in the point cloud. - The
neural network component 210 comprises a DNN that has a plurality of layers including an input layer, a plurality of hidden layers (e.g., greater than 5 layers, greater than 10 layers, greater than 50 layers, etc.), and an output layer. Responsive to receipt of input features for a point in the lidar point cloud at the input layer of the DNN, the input features are propagated through the hidden layers of the DNN. The DNN is configured to output, for each of the points in the point cloud, one or more probabilities, each probability indicating a likelihood that the point is representative of an object of a certain type. By way of example, the DNN can be configured such that the output layer is a softmax layer that outputs a first probability that the input point (e.g., the point represented by the input features provided at the input layer of the DNN) represents a first type of object, a second probability that the input point represents a second type of object, etc. through an nth probability that the input points represents an nth type of object, where then probabilities sum to unity. In an exemplary embodiment, the output layer outputs a first probability that the input point is representative of vegetation, a second probability that the input point is representative of ground cover, and a third probability that the input point is representative of any other type of object (e.g., cars, trucks, pedestrians, bicycles, etc.). As discussed below in greater detail with respect toFIG. 4 , the DNN is learned based upon labeled training data (e.g., labeled lidar point cloud data) such that the output of the DNN indicates a high probability (e.g., greater than 80%, greater than 90%, greater than 95%, etc.) that the input point is ground cover when the input point is actually representative of ground cover, a high probability that the input point is vegetation when the input point is actually representative of vegetation, etc. - The lidar
post processing component 212 receives the output of the DNN (e.g., the n probabilities indicated by the output layer of the DNN) from theneural network component 210 and outputs, based upon the output of the DNN, a segmentation of the lidar point cloud (e.g., as received from the lidar sensor system 101) that indicates which points in the point cloud are representative of same objects in the driving environment of theautonomous vehicle 100. In an exemplary embodiment, the lidarpost processing component 212 performs a segmentation of the lidar point cloud by excluding points from consideration that are indicated by the output of the DNN as being likely to be representative of vegetation and/or ground. Stated differently, the lidarpost processing component 212 performs a segmentation of only those points in the point cloud that are indicated by the output of the DNN as not being likely to represent ground or vegetation. By excluding points representative of ground and vegetation from consideration by a segmentation algorithm, the lidarpost processing component 212 can reduce errors associated with poor performance of the segmentation algorithm in identifying ground and vegetation features. For instance, excluding points representative of ground cover from consideration by a segmentation algorithm can reduce errors caused by an inability of the algorithm to distinguish between the ground and objects in contact with the ground (e.g., pedestrians' feet, car tires, or other object in contact with the ground). In another example, excluding points representative of vegetation from consideration by a segmentation algorithm can reduce errors caused by an inability of the algorithm to identify that scattered points in the point cloud are representative of a same vegetative object, such as a bush, a leafy tree branch, etc. - The lidar
post processing component 212 identifies a first group of points in the point cloud that are indicated by the output of the DNN as being likely to represent ground, and/or a second group of points in the point cloud that are indicated by the output of the DNN as being likely to represent vegetation. In an embodiment, the lidarpost processing component 212 can identify points that are likely to be representative of ground and/or vegetation based upon the output of the DNN and a probability threshold. For instance, the lidarpost processing component 212 can label a point as being representative of ground based upon the output of the DNN indicating that the point has a probability of being ground that is above the probability threshold. In various examples, the probability threshold may be 75%, 90%, 95%, or other desirable threshold. Responsive to identifying a point as being likely to be representative of ground or vegetation (e.g., based upon the probability threshold), the lidarpost processing component 212 assigns a label to the point that indicates ground or vegetation, accordingly. - Upon determining which points in the lidar point cloud are likely to be representative of ground and/or vegetation, the lidar
post processing component 212 generates a segmentation of points in the point cloud that are not labeled as ground or vegetation. The segmentation comprises data indicative of which points are representative of which objects in the driving environment of theautonomous vehicle 100. By way of an example, generating the segmentation comprises assigning respective labels to each of the points in the point cloud, wherein each of the labels indicates a group to which its point belongs, each group being representative of a different object in the driving environment. The lidarpost processing component 212 can generate the segmentation according to any suitable segmentation algorithm. In an example, the lidarpost processing component 212 generates the segmentation based upon output of a neural network that is trained to output data indicative of whether points in a lidar point cloud are representative of same objects. In another example, the segmentation is generated based upon an algorithm that considers distance between points in the point cloud, relative angles formed by points in the point cloud, convex hulls of subsets of points in the point cloud, etc. Other approaches to generating the segmentation of the points not labeled vegetation or ground are also contemplated. - Responsive to generating the segmentation, the lidar
post processing component 212 outputs the segmentation to theobject recognition system 118. Theobject recognition system 118 outputs data indicative of one or more objects in the driving environment to thecontrol system 120 based upon the segmentation generated by thelidar segmentation system 202. By way of example, theobject recognition system 118 outputs data indicative of positions and/or types of objects in the driving environment. In exemplary embodiments, theobject recognition system 118 outputs data indicative of positions and/or types of objects in the driving environment of theautonomous vehicle 100 based upon the segmentation output by thelidar segmentation system 202 and output of one or more of the additional object recognizer components 204-206. Subsequently, thecontrol system 120 of thevehicle 100 can output control signals to at least one of theengine 106, thebraking system 108, or thesteering system 110 based upon the output of theobject recognition system 118, such that the autonomous vehicle is controlled based at least in part upon the segmentation of the lidar point cloud generated by thelidar segmentation system 202. - In various embodiments, it may be desirable to control operation of an autonomous vehicle based at least in part on labeled data that pertains to a potential operating environment of the vehicle (e.g., a city or other region in which the autonomous vehicle is to drive). For instance, it may be desirable to control operation of an autonomous vehicle based upon a semantic map that includes data pertaining to route information (e.g., locations of streets, directions of traffic on streets, road closures, etc.) and locations of interest (e.g., restaurants, fueling/charging stations, entertainment venues, etc.). In another example, it may be desirable to control operation of an autonomous vehicle based upon a heightmap of the driving environment of the autonomous vehicle.
- Referring once again to
FIG. 1 , thecomputing system 112 of theautonomous vehicle 100 further includes adata store 122 that stores labeleddata 124. The labeleddata 124 includes data pertaining to a potential operating environment of the autonomous vehicle 100 (e.g., an area that includes the surroundings of thevehicle 100 that make up the driving environment of the vehicle 100). The labeleddata 124 is generated independent of the sensor systems 101-104 of theautonomous vehicle 100 and is labeled based upon ground truth data pertaining to the potential operating environment (e.g., based upon human input). The labeleddata 124 includes asemantic map 126 of a potential operating environment of the autonomous vehicle 100 (e.g., an area that includes the surroundings of thevehicle 100 that make up the driving environment of the vehicle 100) and aheightmap 128 of the potential operating environment. Theheightmap 128 comprises data indicating a height for each of a plurality of points in a two-dimensional space that comprises an overhead view of the potential operating environment of theautonomous vehicle 100. In an exemplary embodiment, theautonomous vehicle 100 can identify a fastest available route to a destination based upon thesemantic map 126. - It may further be desirable to control operation of the
autonomous vehicle 100 based jointly upon the labeleddata 124 and the sensor signals output by the sensor systems 101-104 of theautonomous vehicle 100. For example, it may be desirable that theautonomous vehicle 100 identify that sensor data is representative of a same object in the driving environment that is also represented in the labeled data, and perform one or more functions based upon the sensor data and the labeleddata 124 being representative of the same object. In order to control operation of theautonomous vehicle 100 based jointly upon sensor data and the labeleddata 124, the sensor data and the labeleddata 124 are aligned to ensure that sensor signals are accurately matched to corresponding information in the labeleddata 124. In an example, the lidar data output by thelidar sensor system 101 is aligned with theheight map 128. The lidar data and theheight map 128 are aligned such that thecomputing system 112 can readily determine that first data in theheight map 128 is representative of a first location of a first point in the lidar data. By way of example, thecomputing system 112 can compute a coordinate transform from a coordinate system of the lidar point cloud to a coordinate system of the height map 128 (and vice-versa). Hence, thecomputing system 112 can match coordinates of a point in the point cloud to coordinates in theheight map 128 to determine a ground height at a position of the point. - Alignment between sensor data and the labeled
data 124 can be initially achieved during calibration of systems of theautonomous vehicle 100 prior to operation of theautonomous vehicle 100 in the driving environment. By way of example, in a calibration environment thelidar sensor system 101 generates a lidar point cloud, wherein the lidar point cloud is representative of a known geometry of the calibration environment. Thecomputing system 112 of thevehicle 100 can compute an alignment between labeleddata 124 and the lidar point cloud based upon the geometry of the calibration environment being known. However, theautonomous vehicle 100 is subject to misalignment of the sensor data and the labeleddata 124 over time as theautonomous vehicle 100 operates in a driving environment. Conventionally, in order to maintain alignment of sensor data and labeled data, the autonomous vehicle must be periodically recalibrated in the environment having the known geometry, requiring downtime of the autonomous vehicle and removal of the autonomous vehicle from the driving environment. - In exemplary embodiments, the
autonomous vehicle 100 is configured to align sensor data from one or more of the sensor systems 101-104 with labeleddata 124 in real-time during operation of thevehicle 100 in a driving environment. Referring now once again toFIG. 2 , thelidar segmentation system 202 further comprises adata alignment system 214 that computes an alignment between lidar data output by thelidar sensor system 101 and the labeleddata 124 based upon output of theneural network component 210. In an exemplary embodiment, thedata alignment system 214 receives output of the DNN from theneural network component 210. Thedata alignment system 214 identifies points in the point cloud for which the output of the DNN indicates a high probability (e.g., greater than 90% probability, greater than 95% probability, greater than 99% probability) that the point is representative of ground. Thedata alignment system 214 then computes an alignment between the coordinate system of the lidar point cloud and the coordinate system of theheightmap 128 based upon the identified points and theheightmap 128. In general, the alignment computed by thedata alignment system 214 based upon the identified points is more accurate than an alignment computed based upon a group of points in the lidar point cloud that also includes points that are not representative of ground. - Responsive to receipt of alignment data from the
data alignment system 214, the lidarpost processing component 212 can generate a segmentation of the lidar data based upon the labeleddata 124 or update an existing segmentation of the lidar data based upon the labeleddata 124. In an example, the lidarpost processing component 212 receives a coordinate transform from thedata alignment system 214. The coordinate transform indicates a transformation from a coordinate system of the lidar data to a coordinate system of at least one dataset in the labeleddata 124. The lidarpost processing component 212 employs the coordinate transform to update a segmentation based upon characteristics of points in the lidar point cloud that are indicated in the labeleddata 124. - For example, the lidar
post processing component 212 can identify a point in the lidar point cloud for which the output of the DNN indicates a moderate probability (e.g., 40%-80%, 50%-70%, or 55%-65%) of the point being representative of ground cover. For instance, referring again briefly toFIG. 3B , apoint 326 near the bottom of a tire of thevehicle 302 in the drivingenvironment 300 can be indicated in the output of the DNN as having a probability of 55% of being representative of ground cover (e.g., due in part to its proximity to the ground surface 324). The lidarpost processing component 212 can compute, based upon the coordinate transform, coordinates of the identified point in theheightmap 128. Based upon a height of the identified point indicated in theheightmap 128, the lidarpost processing component 212 can determine whether the identified point is or is not representative of ground. For example, if the height of the point as indicated in the lidar point cloud is significantly greater than the height indicated in the heightmap 128 (e.g., 20 centimeters greater, 10 centimeters greater, 5 centimeters greater), the lidarpost processing component 212 can assign a label to the point indicating that the point is representative of an object other than ground. In another example, if the height of the point as indicated in the lidar point cloud is substantially similar to the height indicated in the heightmap 128 (e.g., within 20 centimeters, within 10 centimeters, within 5 centimeters), the lidarpost processing component 212 can assign a label to the point indicating that the point is representative of ground cover. It is to be understood that there may be uncertainty associated with heights indicated in theheightmap data 128. In exemplary embodiments, theheightmap 128 includes data indicative of an uncertainty threshold (e.g., a range of potential heights) for one or more of the heights indicated in theheightmap 128. The lidarpost processing component 212 can assign a label to a point in the lidar point cloud based upon the uncertainty threshold, the label indicating that the point is representative of ground cover. For instance, if a height of a point as indicated by the lidar point cloud is within a range of potential heights indicated in theheightmap 128, the lidarpost processing component 212 assigns a label to the point indicating that the point is representative of ground. - It is to be understood that the lidar
post processing component 212 can assign the labels to points in the lidar point cloud based upon the alignment, as described above, prior to performing a segmentation of the lidar point cloud. In an exemplary embodiment, thedata alignment system 214 receives the output of the DNN from theneural network component 210, and computes the alignment as described above. The lidarpost processing component 212, based upon the alignment and theheightmap 128, evaluates points in the lidar point cloud that potentially represent ground cover, and assigns labels to those points with heights consistent with theheightmap 128. The lidarpost processing component 212 can then generate the segmentation of the lidar point cloud based upon the labeled points being representative of ground. - By way of example, and not limitation, the lidar
post processing component 212 can evaluate points in the lidar point cloud that are indicated by the output of the DNN as having a probability of representing ground cover that is above a threshold amount (e.g., greater than 30%, greater than 40%, greater than 50%) against heights indicated in theheightmap 128. To each point indicated by the lidar point cloud as having a height within a range of potential heights indicated in theheightmap 128, the lidarpost processing component 212 assigns a label indicating that the point is representative of ground. The lidarpost processing component 212 can then perform a segmentation of the remaining unlabeled points in the lidar point cloud based upon any suitable segmentation algorithm. Thecontrol system 120 can subsequently output control signals to at least one of theengine 106, thebraking system 108, or thesteering system 110 to control operation of theautonomous vehicle 100 based upon the segmentation of the lidar point cloud. - With reference now to
FIG. 4 , anexemplary computing system 400 that is configured to learn theneural network component 210 is illustrated. Thecomputing system 400 includes aprocessor 402 andmemory 404, wherein thememory 404 includes instructions that are executed by theprocessor 402. Thecomputing system 400 additionally includes adata store 406 that comprises labeledtraining data 408. Thememory 404 includes aneural network learner 410 that learns theneural network component 210. Theneural network learner 410 receives the labeledtraining data 408, wherein the labeledtraining data 408 comprises lidar point cloud data (e.g., parameterized as described above with respect to input features of the DNN), wherein the lidar point cloud data further includes ground truth data indicating which points are representative of vegetation, and which points are representative of ground in the lidar point cloud data. Theneural network learner 410 learns theneural network component 210 such that theneural network component 210 outputs a high ground probability for points in the point cloud that are representative of ground, a high vegetation probability for points representative of vegetation, and a low probability of vegetation and ground for points representative of neither vegetation nor ground. While not shown, theneural network component 210 can be validated based upon a test training data set, which is also labeled, to ensure that theneural network component 210 produces outputs with acceptable characteristics (e.g., such that theneural network component 210 outputs a high vegetation probability for points representative of vegetation and a low vegetation probability for points that are not representative of vegetation). -
FIGS. 5-7 illustrate exemplary methodologies relating to controlling an autonomous vehicle based upon data output by a lidar data segmentation system. While the methodologies are shown and described as being a series of acts that are performed in a sequence, it is to be understood and appreciated that the methodologies are not limited by the order of the sequence. For example, some acts can occur in a different order than what is described herein. In addition, an act can occur concurrently with another act. Further, in some instances, not all acts may be required to implement a methodology described herein. - Moreover, the acts described herein may be computer-executable instructions that can be implemented by one or more processors and/or stored on a computer-readable medium or media. The computer-executable instructions can include a routine, a sub-routine, programs, a thread of execution, and/or the like. Still further, results of acts of the methodologies can be stored in a computer-readable medium, displayed on a display device, and/or the like.
- Referring now to
FIG. 5 , anexemplary methodology 500 for controlling operation of an autonomous vehicle is illustrated. Themethodology 500 starts at 502, and at 504 lidar data is received, wherein the lidar data comprises a plurality of points representative of positions of objects in a driving environment of an autonomous vehicle. At 506, a label is assigned to a first point in the points based upon output of a DNN. By way of example, responsive to receiving input features pertaining to the first point, the DNN outputs respective probabilities that the first point is representative of ground cover, vegetation, or another type of object. In a non-limiting example, the label is assigned to the first point responsive to determining that a probability of the first point being either vegetation or ground as indicated in the output of the DNN is greater than a pre-defined threshold value. The label assigned to the first point indicates that the first point is representative of ground cover or vegetation in the driving environment. At 508, a segmentation of the lidar data is generated based upon the first point being excluded from the segmentation. By generating the segmentation based upon excluding the first point from the segmentation, errors associated with poor performance of a segmentation algorithm relative to ground and vegetation features are avoided. At 510, at least one of an engine, a braking system, or a steering system of the autonomous vehicle is controlled based upon the segmentation generated at 508. For example, an object recognition system of the autonomous vehicle can generate a prediction of future behavior of one or more objects in the driving environment based upon the segmentation, and can control steering of the autonomous vehicle based upon such prediction. Themethodology 500 completes at 512. - Turning now to
FIG. 6 , anexemplary methodology 600 that facilitates learning a DNN is illustrated, wherein the DNN is configured for use in a lidar segmentation system of an autonomous vehicle. Themethodology 600 starts at 602, and at 604 labeled training data is received, wherein the training data comprises labeled lidar point cloud data. At 606, a DNN is learned based upon the training data. The DNN is configured such that responsive to receiving a point in a lidar point cloud as input (e.g., when the DNN receives a vector of input features representative of the point at an input layer of the DNN), the DNN outputs a plurality of probabilities, each probability indicative of a likelihood that the point represents a different respective type of object. For instance, responsive to receiving a point in a lidar point cloud as input, the DNN outputs a first probability that the point is representative of vegetation, a second probability that the point is representative of ground, and a third probability that the point is representative of some other type of object, where the three probabilities sum to unity. The DNN is learned at 606 such that the output of the DNN indicates a high probability that the point is representative of vegetation when the input point is a point representative of vegetation, the output of the DNN indicates a high probability that the point is representative of ground when the input point is a point representative of ground, and the output of the DNN indicates a high probability that the point is representative of some other type of object when the input point is a point representative of neither ground nor vegetation. Themethodology 600 completes at 608. - Referring now to
FIG. 7 , anexemplary methodology 700 that facilitates controlling operation of an autonomous vehicle based upon sensor data and labeled data is illustrated. Themethodology 700 begins at 702 and at 704, lidar data is received, wherein the lidar data comprises a plurality of points representative of positions of objects in a driving environment of an autonomous vehicle. At 706, an alignment is computed between the lidar data and a heightmap based upon a first point in the points, wherein the first point is selected based upon output of a DNN indicating that the first point is representative of ground. At 708, at least one of an engine, a braking system, or a steering system of the autonomous vehicle is controlled based upon the lidar data and the heightmap, wherein the alignment computed at 706 facilitates joint consideration of the lidar data and the heightmap. By way of example, a segmentation of the lidar data can be generated based at least in part upon the heightmap being aligned with the lidar data. Themethodology 700 ends at 710. - Referring now to
FIG. 8 , a high-level illustration of anexemplary computing device 800 that can be used in accordance with the systems and methodologies disclosed herein is illustrated. For instance, thecomputing device 800 may be or include thecomputing system 112 or thecomputing system 400. Thecomputing device 800 includes at least oneprocessor 802 that executes instructions that are stored in amemory 804. The instructions may be, for instance, instructions for implementing functionality described as being carried out by one or more modules and systems discussed above or instructions for implementing one or more of the methods described above. Theprocessor 802 may be a GPU, a plurality of GPUs, a CPU, a plurality of CPUs, a multi-core processor, etc. Theprocessor 802 may access thememory 804 by way of asystem bus 806. In addition to storing executable instructions, thememory 804 may also store point cloud data, vectors of output features for points in a point cloud, segmentation data, training data, etc. - The
computing device 800 additionally includes adata store 808 that is accessible by theprocessor 802 by way of thesystem bus 806. Thedata store 808 may include executable instructions, sensor data, output feature vector data, training data, segmentation data, etc. Thecomputing device 800 also includes aninput interface 810 that allows external devices to communicate with thecomputing device 800. For instance, theinput interface 810 may be used to receive instructions from an external computer device, etc. Thecomputing device 800 also includes anoutput interface 812 that interfaces thecomputing device 800 with one or more external devices. For example, thecomputing device 800 may transmit control signals to theengine 106, thebraking system 108, and/or thesteering system 110 by way of theoutput interface 812. - Additionally, while illustrated as a single system, it is to be understood that the
computing device 800 may be a distributed system. Thus, for instance, several devices may be in communication by way of a network connection and may collectively perform tasks described as being performed by thecomputing device 800. - Various functions described herein can be implemented in hardware, software, or any combination thereof. If implemented in software, the functions can be stored on or transmitted over as one or more instructions or code on a computer-readable medium. Computer-readable media includes computer-readable storage media. A computer-readable storage media can be any available storage media that can be accessed by a computer. By way of example, and not limitation, such computer-readable storage media can comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other medium that can be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer. Disk and disc, as used herein, include compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk, and Blu-ray disc (BD), where disks usually reproduce data magnetically and discs usually reproduce data optically with lasers. Further, a propagated signal is not included within the scope of computer-readable storage media. Computer-readable media also includes communication media including any medium that facilitates transfer of a computer program from one place to another. A connection, for instance, can be a communication medium. For example, if the software is transmitted from a website, server, or other remote source using a coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio and microwave are included in the definition of communication medium. Combinations of the above should also be included within the scope of computer-readable media.
- Alternatively, or in addition, the functionally described herein can be performed, at least in part, by one or more hardware logic components. For example, and without limitation, illustrative types of hardware logic components that can be used include Field-programmable Gate Arrays (FPGAs), Program-specific Integrated Circuits (ASICs), Program-specific Standard Products (ASSPs), System-on-a-chip systems (SOCs), Complex Programmable Logic Devices (CPLDs), etc.
- What has been described above includes examples of one or more embodiments. It is, of course, not possible to describe every conceivable modification and alteration of the above devices or methodologies for purposes of describing the aforementioned aspects, but one of ordinary skill in the art can recognize that many further modifications and permutations of various aspects are possible. Accordingly, the described aspects are intended to embrace all such alterations, modifications, and variations that fall within the spirit and scope of the appended claims. Furthermore, to the extent that the term “includes” is used in either the detailed description or the claims, such term is intended to be inclusive in a manner similar to the term “comprising” as “comprising” is interpreted when employed as a transitional word in a claim.
Claims (20)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US17/226,123 US20210223402A1 (en) | 2018-08-03 | 2021-04-09 | Autonomous vehicle controlled based upon a lidar data segmentation system |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/054,088 US11022693B1 (en) | 2018-08-03 | 2018-08-03 | Autonomous vehicle controlled based upon a lidar data segmentation system |
US17/226,123 US20210223402A1 (en) | 2018-08-03 | 2021-04-09 | Autonomous vehicle controlled based upon a lidar data segmentation system |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/054,088 Continuation US11022693B1 (en) | 2018-08-03 | 2018-08-03 | Autonomous vehicle controlled based upon a lidar data segmentation system |
Publications (1)
Publication Number | Publication Date |
---|---|
US20210223402A1 true US20210223402A1 (en) | 2021-07-22 |
Family
ID=76094568
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/054,088 Active 2038-08-31 US11022693B1 (en) | 2018-08-03 | 2018-08-03 | Autonomous vehicle controlled based upon a lidar data segmentation system |
US17/226,123 Pending US20210223402A1 (en) | 2018-08-03 | 2021-04-09 | Autonomous vehicle controlled based upon a lidar data segmentation system |
Family Applications Before (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/054,088 Active 2038-08-31 US11022693B1 (en) | 2018-08-03 | 2018-08-03 | Autonomous vehicle controlled based upon a lidar data segmentation system |
Country Status (1)
Country | Link |
---|---|
US (2) | US11022693B1 (en) |
Families Citing this family (6)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US11204605B1 (en) * | 2018-08-03 | 2021-12-21 | GM Global Technology Operations LLC | Autonomous vehicle controlled based upon a LIDAR data segmentation system |
US11594011B2 (en) * | 2019-01-30 | 2023-02-28 | Baidu Usa Llc | Deep learning-based feature extraction for LiDAR localization of autonomous driving vehicles |
WO2021021672A2 (en) * | 2019-07-26 | 2021-02-04 | Deka Products Limited Partnership | System and method for free space estimation |
US11556000B1 (en) | 2019-08-22 | 2023-01-17 | Red Creamery Llc | Distally-actuated scanning mirror |
CN110687549B (en) * | 2019-10-25 | 2022-02-25 | 阿波罗智能技术(北京)有限公司 | Obstacle detection method and device |
US11341650B2 (en) * | 2020-03-19 | 2022-05-24 | At&T Intellectual Property I, L.P. | Method for accelerating three-dimensional object segmentation with point cloud simplifications |
Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100021052A1 (en) * | 2006-06-02 | 2010-01-28 | Carnegie Mellon University | System and method for generating a terrain model for autonomous navigation in vegetation |
US20120027298A1 (en) * | 2010-07-27 | 2012-02-02 | Aerotec, Llc | Method and Apparatus for Direct Detection, Location, Analysis, Identification, and Reporting of Vegetation Clearance Violations |
US20140002866A1 (en) * | 2012-06-28 | 2014-01-02 | Xerox Corporation | Method and apparatus for object assisted image editing and transmission of scanned documents |
US8886387B1 (en) * | 2014-01-07 | 2014-11-11 | Google Inc. | Estimating multi-vehicle motion characteristics by finding stable reference points |
US20150269438A1 (en) * | 2014-03-18 | 2015-09-24 | Sri International | Real-time system for multi-modal 3d geospatial mapping, object recognition, scene annotation and analytics |
US20180173971A1 (en) * | 2016-12-19 | 2018-06-21 | Waymo Llc | Pedestrian detection neural networks |
US20180300620A1 (en) * | 2017-04-12 | 2018-10-18 | Ford Global Technologies, Llc | Foliage Detection Training Systems And Methods |
US20190197350A1 (en) * | 2017-12-26 | 2019-06-27 | Samsung Electronics Co., Ltd. | Apparatus for performing neural network operation and method of operating the same |
US20200033880A1 (en) * | 2018-07-30 | 2020-01-30 | Toyota Research Institute, Inc. | System and method for 3d scene reconstruction of agent operation sequences using low-level/high-level reasoning and parametric models |
Family Cites Families (10)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100026555A1 (en) | 2006-06-09 | 2010-02-04 | Whittaker William L | Obstacle detection arrangements in and for autonomous vehicles |
US9188980B2 (en) | 2008-09-11 | 2015-11-17 | Deere & Company | Vehicle with high integrity perception system |
EP2668008A4 (en) | 2011-01-28 | 2018-01-24 | Intouch Technologies, Inc. | Interfacing with a mobile telepresence robot |
US10520482B2 (en) * | 2012-06-01 | 2019-12-31 | Agerpoint, Inc. | Systems and methods for monitoring agricultural products |
US8825260B1 (en) | 2013-07-23 | 2014-09-02 | Google Inc. | Object and ground segmentation from a sparse one-dimensional range data |
US9330435B2 (en) * | 2014-03-19 | 2016-05-03 | Raytheon Company | Bare earth finding and feature extraction for 3D point clouds |
US10137890B2 (en) * | 2016-06-28 | 2018-11-27 | Toyota Motor Engineering & Manufacturing North America, Inc. | Occluded obstacle classification for vehicles |
WO2018108832A1 (en) | 2016-12-14 | 2018-06-21 | Starship Technologies Oü | Robot, system and method detecting and/or responding to transitions in height |
WO2018125938A1 (en) | 2016-12-30 | 2018-07-05 | DeepMap Inc. | Enrichment of point cloud data for high-definition maps for autonomous vehicles |
US10438371B2 (en) * | 2017-09-22 | 2019-10-08 | Zoox, Inc. | Three-dimensional bounding box from two-dimensional image and point cloud data |
-
2018
- 2018-08-03 US US16/054,088 patent/US11022693B1/en active Active
-
2021
- 2021-04-09 US US17/226,123 patent/US20210223402A1/en active Pending
Patent Citations (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20100021052A1 (en) * | 2006-06-02 | 2010-01-28 | Carnegie Mellon University | System and method for generating a terrain model for autonomous navigation in vegetation |
US20120027298A1 (en) * | 2010-07-27 | 2012-02-02 | Aerotec, Llc | Method and Apparatus for Direct Detection, Location, Analysis, Identification, and Reporting of Vegetation Clearance Violations |
US20140002866A1 (en) * | 2012-06-28 | 2014-01-02 | Xerox Corporation | Method and apparatus for object assisted image editing and transmission of scanned documents |
US8886387B1 (en) * | 2014-01-07 | 2014-11-11 | Google Inc. | Estimating multi-vehicle motion characteristics by finding stable reference points |
US20150269438A1 (en) * | 2014-03-18 | 2015-09-24 | Sri International | Real-time system for multi-modal 3d geospatial mapping, object recognition, scene annotation and analytics |
US20180173971A1 (en) * | 2016-12-19 | 2018-06-21 | Waymo Llc | Pedestrian detection neural networks |
US20180300620A1 (en) * | 2017-04-12 | 2018-10-18 | Ford Global Technologies, Llc | Foliage Detection Training Systems And Methods |
US20190197350A1 (en) * | 2017-12-26 | 2019-06-27 | Samsung Electronics Co., Ltd. | Apparatus for performing neural network operation and method of operating the same |
US20200033880A1 (en) * | 2018-07-30 | 2020-01-30 | Toyota Research Institute, Inc. | System and method for 3d scene reconstruction of agent operation sequences using low-level/high-level reasoning and parametric models |
Also Published As
Publication number | Publication date |
---|---|
US11022693B1 (en) | 2021-06-01 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10884411B1 (en) | Autonomous vehicle controlled based upon a lidar data segmentation system and an aligned heightmap | |
US20210223402A1 (en) | Autonomous vehicle controlled based upon a lidar data segmentation system | |
Ma et al. | Artificial intelligence applications in the development of autonomous vehicles: A survey | |
Badue et al. | Self-driving cars: A survey | |
US11131993B2 (en) | Methods and systems for trajectory forecasting with recurrent neural networks using inertial behavioral rollout | |
US10803328B1 (en) | Semantic and instance segmentation | |
US10814871B2 (en) | Computing system for assigning maneuver labels to autonomous vehicle sensor data | |
US11783180B1 (en) | Object detection neural network | |
US11915427B2 (en) | Conflict resolver for a lidar data segmentation system of an autonomous vehicle | |
US11853061B2 (en) | Autonomous vehicle controlled based upon a lidar data segmentation system | |
US11584377B2 (en) | Lidar based detection of road surface features | |
WO2022216660A1 (en) | Verifying reliability of data used for autonomous driving | |
US20220043446A1 (en) | Ranking Agents Near Autonomous Vehicles By Mutual Importance | |
CN109633686B (en) | Method and system for detecting ground obstacle based on laser radar | |
RU2744012C1 (en) | Methods and systems for automated determination of objects presence | |
EP4160146A1 (en) | Quadtree based data structure for storing information relating to an environment of an autonomous vehicle and methods of use thereof | |
US11873011B2 (en) | Labeling lane segments for behavior prediction for agents in an environment | |
US20210302583A1 (en) | Vapor detection in lidar point cloud | |
US11449067B1 (en) | Conflict resolver for a lidar data segmentation system of an autonomous vehicle | |
US11037324B2 (en) | Systems and methods for object detection including z-domain and range-domain analysis | |
US11079767B2 (en) | Lidar based recognition of ride hailing gestures for autonomous vehicles | |
US20230058731A1 (en) | Determining occupancy using unobstructed sensor emissions | |
Chipka et al. | Estimation and navigation methods with limited information for autonomous urban driving | |
Li et al. | An overview of factors influencing the mass adoption of self-driving vehicles | |
US20220289237A1 (en) | Map-free generic obstacle detection for collision avoidance systems |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: GM GLOBAL TECHNOLOGY OPERATIONS LLC, MICHIGAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:ALLAIS, ANDREA;XIE, WILLIAM GONGSHU;CHAMBERS, MICAH CHRISTOPHER;AND OTHERS;REEL/FRAME:055872/0530 Effective date: 20180802 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: APPLICATION DISPATCHED FROM PREEXAM, NOT YET DOCKETED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |