WO2021231043A1 - Method and system for detecting a pile - Google Patents
Method and system for detecting a pile Download PDFInfo
- Publication number
- WO2021231043A1 WO2021231043A1 PCT/US2021/028067 US2021028067W WO2021231043A1 WO 2021231043 A1 WO2021231043 A1 WO 2021231043A1 US 2021028067 W US2021028067 W US 2021028067W WO 2021231043 A1 WO2021231043 A1 WO 2021231043A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- pile
- histogram
- machine
- cluster
- image
- Prior art date
Links
- 238000000034 method Methods 0.000 title claims abstract description 54
- 239000000463 material Substances 0.000 claims abstract description 89
- 238000013528 artificial neural network Methods 0.000 claims abstract description 21
- 238000003384 imaging method Methods 0.000 claims abstract description 18
- 238000013135 deep learning Methods 0.000 claims abstract description 17
- 230000015654 memory Effects 0.000 claims description 11
- 239000012925 reference material Substances 0.000 claims 1
- 238000001514 detection method Methods 0.000 description 11
- 230000011218 segmentation Effects 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 4
- 230000008901 benefit Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000010276 construction Methods 0.000 description 3
- 238000005065 mining Methods 0.000 description 3
- 239000000203 mixture Substances 0.000 description 3
- 230000009466 transformation Effects 0.000 description 3
- 238000013459 approach Methods 0.000 description 2
- 238000009412 basement excavation Methods 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000002372 labelling Methods 0.000 description 2
- 238000013507 mapping Methods 0.000 description 2
- 238000005259 measurement Methods 0.000 description 2
- 238000012935 Averaging Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000015572 biosynthetic process Effects 0.000 description 1
- 239000004927 clay Substances 0.000 description 1
- 238000002485 combustion reaction Methods 0.000 description 1
- 230000003750 conditioning effect Effects 0.000 description 1
- 238000013480 data collection Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 239000012530 fluid Substances 0.000 description 1
- 238000005755 formation reaction Methods 0.000 description 1
- 238000009472 formulation Methods 0.000 description 1
- 239000000446 fuel Substances 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 229910052500 inorganic mineral Inorganic materials 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 239000011707 mineral Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000008447 perception Effects 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 239000011435 rock Substances 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
- 239000004576 sand Substances 0.000 description 1
- 230000003936 working memory Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/10—Segmentation; Edge detection
- G06T7/11—Region-based segmentation
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/42—Simultaneous measurement of distance and other co-ordinates
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/88—Lidar systems specially adapted for specific applications
- G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
- G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S7/00—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00
- G01S7/48—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00
- G01S7/4802—Details of systems according to groups G01S13/00, G01S15/00, G01S17/00 of systems according to group G01S17/00 using analysis of echo signal for target characterisation; Target signature; Target cross-section
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/2163—Partitioning the feature space
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/22—Matching criteria, e.g. proximity measures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/241—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches
- G06F18/2413—Classification techniques relating to the classification model, e.g. parametric or non-parametric approaches based on distances to training or reference patterns
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/50—Depth or shape recovery
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/60—Analysis of geometric attributes
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
- G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/50—Context or environment of the image
- G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V20/00—Scenes; Scene-specific elements
- G06V20/60—Type of objects
- G06V20/64—Three-dimensional objects
- G06V20/647—Three-dimensional objects by matching two-dimensional images to three-dimensional objects
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10004—Still image; Photographic image
- G06T2207/10012—Stereo images
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10024—Color image
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/10—Image acquisition modality
- G06T2207/10028—Range image; Depth image; 3D point clouds
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20081—Training; Learning
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20084—Artificial neural networks [ANN]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20112—Image segmentation details
- G06T2207/20132—Image cropping
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20112—Image segmentation details
- G06T2207/20156—Automatic seed setting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/30—Subject of image; Context of image processing
- G06T2207/30248—Vehicle exterior or interior
- G06T2207/30252—Vehicle exterior; Vicinity of vehicle
- G06T2207/30261—Obstacle
Definitions
- the present disclosure relates generally to autonomous systems, and, more particularly, to a method and system for autonomously detecting a material pile.
- autonomous machines such as bulldozers, off-highway trucks, wheel loaders, and excavators
- the ability to detect piles of material is an important functionality for autonomous systems of many different types of machines.
- autonomous trucks need to be able to detect and localize piles so that they can plan a path and navigate towards a pile or around piles.
- Autonomous wheel loaders and excavators need to be able to detect piles so that they will know how to approach the pile and position the bucket to ensure the loading capacity of the bucket is maximized.
- Autonomous dozers need to detect piles in order to identify the pile to be spread.
- Detecting piles is also important for obstacle detection and safety. If a travel path of an autonomous machine is through or over a pile, the machine may flip, slide or become stuck, which not only presents a safety concern, but also takes time and money to rectify the situation.
- the method includes receiving at a controller a plurality of signals indicative of one or more parameters including a ground speed of the machine, a target ground speed of the machine, a load on a work implement of the machine, an output speed of a torque converter of the machine, a machine pitch, a machine steering command, a machine heading, and a heading of a slot the machine is in.
- the controller standardizes and normalizes each signal from the plurality of signals in order to create values for each of the one or more parameters that all fall within a common range, wherein the common range is representative of a range from minimum to maximum values for each of the one or more parameters.
- the controller also determines a variation of each of the values for the one or more parameters over each of a plurality of time periods to calculate relative rates of change for the one or more parameters, weights each of the values for each of the one or more parameters as a function of the relative rates of change, adds up the weighted values of the parameters, and applies a sigmoid function to the weighted values of the parameters in order to limit the effect any one of the parameters has on an output indicative of behavior characteristic of the machine having pushed a pile of dirt into contact with another pile of dirt.
- the present disclosure is related a method for detecting a pile of material by an autonomous machine.
- the method includes receiving, using a processor on the machine, a three- dimensional point cloud indicative of an environment having a pile of material, performing a ground surface estimation on the three-dimensional point cloud to identify non-ground points, grouping the non-ground points into one or more clusters of points based on the proximity of the points to each other, creating a normalized height histogram for each of the one or more clusters, comparing the normalized height histogram of each cluster to a generalized pile histogram, and identifying a cluster from the one or more clusters as a pile based on the similarity between the normalized height histogram of the cluster and the generalized pile histogram.
- the present disclosure is related to an autonomous vehicle having one or more traction devices connected to and supporting a frame, a power source mounted to the frame and configured to drive the traction devices, a 3D sensing device capable of generating a 3D point cloud indicative of an environment having a pile of material, and a processor.
- the processor being configured to receive a three-dimensional point cloud indicative of an environment having a pile of material, perform a ground surface estimation on the three-dimensional point cloud to identify non-ground points, group the non-ground points into one or more clusters of points based on the proximity of the points to each other, create a normalized height histogram for each of the one or more clusters, compare the normalized height histogram of each cluster to a generalized pile histogram, and identify a cluster from the one or more clusters as a pile based on the similarity between the normalized height histogram of the cluster and the generalized pile histogram.
- the present disclosure is related a method for detecting a pile of material by an autonomous machine that includes receiving from an imaging device, using a processor on the machine, a two-dimensional image of an environment having a pile of material, autonomously detecting an image of the material pile in the two-dimensional image using a deep-learning neural network stored in a memory on the machine and previously trained to detect an image of a material pile in a two- dimensional image, and calibrating the imaging device with respect to a coordinate frame of the machine to determine the distance of the pile of material from the machine.
- FIG. l is a side view of an exemplary machine
- FIG. 2 is a schematic representation of an exemplary system for detecting piles of the machine of FIG. 1;
- FIG. 3 is a flowchart of an exemplary method of detecting a pile
- FIG. 4 is an illustration of an image of an environment having piles of material with the piles identified with bounding boxes;
- FIG. 5 is an illustration of an image of an environment having piles of material with the piles identified with pixelwise segmentation masks
- FIG. 6 is a flowchart of another exemplary method of detecting a pile
- FIG. 7 is an illustration of a 3D point cloud indicative of an environment having a pile of material.
- FIG. 8 is an illustration of the output of a ground surface estimation algorithm on the 3D point cloud of FIG. 7.
- FIG. 1 illustrates an exemplary embodiment of an autonomous machine 10.
- the autonomous machine 10 is a wheel loader. It is contemplated, however, that the autonomous machine 10 could be a variety of machines.
- the autonomous machine 10 may be a machine used in the construction and/or mining industry, such as an excavator, a dozer, and an off-highway truck. Any autonomous machine that would benefit from the disclosed systems and methods for detecting a pile may be used.
- the machine 10 may include, among other things, a power source 12, one or more traction devices 14 (e.g., wheels or tracks), a work tool 16, one or more lift actuators 18, and one or more tilt actuators 20.
- the lift actuators 18 and the tilt actuators 20 may connect the work tool 16 to a frame 22 of the machine 10.
- the lift actuators 18 may have one end connected to the frame 22 and an opposite end connected to a structural member 24, which may be connected to the work tool 16.
- the work tool 16 may be connected to the structural member 24 via a pivot pin 26.
- the lift actuators 18 may be configured to lift or raise the work tool 16 to a desired height above a ground surface 28.
- the tilt actuators 20 may have one end connected to the frame 22 and an opposite end connected to a linkage member 30, which may be connected to the work tool 16.
- the power source 12 may be supported by the frame 22 of the machine 10 and may include an engine (not shown) configured to produce a rotational power output and a transmission (not shown) that converts the power output to a desired ratio of speed and torque.
- the rotational power output may be used to drive a pump (not shown) that supplies pressurized fluid to the lift actuators 18, the tilt actuators 20, and/or to one or more motors (not shown) associated with traction devices 14.
- the engine of the power source 12 may be a combustion engine configured to bum a mixture of fuel and air.
- the transmission of the power source 12 may take any form known in the art, for example a power shift configuration that provides multiple discrete operating ranges, a continuously variable configuration, or a hybrid configuration.
- the power source 12, in addition to driving the work tool 16, may also function to propel the machine 10, for example via one or more traction devices 14.
- the work tool 16 may include any device used to perform a particular task such as, for example, a bucket, a fork arrangement, a blade, a shovel, or any other task-performing device known in the art.
- the work tool 16 may alternatively or additionally rotate, slide, swing open/close, or move in any other manner known in the art.
- the lift and tilt actuators 18, 20 may be extended or retracted to repetitively move work tool 16 during an excavation cycle.
- the machine 10 may be used to move or add to a material pile 34 and/or plot a path to or around the material pile 34.
- the material pile 34 may constitute a variety of different types of materials.
- the material pile 34 may consist of loose sand, dirt, gravel, clay, rocks, mineral formations, etc.
- the work tool 16 is a bucket having a tip 38 configured to penetrate the material pile 34.
- the machine 10 includes a system 40 for autonomously detecting material piles 34.
- the system 40 includes one or more sensors 42 configured to provide a two-dimensional (2D) image of an environment having a pile of material 34 and/or a three-dimensional (3D) map space representation indicative of an environment having a pile of material 34.
- the machine 10 may include a 2D imaging device 44, such as a mono camera, thermal camera, video camera, stereo camera, or some other imaging device (e.g., an image sensor).
- the machine 10 may also include a 3D sensing device 46, such as for example, a LIDAR (light detection and ranging) device, a RADAR (radio detection and ranging) device, a SONAR (sound navigation and ranging) device, a stereo camera, or any other device capable of providing a 3D map space representation (i.e. a 3D point cloud) indicative of an environment having a material pile 34.
- a 3D sensing device 46 such as for example, a LIDAR (light detection and ranging) device, a RADAR (radio detection and ranging) device, a SONAR (sound navigation and ranging) device, a stereo camera, or any other device capable of providing a 3D map space representation (i.e. a 3D point cloud) indicative of an environment having a material pile 34.
- the one or more sensors 42 may be any suitable device known in the art for creating a 2D image and/or a 3D map space representation, or other suitable output for use in detecting a pile.
- the one or more sensors 42 may generate the image and/or a 3D map space representation of the material pile 34 and communicate the image and/or a 3D map space representation to an on-board processor 48 for subsequent conditioning.
- the one or more sensors 42 are communicatively coupled to the processor 48 in any suitable manner.
- the processor 48 may be configured in a variety of ways.
- the processor 48 may embody a single microprocessor or multiple microprocessors.
- the processor 48 may be dedicated to the function of pile detection or may provide additional functionality to the machine 10, such as an engine control module (ECM).
- ECM engine control module
- the system 40 also includes memory 50.
- the memory 50 may be integral to the processor 48 or remote but accessible by the processor 48.
- the memory 50 may be a read only memory (ROM) for storing a program(s), a neural network, or other information, a random access memory (RAM) which serves as a working memory area for use in executing the program(s) stored in the memory 50, or a combination thereof.
- the processor 48 may be configured to refer to information stored in the memory 50 and the memory 50 may be configured to store various information determined by the processor 48.
- the machine 10 may also be outfitted with a communication device 52 (FIG. 1) that allows communication of the image and/or a 3D map space representation to an off-board entity or from an off-board entity.
- a communication device 52 (FIG. 1) that allows communication of the image and/or a 3D map space representation to an off-board entity or from an off-board entity.
- the machine 10 may communicate with a remote-control operator and/or a central facility (not shown) via the communication device 52.
- FIG. 3 illustrates an exemplary method 100 used by the machine 10 for detecting the material pile 34.
- the method 100 uses a mono-camera(s) 44 or other imaging device(s)(e.g. Ethernet image sensor) capable of creating a 2D image and the processor 48 on the machine 10 with support to run a deep learning network for inference.
- the method 100 involves inputting 2D images into the deep learning neural network, which was trained to perform the task of pile detection.
- the method 100 includes the initial steps of collecting and annotating 2D image data 102 and then using the annotated image data to train a deep learning neural network 104. For the step of data collection and annotation 102, 2D images are obtained of a variety of piles.
- the mono- camera ⁇ ) or other imaging device(s) used to obtain the 2D images may be one or more of the sensors 42 on the machine 10, may be one or more cameras or other imaging devices independent of the machine 10, or a combination thereof.
- the 2D images would include a variety of material piles of different shapes, sizes, and material types and the images may be obtained from a variety of construction and mining sites.
- the 2D images obtained in step 102 are labelled using an annotation tool. Any suitable annotation tool or annotation technique known in the art for labelling portions of images may be used. FIG.
- the annotated image 120 includes a base ground surface 121, a first material pile 122, a second material pile 124, a person standing 126, and a boulder 128.
- the first material pile 122 is labeled by the annotation tool by surrounding it in a first bounded box 130.
- the second material pile 124 is surrounded by a second bounded box 132.
- the bounded boxes 130, 132 identify those portions of the image 120 that are material piles.
- FIG. 4 illustrates the annotated image 120 utilizing a different annotation method.
- the annotated image 120 includes a base ground surface 121, a first material pile 122, a second material pile 124, a person standing 126, and a boulder 128.
- the first material pile 122 is labeled by the annotation tool by covering the portions of the image depicting the first material pile 122 with a first pixelwise segmentation mask 140.
- the portions of the image depicting the second material pile 124 are covered with a second pixelwise segmentation mask 142.
- the pixelwise segmentation masks 140, 142 identify those portions of the image 120 that are material piles.
- Deep learning neural networks are known in the art. Any suitable deep learning neural network may be used. For example, if bounding box labelling is used, a network such as MobileNetSSD-V2 may be used. If pixelwise segmentation masks are used to label the material piles, a network such as SegNet may be used. Further, the methodology of how to train a deep learning neural network is also know in the art and is not discussed in detail.
- the processor 48 may be any processor capable of running the deep learning neural network.
- the processor 48 may be an accelerated ECM that is FPGA based or GPU based.
- An additional part of deployment includes calibration of the camera 44 or 2D imaging device on the machine 10 so that the distance between objects captured in the image and the machine can be determined or estimated.
- the machine 10 may include an inertial measurement unit (IMU)(not shown) which can be used to define a coordinate frame of the machine 10.
- IMU inertial measurement unit
- An extrinsic calibration of the camera 44 with respect to the coordinate frame of the machine 10 can be measured using a device such as a Universal Total Station.
- the system 40 is configured to autonomously detect piles.
- the camera 44 or 2D imaging device generates images and communicates the images to the processor 48.
- the images are passed to the trained neural network in the processor 48.
- the trained neural network examines each image received and detects whether a material pile 34 is shown in any given image frame.
- centroid (center) of the material pile 34 is calculated by averaging the position of all the pixels that belong to the material pile 34.
- the position can be transformed using the camera projection matrix and the extrinsic calibration, into the machine frame, so that the autonomy system would be able to know the position and location of the material pile 34, with respect to the machine 10
- FIG. 6 illustrates another exemplary method 200 used by the machine 10 for detecting the material pile 34.
- the method 100 uses a system including the 3D sensing device 46 and the processor 48 on the machine 10 with support to perform accelerated computing.
- the method 200 uses a geometric technique for detecting a pile based on a 3D point cloud.
- a 3D point cloud is a set of data points (XYZ coordinates) in space.
- the method 200 includes inputting a 3D point cloud to the processor 48, at step 202.
- the 3D point cloud can come directly from the 3D sensing device 46, such as a stereo camera, LIDAR, etc. and/or can come from any type of 3D mapping space representation, such as an octmap or height map.
- the 3D mapping space representation may be generated remote from the machine 10 and uploaded to the processor 48, such as for example, from an off-board entity via the communication device 52
- FIG. 7 illustrates an example of a 3D point cloud 300 of same scene as shown in annotated image 120.
- the 3D point cloud depicts the base ground surface 121, the first material pile 122, the second material pile 124, the person standing 126, and the boulder 128.
- a ground surface estimation can be performed on the 3D point cloud, at step 204.
- GSE is known in the art. Any suitable GSE technique or algorithm may be used.
- a GSE algorithm identifies and extracts the major ground surface from a 3D point cloud via a series of mathematical and geometric calculations such as normal calculation, unevenness estimation, plane fitting, and random sampling consensus (RANSAC).
- step 206 the output of the GSE assigns a label to the points which are considered ground and a different label to the points which are considered non-ground (i.e., everything above or below the ground surface).
- FIG. 8 is an example of the output of the GSE 302 for the 3D point cloud of FIG. 7.
- the ground points 304 are shown in a lighter shade or color, while the non-ground points 306 are shown in a darker shade or color.
- step 208 with each point classified as ground and non ground, only the non-ground points are selected for further processing.
- the non-ground points are then grouped into clusters based on a point-to-point proximity metric (i.e., non-ground points in close proximity to each other are grouped as part of the same cluster).
- Clustering algorithms based on a proximity metric are known in the art. Any suitable clustering algorithm may be used to group the non-ground points into clusters.
- step 210 for each cluster, the processor 48 calculates a histogram of the normalized height of the points from the ground surface. For example, the lowest point in the cluster is assigned a value of zero and the highest point in the cluster is assigned a value of 1. All the points with heights between the highest and lowest point are assigned numeric values between zero and 1 in proportion to their height relative to the highest and lowest points. A histogram of the normalized height data is then created.
- step 212 the processor 48 compares the histogram to a pile descriptor template that has been previously created and is accessible by the processor 48. Regarding the pile descriptor template, material piles generally have a similar shape regardless of the type of material.
- the normalized height histogram of one material pile is generally similar to the normalized height histogram of other material piles.
- a generalized normalized histogram of a material pile is created from previous data collected on other piles and made available to the processor 48.
- the generalized pile histogram is used as a representation of most piles (i.e., a pile descriptor template).
- multiple generalized pile histograms can be created and each can be compared to the histogram of the normalized height of the points from the ground surface for each cluster.
- a similarity score is a way to represent how confident the algorithm is that the normalized height histogram matches the descriptor template.
- a similarity score may be a numerical value indicating how well the normalized histogram matches the pile descriptor template.
- the similarity score can be a value between 0 to 100, where 0 represents not matching at all, while 100 represents a perfect match. Calculating a similarity score between data sets is known in the art. Any suitable methodology for calculating a similarity score between the histogram of the normalized height of the points from the ground surface for each cluster and the pile descriptor template may be used.
- the processor 48 identifies the clusters with the highest similarity scores and treats those clusters as being material piles.
- a similarity score threshold may be set and the processor compares the similarity score of each cluster with the similarity score threshold. Any cluster having a similarity score at or greater than threshold is considered by the processor 48 to be a material pile.
- the location of the material pile 34 is determined. The location of the piles may be determined in any suitable manner. For example, the centroid (center) of the pile can be calculated and then transformed, if needed, into the machine or world frame.
- each of the points has 3D cartesian coordinates (x,y,z) in space that allow the location of these points to be readily identified.
- a transformation to the world frame or machine frame may be needed. If the 3D point data is in the world frame or machine frame, no transformation would need to be done to allow the location of the piles to be determined. If, however, the 3D point data is in the sensing device frame, then an extrinsic calibration with respect to the coordinate frame of the machine 10 can be measured using a device such as a Universal Total Station.
- the machine 10 may utilize a combination or fusion of both the method 100 which involves inputting 2D images into the deep learning neural network trained to perform the task of pile detection and the method 200 which uses a geometric technique for detecting a pile based on a 3D point cloud.
- the one or more sensors 42 one the machine 10 may include both a camera 44 or some other 2D imaging device and a 3D sensing device 46, such as LIDAR or a stereo camera. If the 3D sensing device 46 is a stereo camera, it automatically proves the 3D point cloud and the 2D image as one message and can be used for both methods.
- each detection method 100, 200 is run independently with any material piles detected by either method being treated as a pile by the machine 10.
- each detection method 100, 200 is assigned a confidence weighting and the output of the two methods 100, 200 are fused.
- any pile detected in the method 200 can be projected back into the image, and the two predictions can be fused to improve the prediction accuracy.
- the different pile detection methods 100, 200 may use different sensors (e.g., a mono camera and a LIDAR), and thus, may provide detected piles in the respective sensor coordinate frame.
- a coordinate transformation to the pile detected in one image sensor e.g.
- a 3D image sensor can be performed to place the points onto the image frame of the 2D imaging device.
- the pile detected by the 3D sensing device is overlaid on the pile detected by the 2D imaging device and it can be determined, using methods known in the art, that the two sensors are detecting the same pile.
- the systems and methods for detecting piles may be used on any autonomous machine, such as a wheel loader, an excavator, an off- highway truck, and a dozer, that may encounter material piles.
- Many autonomous systems currently utilize both a camera and a 3D sensing device, such as LIDAR.
- the disclosed systems and methods provide an effective way to detect materials piles using the sensors already on most autonomous systems along with a processor 48 on the machine 10 capable of performing the accelerated computing required for utilizing the geometric technique for detecting a pile based on a 3D point cloud of the method 200 of FIG. 5 and for running the trained neural network and image comparison of the method 100 of FIG. 2.
- the pile detection methods disclosed provide benefits to productivity measurement and monitoring.
- Site supervisors are interested in knowing how much material the pile contains and how much material has been moved from the pile.
- Traditional methods involve hand and paper calculations and estimates.
- perception systems automatically calculate the volume of piles and the amount of material moved. Before these metrics can be calculated, the pile must first be automatically detected. While the system is illustrated as implemented on an autonomous wheel loader, it may be used on other types of machines, autonomous or not.
Landscapes
- Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Data Mining & Analysis (AREA)
- Multimedia (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Remote Sensing (AREA)
- Radar, Positioning & Navigation (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Biology (AREA)
- Electromagnetism (AREA)
- General Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Geometry (AREA)
- Length Measuring Devices By Optical Means (AREA)
- Image Analysis (AREA)
Abstract
Description
Claims
Priority Applications (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
DE112021002328.4T DE112021002328T5 (en) | 2020-05-11 | 2021-04-20 | METHOD AND SYSTEM FOR DETECTING A STOCK STOCK |
CA3177321A CA3177321A1 (en) | 2020-05-11 | 2021-04-20 | Method and system for detecting a pile |
AU2021272867A AU2021272867A1 (en) | 2020-05-11 | 2021-04-20 | Method and system for detecting a pile |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US16/871,419 | 2020-05-11 | ||
US16/871,419 US11462030B2 (en) | 2020-05-11 | 2020-05-11 | Method and system for detecting a pile |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2021231043A1 true WO2021231043A1 (en) | 2021-11-18 |
Family
ID=78412835
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/US2021/028067 WO2021231043A1 (en) | 2020-05-11 | 2021-04-20 | Method and system for detecting a pile |
Country Status (5)
Country | Link |
---|---|
US (1) | US11462030B2 (en) |
AU (1) | AU2021272867A1 (en) |
CA (1) | CA3177321A1 (en) |
DE (1) | DE112021002328T5 (en) |
WO (1) | WO2021231043A1 (en) |
Families Citing this family (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP4185990A1 (en) * | 2020-08-21 | 2023-05-31 | Waymo Llc | Object-centric three-dimensional auto labeling of point cloud data |
CN114325755B (en) * | 2021-11-26 | 2023-08-01 | 江苏徐工工程机械研究院有限公司 | Retaining wall detection method and system suitable for automatic driving vehicle |
CN115032608B (en) * | 2022-08-15 | 2022-11-01 | 杭州宇称电子技术有限公司 | Ranging sensor data optimization method and application thereof |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20180236664A1 (en) * | 2017-02-21 | 2018-08-23 | Fanuc Corporation | Workpiece pick up system |
US20190005667A1 (en) * | 2017-07-24 | 2019-01-03 | Muhammad Zain Khawaja | Ground Surface Estimation |
US20190026531A1 (en) * | 2017-07-21 | 2019-01-24 | Skycatch, Inc. | Determining stockpile volume based on digital aerial images and three-dimensional representations of a site |
US20190352110A1 (en) * | 2016-10-20 | 2019-11-21 | Intelligrated Headquarters, Llc | 3d-2d vision system for robotic carton unloading |
US20200063399A1 (en) * | 2018-08-22 | 2020-02-27 | Deere & Company | Control system for a work machine |
Family Cites Families (12)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7298895B2 (en) * | 2003-04-15 | 2007-11-20 | Eastman Kodak Company | Method for automatically classifying images into events |
US9222771B2 (en) * | 2011-10-17 | 2015-12-29 | Kla-Tencor Corp. | Acquisition of information for a construction site |
US9990712B2 (en) * | 2015-04-08 | 2018-06-05 | Algotec Systems Ltd. | Organ detection and segmentation |
US9587369B2 (en) | 2015-07-02 | 2017-03-07 | Caterpillar Inc. | Excavation system having adaptive dig control |
TWI581213B (en) * | 2015-12-28 | 2017-05-01 | 力晶科技股份有限公司 | Method, image processing system and computer-readable recording medium for item defect inspection |
US9702115B1 (en) | 2016-01-08 | 2017-07-11 | Caterpillar Inc. | Autonomous method for detecting a pile |
WO2018201180A1 (en) | 2017-05-02 | 2018-11-08 | PETRA Data Science Pty Ltd | Automated, real time processing, analysis, mapping and reporting of data for the detection of geotechnical features |
US10593108B2 (en) * | 2017-10-31 | 2020-03-17 | Skycatch, Inc. | Converting digital aerial images into a three-dimensional representation utilizing processing clusters |
JP7211735B2 (en) * | 2018-08-29 | 2023-01-24 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカ | CONTRIBUTION DETERMINATION METHOD, CONTRIBUTION DETERMINATION DEVICE AND PROGRAM |
US10872269B2 (en) * | 2018-10-26 | 2020-12-22 | Volvo Car Corporation | Methods and systems for the fast estimation of three-dimensional bounding boxes and drivable surfaces using LIDAR point clouds |
CN110053943B (en) | 2019-05-21 | 2021-04-23 | 精英数智科技股份有限公司 | Monitoring method for artificial intelligence video identification belt coal pile |
GB2591332B (en) * | 2019-12-19 | 2024-02-14 | Motional Ad Llc | Foreground extraction using surface fitting |
-
2020
- 2020-05-11 US US16/871,419 patent/US11462030B2/en active Active
-
2021
- 2021-04-20 WO PCT/US2021/028067 patent/WO2021231043A1/en active Application Filing
- 2021-04-20 DE DE112021002328.4T patent/DE112021002328T5/en active Pending
- 2021-04-20 AU AU2021272867A patent/AU2021272867A1/en active Pending
- 2021-04-20 CA CA3177321A patent/CA3177321A1/en active Pending
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20190352110A1 (en) * | 2016-10-20 | 2019-11-21 | Intelligrated Headquarters, Llc | 3d-2d vision system for robotic carton unloading |
US20180236664A1 (en) * | 2017-02-21 | 2018-08-23 | Fanuc Corporation | Workpiece pick up system |
US20190026531A1 (en) * | 2017-07-21 | 2019-01-24 | Skycatch, Inc. | Determining stockpile volume based on digital aerial images and three-dimensional representations of a site |
US20190005667A1 (en) * | 2017-07-24 | 2019-01-03 | Muhammad Zain Khawaja | Ground Surface Estimation |
US20200063399A1 (en) * | 2018-08-22 | 2020-02-27 | Deere & Company | Control system for a work machine |
Also Published As
Publication number | Publication date |
---|---|
DE112021002328T5 (en) | 2023-02-09 |
CA3177321A1 (en) | 2021-11-18 |
AU2021272867A1 (en) | 2022-12-15 |
US20210350114A1 (en) | 2021-11-11 |
US11462030B2 (en) | 2022-10-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11462030B2 (en) | Method and system for detecting a pile | |
US11709495B2 (en) | Systems and methods for transfer of material using autonomous machines with reinforcement learning and visual servo control | |
US11378964B2 (en) | Systems and methods for autonomous movement of material | |
JP7166108B2 (en) | Image processing system, display device, image processing method, trained model generation method, and training data set | |
US9322148B2 (en) | System and method for terrain mapping | |
US9014925B2 (en) | System and method for determining a ripping path | |
US11126188B2 (en) | System and method for maintaining a work surface at a worksite | |
US9487929B2 (en) | Systems and methods for adjusting pass depth in view of excess materials | |
JP7365122B2 (en) | Image processing system and image processing method | |
AU2017276225B2 (en) | Systems and methods for preparing a worksite for additive construction | |
US9703290B1 (en) | Method for operating machines on worksites | |
CN111857124A (en) | System and method for machine control | |
WO2021002245A1 (en) | System including work machine and work machine | |
Guan et al. | Ttm: Terrain traversability mapping for autonomous excavator navigation in unstructured environments | |
AU2014277669A1 (en) | Terrain mapping system using virtual tracking features | |
JP2021188258A (en) | System for shovel | |
WO2021002249A1 (en) | Manufacturing method of trained work classification estimation model, data for training, method executed by computer, and system including work machine | |
US20230243130A1 (en) | Excavation plan creation device, working machine, and excavation plan creation method | |
US11118453B2 (en) | System and method of layering material | |
US10377125B2 (en) | Control systems and methods to optimize machine placement for additive construction operations | |
KR20210000593A (en) | Apparatus for generating environment data neighboring construction equipment and construction equipment including the same | |
US11879231B2 (en) | System and method of selective automation of loading operation stages for self-propelled work vehicles | |
US20220002970A1 (en) | Excavator | |
EP3825477A1 (en) | Environment cognition system for construction machinery | |
KR20220140297A (en) | Sensor fusion system for construction machinery and sensing method thereof |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 21803035 Country of ref document: EP Kind code of ref document: A1 |
|
ENP | Entry into the national phase |
Ref document number: 3177321 Country of ref document: CA |
|
ENP | Entry into the national phase |
Ref document number: 2021272867 Country of ref document: AU Date of ref document: 20210420 Kind code of ref document: A |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 21803035 Country of ref document: EP Kind code of ref document: A1 |