EP4666040A1

EP4666040A1 - Method of mapping bulk material in a bin using machine learning

Info

Publication number: EP4666040A1
Application number: EP24755788.7A
Authority: EP
Inventors: Nathan James Hoel; Benjamin Mark WILLARD; Lei LEI
Original assignee: Binsentry Inc
Current assignee: Binsentry Inc
Priority date: 2023-02-17
Filing date: 2024-01-23
Publication date: 2025-12-24
Also published as: CN120712460A; WO2024168419A1; IL322744A; AU2024222986A1; JP2026506050A; KR20250166118A

Abstract

Disclosed is a method of mapping a bulk material in a storage bin having a wall for containing the bulk material. The method entails capturing a plurality of training images of various topologies of bulk material in the storage bin using a depth-map-capturing sensor such as a time-of-flight camera, each of the plurality of training images being defined by an array of pixels. An artificial neural network is trained on the plurality of training images to develop an artificial intelligence model capable of discriminating between the wall and a surface of the bulk material. The method further entails capturing one or more images of bulk material to be mapped in the bin or another similarly shaped bin using the depth-map-capturing sensor (e.g. time-of-flight camera) and then mapping the bulk material in the bin or another similarly shaped bin using the artificial intelligence model to discriminate between the wall and the bulk material in the bin or another similarly shaped bin.

Description

METHOD OF MAPPING BULK MATERIAL IN A BIN USING MACHINE LEARNING

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application claims priority from US Provisional Patent Application 63/485,701 filed February 17, 2023.

TECHNICAL FIELD

[0002] The present invention relates generally to technologies for sensing bulk material in a bulk storage container and, more particularly, to techniques for mapping a topology of the bulk material in the bulk storage container.

BACKGROUND

[0003] Various technologies are known for monitoring and/or measuring bulk material stored inside of a bulk storage container. In the agricultural industry, it is highly desirable to monitor various attributes of grain, seed or other such agricultural products within a storage bin. For example, it is known to measure temperature and/or moisture of grain or seed to prevent spoilage and to know when the grain is ready to be delivered to an end-user. It is also highly desirable to accurately measure the volume of grain or feed within a bin. Various LIDAR and camera-based sensor technologies have been disclosed in order to map a topology of a bulk material within a bin. An example of a bin level monitoring system is disclosed in W02020102879A1 , which is hereby incorporated by reference. The bin level monitoring system of W02020102879A1 comprises an optical sensor such as a LIDAR sensor or a time-of-flight (ToF) camera for sensing a feed level inside a feed bin, a circuit board communicatively connected to the sensor for receiving a level signal from the sensor and for processing the level signal to generate bin level data, a battery for powering the circuit board and sensor, an enclosure for enclosing the circuit board and a radio transmitter for transmitting the bin level data.

[0004] Discriminating between the wall of the container and the upper surface of the bulk material remains a technical challenge that has until now limited the precision of these prior-art sensor technologies. A technical solution to this problem would be highly desirable in order to provide more accurate monitoring of bulk material in bins.

SUMMARY

[0005] In general, the present invention provides a method and system for mapping a bulk material in a bin by using machine learning to develop an artificial intelligence model capable of discriminating the wall of the bin from the bulk material.

[0006] One inventive aspect of the disclosure is a method of mapping a bulk material in a storage bin having a wall for containing the bulk material. The method entails capturing a plurality of training images of various topologies of bulk material in the storage bin using a depth-map-capturing sensor such as a time-of-flight camera, each of the plurality of training images being defined by an array of pixels. An artificial neural network is trained on the plurality of training images to develop an artificial intelligence model capable of discriminating between the wall and a surface of the bulk material. The method further entails capturing one or more images of bulk material to be mapped in the bin or another similarly shaped bin using the depth-map-capturing sensor (e.g. time- of-flight camera) and then mapping the bulk material in the bin or another similarly shaped bin using the artificial intelligence model to discriminate between the wall and the bulk material in the bin or another similarly shaped bin.

[0007] Another inventive aspect of the disclosure is a system for mapping a bulk material in a storage bin having a wall for containing the bulk material. The system includes a depth-map-capturing sensor such as a time-of-flight camera for capturing a plurality of training images of various topologies of bulk material in the storage bin, each of the plurality of training images being defined by an array of pixels. The system includes a processor for training an artificial neural network on the plurality of training images to develop an artificial intelligence model capable of discriminating between the wall and a surface of the bulk material. The depth-map-capturing sensor (e g. time-of-flight camera) subsequently captures one or more images of bulk material to be mapped in the bin or another similarly shaped bin. The processor is further configured to map the bulk material in the bin or another similarly shaped bin using the artificial intelligence model to discriminate between the wall and the bulk material in the bin or another similarly shaped bin.

[0008] The foregoing presents a simplified summary of the invention in order to provide a basic understanding of some aspects of the invention. This summary is not an exhaustive overview of the invention. It is not intended to identify essential, key or critical elements of the invention or to delineate the scope of the invention. Its sole purpose is to present some concepts in a simplified form as a prelude to the more detailed description that is discussed later. Other aspects of the invention are described below in relation to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Further features and advantages of the present technology will become apparent from the following detailed description, taken in combination with the appended drawings, in which:

[0010] FIG. 1 is a schematic depiction of a system for mapping a bulk material in a storage bin in accordance with an embodiment of the present invention.

[0011] FIG. 2 is a flowchart depicting a method of mapping a bulk material in a storage bin in accordance with an embodiment of the present invention.

[0012] FIG. 3 is a flowchart depicting optional further steps of the method.

[0013] FIG. 4 is an example of an image where the pixels represent amplitude data.

[0014] FIG. 5 is an example of an image where the pixels represent distance data.

[0015] It will be noted that throughout the appended drawings, like features are identified by like reference numerals.

DETAILED DESCRIPTION

[0016] FIG. 1 schematically depicts a novel system for mapping a bulk material in a storage bin in accordance with an embodiment of the present invention. In the embodiment depicted in FIG. 1 , the system, which is denoted by reference numeral 10, is designed to map a bulk material 20 in a storage bin 30 having a wall 32 for containing the bulk material. The storage bin 30 may be any container, silo, receptacle or storage structure used to store a solid particulate or bulk material such as for example grain, seeds, or other such material whether agricultural or not. The system 10 includes a depth-map-capturing sensor such as a time-of-flight camera 40 for capturing a plurality of training images of various topologies of bulk material in the storage bin, each of the plurality of training images being defined by an array of pixels. Camera or sensor image data from the depth-map-capturing sensor (e.g. time-of-flight camera) 40 may be transmitted to a server or other computing device for image processing as further explained below. The camera or sensor image data may be transmitted wirelessly, e.g. via a cellular data network using for example an antenna 50 on the bin. Alternatively, the camera or sensor image data may be transmitted via any other wired or wireless telecommunication means. In this particular example, the camera or sensor image data is transmitted to a cellular base transceiver station 60 that has a gateway to the internet 70. The data packets of the camera or sensor image data are then transmitted to the server or other computing device for image processing. In the illustration of FIG. 1 , a server cluster 80 or cloud implementation is shown although it will be appreciated that a single server or computing device may be used in a simpler implementation. The system 10 thus includes at least one server or computing device 100 each having a processor or CPU 110 (or multiple processors or CPUs) for training an artificial neural network 150 by using the plurality of training images to develop an artificial intelligence model capable of discriminating between the wall and a surface of the bulk material. For completeness, it will be understood that the server or computing device 100 also includes a memory 120, communication port 130 and input/output (I/O) device 140 which cooperate with the processor 100 to implement the artificial neural network for image processing. Once the neural network has been duly trained using training images, the time-of-flight (TOF) camera 40 subsequently captures one or more images of a volume of bulk material to be mapped in the bin. The processor is further configured to map the volume of bulk material in the bin using the artificial intelligence model to discriminate between the wall and the bulk material in the bin or any other similarly shaped bin. It will be appreciated that in alternate embodiments the computations for training the model need not be performed by the computing device 100; rather, in alternate embodiments, it is possible to perform these computations in whole or in part in a processor or computer located in the bin. Alternatively, an edge computing paradigm may be employed to perform computations in whole or in part by computing resources closer to the bin. Still alternatively, a cloud computing paradigm may be employed in whole or in part for training the model.

[0017] In one embodiment, the system uses the TOF camera 40 to measure the surface of the bulk material and the bin wall. The system uses machine learning (i.e. an artificial intelligence) to identify the bulk material and the bin wall. In one embodiment, the processor is configured to computationally remove the bin wall and to fit the surface of the bulk material to a 3D model of the bin.

[0018] In one embodiment, the depth-map-capturing sensor is a time-of-flight (TOF) camera 40. The TOF camera 40 may be any suitable device that emits modulated light to illuminate the feed in the bin and the inner surface of the wall of the bin. The TOF camera 40 is configured to capture the reflected light from the feed in the bin and the inner surface of the wall of the bin. The TOF camera 40 is configured to exploit a phase shift between the illumination and the reflection. Measuring this phase shaft enables the TOF camera 40 to calculate distance. The TOF camera 40 may use, for example, a solid-state laser or a light-emitting diode (LED) operating in the near-infrared range (i.e. around 850nm). The TOF camera 40 may include an imaging sensor having a plurality of pixels in an array that respond to the reflected infrared (IR) light to generate an electrical current for each pixel. To detect phase shifts between the illumination and the reflection, the light source may be pulsed or otherwise modulated by a continuous-wave such as a sinusoidal wave or a square wave. The TOF camera 40 may for example have its own illumination source in the form a matrix of LEDs that emit modulated infrared light. Based on the detection of the reflected waves, two images can be formed: (i) a distance image which is computed based on the phase shift between the emitted and reflected signals, and (ii) an amplitude image which is computed based on the amplitude of the reflected signal at each pixel location. Distance can be measured for every pixel in a 2D addressable array to produce a depth map. A depth map may be conceptualized as a collection of 3D points in which the brighter the intensity of the pixel, the closer is the location of the corresponding physical point in 3D space. Alternatively, a depth map can be rendered in a three-dimensional space as a collection of points (a point-cloud). The 3D points can be mathematically connected to form a mesh onto which a surface can be mapped.

[0019] This system enables a novel method of mapping a bulk material in a storage bin as shown in the flowchart in FIG. 2. The method 200 entails a step 210 of capturing a plurality of training images of various topologies of bulk material in the storage bin using a depth-map-capturing sensor such as a time-of-flight camera, each of the plurality of training images being defined by an array of pixels. The method 100 further entails a step 220 of training an artificial neural network on the plurality of training images to develop an artificial intelligence model capable of discriminating between the wall and a surface of the bulk material. The method 200 further entails a step 230 of capturing one or more images of bulk material to be mapped in the bin or another similarly shaped bin using the depth-map-capturing sensor (e.g. time-of-flight camera). The method 200 further entails a step 240 of mapping the bulk material in the bin or another similarly shaped bin using the artificial intelligence model to discriminate between the wall and the bulk material in the bin or another similarly shaped bin. For clarity, it will be understood that the training of the model may be performed using a first bin and then the model may be used in a second bin to identify the topology of the bulk material in the second bin. In a variant, the model may be trained using a first set of bins and then the model can be deployed for use in one or more other bins, i.e. it may be deployed for use in a single bin or in a second set of bins.

[0020] In one embodiment, as shown in FIG. 3, the method may include optional further steps. The step 240 of mapping the volume of the bulk material may include a step 250 of, after discriminating the wall of the bin from the bulk material in the bin or another similarly shaped bin, removing the pixels representing the wall and a further step 260 of fitting the pixels representing the bulk material to a predetermined 3D model of the bin or another similarly shaped bin. [0021] In one embodiment, training the artificial neural network on the plurality of training images is performed by using both amplitude data and distance data for the pixels. In another embodiment, training the artificial neural network on the plurality of training images is performed by using only amplitude data for the pixels. In another embodiment, training the artificial neural network on the plurality of training images is performed by using only distance data for the pixels.

[0022] The method may be performed using a deep convolutional neural network (DCNN) although other types of artificial neural networks may be used to achieve similar results. The deep convolutional neural network may be, for example, a Region-based Convolutional Neural Network (R-CNN), a Fast R-CNN, GoogleNet, VGGNet, ResNet (Residual Neural Network). The DCNN employs layering to process depth and amplitude data of the captured TOF camera images. The deep convolutional neural network receives training images (i.e. human-marked images acting as a ground truth) as an input and uses them to train a classifier. The DCNN typically has four types of layers: convolution, activation, pooling, and fully connected. In the convolution layer, the DCNN applies a convolution filter to the image to extract features of the image. The convolution involves multiplying weights with inputs from the neural network. During this multiplication, a kernel (for a 2D array of weights) or a filter (for a 3D matrix) passes over the image. During the convolution, each filter multiplies the weights with different input values which are then summed to yield a specific value for each filter position. The convolution maps are then processed by a nonlinear activation layer, e.g. a Rectified Linear Unit (ReLu), which replaces any negative values in the images with zeros. In the pooling layer, the image size is sequentially reduced. For example, for each group of 4 pixels, the pixel having the maximum value is retained (this is called max pooling), or only the average is retained (average pooling). After multiple convolutions and pooling, the result is a multi-layer perceptron, i.e. a fully connected neural network. The DCNN can then receive a new captured image from the TOF camera containing the amplitude and depth data for the pixels of the new captured image. The fully connected neural network has thus been trained to can then recognize whether the pixels represent the surface of bulk material or the bin wall. [0023] FIG. 4 is an example of an image captured by the TOF camera where the pixels represent amplitude data. Using the amplitude data, the DCNN is enable to discriminate between the wall of the bin 310 and the bulk material 300.

[0024] FIG. 5 is an example of an image captured by the TOF camera where the pixels represent distance data. Using the depth data, the DCNN is enable to discriminate between the wall of the bin 310 and the bulk material 300.

[0025] The bin in the illustrated embodiment is a cylindrical bin having a cylindrical wall defining a fixed radius of curvature. It will be appreciated that the inventive concept may be adapted or modified to be used with bins having other shapes.

[0026] In one embodiment, the artificial intelligence model comprises a plurality of material-specific sub-models for different types of bulk material. For example, one Al submodel may be developed for grain and another Al sub-model may be developed for seeds. The processor may receive data indicative of bulk material type and then apply the material-specific sub-model to the images in order to better discriminate the wall from the bulk material. The bulk material type may be characterized by the material’s color, granularity, reflectivity, or other properties. The bulk material type may also be characterized by its angle of repose, e.g. a bulk material property indicative of how a particular bulk material piles up as a function of its particle shape and coefficient of friction.

[0027] The method(s) can be implemented in hardware, software, firmware or as any suitable combination thereof. That is, if implemented as software, the computer-readable medium comprises instructions in code which when loaded into memory and executed on a processor of a computing device causes the computing device to perform any of the foregoing method steps. These method steps may be implemented as software, i.e. as coded instructions stored on a computer readable medium which performs the foregoing steps when the computer readable medium is loaded into memory and executed by the microprocessor of the mobile device. A computer readable medium can be any means that contain, store, communicate, propagate or transport the program for use by or in connection with the instruction execution system, apparatus or device. The computer- readable medium may be electronic, magnetic, optical, electromagnetic, infrared or any semiconductor system or device. For example, computer executable code to perform the methods disclosed herein may be tangibly recorded on a computer-readable medium including, but not limited to, a floppy-disk, a CD-ROM, a DVD, RAM, ROM, EPROM, Flash Memory or any suitable memory card, etc. The method may also be implemented in hardware. A hardware implementation might employ discrete logic circuits having logic gates for implementing logic functions on data signals, an application-specific integrated circuit (ASIC) having appropriate combinational logic gates, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0028] For the purposes of interpreting this specification, when referring to elements of various embodiments of the present invention, the articles “a”, “an”, “the” and “said” are intended to mean that there are one or more of the elements. The terms “comprising”, “including”, “having”, “entailing” and “involving”, and verb tense variants thereof, are intended to be inclusive and open-ended by which it is meant that there may be additional elements other than the listed elements.

[0029] This new technology has been described in terms of specific implementations and configurations which are intended to be exemplary only. Persons of ordinary skill in the art will appreciate that many obvious variations, refinements and modifications may be made without departing from the inventive concepts presented in this application. The scope of the exclusive right sought by the Applicant(s) is therefore intended to be limited solely by the appended claims.

Claims

CLAIMS:

1 . A method of mapping a bulk material in a storage bin having a wall for containing the bulk material, the method comprising: capturing a plurality of training images of various topologies of bulk material in the storage bin using a depth-map-capturing sensor, each of the plurality of training images being defined by an array of pixels; training an artificial neural network on the plurality of training images to develop an artificial intelligence model capable of discriminating between the wall and a surface of the bulk material; capturing one or more images of bulk material to be mapped in the bin using the depth-map-capturing sensor; mapping the bulk material in the bin using the artificial intelligence model to discriminate between the wall and the bulk material in the bin or in another similarly shaped bin.

2. The method of claim 1 wherein mapping the volume of the bulk material comprises: after discriminating the wall of the bin from the volume of the bulk material in the bin or another similarly shaped bin, removing the pixels representing the wall; and fitting the pixels representing the bulk material to a predetermined 3D model of the bin or another similarly shaped bin.

3. The method of claim 1 wherein training the artificial neural network on the plurality of training images is performed by using both amplitude data and distance data for the pixels.

4. The method of claim 2 wherein training the artificial neural network on the plurality of training images is performed by using both amplitude data and distance data for the pixels.

5. The method of claim 1 wherein training the artificial neural network on the plurality of training images is performed by using amplitude data for the pixels.

6. The method of claim 1 wherein training the artificial neural network on the plurality of training images is performed by using distance data for the pixels.

7. The method of claim 1 wherein the neural network is a deep convolutional neural network.

8. The method of claim 1 wherein the depth-map-capturing sensor is a time-of-flight (TOF) camera.

9. The method of claim 1 wherein the bin is a cylindrical bin having a cylindrical wall defining a fixed radius of curvature.

10. The method of claim 1 wherein the artificial intelligence model comprises a plurality of material-specific sub-models for different types of bulk material.

11. A system for mapping a bulk material in a storage bin having a wall for containing the bulk material, the system comprising: a depth-map-capturing sensor for capturing a plurality of training images of various topologies of bulk material in the storage bin, each of the plurality of training images being defined by an array of pixels; a processor for training an artificial neural network on the plurality of training images to develop an artificial intelligence model capable of discriminating between the wall and a surface of the bulk material; wherein the depth-map-capturing sensor subsequently captures one or more images of bulk material to be mapped in the bin or in another similarly shaped bin; wherein the processor is further configured to map the bulk material in the bin or another similarly shaped bin using the artificial intelligence model to discriminate between the wall and the bulk material in the bin or another similarly shaped bin.

12. The system of claim 11 wherein the processor is configured to map the bulk material by: after discriminating the wall of the bin from the bulk material in the bin or another similarly shaped bin, removing the pixels representing the wall; and fitting the pixels representing the bulk material to a predetermined 3D model of the bin or another similarly shaped bin.

13. The system of claim 11 wherein the processor trains the artificial neural network on the plurality of training images is performed by using both amplitude data and distance data for the pixels.

14. The system of claim 12 wherein the processor trains the artificial neural network on the plurality of training images is performed by using both amplitude data and distance data for the pixels.

15. The system of claim 11 wherein the processor trains the artificial neural network on the plurality of training images is performed by using amplitude data for the pixels.

16. The system of claim 11 wherein the processor trains the artificial neural network on the plurality of training images is performed by using distance data for the pixels.

17. The system of claim 11 wherein the neural network is a deep convolutional neural network.

18. The system of claim 11 wherein the depth-map-capturing sensor is a time-of- flight (TOF) camera.

19. The system of claim 11 wherein the bin is a cylindrical bin having a cylindrical wall defining a fixed radius of curvature.

20. The system of claim 11 wherein the artificial intelligence model comprises a plurality of material-specific sub-models for different types of bulk material.