EA045770B1 - DEVICE FOR ANALYSIS OF PAYLOAD TRANSPORTED IN A CARGO CONTAINER OF A VEHICLE - Google Patents

Description

Technical field

The present invention generally relates to the processing of acquired images and, more particularly, to the acquisition and processing of images of a vehicle for the analysis of payload carried in the cargo container of the vehicle.

State of the art

Large vehicles are typically used to transport a payload on an open platform of the vehicle. For example, in mining operations, quarry loaders and excavators load the ore payload onto a quarry truck for transport to the processing site. The nature and volume of the ore payload is often important because downstream processing can be designed for a payload that does not contain large boulders or other unwanted materials, such as broken teeth, that could cause equipment damage during subsequent payload processing. Another important aspect may be the degree of fragmentation or particle size distribution of the ore in the payload. In mining operations, due to the large size of the equipment involved in loading the mined ore and the high capital costs of it, payload monitoring is able to ensure the safe and/or efficient operation of the equipment involved. There remains a need for methods and systems for detecting boulders and estimating particle size distribution.

Disclosure of the invention

According to one disclosed aspect, a device is provided for analyzing a payload transported in a cargo container of a vehicle. The device contains a camera located with the ability to sequentially obtain images of vehicles crossing the camera's field of view. The device also includes at least one processor in communication with the camera, wherein said at least one processor is operatively configured to select at least one image from the sequentially acquired images in response to the likelihood that the vehicle and cargo cargo container are in within the field of view of the at least one image, and that the image data associated with the at least one image satisfies the criterion of suitability for further processing. Further processing includes processing, by said at least one processor, the selected image to identify a payload region of interest in the image and performing an analysis of the payload within the identified payload region of interest based on image data associated with the at least one image.

The at least one processor may be operative to select said at least one image by generating three-dimensional point cloud data for sequentially acquired images, determining a point density of said point cloud data, and comparing that point density to a threshold point density to determine whether the specified suitability criterion is met.

Said at least one processor may be operatively configured to pre-process said three-dimensional point cloud data for a selected image prior to analyzing the payload, said pre-processing may include at least one of the following: removing those coordinates from the cloud points that are below the expected height of the load-supporting base of the cargo container relative to the surrounding earth's surface; and removing the coordinates of those from the point cloud that are outside a subregion of the point cloud within the point cloud, wherein said subregion of the point cloud is smaller than the point cloud.

The at least one processor may be further configured to select, for further processing, one of the following: an image having the highest dot density, a first image having a dot density greater than a threshold dot density, and a plurality of images that have a dot density , exceeding the threshold point density if it is determined that a set of images meets the suitability criterion.

The processor may be further configured to generate a confidence level when processing a selected image to identify a payload region of interest, wherein the confidence level expresses a quantitative estimate of the confidence that the identified region of interest contains a payload, and wherein said confidence level may be used at least in part to determine whether the eligibility criterion is met for the selected image.

The at least one processor may be operative to select a plurality of images from sequentially acquired images, each of the plurality of images providing a different view of the payload, and the at least one processor may be operative to perform additional processing for each of the specified set of images for analysis

- 1 045770 payload.

The camera may be positioned above the vehicle and the field of view may be oriented downward to obtain images of the top surface of the payload, which is exposed due to the open top side of the cargo container.

The at least one processor may comprise an embedded processor in communication with the camera, the embedded processor may be operative to initiate transmission of image data for a selected image to a remote processor, and further processing is performed by the remote processor.

The embedded processor may include a WAN interface, and the embedded processor may be operative to download the selected image to the remote processor via the WAN.

The at least one processor may be operative to initiate generation of a warning signal in response to payload analysis results that meet a warning criterion.

The device may further comprise a warning annunciator operably configured to generate an audible or visual signal to alert the operator.

The at least one processor may be operative to process the first and second two-dimensional images from different perspective viewpoints to create a three-dimensional point cloud containing three-dimensional coordinates of the vehicle and the cargo container.

The camera may include one of the following: first and second image sensors that are offset to respectively obtain first and second two-dimensional images from different perspective viewpoints; and one image sensor operatively configured to obtain first and second images spaced apart in time such that movement of the vehicle while crossing the field of view provides different perspective viewpoints for the first and second images.

The at least one processor may be operatively configured to process one of the corresponding two-dimensional images to identify a payload region of interest in a two-dimensional representation and to generate payload analysis results by processing two-dimensional data within the payload region of interest, wherein said at least one one processor is operatively configured to use a three-dimensional point cloud to generate scaling information for payload analysis.

The at least one processor may be operatively configured to process the selected image to identify a payload region of interest using a trained neural network to produce output that localizes the region of interest within the selected image.

The apparatus may include a function for training a neural network using at least one of a set of images of representative shipping containers that have been previously labeled by a human, and an unsupervised learning algorithm implemented to extract patterns in the image data.

The neural network may comprise a masking region-based convolutional neural network.

The at least one processor may be operatively configured to process the selected image by at least one of: processing the image data to enhance shadow areas before performing payload analysis; performing cleansing on the selected image to correct image distortions caused by imaging optics associated with the camera before identifying a payload region of interest; and decimating the pixels of the original selected image to obtain a decimated image having a reduced number of pixels before identifying the payload region of interest.

The output of the neural network provides the ability to identify boundary pixels demarcating a payload region of interest within the decimated image, and performing payload analysis may include identifying corresponding boundary pixels in the original selected image and processing portions of the original selected image within the corresponding boundary pixels.

The at least one processor may be operative to determine the dimensions of a cargo container of a vehicle by one of the following: determining a vehicle identifier associated with the selected image and reading from a database parameters defining the dimensions of a cargo container for the identified vehicle; and performing further processing on the vehicle with the empty cargo container and determining dimensions of the cargo container based on the empty cargo container.

The at least one processor may be operatively configured to perform vehicle identification by one of the following: processing at least one of the sequential images to extract a vehicle identifier identified on the vehicle within the camera's field of view; receiving an identifier from a radio-frequency identification (RFID) sensor positioned to read an RFID tag located on the vehicle; and processing at least one of the acquired sequential images using a neural network that has been previously trained to generate a vehicle identifier based on the acquired image.

The processor may be operatively configured to perform payload analysis by determining payload volume by determining the payload height of an interior region of the cargo container based on three-dimensional coordinates for points within the payload region of interest and calculating the payload volume based on the payload height and determined dimensions of the cargo container.

The processor may be operative to perform analysis of the payload by identifying a foreign object within the payload.

The processor may be operative to identify a foreign object by processing infrared images of the payload, wherein said foreign object is identified by detecting electromagnetic radiation having wavelengths in the infrared range.

The processor may be operative to perform payload analysis by calculating a payload displacement, and the processor may further be operative to generate a load imbalance warning if the payload displacement exceeds a certain maximum payload displacement.

The processor may be operative to perform payload analysis by analyzing segmentations in a region of interest of the payload to determine the dimensions of distinguishable portions of the payload.

The processor may be operative to initiate generation of an alert signal in response to the at least one distinguishable portion exceeding a threshold size or being identified as a non-payload object.

The payload may contain a mined ore payload, and the segmentation analysis may include one of the following: fragmentation analysis, which serves to identify distinguishable parts as one of a rock part, a fine part, or a gap between parts, the distribution of the load within cargo container dimensions and moisture analysis, which serves to classify the level of moisture associated with the payload.

The vehicle may be one of the following: a dump truck, a rail truck, a barge, a mine cart, a load, haul, dump, or LHD vehicle, or a mine skip.

Other aspects and features will become apparent to those skilled in the art upon reading the following description of the specific embodiments disclosed in conjunction with the accompanying drawings.

Brief description of drawings

In the drawings which illustrate the disclosed embodiments, FIGS. 1A is a perspective view of a payload analysis apparatus according to the first disclosed embodiment;

in fig. 1B is a perspective view of a camera used in the apparatus of FIG. 1A;

in fig. 1C is a perspective view of a payload analysis apparatus according to yet another disclosed embodiment;

in fig. 1D is a perspective view of another embodiment of a camera that can be used in the device of FIG. 1A or 1C;

in fig. 1E is a perspective view of an underground work site according to yet another disclosed embodiment;

in fig. 1F is a perspective view of an embodiment of an unmanned aerial vehicle for mounting and positioning the camera of FIG. 1A;

in fig. 2 is a block diagram of a payload analysis system comprising elements of the apparatus shown in FIG. 1;

in fig. 3 is a block diagram depicting code blocks for controlling an embedded processor of the system shown in FIG. 2, to provide image acquisition functions;

in fig. 4 shows an example of an image captured by the camera shown in FIG. 1B;

in fig. 5 is a block diagram illustrating code blocks for controlling the embedded processor to determine whether an eligibility criterion has been met;

- 3 045770 in Fig. 6 is an additional perspective view of the device shown in FIG. 1A;

in fig. 7 is a block diagram depicting code blocks for controlling the remote processor circuit of the system shown in FIG. 2, to identify the payload region of interest;

in fig. 8 is an example image showing a vehicle mask, a cargo container mask, and a payload mask;

in fig. 9 shows an example of fragmentation analysis results for the payload image shown in FIG. 8;

in fig. 10 shows a functional diagram of a neural network architecture for identifying a payload region of interest;

in fig. 11 is a block diagram depicting code blocks for controlling remote processing circuitry to perform alternative payload analysis; and in fig. 12 is a screenshot of a dashboard for displaying results according to one disclosed embodiment.

Carrying out the invention

Referring to FIG. 1A, in which the payload analysis apparatus of the first disclosed embodiment is shown generally at 100. The payload 102 is transported in a cargo container 104 of a vehicle 106 passing under an arch 108 located at a work site 126, such as a mine or quarry. In this embodiment, the illustrated vehicle 106 is a mining truck, and the payload 102 contains excavated ore from a work site 126, such as a mine or quarry. In other embodiments, the cargo container may be associated with another type of vehicle, such as a railcar, barge, or other marine vehicle. Alternatively, the shipping container 104 may be a cart or skip, such as those used in a quarry or underground mining operation. The device 100 includes a camera 110 mounted on the arch 108 and positioned to sequentially acquire images of vehicles crossing the camera's field of view 112. In this embodiment, the camera 110 is mounted above the vehicle and the field of view 112 is oriented downward to capture images of the top surface of the payload 102, which is exposed due to the open top side of the cargo container 104. In the illustrated embodiment, device 100 includes lamps 114 and 116 directed downwards to illuminate the field of view 112. Lamps 114 and 116 may be implemented using LED light sources with increased structural strength.

In this embodiment, the device 100 further includes a distribution box 118 mounted on the vertical arch member 120. Power, signal and control cables associated with camera 110 (not shown in FIG. 1A) are routed back to junction box 118. Switch box 118 also supplies power to lamps 114 and 116. In the illustrated embodiment, device 100 further includes RFID - an RFID reader 122 for reading an RFID tag 124 attached to the vehicle 106. A code associated with the RFID tag 124 on the vehicle 106 can be read by the RFID reader 122 to identify the vehicle 106. The RFID tag 124 may be encoded with an identifier that is uniquely associated with a specific vehicle operating at the work site 126. In some embodiments, vehicle 106 may be an automated driverless vehicle. For example, self-driving vehicles may be used in some locations, such that the vehicle 106 will automatically navigate through the arch 108.

Camera 110 is shown separately in FIG. 1B, and in this embodiment, it includes a first image sensor 130 and a second image sensor 132, which are mounted within a robust housing 134 and offset from each other by a distance D to respectively obtain first and second two-dimensional images from different perspective viewpoints. Image sensors 130 and 132 may be implemented using color sensors with Full HD resolution. The use of two spaced sensors 130 and 132 facilitates the generation of three-dimensional information by implementing stereoscopic image processing technologies.

Lamps 114 and 116 generally will operate at least at night or in low light conditions to assist in generating suitable images. In some embodiments, the camera may be sensitive to visible light wavelengths, while in other embodiments, the camera may be sensitive to thermal wavelengths or other hyperspectral wavelengths outside the visible portion of the spectrum. For example, some objects, such as metal objects in payload 102, interact in a specific manner with thermal wavelengths, which facilitates the identification of such objects.

Although camera 110 in FIG. 1A is shown oriented downward; in other embodiments, the camera may be oriented differently. For example, as shown in FIG. 1C, first chamber 152

- 4 045770 is installed on the post 154 to the left of the vehicle 106 to obtain images of the field of view

156 from a first perspective 158. A second camera 160 is mounted on a post 162 to the right of the vehicle 106 to obtain images of the field of view 156 from a second perspective 164. Referring to FIG.

1D, which shows another example of a camera generally designated 170. Camera 170 may be used in place of camera 110 shown in FIG. 1A, or cameras 152 and 160 shown in FIG. 1C.

In some embodiments, camera supports other than the arch 108 may be used. For example, in embodiments related to underground mining, the vehicle used may be smaller than the vehicle 106 shown in FIG. 1A, and arch 108 may be suitably scaled or eliminated entirely. Referring to FIG. 1E, which shows an example of an underground mining site including a chamber 180 mounted on the roof portion of the site. The camera 180 in this embodiment is configured as a unit with built-in LED lamps 182 located around the camera. The load, haul, dump (LHD) vehicle 184 is configured to operate at an underground work site and contains a relatively large bucket 186, which is configured to both load and transport the payload 188 within the work site. .

As shown in FIG. 1F, in yet another embodiment, camera 190 may be mounted on drone 192, and the drone may be directed to an area above the vehicle to position the camera to capture images. Alternatively, the drone 192 may be attached by a cable 194 that will hold the drone so that it hangs motionless at a specific location and position the camera 190 to capture images of a vehicle passing below.

A block diagram of the payload analysis system 102 is shown in FIG. 2 at 200. System 200 contains elements of device 100, including camera 110, junction box 118, lamps 114 and 116, and RFID reader 122. Switch box 118 supplies operating power to camera 110 via power wire 202 and also provides power and control. for lamps 114 and 116 and RFID reader 122. In this embodiment, camera 110 includes an embedded processor 204 in communication with memory 206 and input/output device 208, all of which are mounted within housing 134 (shown in FIG. 1B). The memory 206 provides storage of instructions for controlling the embedded processor 204 to acquire sequential images, and also provides storage of acquired image data.

The input/output device 208 communicates with the embedded processor 204 and provides an image sensor interface 210 including inputs 212 for receiving image data from the first and second image sensors 130 and 132 . The I/O device 208 further includes a communication interface 214, such as an Ethernet interface. Communication interface 214 has a port 216 that is connected via a reverse data cable 218 to a junction box 118. The junction box 118 may contain a modem, router, or other network equipment that facilitates connection to the network 220. The network 220 may be a local area network (LAN) implemented for local data transmission at workstation 126. Alternatively, junction box 118 may transmit signals via data cable 218 to a wide area network such as the Internet. In some embodiments, in which a wired connection to network 220 is not available, junction box 118 may contain a cellular transceiver, and the connection to network 220 may be via a cellular data network or other wireless network connection.

In the embodiment shown in FIG. 2, system 200 also includes a remote processor circuit 230 that includes a microprocessor 232 in communication with an input/output (I/O) device 234. I/O device 234 provides a communications interface 236 for transmitting and receiving data over network 220. Microprocessor 232 communicates with memory 238 for storing data and instruction codes. In this embodiment, microprocessor 232 is also in communication with mass storage device 240 for storing image data and for archiving payload analysis results. In the illustrated embodiment, remote processor circuitry 230 further provides processing via a graphics processing unit (GPU) 242, which can be used to provide increased processing power for image-intensive tasks. Thus, microprocessor 232 may be implemented as a graphics processor or remote processor circuit 230, and may further include a GPU coprocessor to relieve the microprocessor of some processing tasks.

In those embodiments in which the network 220 is a local area network, the remote processor circuitry 230 may be located in an operations center associated with the worksite 126. In other embodiments in which the network 220 is a wide area network, the remote processing circuitry 230 may be located in a remote processing center provided for image processing for multiple job sites. Alternatively, the remote processor circuit 230 may be provided in the form of an on-demand cloud computing platform provided by providers such as Amazon Web Services (AWS).

- 5 045770

The system 200 further includes a processor circuit 250 including a microprocessor 252, a memory 254, and an input/output device 256. I/O device 256 provides a communications interface 258 that is capable of receiving data via network 220. I/O device 256 also includes an interface 260 for initiating a visual warning through a display 262 or an audible warning through an annunciator 264, such as a loudspeaker or other audible alarm device. Processing circuitry 250 may be located in the operations center of the work site 126, where payload analysis results may be displayed along with alerts or warnings. Alternatively, processor circuitry 250 may be located in the cabin of vehicle 106 and connected wirelessly to network 220. In self-propelled or other non-powered vehicles, such as rail freight container 104, the warning signal may be processed differently to in order to cause the vehicle to be diverted or flagged so that further action can be taken.

Although the embodiment of system 200 shown in FIG. 2 includes an onboard processor 204 in camera 110 and a separate remote processor circuit 230; in other embodiments, the system may combine onboard and remote processors. Thus, the functions described below as being performed by remote processor circuit 230 may be performed by embedded processor 204 or any other combination of processor circuits.

Referring to FIG. 3, which is a block diagram depicting blocks of code for controlling the embedded processor 204 to provide image acquisition functions, generally designated 300. These blocks generally represent codes that can be read from memory 206 into a computer readable medium for controlling the embedded processor 204 to provide various functions. The actual code for implementing each block may be written in any suitable programming language, such as C, C++, C#, Java, and/or assembly code.

The image acquisition process 300 begins at block 302, which controls the embedded processor 204 to initiate sequential acquisition of images by camera 110 within the camera field of view 112. Thus, block 302 may control one or both of image sensors 130 and 132 to produce sequential images that may be received through image sensor interface 210 and stored in memory 206. An example of the acquired image is shown at 400 in FIG. 4, in which the image is limited to a field of view 112 shown as a dotted outline. In practice, images obtained with image sensor 130 will have a slightly different perspective than images obtained with image sensor 132. Thus, example image 400 represents one of the images obtained by either image sensor 130 or image sensor 132.

Block 304 then controls embedded processor 204 to determine whether the vehicle and cargo container are likely to be within the field of view. The embedded processor 204 is also controlled to determine whether the acquired image data meets the eligibility criterion for further processing. Thus, camera 110 may continuously acquire images of field of view 112, which may or may not contain vehicle 106 within the field of view. If there is a possibility that a vehicle is within the field of view 112, then the embedded processor 204 makes a further determination as to whether the image data is suitable for further processing. For example, the vehicle 106 may be only partially located within the field of view 112 in some images, and it is possible that more suitable images will be or have already been acquired that include a clear view of the vehicle 106, the payload 102, and the ground surfaces 402 surrounding the vehicle. means.

If it is determined at block 304 that the image meets the eligibility criterion, then the embedded processor 204 is controlled to proceed to block 306. Block 306 controls the embedded processor 204 to initiate reading the selected image data from the memory 206 and transmitting it through the communication interface 214 and the network 220 to the remote location. processor circuit 230. The selected image data may be tagged or otherwise associated with a vehicle ID generated by the RFID reader 122 as a result of reading the RFID tag 124 on the vehicle 106. For example, the selected image may have a vehicle ID embedded in a metadata field images, and transmitted along with the image data.

Although in this embodiment the vehicle ID is read from the RFID tag 124, in other embodiments the vehicle ID may be generated in a different manner. For example, a vehicle identifier may be marked on the vehicle within the field of view 112 of camera 110 and determined by processing one of the acquired images to extract the identifier from the image. Alternatively, one of the resulting images can be processed using a neural network that has previously been trained to generate identity outputs for the resulting image. A neural network can be trained using a set of labeled images of vehicles used on site 126 to enable identification of people

- 6 045770 god of used vehicles using a neural network.

Referring again to FIG. 3, in which a block diagram depicting blocks of code for controlling the microprocessor 232 of the remote processor circuit 230 for receiving image data is shown at numeral 320. Block 322 controls the microprocessor 232 for receiving image data from the camera 110. In some embodiments, images from of both image sensors 130 and 132 may be transmitted by device 100 and received by remote processor circuit 230. Unit 324 further controls microprocessor 232 to write image data to mass storage device 240. In some embodiments, image data may be received at a rate that is too high to allow further processing in real time. Storing image data in the mass storage device 240 facilitates queuing image data awaiting further processing. In most cases, immediate provision of payload analysis results may not be necessary; they can be delayed by a minute or more and still provide effective and timely notification to the on-site operations center 126 work.

Referring again to FIG. 3, in which a block diagram depicting blocks of code for controlling the microprocessor 232 of the remote processor circuit 230 to perform additional processing is shown at general number 330. The additional processing process 330 begins at block 332, which controls the microprocessor 232 to read image data for further processing. next image from mass storage device 240 to memory 238. Unit 334 then controls microprocessor 232 to process the image data to identify a payload region of interest within the image. Referring again to FIG. 4, in which the payload area of interest is designated 404 and is shown using a dotted line surrounding the ore in the cargo container 104.

Process 320 then continues at block 336. Optionally, upon identifying a payload region of interest 404, the microprocessor 232 may be controlled to generate a confidence level quantifying the confidence that the identified region of interest contains a payload. In this case, block 336 controls microprocessor 232 to further determine whether an additional processing criterion is met for the selected image based on the confidence level associated with the identified payload region of interest 404. If it is determined at block 336 that the payload region of interest 404 does not meet the additional processing criterion, microprocessor 232 is controlled to move back to block 332 to process the next image. If the additional processing criterion is determined to be met at block 326, microprocessor 232 is controlled to proceed to block 338 to process the next image in the queue.

Process 330 then continues to block 338, which controls microprocessor 232 to perform payload analysis for payload region of interest 404. Payload analysis may involve any of several different analysis processes. Analyzing the payload may include, for example, determining whether there are any distinguishable payload parts, such as large boulders or foreign objects, in the payload 102. Referring to FIG. 4, boulder 406 may be detected and determined to be larger than the ore threshold size. The threshold size may be set based on the maximum size of a boulder that can be safely handled by the processing equipment that is used in subsequent steps and receives the payload from the vehicle 106. Similarly, distinguishable parts other than boulders can be identified. Foreign objects, such as metal parts, may break off from the excavator, and the microprocessor 232 of the remote processing circuit 230 may be operative to trigger a warning signal in such cases. Block 338 then controls microprocessor 232 to move back to block 332, and the process is repeated for the next image in the queue. In one embodiment, foreign objects may be detected by processing image data acquired by camera 110. A machine learning approach may be used to detect common foreign objects, which may include metal tools and/or parts of the teeth of a loader used to load payload 104 into the vehicle 106. In some embodiments, it is possible to image using wavelengths in the infrared portion of the spectrum, since higher levels of infrared radiation in the payload 104 may indicate a metal or other foreign object in the payload that is different in temperature from the surrounding payload cargo

In this embodiment, the remote processor circuit 230 performs said additional processing. Identification of a payload region of interest and/or subsequent analysis of the payload may be processor intensive and may not be completed until additional image data is acquired by camera 110. In other embodiments, embedded processor 204 in device 100 may be configured to provide the required performance processing to perform region of interest identification and analysis

- 7 045770 payload in essentially real time. In such cases, payload analysis can be simplified to focus on a single function, such as boulder detection, to reduce processing requirements for the embedded processor 204.

In some embodiments, the distribution box 118 may provide continuous power to the lamps 114 and 116 in low light conditions to ensure that the vehicle 106 is detected and sufficient illumination is available for imaging purposes. In other embodiments, lamps 114 and 116 may be powered through distribution box 118 only when a vehicle is present. As an example, RFID reader 122 may be spaced apart from truss 108 such that when a vehicle is detected before passing under camera 110, a signal is transmitted via data cable 218 to I/O 208. Embedded processor 204 may optionally be configured to initiate power to the lamps 114 and 116 before a vehicle passes under the camera 110. To avoid startling the driver of the vehicle 106 due to the sudden activation of the lamps 114 and 116, the brightness level may be gradually increased after the vehicle is detected and then decreased after receiving the necessary images.

An example process for implementing block 304 of process 300 is shown in FIG. 5. For a camera 110 having two spaced apart image sensors 130 and 132, an embedded processor 204 may be configured to process the acquired first and second two-dimensional images from different perspective viewpoints to generate a three-dimensional point cloud containing three-dimensional coordinates of the vehicle. means and cargo container. The embedded processor 204 may implement a stereoscopic process that compares the first and second images to find matching features, and uses the offset or disparity between the matching features to determine three-dimensional coordinates for the matching features. A set of three-dimensional coordinates can be called a three-dimensional point cloud.

Generating the 3D point cloud information allows for convenient scaling of images to determine the physical dimensions associated with the payload 102. In other embodiments, processing may be based on the 2D image information, along with additional scaling information. For example, if the dimensions of the cargo container 104 of the vehicle 106 are known, then the two-dimensional image can be scaled based on the edges of the cargo container. In some embodiments, if one of the image sensors 130 and 132 becomes inoperable due to lens dirt or other problems, processing may be performed based on the two-dimensional information.

Process 304 begins at block 500, which controls embedded processor 204 to generate three-dimensional point cloud data based on the first and second images acquired by image sensors 130 and 132. Block 502 then controls embedded processor 204 to read elevation coordinates for each three-dimensional point in the point cloud data. Block 504 then controls the embedded processor 204 to read the first coordinate in the point cloud data and determine whether the corresponding altitude coordinate exceeds the minimum expected height 506 of the cargo container 104.

As shown in FIG. 6 for job site 126, the generated 3D point cloud will contain the 3D coordinates of the vehicle and cargo container in a 600x, y, z coordinate system. The three-dimensional point cloud may also contain three-dimensional coordinates for other objects, such as the earth's surface and parts of the vertical arch element 120. An expected minimum height 506 may be established for a work site 126 based on the known height of vehicles operating at that work site (ie, at height Z ₁ in FIG. 6). The use of a minimum expected height 506 may be particularly useful in cases where the vehicle 106 is a very large mining truck. Although other vehicles, such as light trucks, may cross the field of view 112, the resulting images may be discarded by the embedded processor 204 of the camera 110 based on the lack of point cloud data above the minimum expected height 506. In this case, the minimum expected height 506 for the location mining operations can be set higher than that of non-mining dump trucks of conventional cargo vehicles typically used in mining operations. If the altitude coordinate at block 504 does not exceed the minimum expected height 506, then the embedded processor 204 is controlled to proceed to block 508. Block 508 controls the embedded processor 204 to remove the 3D coordinates from the point cloud data. Thus, any 3D coordinates in the point cloud that have a Z coordinate less than Z1 will be excluded from consideration.

Process 304 then continues at block 512, which controls microprocessor 204 to determine whether the x and y coordinate values fall within point cloud subregion 514. Referring again to FIG. 6, in which the point cloud subregion 514 is shown as an outline, and in this embodiment extends only within the central portion of the field of view 112. Thus, the point cloud subregion 514 is defined as a cubic volume represented in an x, y, z coordinate system. 8 045770 designated by number 600, and passing between the coordinates x1,y1,z1 and _x2 , _y2 ,z2. The size of said cubic volume may be set as a fraction, such as 1/3 or 1/4, of the field of view 112. If the x and y coordinates of the point are not within the point cloud subregion 514 and this is determined at block 512, then the embedded processor 204 is controlled. to proceed to block 508 where the point is removed from the point cloud data. Block 508 then controls embedded processor 204 to go to block 516. If block 512 determines that the x and y coordinates of a point are within point cloud subregion 514, it stores that point in the point cloud data and controls embedded processor 204 to go to block 516. Block 516 controls the embedded processor 204 to determine whether the last coordinate in the point cloud data has been processed. If not, block 516 controls the embedded processor 204 to move to block 510, which controls the embedded processor to read the next altitude coordinate and go through blocks 502-516 again. If it is determined at block 516 that the last coordinate in the point cloud data has been processed, the embedded processor 204 is controlled to proceed to block 518. Thus, blocks 504 and 512 pre-process the point cloud data and provide the effect of reducing the number of points with leaving only the points that fall within the point cloud subregion 514, which is also generally centered with respect to the arch 108 and camera 110. This preprocessing significantly reduces the number of coordinate points in the point cloud data.

Block 518 then controls embedded processor 204 to calculate point density (PD) for the remaining points in the point cloud data. Point density can be defined as the number of coordinate points per unit volume. To estimate the PD for a point cloud, various types of approximation can be used, and functions for efficiently estimating the PD are generally available and can be easily implemented on the embedded processor 204. Block 520 then controls the embedded processor 204 to determine whether the calculated PD value is greater than a threshold value. PD522. For example, the PD522 threshold value may be predetermined based on the type of payload analysis being performed. The PD522 threshold may be set lower if it is only required to perform boulder detection, while a full fragmentation analysis may require a higher PD threshold.

If the calculated PD value is not greater than the threshold PD522, as determined at block 520, then the embedded processor 204 is controlled to move to block 524, where the next acquired image is selected, and the embedded processor is controlled to retrace the blocks 500518. If it is determined at block 520 that the calculated PD value is greater than the threshold value PD522, then the embedded processor 204 is controlled to proceed to block 526. Block 526 controls the embedded processor 204 to select an image for further transmission to the remote processor circuit 230 in block 306 of the process 300 shown in FIG. 3. Process 304 then continues to block 528, which controls the embedded processor 204 to interrupt processing of further images until the timeout period has expired. In one embodiment, the waiting period may be selected to provide sufficient time for vehicle 106 to exit field of view 112 and may be based on the expected speed of the vehicle. Block 528 has the effect of preventing additional images of the same vehicle 106 from being processed after an image meeting the eligibility criterion has been selected and transmitted to remote processing circuitry 230. If there is no vehicle within the field of view 112 of camera 110, then pre-processing at blocks 502-504 based on the minimum expected height 506 will result in a very low calculated PD value for the point cloud data due to the exclusion of ground level coordinates 402. Although in this embodiment, only one image having sufficient density can be selected for further processing points, in other embodiments more than one image may be transmitted for further processing purposes.

In the embodiment described above, camera 110 is configured to acquire first and second images using two physically spaced image sensors 130 and 132, and in other embodiments, the camera may have a single image sensor. In such embodiments, said single image sensor may be configured to obtain first and second images spaced apart in time. Thus, the movement of the vehicle 106 as it traverses the field of view 112 will provide images from two different perspective viewpoints, which can be used to create a three-dimensional point cloud.

Stereoscopic processes for generating 3D data rely on texture to facilitate point identification to determine disparity between images. Thus, the density of a 3D point cloud is a useful representation of the texture of the resulting image. Thus, there will be a significant difference in the density of the point cloud if there is no vehicle within the field of view 112, which facilitates the evaluation of the fitness criterion based on the density of the point cloud. Alternative methods for generating a 3D point cloud may be less dependent on texture and thus less sensitive to whether

- 9 045770 vehicle within field of view 112. In this case, prior knowledge of the geometry of the expected vehicles can be used to determine whether the resulting image meets the suitability criterion. For example, a two-dimensional horizontal plane drawn through a three-dimensional point cloud at a sufficient height above the ground should provide features that indicate the typical aspect ratio of a mining truck. This allows the images to be processed quickly enough to determine whether typical vehicle geometry is present within the field of view 112 and to determine whether the vehicle is of interest or is another type of vehicle, such as a light truck.

The embodiment of the image acquisition process 300 described above results in the selection of one image that meets the suitability criterion. In other embodiments, the embedded processor 204 of the camera 110 may be operative to select multiple images from sequentially acquired images that meet a suitability criterion, with each selected image providing a different view of the payload 102. Thus, block 306 of process 300 may provide transmitting image data for a plurality of selected images of the vehicle 106 to the remote processing circuit 230 for further processing. If several generally suitable images are available, additional processing may be implemented to clean up the images to remove shadows or other image quality defects. Further processing may involve using multiple selected images to generate a payload region of interest and/or payload analysis that aggregates or otherwise combines data from more than one image to generate results with an increased level of accuracy or confidence.

Although the embodiments described above use stereoscopic image processing technologies to generate 3D point cloud data from 2D images, in other embodiments 3D point cloud data may be generated using other technologies such as LIDAR (Light Detection and Ranging). , detection and ranging using light), time-of-flight camera, scanning laser, etc. For example, a LIDAR sensor may be used to obtain three-dimensional point cloud data within the field of view 112. The LIDAR sensor may be combined with a two-dimensional camera that provides two-dimensional image data to identify an area of interest.

An example process for implementing block 334 related to process 330 for identifying a region of interest is shown in FIG. 7. As described above, image sensors 130 and 132 in camera 110 acquire first and second two-dimensional images, and embedded processor 204 further generates three-dimensional point cloud data based on said first and second images. At block 306 of the image acquisition process 300, when transmitting an image for further processing, at least one of the first and second two-dimensional images may be transmitted to the remote processing circuit 230, where the data is written to the mass storage device 240. In some embodiments, the point cloud data may be deleted by the embedded processor 204 after process 304 of FIG. 5 and transmitting the first and second two-dimensional images to remote processing circuitry 230. In the case of high density (HD) image sensors 130 and 132, the two-dimensional images may be transmitted in full HD resolution, including color information. Typically, both of the first and second two-dimensional images are transmitted and recorded in the mass storage device 240. Although further processing performed by remote processor circuit 230 may require 3D information, the processing associated with generating 3D point cloud data from the first and second high-density 2D images is not expected to incur significant overhead relative to such further processing. . Accordingly, the three-dimensional point cloud can be regenerated by the remote processing circuit 230 based on the first and second two-dimensional images. In other embodiments, point cloud data generated by embedded processor 204 may be stored and transmitted to remote processor circuitry 230. In an embodiment in which further processing is performed by the embedded processor 204, the point cloud data may be stored for further processing by the embedded processor.

Process 334 begins with block 700, which controls microprocessor 232 of remote processing circuit 230 to select one of the two-dimensional images for processing to identify a payload region of interest. Block 702 then controls microprocessor 232 to preprocess the two-dimensional image data. The preprocessing at block 702 may include one or more optional image processing functions. For example, the two-dimensional image data may be cleaned to compensate for image distortion created by the imaging optics associated with the image sensors 130 and 132. When imaging over a large field of view 112, geometric distortions due to imperfections and alignment errors of the imaging optics permeate the image data and can be compensated for by making adjustments to the image data. Various models are available for general distortion correction that can be implemented to clean up image data based on the parameters of the imaging optics or other calibration data determined during manufacturing.

In some embodiments, the two-dimensional image data may be decimated to generate a smaller image data file to identify a payload region of interest. Reducing the resolution of the image data provides the ability to facilitate faster processing compared to a Full HD resolution image data file. In one embodiment, the high-density image may be reduced to a quarter of its original size for the purpose of identifying a region of interest.

Block 704 then controls microprocessor 232 to process the two-dimensional image to identify a payload region of interest using the trained neural network. In some embodiments, the neural network may be trained using a set of labeled training images. The set of images may include images in which representative vehicles, representative cargo containers, and representative payloads 102 can be identified by labeled boundaries within the respective images. In some of the training images, the cargo container may not be carrying a payload, and thus the payload will not be identified by the marked boundary. If several different types of vehicles containing cargo containers are operating at a 126 work site, then suitable labeled images can be included to train the neural network so that it is able to make general inferences to enable the identification of different vehicles.

Training of the neural network may be conducted in a supervised training process performed prior to deployment of the system 200 at the site 126. In this way, a set of labeled training images can be pre-labeled by a human operator and used in training activities. The human operator can also define control parameters for training the neural network, which can be adjusted to optimize the performance of the neural network. The trained neural network may be defined by a data set 706 that establishes the neural network architecture and defines appropriate parameters and/or weights that configure the architecture to perform the function of identifying a payload region of interest.

Thus, block 704 controls microprocessor 232 to receive pre-processed two-dimensional image data and generate output for identifying a region of interest based on said neural network data set 706. In one embodiment, the output may be a set of masks or bounding regions, as shown in FIG. 8. As shown in Figure 8, an image 800 having a vehicle 106 in the field of view 112 has a vehicle bounding box 802, a cargo container bounding box 804, and a payload mask 806, each indicated in the image by broken lines. . The vehicle bounding box 802 identifies the image 800 as containing a vehicle with a confidence level of 0.99 (ie, 99%). The cargo container bounding box 804 identifies boundaries associated with the cargo container with the same confidence level of 0.99. Finally, payload mask 806 identifies the payload region of interest with a confidence level of 0.92. The bounding boxes 802 and 804 generated by the neural network may be formed by boundary pixels demarcating given regions within the decimated image.

Process 324 then continues to block 708, which controls microprocessor 232 to determine whether the identified masks meet the criteria for further processing. For example, confidence threshold levels may be set for each of the bounding boxes 802 and 804. If the confidence level associated with the vehicle bounding box 802 is below a threshold (eg, 0.85), then the image may not be an image of a truck, or it may not contain a vehicle at all, so that selection and transmission by camera 110 could be wrong. Likewise, if the vehicle bounding box 802 has a high corresponding confidence level, but the container bounding box 804 does not meet the threshold confidence level, then image problems may arise that will prevent successful further processing. Additional processing criteria may also include logical definitions that are used to prevent processing of inappropriate acquired images. For example, if the cargo container bounding box 804 is positioned outside or partially outside the vehicle bounding box 802, this may indicate an inappropriate image that may lead to erroneous results if further processed. Likewise, if the payload mask 806 is positioned outside or partially outside the cargo container bounding box 804, this may also indicate an inappropriate image.

If a failure to meet the established confidence thresholds is determined at block 708, then

- 11 045770 controls the microprocessor 232 to move to block 710, where the selected image is marked as unsuitable for further processing. Block 710 may control microprocessor 232 to mark corresponding 2D and 3D point cloud data in mass storage device 240 so that the data can be deleted.

If the set confidence thresholds are determined to be met at block 708, microprocessor 232 is controlled to proceed to block 712. Block 712 controls microprocessor 232 to perform post-processing of the image data within the region of interest. Post-processing may include processing the image data to enhance shadowed areas that are caused by the sides of the cargo container 104 obscuring some portions of the payload 102. For example, color intensity manipulation may be implemented using a neural network to provide clearer input data for analysis. payload. In those embodiments in which the payload mask 806 is set based on the image data that has been thinned out in block 702, post-processing may be performed based on the original selected high-density image data stored in the mass storage device 240. Thus, post-processing will include first mapping the boundary pixels of the payload mask 806 defined for the sparse image to the pixels of the original high optical density image before performing image processing.

Block 714 then controls microprocessor 232 to perform payload analysis on the post-processed image data. In one embodiment, microprocessor 232 may be operative to perform payload analysis by performing segmentation analysis on a payload region of interest 806 to determine the dimensions of distinguishable portions of the payload. For example, payload analysis may include performing a fragmentation analysis on the payload, as described in Cooperative Patent Application No. 15/752430 by Tafazoli Bilandi et al., entitled Method and Apparatus for Identifying Fragmented Pieces of Material within an Image, which is included in this document by reference in its entirety.

Referring to FIG. 9, which is an example of a fragmentation analysis result within the identified payload mask 806 shown in FIG. 8 is shown at 900. In general, fragmentation analysis performed according to the methods disclosed in application 15/752430 involves processing pixel data using a convolutional neural network, which (processing) indicates whether pixels are located on the edge of the fragmented part of the material, inside from the edge of a fragmented piece of material or in spaces between fragmented pieces of material. In some disclosed embodiments, the determination is made at least in part based on information about two-dimensional and three-dimensional disparities. The resulting pixel classification can then be further processed to associate identified edges with fragmented parts of the material and provide scaling of fragment sizes.

Referring again to FIG. 7, whereby after the payload has been analyzed at block 714, process 324 continues at block 716, which controls microprocessor 232 to determine whether the warning criterion has been met. In the example shown in FIG. 9, identified fragment 902 could be identified as a boulder whose size exceeds the threshold size of ore for processing in other equipment on site. If the warning criterion is determined to be met at block 716, microprocessor 232 is controlled to proceed to block 718. Block 718 controls microprocessor 232 to transmit the warning message to processor circuit 250 via network 220 shown in FIG. 2. The warning message, when received by the communication interface 258 of the processor circuit 250, causes the microprocessor 252 to generate a warning signal at the operation center of the work site 126. The processor circuit 250 is operatively configured to cause the alarm to generate either an audible warning through the loudspeaker 264 or a visual warning through the display 262 to alert the operator to the presence of a boulder in the payload of the vehicle.

If a failure to meet the warning criterion is determined at block 716, microprocessor 232 is controlled to proceed to block 720. Block 720 controls microprocessor 232 to perform, as necessary, appropriate steps for displaying or transmitting payload analysis results. For example, payload fragmentation analysis records can be stored for later access by mining engineers on site 126 for use in making mining decisions. The results may indicate, for example, that the ore currently being mined is not optimal and that mining engineers may be moving mining resources to another stope.

Referring to FIG. 10, in which a neural network architecture for identifying a payload region of interest is shown as a block diagram at 1000. The blocks represent functions implemented by code blocks that control the microprocessor 232 to perform processing tasks for identifying a payload region of interest. The implementation of the neural network is based on the architecture proposed in the publication Mask R-CNN, authors Kaiming H. et al., 2017,

- 12 045770 which is incorporated herein by reference in its entirety. The Mask R-CNN architecture can be implemented to efficiently detect objects in an image while simultaneously generating a segmentation mask for each object instance. This end-to-end architecture is a multi-stage, multi-head neural network that provides predictions for multiple instances of object types, their bounding boxes, and corresponding masks or boundaries.

The neural network 1000 includes a four-layer feature pyramid network (FPN), such as that described in Feature Pyramid Networks for Object Detection by Tsung-Yi Lin et al., 2017, which is incorporated herein by reference in its entirety. . The pyramidal feature network is shown summarized in blocks 1006 and 1010, and the preprocessed image data 1002 is provided as an input 1004 to the residual neural network (ResNet) 1006 of the pyramidal feature network. The residual neural network 1006 generates features using a backbone network, such as the residual neural network 101 described in Deep Residual Learning for Image Recognition by Kaiming He et al., 2015. Backbone networks are pretrained on publicly available natural image datasets such as like ImageNet, are capable of classifying images into object categories.

Data from the outputs 1008 of the residual neural network 1006 are fed to a feature pyramidal network block 1010, which generates data on a plurality of outputs 1012 ranging from low-level, highly granular features to high-level semantic representations of the input image 1002. The feature pyramidal network block 1010 combines bottom-up and top-down feature maps are received from the residual neural network 1006 and generates enhanced feature maps at outputs 1012. Data from outputs 1012 may be further used by neural network 1000 to localize and segment objects of interest.

For each of the downstream paths of the feature pyramid network 1010, a lightweight region proposal network (RPN) 1014 finds those regions in the feature maps generated by the feature pyramid network 1010 that potentially contain a single feature of interest. A network of 1014 regional proposals ranks a set of anchors for each location within each level of the feature map pyramid. At each level, a fixed step is used to select some locations, and a set of anchors is defined for each location. Each anchor set includes horizontal and vertical frames at different scales (usually three scales, each with three anchors). To map these regions to the corresponding location in the original image, a specified set of anchors is first defined. Predicted regions are assigned to reference anchors based on the overlap between a pair of anchors and regions. Proposals are filtered by their rank, maximum expected regions, and overlap with the reference using a non-maximum suppression (NMS) approach. The remaining regions must be converted to a fixed size such that multiple network heads can be attached to the specified set of features. A region of interest align (ROI align) process 1016 is used to collect all regions based on their scores. The region-of-interest alignment approach will generate a fixed-size output in which each pixel is generated by sampling within the region of the feature map that corresponds to that output pixel. All sample points are averaged and the resulting average value will be assigned to the output pixel.

Depending on the size of the proposals, one of the feature maps generated at the outputs 1012 by the feature pyramid network 1010 represents a range of feature sizes that will be used to align regions of interest. Data from the outputs 1016 is fed to the fully connected layer or frame head 1018 to generate a feature vector of a certain size for each of the regions. This list of vectors is used in two branches to generate class membership probability 1020 and bounding box coordinates 1022 for each region. The output of the region of interest alignment process 1016 and the resulting results 1020, 1022 are further processed on a per-class basis to generate final detection results 1024 for each class. This process filters proposals based on probability estimates and calculates non-maximum suppression (NMS) for each class, using non-maximum suppression to ensure identification of a particular object only once.

The output of the feature map at the outputs 1012 of the feature pyramid network 1010 is converted into a fixed-size matrix according to the final detection results. An approach similar to the region of interest alignment process is used, and the results are fed to a series of convolutional neural network layers to adjust the number of output channels. A series of deconvolution layers then recover the spatial information, and 1-D convolutions reduce the number of channels to match the total number of classes (ie, in this case, the payload classes 102, which are identified by the payload mask 806).

- 13 045770

The generated masks for each class are then resized to match the dimensions of the original image at block 1030. Each mask is generated by clipping the prediction maps at 0.5.

Another embodiment for implementing the payload analysis block 714 of FIG. 7 is described in more detail with reference to FIG. 11. The process shown in FIG. 11 may be performed as an alternative to or in addition to the fragmentation process described above. Block 1100 controls microprocessor 232 to determine the dimensions of the cargo container 104 of the vehicle 106. In one embodiment, the dimensions of the cargo container 104 may be determined by performing the additional processing steps described above while the vehicle 106 is known to have an empty cargo container. 104. In this case, the actual dimensions of the cargo container can be determined based on the three-dimensional coordinates associated with points inside the vehicle mask 802 and payload mask 806 selected to determine the interior dimensions of the container. Thus, the determined dimensions of the container for the vehicle 106 will be predetermined at some point in time and stored in the vehicle database 1102. The vehicle database 1102 may be stored in mass storage device 240 or other location in communication with remote processor circuitry 230 . Thus, in this embodiment, block 1100 controls microprocessor 232 to determine the vehicle identifier 1104 associated with the selected image being processed. As stated above, the RFID reader 122 of the device 100 shown in FIG. 1 may be used to read a vehicle identifier associated with selected images that are transmitted by camera 110 to remote processing circuitry 230. Vehicle identifier 1104 is then used to locate container size data in vehicle data database 1102.

In other embodiments, values for vehicles used at work station 126 may be predetermined from vehicle specifications or conventionally measured and stored in a vehicle database 1102 referencing vehicle identification numbers. As described above, block 1100 controls microprocessor 232 to determine vehicle ID 1104, and corresponding container dimensions can be found in database 1102. In some embodiments, microprocessor 232 may be configured to remove images of a vehicle or to flag results as associated with an unidentified vehicle if identification of the vehicle fails.

After determining the dimensions of the cargo container 104 of the vehicle 106 associated with the selected image currently being processed, the microprocessor 232 is controlled to proceed to block 1106. Block 1106 controls the microprocessor 232 to determine the height of the payload filling of the interior area of the container 104 based on the three-dimensional coordinates of the points inside payload region of interest (ie, payload mask 806). Block 1106 controls microprocessor 232 to select a plurality of points on a two-dimensional image that lie within the payload mask 806 and to determine three-dimensional coordinates for these points that enable the determination of corresponding payload fill height points. This may include selecting coordinates based on 3D point cloud data that correspond to a selected set of points. As such, block 1106 determines the cargo height distribution within the cargo container 104.

Block 1108 then controls microprocessor 323 to use the distribution of cargo heights within the dimensions of cargo container 104 to calculate the offset of the cargo from a center line extending longitudinally through the cargo container. Lateral movement of the load can potentially lead to instability of the vehicle 106. Longitudinal movement of the load is less problematic due to the length of the vehicle's wheelbase in that direction. In one embodiment, the cargo displacement may be expressed as a percentage of the lateral dimension of the cargo container 104. Load displacement may be of interest to the field operator 126 when detecting vehicles that have uneven load distribution. In some embodiments, load shifting may be associated with a shovel or other heavy equipment that loads vehicle 106 so that uneven loading of specific operators can be detected and corrected. In the illustrated process embodiment, block 1110 then directs microprocessor 232 to determine whether the load distribution is uneven (ie, the load offset is greater than a maximum predetermined percentage). When the maximum load displacement is exceeded, block 1110 controls microprocessor 232 to proceed to block 1112 where a warning signal is generated and processed generally as described above. If it is determined at block 1110 that the maximum load displacement has not been exceeded, microprocessor 232 is controlled to proceed to block 1114.

Block 1114 controls microprocessor 232 to calculate the total payload volume. The total volume of the payload in the transverse direction is limited by the payload mask 806 at

-

Claims (29)

the surface of the payload and the dimensions of the container below the surface of the payload. Thus, these boundaries and payload height points can be used to generate a relatively accurate estimate of the total payload volume transported in the cargo container 104. Block 1116 then controls microprocessor 232 to transmit the calculated payload volume to the work site 126 or other location where information related to on-site operations is displayed. The operations center processor circuit 250 shown in FIG. 2 may receive data through network 220 from multiple cameras 110. For example, in one embodiment, a work site may contain multiple roads leading from the site, and each of them may contain a corresponding device 100 and camera 110. As shown in FIG. 12, in one embodiment, a dashboard 1200 may be displayed by the operations center processor circuit 250 on a display 262. The dashboard 1200 displays status information for four different locations 1202, 1204, 1206, and 1208. At location 1208, a dump truck is not currently detected or present. Each of the identified locations is characterized by status information associated with the detected truck and its associated payload. Each status display includes a highly visible warning area 1210 that indicates whether a boulder or any other foreign object is detected in the payload. Other areas 1212 of the dashboard 1200 display additional status information, such as the average number of fragmentations over the last 12 hours, the average total volume calculated based on the payload volumes determined in block 1110 of process 714, and the cargo offset calculated in block 1106 of process 714 . Although specific embodiments have been described and illustrated above, such embodiments are to be considered illustrative only and are not limiting of the disclosed embodiments as interpreted by the appended claims. CLAIM 1. A device for analyzing payload transported in a cargo container of a vehicle, containing: a camera positioned to sequentially obtain images of vehicles crossing the camera's field of view; at least one processor in communication with said camera, wherein said at least one processor is operatively configured to select at least one image from the sequentially acquired images in response to: quickly determining the probability that the vehicle and the cargo container will be in the field of view in at least one image; and matching the image data associated with the at least one image with the criterion of suitability for further processing; wherein said further processing includes ensuring that said at least one processor: processing the selected at least one image to identify a payload region of interest in the image; and performing payload analysis within the identified payload region of interest based on image data associated with the at least one image. 2. The device of claim 1, wherein said at least one processor is operatively configured to select said at least one image by: generating 3D point cloud data for sequentially acquired images; determining the point density of point cloud data; and comparing the point density with a threshold point density to determine compliance with the suitability criterion. 3. The apparatus of claim 2, wherein said at least one processor is operative to pre-process 3D point cloud data for a selected image prior to analyzing the payload, said pre-processing including at least one of the following: removing the coordinates of those points of the cloud that are located below the expected height of the supporting base of the cargo container relative to the surrounding surface of the earth; and/or removing the coordinates of those point cloud points that are outside a point cloud subregion within the point cloud, wherein said point cloud subregion is smaller than the point cloud. 4. The apparatus of claim 2, wherein said at least one processor is further operative to select for further processing at least one of the following if the plurality of images are determined to meet the suitability criterion: image having the highest pixel density; a first image having a dot density greater than a threshold dot density; or - 15 045770 sets of images that have a point density greater than the threshold point density. 5. The apparatus of claim 1, wherein the processor is further operative to generate a confidence level during processing of the selected image to identify a payload region of interest, wherein said confidence level quantifies the confidence that the identified region of interest contains a payload, and said the confidence level is used at least in part to determine whether the selected image meets the suitability criterion. 6. The apparatus of claim 1, wherein said at least one processor is operatively configured to select a plurality of images from sequentially acquired images, wherein each of said plurality of images provides a different view of the payload, and said at least one processor is operatively configured to configured to perform additional processing on each of the plurality of images to obtain an analysis of the payload. 7. The apparatus of claim 1, wherein the camera is positioned above the vehicle and the field of view is oriented downward to obtain images of the top surface of the payload exposed by the open top side of the cargo container. 8. The apparatus of claim 1, wherein said at least one processor comprises an embedded processor in communication with the camera, wherein said embedded processor is operative to initiate transmission of image data for the selected image to a remote processor, where further processing is performed by said remote processor. processor. 9. The apparatus of claim 8, wherein the embedded processor comprises a WAN interface, wherein said embedded processor is operative to transmit the selected image to the remote processor via said WAN. 10. The apparatus of claim 1, wherein said at least one processor is operative to initiate an alert signal in response to payload analysis results that meet an alert criterion. 11. The device of claim 10, further comprising a warning annunciator operably configured to generate an audible or visual signal to warn the operator. 12. The device of claim 1, wherein said at least one processor is operative to process the first and second two-dimensional images from different perspective viewpoints to generate a three-dimensional point cloud containing three-dimensional coordinates of the vehicle and the cargo container. 13. The device of claim 12, wherein the camera comprises one of the following: first and second image sensors that are offset to obtain corresponding first and second two-dimensional images from different perspective viewpoints; and one image sensor operatively configured to obtain first and second images spaced apart in time such that movement of the vehicle as it traverses the field of view provides different perspective viewpoints for the first and second images. 14. The apparatus of claim 12, wherein said at least one processor is operatively configured to process one of the respective two-dimensional images to identify a payload region of interest in a two-dimensional representation and perform an analysis of the payload by processing the two-dimensional data in the payload region of interest, wherein said at least one processor is operatively configured to use a three-dimensional point cloud to generate scaling information for payload analysis. 15. The apparatus of claim 1, wherein said at least one processor is operatively configured to process the selected image to identify a payload region of interest using a trained neural network to produce output indicating the location of the region of interest within the selected image. 16. The device of claim 15, further comprising a function for training a neural network using at least one of the following: a set of images of representative cargo containers that have previously been tagged by humans; and an unsupervised learning algorithm implemented to extract patterns in image data. 17. The apparatus of claim 15, wherein the neural network comprises a masking region-based convolutional neural network. 18. The device of claim 15, wherein said at least one processor is operatively configured to process the selected image by at least one of - 16 045770 the following: processing the image data to enhance shadowed areas before performing payload analysis; performing cleanup on the selected image to correct image distortion caused by imaging optics associated with the camera before identifying a payload region of interest; and sparse the pixels of the original selected image to obtain a sparse image with a reduced number of pixels before identifying the payload region of interest. 19. The apparatus of claim 18, wherein the output of the neural network identifies boundary pixels demarcating a payload region of interest within the sparse image, wherein performing payload analysis includes identifying corresponding boundary pixels within the original selected image and processing portions of the original selected image into within the corresponding boundary pixels. 20. The device of claim 1, wherein said at least one processor is operative to determine the dimensions of a cargo container of a vehicle by one of the following: determining a vehicle identifier associated with the selected image and reading from the database parameters defining the dimensions of a cargo container for the identified vehicle; and performing further processing on the vehicle with the empty cargo container and determining dimensions of the cargo container based on the empty cargo container. 21. The device of claim 20, wherein said at least one processor is operatively configured to identify the vehicle by one of the following: processing at least one of the sequential images to extract a vehicle identifier indicated on the vehicle within the camera's field of view; receiving an identifier from a radio-frequency identification (RFID) sensor located to read an RFID tag placed on the vehicle; or processing at least one of the sequentially acquired images using a neural network that has been previously trained to identify the vehicle based on the acquired image. 22. The apparatus of claim 20, wherein the processor is operably configured to perform payload analysis by determining payload volume by determining the payload fill height of an interior region of the cargo container based on three-dimensional coordinates for points within the payload region of interest and calculating the payload volume based on payload fill height and specific cargo container dimensions. 23. The apparatus of claim 1, wherein the processor is operative to perform analysis of the payload by identifying a foreign object in the payload. 24. The apparatus of claim 23, wherein the processor is operative to identify a foreign object by processing infrared images of the payload, the foreign object being identified by detecting electromagnetic radiation having wavelengths in the infrared portion of the spectrum. 25. The device according to claim 1, in which the processor is functionally configured to: performing payload analysis by calculating load displacement; and generating an uneven loading warning if the load displacement exceeds a predetermined maximum load displacement. 26. The apparatus of claim 1, wherein the processor is operative to perform payload analysis by performing segmentation analysis on a region of interest of the payload to determine the dimensions of distinguishable portions of the payload. 27. The apparatus of claim 26, wherein the processor is operative to initiate generation of a warning signal in response to the at least one distinguishable portion exceeding a threshold size or being identified as a non-payload object. 28. The apparatus of claim 27, wherein the payload comprises a mined ore payload and wherein the segmentation analysis comprises one of the following: fragmentation analysis, which provides identification of distinguishable parts as one of a rock part, a fine part, or a gap between parts; distribution of cargo within the dimensions of the cargo container; and/or moisture analysis, which provides a classification of the moisture level associated with the payload. 29. The apparatus of claim 1, wherein the vehicle is one of a mining truck, a rail car, a barge, a trolley, a loading, hauling and unloading vehicle, or a mine skip. -