AU9522598A

AU9522598A - Template-based loading strategy using perceptual feedback

Info

Publication number: AU9522598A
Application number: AU95225/98A
Authority: AU
Original assignee: Carnegie Wave Energy Ltd
Current assignee: Carnegie Wave Energy Ltd
Priority date: 1997-12-19
Filing date: 1998-12-03
Publication date: 1999-07-08

Description

Our Ref: 709910 P/00/011 Regulation 3:2

AUSTRALIA

Patents Act 1990

ORIGINAL

COMPLETE SPECIFICATION STANDARD PATENT Applicant(s): Carnegie Mellon University 5000 Forbes Avenue Pittsburgh Pennsylvania 15213 UNITED STATES OF AMERICA Address for Service: Invention Title: DAVIES COLLISON CAVE Patent Trade Mark Attorneys Level 10, 10 Barrack Street SYDNEY NSW 2000 Template-based loading strategy using perceptual feedback The following statement is a full description of this invention, including the best method of performing it known to me:- 5020 Template-Based Loading Strategy Using Perceptual Feedback Technical Field This invention relates generally to a method and apparatus for selecting a load site in a containment bin and, more particularly, to a method and apparatus for correlating the distribution of material in the bin with a template of the ideal load distribution for selecting the optimum load site.

Background Art As used in this patent specification the phrase "earthmoving machine" and various approximations thereof refer to excavators, wheel loaders, track-type tractors, compactors, motor graders, agricultural machinery, pavers, asphalt layers, and the like, which exhibit both (1) mobility over or through a work site, and the capacity to alter the topography or geography of a work site with a tool or operative portion of the machine such as a bucket, shovel, blade, ripper, compacting wheel and the like. Currently, autonomous earthmoving machinery is being developed that eliminates the need for a human operator at the controls. This reduces cost, and allows operation in environments that are harsh or difficult for humans, e.g. in an environment with poor visibility.

Autonomous loading systems require information regarding the work site including the location to load materials. There are situations when it is important to distribute the load in a specific configuration. For example, some dump trucks can become unstable while being driven if the load is not evenly distributed in the bed, while other trucks require a load distributed toward the front of the bed.

To achieve the desired result during autonomous operation, the machinery's control system must be capable of determining optimal locations for loading materials. Information regarding the location of the bin or receptacle, the distribution of the materials currently in the receptacle, and the desired distribution for loading the receptacle is required. A rangefinding sensor system is often used to provide range data to the control system to compute location, dimensions, and orientation of the receptacle. Currently, however, no method or apparatus for determining the distribution of materials in the receptacle, comparing it to the desired distribution, and generating information for the control system to use in directing the machinery to load materials, is available. It is also important to be able to distribute loads in a variety of distribution patterns, depending on the requirements of the receptacle into which the material is loaded.

Accordingly, the present invention is directed to overcoming one or more of the problems as set forth above.

Disclosure of the Invention In one embodiment of the present invention, an imaging sensor provides data corresponding to a digital map of a load of material in a receptacle to data processing means such as a digital computer with software. The load map data includes the height and shape of the material in the receptacle. A template corresponding to the ideal distribution pattern for one or more sections in the receptacle is also provided. Various shapes may be chosen for the templates and the data preprogrammed in the digital computer or calculated interactively based on user input. The templates for the desired load distribution can vary from simple to more complex patterns.

The load map and templates are divided into similarly sized grid portions and a number representative of the height for each grid portion is computed. The number of grid portions depends on the desired resolution and data processing capabilities. A value representing the correlation between the ideal and the actual distribution of the load is calculated for each grid portion, and the optimal location to place the next load of material is selected from the correlation values.

The values calculated by the correlation algorithm are also used for higher level planning, including selecting alternative and future loading sites. The computation for the correlation value for a particular xy location takes into account the height of the material in the grid portion corresponding to the particular xy location as well as the height of the material in the surrounding grid portions. The processing means associated with the present device will also estimate data for grid portions of the load map for which height data is unavailable due to problems such as occlusion by the receptacle or other objects in the environment, or noise in the sensor signal. In order to reduce processing time to compute the correlation values, the present invention may include instructions to process only certain portions of the grid, such as the center grid portions.

Brief Description of Drawings Fig. 1 is a perspective view of a receptacle or bin into which materials are loaded; Fig. 2 is a perspective view of a load map of the contents of a receptacle; Fig. 3 is a perspective view of a template for a desired load distribution in a receptacle.

Fig. 4 is a cross-sectional view of a load map of the contents of a receptacle; Fig. 5 is a cross-sectional view of a template for ideal load distribution; Fig. 6 is a plot of correlation values indicating the difference between the actual and ideal load distributions for a cross section of the load; Fig. 7 is a cross-sectional view of a load map of the contents of a receptacle; Fig. 8 is a cross-sectional view of a template for ideal load distribution; Fig. 9 is a plot of correlation values indicating the difference between the actual and ideal load distributions for a cross section of the load; Fig. 10 is a top view of an excavator and a dump truck at a work site; and Fig. 11 is a block diagram of the components associated with a system wherein the present invention may be utilized.

Best Mode for Carrying Out the Invention Referring now to the drawings, Fig. 1 shows a partially filled bin or receptacle in the form of a dump truck bed 20 with a load of material 22 partially filling the truck bed 20. In the present invention, the distribution of the load in the dump truck bed 20 is determined by using a sensor system (not shown), such as an imaging sensor or a rangefinding sensor system, to provide data on the existing load in a bin or receptacle. Fig. 2 shows a load map 26 of material 22 in the dump truck bed 20 superimposed on the xyz axes of a Cartesian coordinate system 28. Fig. 2 illustrates the load divided into portions 30 in a grid pattern along the x and y axes of the coordinate system 28. The grid portions 30 are sized according to the level of resolution required to achieve the desired load distribution. In the present invention, the grid may have the same number of divisions along both the x and y axes, or alternatively, a different number of divisions along the x and y axes. Fig. 3 shows an example of a desired load distribution 32 suitable for a dump truck requiring a larger portion of the load distributed toward the front of the truck. The ideal load distribution 32 is also divided into grid portions 34 of similar size to the grid portions 30 for the load map 26 shown in Fig.

2. The template for the ideal load distribution may apply to a portion of the receptacle or to the entire receptacle.

Values representative of the height of the material in each load map grid portion 30, 34 and the desired height of material according to the template distributions corresponding to the grid portions are then calculated and stored in memory.

To save computational time, the values representative of heights for the template distributions could be preprogrammed or calculated only once during initialization. The representative values may be any number that gives an indication of the height of material in the grid portions, such as the average height, maximum height, or minimum height. A two-dimensional table of correlation values is then computed based on numbers representative of the height of the material in each grid portion of the template and load map distributions using a correlation function such as the following embodiment of a correlation function: C(x, y) Z_ Im(x+ u, y+ t(u, v)l where: m represents the table containing a number representative of the height for each grid portion of the load map; 1 0 t represents the table containing a number representative of the height for each grid portion of the ideal or template load; J is the number of grid portions before the x-value that should be included in the region considered to calculate the correlation value; K is the number of grid portions after the x-value that should be included in the region considered to calculate the correlation value; L is the number of grid portions before the y-value that should be included in the region considered to calculate the correlation value; M is the number of grid portions after the y-value that should be included in the region considered to calculate the correlation value; and C represents correlation values between the actual and the ideal load distributions. Low numbers of C correspond to optimal grid locations for loading.

This example correlation function requires each axis of the grid to be divided into at least (J K) portions in the x-axis and (L M) portions along the y-axis, where J, K, L, and M are numbers greater than or equal to 0. The grid may have a greater number of divisions along each axis, regardless of the dimensions of the bin or receptacle along the x and y axes. For example, the grid can be divided into 3 by 4, 5 by 5, 8 by 8, etc. portions. There is no set number of divisions, but rather, the number is estimated and entered by the operator based on the size of the receptacle and the amount of material to be loaded during each loading cycle. The number of divisions and the values for J, K, L, and M may also be determined by the processing means according to a formula programmed in software. The processing means associated with the present device will also estimate data for portions of the load map for which height data is unavailable due to problems such as noise in the sensor signal. There are several ways to fill in the missing data including using an average, minimum, or maximum value of the height values from the grid portions immediately surrounding the subject grid portion.

The two-dimensional table of values that is computed with the example correlation function is used to find the optimal location for loading as well as for planning alternative and future loading locations. In the preferred correlation function, the correlation values having the lowest value correspond to the grid portion that is the optimal loading site, however, different correlation functions may use different criteria for determining the optimal correlation values. It is important to note that in the preferred embodiment of the correlation function, the correlation value for each grid portion is based on the difference between the actual and the ideal distribution height not only for one grid portion, but for the area surrounding the grid portion as well. If the amount of material to be loaded is less than or equal to the space available in one of the grid portions, however, logic in the software may indicate that the surrounding grid portions do not have to be compared to save computational time.

One problem with the correlation function arises during computations for the grid portions along the border and at the corners of the receptacle. This is because there is no data for portions outside the grids for the actual distributions. One way to handle the problem is to create a buffer of grid portions around the periphery having average elevation values equal to center grid portion. This solution increases computation requirements and may be unrealistic because it is rare for material to be loaded so near the edges of the receptacle. Another approach tothe problem involves checking only the inner grid portions by choosing values for J, K, L, and M so that the values for and (y+M) never lie outside the template or load map. This solution saves computation time and storage requirements.

The correlation formula is independent of the shape of the ideal load distribution template, and different templates may be selected without requiring a change in the correlation formula. The template shape may be as simple or as complex as required and different templates may be used for different portions of the receptacle. The templates may be developed for any type of receptacle, with dump trucks being one example.

Industrial Applicability Fig. 4 illustrates a cross-section of a load map that is unevenly distributed in a receptacle from the front 42 to the back 44, and from the bottom 46 to the top 48 of the receptacle.

To apply the present invention to attain the desired distribution of material, a template for the ideal distribution is specified as shown in Fig. 5 where a cross section of a flat load distribution is desired. The flat line 50 lies along the x axis, with no elevation in the z axis. In the preferred embodiment of the correlation formula discussed hereinabove, a template of all zero elevation values results in t(u, v) 0 for all values of u and v.

Therefore, the calculated correlation values will correspond to the elevations for the grid portions of the load map. From these values, the lowest number is selected as the optimal loading location.

For example, Fig. 6 shows correlation values displayed below the x axis for the cross section of the load map in Fig. 4 and the flat distribution template in Fig. 5. The low value of 1 is selected as the optimal loading location.

Another example applying the present invention is shown in Figs. 7, 8, and 9. Fig. 7 shows a cross-section of a load map 70 and Fig. 8 shows a cross-section of the template for the ideal distribution 80 as being heavily loaded toward one end. The correlation values that result for these cross-sections are displayed below the x axis in Fig. 9, and the grid portion with the correlation value equal to 4 is chosen as the ideal loading location. This results in the material being loaded onto the slope between section 72 and section 74 to fill section 74. The shape of this template is useful in situations when there is greater support on one side of a receptacle than on the other, such as when the rear axle of a dump truck is located close to the center of the bed and the load must be distributed toward the front of the bed for stability.

When a receptacle is empty, loading will begin in the location with the minimum correlation value. Alternatively, the system may default to a particular location at the beginning of a new loading sequence.

Fig. 10 illustrates an earthmoving site where an excavator 100 is loading excavated material into the bed of a dump truck 110. When the excavator 100 is operating autonomously, the present invention is used to determine where to load the materials to achieve the desired distribution in the bed of the truck 110. Some of the components that may be included in such a system are shown in the block diagram in Fig. 11. The control system 120 receives input from one or more sensor systems 122, such as a scanning imaging sensor or a scanning rangefinding sensor. Information regarding the level of material in the dump truck 110 and the distribution of the material is provided by the sensor systems 122 when required. For instance, the system may update the load map after every load, or it may wait until after a couple of cycles to update the information. The data processing portion of the control system 120 may consist of a digital computer including a microprocessor, data storage and retrieval means, and data input and output capability. Software 124 to process the load map supplied by the sensor system 122, to compute the average height of the grid portions between the template and the load map, and to calculate the correlation function values, is executed in the microprocessor. Commands for controlling the equipment may also be calculated by software as output signals to the actuators 126 associated with the equipment. Data corresponding to the shape of the template for the ideal load distribution may be stored in memory. If a variety of template shapes are available, an interface 128 for the operator is supplied allowing the desired distribution to be selected. Alternatively, image recognition software may be used to determine the type of receptacle and calculate an appropriate distribution pattern.

The control system may include planning logic for preplanning the loading cycle, or for selecting alternative loading locations if necessary. Guidelines may be implemented to control a series of loading cycles. For instance, the system may be instructed to "always find the lowest region in which to load." This is the simplest template, having elevation values of zero at all locations. If there is a requirement for the receptacle to be filled to a certain level, the sensor system may be used to monitor the material level, and the planning routines in the control system software may halt the equipment when the desired level is reached.

A more complicated set of instructions could include using more than one template for distributing the load. For instance, the instructions may be to "fill the receptacle with 6 bucket loads, but load the bin more heavily toward the front." The decision logic associated with the control system could instruct the equipment to load the first buckets at the front of the receptacle and then determine if the load is higher than a predetermined height of the receptacle. If so, then a template shaped like a ramp with the upper portion truncated or squared off could be used so that the system would then load future buckets in front of the first loads. If two correlation values are tied for the optimum or desired value, then the location corresponding to the closest shape of the ideal template could be chosen, such as the location closest to the front of the truck.

Other aspects, objects and advantages of the present invention can be obtained from a study of the drawings, the disclosure and the appended claims.

Throughout this specification and the claims which follow, unless the context requires otherwise, the word "comprise", or variations such as "comprises" or "comprising", will be understood to imply the inclusion of a stated integer or step or group of integers or steps but not the exclusion of any other integer or step or group of integers or steps.

Claims

1. A method of determining the optimum location to load material in a receptacle, the method comprising the steps of: dividing a template representing an ideal load configuration in the receptacle into grid portions; computing a value representative of the height of the ideal material level in each grid portion; dividing a load map of the material in the receptacle into grid portions; computing a value representative of the height of the material in each grid portion of the load map; generating a correlation value for each grid portion representing the desirability of placing a load in the grid portion; and selecting a numerical value representing an optimal grid portion for the location to load material.

2. The method as set forth in claim 1 wherein correlation values are computed using only grid portions that lie within boundaries of the load map and template in step

3. The method as set forth in claim 1 wherein step further includes utilizing a plurality of templates for the desired load configuration.

4. The method as set forth in claim 1 wherein step further includes developing data for unknown elevations of material in the load map grid portions. 14 The method as set forth in claim 1 wherein the grid portions for the template and the load map are substantially similarly sized.

6. The method as set forth in claim 1 wherein step further comprises calculating each correlation value taking into account the height of the material in surrounding grid portions.

7. An apparatus for determining the optimum location to load material in a receptacle comprising: a numerical template divided into grid portions comprised of values representing the ideal load configuration in the receptacle; a load map of material in the receptacle divided into grid portions corresponding substantially to the template grid portions; and a data processor operable to compute a correlation value for each grid portion based on the correlation between the height of the material in the receptacle and the ideal height of material for the grid portion, and to determine the optimum grid portion for receiving a load.

8. The apparatus as set forth in claim 7 wherein correlation values are computed using only grid portions that lie within boundaries of the load map and template.

9. The apparatus as set forth in claim 7 further comprising a plurality of templates for the desired load configuration, each template having a different distribution pattern associated with it and the data processor operable to select among the templates during the loading. The apparatus as set forth in claim 9 further comprising a data processor operable to develop data for unknown elevations of material in the grid portions.

11. The apparatus as set forth in claim 7 wherein the grid portions for the template and the load map are substantially similarly sized. P:\WPDOCS\MCC\709910 2/12/98 -16-

12. The apparatus as set forth in claim 7 wherein the number of grid portions along the x axis of the load map is different from the number of grid portions along the y axes of the load map.

13. The apparatus as set forth in claim 7 further comprising a data processor operable to calculate a correlation value for each grid portion taking into account the average height of the material in surrounding grid portions.

14. A method of determining the optimum location to load material in a receptacle, substantially as herein described. An apparatus for determining the optimum location to load material in a receptacle, substantially as herein described with reference to the accompanying drawings. DATED this 1st day of December 1998 CARNEGIE MELLON UNIVERSITY By its Patent Attorneys DAVIES COLLISON CAVE