DE19859169A1 - Learning system for controlling autonomous earth moving machines - Google Patents

Abstract

The system has a machine with several mechanical connections joined together at connecting points. These are simultaneously moved on command. A processing system (20-38) carries out at least one script. Each script has at least one variable parameter defining a movement of the machine. A learning algorithm modifies the value of at least one variable parameter on the basis of a desired result and measured conditions affecting the result. An Independent claim is also included for a method for controlling autonomous earth moving machines

Description

The invention relates to a system and a method to control the movement of robotic or remote controlled Machines, especially on a learning algorithm for Modifi the control parameters of remote-controlled machines during Earthmoving work.

Certain types of machines operate during the Operating repeated movements. For example, introduces a hydraulic excavator repeated during earthmoving work Movements such as removal or excavation and loading. Currently goes on to develop systems to automate the Control of earthmoving and other types of machines to reduce the need for human operators to min and the tasks as quickly and accurately as possible to lead. The term "earth moving machine" and the like here on excavators, wheel loaders, chain tractors, motor Ground planes, agricultural machines, paving machines, asphal animal machines and the like, both firstly via a Construction site is mobile and secondly, are able to topography or geography of a construction site with a tool or the Functional part of the machine such as the bucket, the shovel, the Blade, the ripper, the compacting roller and the like to change.

Systems for remote-controlled machines are currently being used developed that "learn" during operation. The "learning process gang "typically involves storing a number of Steps to perform a function such as cut / dig and tipping as well as the repetition of the Steps. The current learning functions are designed that repetitive tasks are repeated to meet the need to reduce operating personnel to perform the same task  to perform several times. The conditions on a construction site however, can change frequently, making a programmed Series of steps with changing conditions less become efficient. For example, on an excavator site the terrain on the excavator front is constantly changing, the crowd and the distribution of the material on the bed one Truck changes when material loads are added the. Furthermore, the properties of the excavated material can change with the progressive exposure of new soil layers, for example large rubble stones, rocks, gravel, loose sand, sticky of. A programmed series of steps that take place on Be efficient at the start of a shift can become less efficient as work progresses.

By B. Song and A. Koivo, "Neural Adaptive Control of Digger ", Proceedings of International Conference on Intel ligent Robots and Systems, Volume 1, pp. 162-167 is an opti Feed torque torque term for adaptation the excavation plan when the texture of the excavation changes known the material. The torque term is replaced by a new one calculated network, which is trained, the inverse Dyna to calculate the mic of the excavator. Although the introduction of the opti torque terms, the tracking and stability ins overall, neural networks require considerable computing time for training; once trained, be pre determinations but calculated very quickly. An additional one The disadvantage of neural networks is that they train again to introduce information about new data, d. H. they do not easily adapt to changes in the autonomous order indication.

For example, from D. Seward, "LUCIE - The Autonomous remote controlled excavators ", Industrial Robot International Quar terly, Volume 19, No. 1, pp. 14-18 are rule-based Systems for controlling work on an excavator site known. These systems typically require a large one Number of rules for processing variable conditions during the excavation work; the rules need to start of the work to be implemented. Such systems are  unable to set or adjust the rules to to cope with unforeseen situations or movement to optimize based on previous experience. It is also unclear how the parameters or thresholds in the Rules are generated.

A system is known from DE 198 04 006.7 A1 and a method for controlling the movement of machines using parameterized scripts. Various Groups of script parameters can vary depending on the work mode or if different events occur the. The parameters of a movement, e.g. B. for a hydraulic excavators, using inverse kinematics, ver tied speed information and various Heuristic calculated. Some of the heuristically found values are used to calculate the soil conditions Parameters used. The angles between the movement comm Components can vary depending on the properties of the excavated material be very different, e.g. B. dry sand and wet Mud, and an incorrect angle can lead to an inaccurate one Guide floor placement. The equations by which this Parameters calculated will not change if they are not be reprogrammed in the system. So if the excavator because of bad assumptions or heuristically found values bad works, then there is currently no way of performance to improve without interrupting the operation of the machine.

The invention has for its object that in the state to overcome existing difficulties in technology. Ins special procedures and systems are to be created be able to track the progress of the work autonomously monitor and program during the Modify operation so that the machine has a wide range of excavation and loading conditions efficiently is working.

In one embodiment of the invention is used for control an autonomous machine a motion planning algorithm used. The motion planning algorithm consists of a framework or script that outlines the general direction of the  Movement picks up while parameters in the script with the kinema table details for a specific machine and groups filled with movements. Calculate a learning algorithm net the script parameters with feedback of the working method and quality of the machine during the previous cycle the current parameter set. The parameters are set in this way represents that the operation and quality of the machine during subsequent working cycles is improved. The new params are learned through the learning algorithm using a predictive function approximators calculated and out evaluates to test different work criteria, e.g. B. the the time required to complete a task and the exact with which the task was carried out. The working or Quality criteria are weighted so that the prediction of the Result of alternative movements emphasizes the quality criteria, that are considered the most important. With the accumulation The algorithm makes use of data from repeated movements must be the history of the results of various movements use to recalculate and refine the parameters and improve the quality of work.

The invention will be explained in more detail with reference to the drawings tert. Show it:

Fig. 1 is a flow diagram of the motion planning scheme in which the learning algorithm according to the invention is used,

Fig. 2 is a perspective view of an excavator when loading a truck, and

Fig. 3 is a plan view of the position of an excavator, an excavator surface or site and an excavation pile on an excavator site in a polar coordinate system.

Fig. 1 shows several components of a preferred embodiment of the system according to the invention with one or more sensor systems 20 , which provide perception information about the environment of the machine, for. B. excavation in an earth moving area. The information supplied by the sensor system 20 is processed by one or more software modules in a detection system 33 , which obtain a certain piece of information about the environment or provide a desired result for the actions of the machine. In an earth moving terrain, for example, the recognition system 33 can perform functions such as recognizing loading containers and determining their location and orientation 23 , determining the desired area 24 to be excavated, determining the desired area for unloading the excavation 26 , detecting obstacles 28. A learning algorithm 30 calculates script parameters using previous results of actions along with the processed information. The parameters are used in scripts 32 that represent patterns or templates that describe how a particular task (lot) is to be performed as a series of steps. The scripts 32 generate commands for control units 22 for positioning the moving components of the machine in order to perform the required tasks.

The learning algorithm 30 uses the information provided by the recognition or detection system 33 about the current initial conditions 31 and the desired results to calculate the next action of the machine. The initial conditions 31 may include any required information about the environment necessary to perform the tasks, such as the shape and location of the excavator site, the height of the unloading truck and the location where the excavation is being loaded, or the initial start configuration of the Machine itself. The desired results can relate to any aspect of the nature of the task, such as the specific volume of the soil covered per load, excavating a maximum amount of soil during the minimum possible time and / or the location for the mass of the loaded material. The learning algorithm 30 returns a proposed machine action to assume that the desired results are best achieved under the current initial conditions.

The learning algorithm 30 initiates a request to the recognition system 33 and requests an initial condition 31 about the environment or the terrain and which result is desired. After the recognition system 33 responds, the learning algorithm 30 returns a proposed action by the machine. The question can be divided into separate parts, e.g. B. the initial state of the environment and the desired result of an action such as lifting, and another part that the desired result for another action, for. B. the loading procedure is taken into account. In this way, the detection system 33 can formulate an answer to the second question while the first action is being carried out. The initial conditions 31 may include any required environmental information required to accomplish the task, such as the shape and location of the excavation site, the truck loading height, and the location where the excavation is loaded. The results of actions 35 may relate to any aspect of performing the task, such as the specific volume of the soil covered per load, excavating a maximum amount of soil in the minimum possible time and / or the location for the mass of the loaded material. The information about the environment and the desired results are available from the sensor system 20 and from the recognition system 33 via the results of the action data 35 .

The optimization routine 38 of the learning algorithm 30 generates or initiates a proposed action for the excavator using the initial conditions 31 , the desired results and the experience from previous actions and results. A function approximator 36 is used by the optimization routine 38 to predict the result of a pending action before it is performed on the machine. The predicted result is then used to calculate the cost of the upcoming action, which relates to how close the predicted and desired result is. The functional approximation can be carried out in various ways; In the preferred embodiment, a memory-based learning model is used, in which all previous actions of the machine are saved exactly and explicitly over the entire life of a remote-controlled machine. Such a memory-based learning model is locally weighted regression. With locally weighted regression, points in a database are assigned a weighting that is proportional to the distance of the points from the upcoming action. Segments or locations of complex, non-linear functions can be approximated by relatively simple algebraic models, such as linear or quadratic equations. An exponential weighting term is typically used and the coefficients for the local model of the data are calculated using the weighted data. Weighting the data gives data points greater impact, which are closer to the upcoming action than points farther away. Segments of complex, non-linear functions can be approximated because the weighting of the local points of a segment allows different coefficients for different areas of the non-linear function.

The database entries, namely the upcoming remote controlled or robotic actions, and the expenditure, namely the The results of the robot actions depend on the task itself. With an autonomous excavator, with the parameterized Scripts used are input or pending Actions a set of script parameters. The expenses are the desired variables for optimization or improvement the actions, such as execution time of a task, machine ref efficiency and / or accuracy in performing the task. The output variables are preferred for every work cycle measured using suitable sensor systems or he sums up.

For example, a simplified implementation of the learning algorithm 30 for an earth moving machine, namely an excavator 50 ( FIG. 2), includes the selection of a limited number of input and output variables for monitoring during the execution of the task. In this example, most of the relevant script parameters for learning algorithm 30 were chosen to determine an excavation cycle as follows:

• the angle between a boom 55 and a horizontal plane 56 which triggers the pivoting to a truck 57 ,
• the angle of rotation about a pivot axis 60 which triggers the movement of a rod 54 for the unloading maneuver,
• the angle of rotation about the pivot axis 60 which triggers the beginning of the opening of the blade 58 ,
• the angle between the rod 54 and the arm 55 which triggers the opening of the bucket 58 ,
• the angle between the rod 54 and the boom 55 for the first half of the unloading process,
• - The angle between the rod 54 and the boom 55 for the second half of the unloading process, and
• - The angle of the blade 58 , which triggers the movement of the rod 54 from the first to the second unloading position.

In the embodiment, the output variables stored for each work cycle include the time (t) to completion of the unloading movement that begins and ends immediately after the excavation has ended when the excavator has unloaded the load and pivoted back to the excavation site, and the location of one Excavation hill 52 in polar coordinates (r, θ) with respect to the excavator. The output space is therefore three-dimensional and the input space is seven-dimensional, so that there are a total of 10 numbers which are stored in the database for each work cycle.

The optimization routine 38 creates an initial set of script parameter to be executed or "candidate". These candidate parameters are then evaluated by the function approximator, which returns a predicted result of the action. This predicted result is used to calculate the effort or rank to perform the given upcoming action. The effort information is then used by the optimization routine 38 to select a new set of candidate script parameters that have a lower effort or a higher rank. This process continues until the optimization routine 38 arrives at an end set of script parameters.

The candidates used by the function approximator 36 group script parameters is chosen approximately 38 component using the optimization. Various optimization routines can be used in the optimization component 38 of the learning algorithm 30 in FIG. 1 in order to choose an upcoming action. If the dimension of the action space is small enough, a rough search of all actions with a somewhat finite resolution can be acceptable. In other cases, when the dimensionality is high, e.g. For example, over 4, some options may be to choose actions arbitrarily or perhaps to make arbitrary changes to the best precedent actions. You can also try an interpolation that creates a new pending or "candidate action" from a linear combination of previous actions in the database. Another possibility is to generate actions arbitrarily and, as a way to prioritize the actions, use an assessment of the reliability of the predicted accuracy. If the desired result can be expressed as an effort function to be minimized or maximized, an algorithm can be used to select an action, which is known as gradient descent.

In a preferred embodiment of the invention, the known "downhill simplex method" is used, in which a locally weighted linear regression function is applied in the approximator 36 for each function evaluation carried out. The initial simplex value is calculated using intelligence above or around the gradient at the starting point. Instead of adding a small value along each dimension of the input space, each term in the gradient (or negative gradient, if minimized) is added to the starting point. If there is a very steep incline in one dimension of the input space, the corresponding simplex vertex can start further downhill and lead to a minimum more quickly. The optimization routine can be started in various different places if there are potential difficulties with local minima. The starting points are selected as the previous machine actions in the database with the best results, such as lowest effort, shortest time or lowest radial error.

The optimization component 38 is executed until it reaches a minimum cost. An example of an expense function that is suitable for the unloading phase of the mass excavation task is the following equation:

c = w ₁ t + w ₂ (loc _des - loc _act ) ² [1]

where w ₁ and w ₂ weighting values on the various terms and loc _des - loc _{act is} the error between the target and the actual position of the excavation pile during the unloading phase of the movement. For the situation shown in FIGS . 2 and 3, the effort function can be written as follows:

c = w _t t + w _r (r _des - r _act ) ² + w _θ (θ _des - e _act ) ² [2].

Therein t is the execution time, r _act and θ _act the actual radius or angle of the excavation pile 52 with respect to a reference point at the excavator point to the center of the excavation pile 52 ( FIG. 3), r _des and θ _des are the desired or desired Coordinates of the excavation pile 52. w _θ before the tangential error term is a weighted term that is used to compare the tangential error versus the radial error using the equation

s = rθ [3]

to dimension and / or otherwise scale, where s is the arc length in the tangential direction at radius r. This special expense function is a linear combination of the various output variables, namely the execution time and the components of the accuracy of the operation, e.g. B. the radial and the tangential position error. These terms can be weighted with adjustable weights or factors, whichever is considered more important. For example, the factors can be selected by determining how much a time second in units of distance is worth, ie if w _t = 2 and w _r = 400, then 0.5 time seconds are equivalent to 5 centimeters of spatial error. The ability to adjust these factors allows a human supervisor to prefer one criterion over the other depending on the type of task.

Examples of other criteria included in the expense function could include: the error between the Target and actual volume of the excavator in the bucket ground that machine efficiency was available compared to the power consumed, and a measure of Uniformity of movement, especially when on the Ma Automatic detection sensors are attached. With this technique lets you choose an action to be performed on the robot should be presented as an optimization problem. There are Many techniques for optimizing a function are already known. With this solution, the learning system chooses from what it already has knows the action that it thinks is best, rather than one arbitrary action.

The optimization routine 38 uses the function appro ximator 36 to predict the result of an upcoming machine action, which is used to calculate a rank for that action. The function approximator 36 uses a locally weighted linear regression algorithm in this embodiment to calculate the predicted results. A weighted scheme is used in the preferred embodiment to emphasize the previously stored parameters that are closer to the "candidate" parameters. The following weighting function is one of many that can be used:

where w _{i is} the weight assigned to the i-th group of parameters in the database, i between 1 and n, n the number of cycles for which data is stored, x _i the i-th data point, ie the group of parameters that was used in the recording of the data point, input the group of one or more "candidate" parameters that are input to the function approximator 36 whose output is predicted, K the kernel width that scales the exponential term, and D is a Euclidean distance function , which leads back the quadratic distance between the input and the ith data point. Points that are very close to the entrance are weighted higher than points that are further away; kernel width affects location and customization. Large kernel widths result in a globally equal weighting of the data points, while a small kernel width only weights the next data points. All input data is normalized between 0 and 1 based on predetermined limits in the range of each input parameter before the weight or factor is calculated. This avoids that a data point with a very different scale dominates the distance calculation. For a database with a large amount of data, the number of cycles included in the calculation can be limited to a selected number. Furthermore, intelligent data structures, such as kd trees, can be used to greatly accelerate data acquisition and weight or factor calculation.

Both the input and the output terms of each data point are multiplied by the weight factors. One way to express this in matrix form is:

Z = WX [5]
v = Wy [6]

Assume there are n data points in the database and m Terms for the entry of each data point, then W is a (n × n) diagonal matrix of the weight factors, X an (n × m) matrix the input term or parameter of each data point and y (n × 1) vector of the output or assigned to each data point Output values. In the case of multiple outputs, for example temporal and spatial accuracy, are several under different y and v vectors exist, but they would  same weight factors because they are a function of Are the distances between the input terms. To the problem solve, not needing the linear model to go through the To go to origin, an additional column of 1's is added Matrix X added, so that there is an (n × (m + 1)) matrix results.

After the Z and v matrix have been calculated, the m + 1 coefficients β of the linear model are found by solving the following equation:

Zβ = v [7]

A numerically stable algorithm for singular or specifying almost singular matrices becomes a solution the above matrix equation for β a singular value resolution algorithm used. For every forecast for every exit different β vectors are calculated.

Once β is calculated, the predicted output is

y '= β ^T input [8]

One of the more important variables to choose from locally weighted linear regression technique is the kernel width K. This can be done automatically by cross-reference or cross-validation happen. During the cross validation, an or removed more data points from the existing database, and the "sub-database" is with these extracted points queried. The mistake between the predicted and the did factual answer is calculated. In this way, the Kernel width K can be chosen which has the lowest cross vali dation error indicates.

The locally weighted linear regression is desirable worthy properties, u. a. easily available gradient information mation, the ability to deal with disruptive data, and no local minimum difficulties (like in neural Networks) because the answer is calculated analytically. More complex regression algorithms, such as the Bayesian  Regression, confidence intervals can also be based on the return predicted output and fault statistics. Quadratic regression algorithms can be used if the functions are not linear, but locally quadratic are.

After the next machine action is determined by the learning algorithm 30 , the scripts 32 are filled with the script parameters, the action is carried out and the results are measured and stored. As this process of collecting more data progresses, the script parameters are updated to improve the work quality of the machine. Different robot learning systems require a period of experimentation or "practice" to explore areas of the site, which can ultimately lead to a more efficient way of working. According to the invention, some initial calibration operations can be performed before the task begins. For example, with an excavator, it is possible to experiment with the excavation movement parameters and then put the ground back in the same place after the test is finished. The trial phase can also provide data that can be used when the result of an action cannot be fully measured during operation. This can happen if the line of sight of a sensor system from the area of interest is blocked and the data is therefore faulty or missing. In this situation, data obtained during calibration can be used to replace the missing data.

There is no "history" or database to look at change at the start or to improve operations can leave if a machine in a new place to Getting started is a solution, data from previous operations and / or machines and assume that the environmental conditions are similar. Another option is human Operate a few cycles initially and the system the relevant parameters and measured results to be saved in the database. Another procedure  is to start with an initial set of parameters start that with machine models and heuristically found Values are calculated. The initial parameters can then be improved after the learning algorithm the wide control takes over.

The learning algorithm according to the invention can also be used with others Ren motion control techniques are combined, for example wise rule-based systems. This hybrid solution leaves Knowing about changeable parts of a given Task that can be coded in rules. Then parameters for such parts of the task can be calculated those for whom the learning process improves work effectiveness can learn.

The invention can be applied to various types of construction and agricultural machinery. An example of a script for a hydraulic excavator is shown below that was designed for

• (1) moving around obstacles detected in the terrain,
• (2) Minimize the time for each bucket load by coupling simultaneous movements between the rod 54 , the boom 55 , the bucket 58 and the movement about the pivot axis 60 of the excavator 50 ( FIG. 2), and
• (3) Minimize spillage of soil on the ground rings around the truck while tipping.

For this script, all parameters are connected angles. The Numbers in bold are new for each bucket load calculated parameters. The parameters defined as commands are calculated based on the geometry of the excavator. The parameters in the rules section of each script step are Triggering parameters using dynamic information or heuristic values can be calculated.  

As an example of how to read the script, consider the first two steps of the pan sub-script. The script begins at step 1 when the lift is complete and the pan command associated with step 1 is 5 °, which is the current pan angle. During this time, the boom angle is monitored while boom 55 is being raised . When the boom 55 passes a certain angle, in this case 14 °, the script step switches from step 1 to step 2 and the pivot command switches from 5 ° to 101 °, causing the excavator 50 to pivot to the designated unloading point. The script continues to send this swivel command until the conditions are met to switch to step 3 and change the swivel command again.

The learning algorithm modifies the parameters if it notes that a change in the effectiveness of the above be written operation improved. The scripts can be found at many different types of complicated motion controls be used in robotic systems where complicated tasks can be broken down into a series of simple steps. The Using variable parameters in general commands is egg an effective way to refine operations. The coupling of  Movement of connections via parameters with external conditions values, including the occurrence of Events as a result of the movement of other connections and internal factors, such as power limits, simplify the Generation of commands or subscripts and enables one flexible operation.

Claims (15)

1. A method for controlling the automated movement of an earth moving machine with a plurality of mechanical connections connected at connection points, which are simultaneously movable on commands generated by at least one script, each script containing at least one variable parameter defining a movement of the machine, with the following steps:
Determining at least one desired result,
Measuring the conditions in the vicinity of the machine relating to any desired result, and
Determine the candidate value for at least one variable parameter using a learning algorithm before executing the script in which the variable parameter is used. 2. The method according to claim 1, characterized in that the candidate value for each variable parameter is a predetermined value that is used during the initial execution of at least one script, further comprising the following steps:
Storing the measured conditions during at least one execution cycle, and
Store the candidate value used for each variable parameter during at least one execution cycle. 3. The method according to claim 2, characterized in that the learning algorithm comprises the following step:
Executing a function approximator ( 36 ) to evaluate each candidate value by predicting the result using at least one stored value for each variable parameter and at least one of the stored conditions from at least one execution cycle. 4. The method according to claim 3, characterized in that the learning algorithm ( 30 ) further includes the following step:
Optimizing an expense function which is dependent on at least one variable which represents the desired result and on at least one corresponding data value from the actual measured conditions. 5. The method of claim 4, wherein executing the func tion approximator ( 36 ) comprises the step of:
Predicting the result of the machine action using a weighted regression algorithm, wherein the weighting factors are calculated using an exponential function that is dependent on the difference between at least one stored variable parameter and the candidate value. 6. The method according to claim 5, characterized in that the step of executing the function approximator ( 36 ) comprises using a locally linear function to approximate a segment of the response. 7. The method of claim 5, characterized in that the step of executing the function approximator ( 36 ) comprises using a locally quadratic function to approximate a segment of the response. 8. A system for controlling the automated movement of a machine, comprising:
a machine with several mechanical connections connected to one another at connection points, which can be moved at the same time on command,
a processing system for executing at least one script, each of which has at least one variable parameter defining a movement of the machine, and
a learning algorithm for modifying the values of at least one variable parameter based on a desired result of a machine action and conditions measured in the environment of the machine that relate to the desired result. 9. System according to claim 8, characterized in that we at least one candidate value for each variable parameter is a predetermined value during the initial execution tion of at least one script and that a Data storage device for storing the measured conditions conditions during at least one execution cycle and at Save the for each variable parameter for at least of an execution cycle used value is provided. 10. The system of claim 9, characterized in that the learning algorithm ( 30 ) comprises a functional approximator ( 36 ) having at least one candidate value for modifying at least one variable parameter by predicting the result using at least one stored value for each variable parameter and evaluates at least one of the stored measured conditions from at least one execution cycle. 11. The system of claim 10, characterized in that the learning algorithm further comprises:
an expense function which is dependent on at least one variable which represents the desired result and at least one corresponding data value from the actually measured conditions. 12. The system of claim 11, characterized in that the functional approximator ( 36 ) further comprises:
a weighted regression algorithm for predicting the result of the machine action, wherein the weighting factors are calculated using an exponential function that is dependent on the difference between at least one stored variable parameter and the candidate value. 13. System according to claim 12, characterized in that the Function approximator also the use of a local linea function to approximate a segment of the response of the Machine contains. 14. System according to claim 12, characterized in that the Function approximator also the use of a local qua dramatic function to approximate a segment of the answer the machine contains. 15. System according to claim 8, characterized in that we at least a fixed command to control the movement of the Machine is provided.