DE102018200060B4 - Method for operating a mobile work machine and mobile work machine - Google Patents

Abstract

Method for operating a mobile work machine (100, 100') having a manipulator (110) movably arranged on a component (120) of the mobile work machine (100, 100'),
wherein a position of the manipulator (110) relative to the component (120) is determined using at least one sensor (111, 112, 113, 114, 114) arranged on the manipulator (110),
wherein a rotation (D) of the component (120) about a vertical axis (H) of the mobile work machine (100, 100') is detected using a rotation rate sensor (121) arranged in a fixed position with respect to the component (120), and
wherein a position of the manipulator (110) within the mobile work machine (100, 100') is determined based on the position of the manipulator (110) relative to the component (120) and the rotation (D) of the component (120) about the vertical axis (H),
characterized in that the rotation (D) of the component (120) about the vertical axis (H) is further detected using at least one camera (125, 126, 127) arranged on the mobile work machine (100, 100').

Description

The present invention relates to a method for operating a mobile work machine having a manipulator, as well as a computing unit for carrying out the method and such a mobile work machine.

State of the art

Kinematics or manipulators of mobile work machines, such as the working arm of an excavator, can be controlled purely hydraulically, but there is a trend towards electrification of hydraulics in mobile work machines, particularly construction machines and excavators in particular. In addition, operation can be improved by means of assistance systems that show a driver on displays how exactly he has to dig, for example, or that specify lines or contours that the driver must not cross. Advanced systems can, for example, also intervene in the control of the hydraulics if the driver crosses a boundary line, or they carry out subtasks automatically.

The basis of such functions is the ability to determine the state of the kinematics or the manipulator and in particular the so-called tool center point (TCP), a reference point for the tool on the manipulator, for example on the excavator arm, exactly in relation to the mobile work machine, for example to the upper carriage of an excavator, and absolutely on the globe.

The position of an excavator's upper carriage as a mobile work machine on the globe is usually determined using a GPS-based system. To determine the tool center point on the globe, the position of the tool center point relative to the GPS reference point on the upper carriage must therefore be determined. There are various approaches to this, such as calculating the tool center point based on the kinematics of the excavator arm and displacement sensors installed on the cylinders, angle sensors installed on the joints or inertial sensors on the individual parts of the excavator arm. Inertial sensors and angle encoders or yaw rate sensors are particularly common.

From the EN 10 2009 018 070 A1 For example, a mobile work machine is known in which positions are determined using inclination sensors and yaw rate sensors. From the EN 10 2012 102 291 A1 For example, a method for operating a mobile work machine is known in which a magnetic compass is used to determine more precise orientations of the mobile work machine or a manipulator of this mobile work machine.

Further state of the art can be found in EN 11 2015 006 905 T5 , the EN 11 2015 000 126 T5 , the EN 11 2014 000 079 B4 , the EN 11 2009 001 466 B4 , the DE 41 10 959 C2 and WO 2016 / 164 975 A1.

Disclosure of the invention

According to the invention, a method for operating a mobile work machine, a computing unit for carrying it out and such a mobile work machine with the features of the independent patent claims are proposed. Advantageous embodiments are the subject of the subclaims and the following description.

The invention is based on a method for operating a mobile work machine that has a manipulator that is movably arranged on a component of the mobile work machine. In the case of an excavator as a mobile work machine, such a component can be an upper carriage of the excavator. In the case of a loading crane on a truck, it can also be, for example, an articulated device on the truck to which the manipulator or crane arm is attached.

In this case, a position of the manipulator relative to the component is determined using at least one sensor arranged on the manipulator. Such sensors include in particular displacement sensors, angle sensors and inertial sensors, in particular rotation rate sensors and/or acceleration sensors. Using a rotation rate sensor arranged in a fixed position relative to the component, a rotation of the component about a vertical axis of the mobile work machine is then detected. A position of the manipulator within the mobile work machine is then determined based on the position of the manipulator relative to the component and the rotation of the component about the vertical axis.

In the following, the method will be described and explained in more detail using an excavator as a mobile work machine with an excavator arm or work arm as a manipulator. However, it goes without saying that such a method can also be used with other mobile work machines with a manipulator.

The sensors mentioned can now be installed on the individual parts of the excavator arms, the yaw rate sensor then on the upper carriage or a part of it as the mentioned component. The sensors can in particular be inertial sensors, which can measure both the accelerations and the yaw rates around certain, preferably all three spatial axes. A resulting measured acceleration can be interpreted as the acceleration due to gravity when the excavator and its kinematics or manipulator are at a standstill, from which an absolute angle of the corresponding inertial sensor to the earth's gravitational field can be deduced.

Since this signal is usually very noisy and can be very distorted by the accelerations caused by the movement of the upper carriage, the angle signal resulting from the acceleration signals can be fused with a rotation rate measured in the direction of the rotation axis of the upper carriage or the vertical axis of the excavator or its integral via a Kalman filter. With knowledge of all the absolute angles of the individual links of the working arm, the relative angles between the respective links can be determined so that (taking the other dimensions into account) the position of the manipulator or the tool center point (TCP) can be determined.

However, since the rotation axis or vertical axis is usually (or at least in most situations) always oriented parallel to the earth's gravitational field, only the corresponding rotation rate measured with the rotation rate sensor on the upper carriage can be used to determine the position of the upper carriage, and thus also the position of the tool center point in space. This signal cannot be supported with an acceleration sensor with respect to the earth's gravitational field. Since the rotation rate sensor usually has an unknown offset and/or noise, any error is continually added up and thus increased by the integration of the rotation rate required for position determination (a so-called drift occurs).

According to the invention, it is now provided that the rotation of the component around the vertical axis is further (i.e. additionally) recorded using at least one camera arranged on the mobile work machine. In this way, the rotation rate signal (i.e. the signal from the rotation rate sensor) can be supported, i.e. improved, or fused with another signal for determining the position, namely the signal from the camera.

Since, for example, excavators or other mobile work machines are often already equipped with rear-view cameras and there is also a trend towards recording the surroundings, future excavators or other mobile work machines will increasingly be equipped with cameras anyway. In order to be able to support the upper carriage rotation rate using a signal from such a camera or to improve the rotation recorded or determined using the rotation rate sensor, the speed of the upper carriage can be determined absolutely in relation to the surroundings and/or relative to the undercarriage and fed, for example, to an existing fusion algorithm (e.g. a Kalman filter).

In this respect, it is also preferred to continue to record a translational movement of the mobile work machine using the at least one camera arranged on the mobile work machine.

As mentioned, a position of the mobile work machine in space can preferably be determined using radio and/or satellite support, in particular using GPS and/or mobile phone triangulation. The determined position of the mobile work machine in space can then be improved using the recorded translational movement. This can be done both with satellite-supported determination of the position and with another method of determining the position, possibly only relatively.

Based on the position of the manipulator within the mobile work machine and the position of the mobile work machine in space, a position of the manipulator (or the tool center point) in space can then be determined in a particularly preferred manner.

It is particularly preferred if the mobile work machine comprises an excavator in which the component on which the manipulator designed as an excavator arm is arranged is part of a superstructure. Alternatively, however, it is also preferred if the mobile work machine comprises a telehandler, a truck with a loading crane or a forestry machine, each of which also has a manipulator or working arm.

It is also useful for such mobile work machines if, as already mentioned, a driver of the mobile work machine is guided based on the determined position of the manipulator, for example by displaying the position of the manipulator or tool center point in relation to specified work areas, which are shown using lines, for example, on a display device (or a screen). This simplifies working with the mobile work machine and increases safety. It is also conceivable that the movement of the manipulator is automatically and actively intervened in order to maintain the specified work area. The proposed use of the camera further increases the advantages of such a guided movement, as the position of the manipulator can be determined much better and more precisely.

A computing unit according to the invention, e.g. a control unit of a mobile work machine, is set up, in particular in terms of programming, to carry out a method according to the invention.

The invention further relates to a mobile work machine with a manipulator which is arranged on a component of the mobile work machine, at least one sensor arranged on the manipulator, a rotation rate sensor arranged in a fixed position with respect to the component, at least one camera and a computing unit according to the invention, that is to say, for example, a control unit or a control device with which the aforementioned steps of the method can be carried out.

The implementation of the method in the form of a computer program is also advantageous, as this causes particularly low costs, especially if an executing control unit is also used for other tasks and is therefore already available. Suitable data storage devices for providing the computer program are in particular magnetic, optical and electrical storage devices, such as hard disks, flash memories, EEPROMs, DVDs, etc. It is also possible to download a program via computer networks (Internet, intranet, etc.).
Further advantages and embodiments of the invention emerge from the description and the accompanying drawings.

It is understood that the features mentioned above and those to be explained below can be used not only in the combination specified in each case, but also in other combinations or on their own, without departing from the scope of the present invention.

The invention is illustrated schematically in the drawing using an embodiment and is described in detail below with reference to the drawing.

Character description

• 1 shows schematically a mobile work machine according to the invention in a preferred embodiment, with which a method according to the invention can be carried out.
• 2 shows schematically a sequence of part of a method according to the invention in a preferred embodiment.
• 3 shows schematically a sequence of a method according to the invention in a further preferred embodiment.
• 4 shows schematically a mobile work machine according to the invention in a further preferred embodiment.

Detailed description of the drawing

In 1 A mobile work machine 100 according to the invention is shown schematically in a preferred embodiment, here in the form of an excavator, with which a method according to the invention can be carried out.

The excavator 100 has a manipulator 110 in the form of an excavator arm or working arm, which is movably arranged or attached to a part of an upper carriage 120 as a component of the mobile work machine. The upper carriage 120 is in turn connected to an undercarriage 130 so as to be rotatable about a vertical axis H of the excavator.

Five sensors 111, 112, 113, 114 and 115 are arranged on the manipulator 110, for example, each on a link of the manipulator 110. These sensors can be inertial sensors, in particular acceleration and yaw rate sensors. A yaw rate sensor 121 is arranged on the upper carriage 120, with which a yaw rate or a rotation D (then by integrating the yaw rate) about the vertical axis H can be detected. Furthermore, a camera 125, here in the form of a rear-view camera, is arranged on the upper carriage 120.

The rear view camera 125 is designed to record the rear area of the excavator 100. For this purpose, it is also aimed at the ground. By appropriately evaluating the image signals, conclusions can be drawn about both the vehicle speed and the rotation speed (i.e. the speed of rotation around the vertical axis) of the excavator or the upper carriage. These signals can be used in a suitable manner to support the measurement signals obtained from the inertial sensors in order to improve the estimation of the position of the tool center point, as described in more detail below. will be explained. Such a method can be carried out in particular by means of a computing unit 140, for example in the form of a control unit of the excavator 100.

One possible method for making camera-based conclusions about the vehicle speed or the rotation of the upper carriage can be seen in the evaluation of the optical flow. Feature-based methods (e.g. matching methods) or identity-based methods (differential techniques, for example using gradients or structure tensors, correlation techniques or filter-based techniques) are available for this. Since real-time capability must be guaranteed, particular reference is made to the matching method, as described, for example, in the "Handbook of Driver Assistance Systems: Basics, Components and Systems for Active Safety and Comfort", Springer-Verlag, 2015, by Hermann Winner, Stephan Hakuli, Felix Lotz and Christina Singer.

In this case, correspondences are only searched for within the image position specified by the image grid. The correspondence is determined by the region that is most similar in terms of a certain dimension or object. The search space can be restricted to distinctive image structures (e.g. distinctive points of the excavator and/or its surroundings that are always recorded). The speed can be determined from the image sequences and the resulting distance between the identical features of successive images evaluated by the algorithm.

In principle, the upper carriage rotation D can be determined by integrating the rotation rate around the z-axis, here the vertical axis H. However, this has the disadvantage that rotation rate sensors, which are based on MEMS technology, for example, have an offset in addition to the measured rotation rate. This would lead to a large drift of the calculated angle after a short time during integration and would deviate significantly from the actual upper carriage rotation.

Accordingly, support has so far been provided with acceleration sensors in order to compensate for the drift. However, this has the disadvantage that no support information is available for the yaw movement, ie the rotation around the vertical axis H, as already explained at the beginning. Assuming a level positioning of the excavator, the yaw axis (z-axis) coincides with the direction of the gravitational acceleration g, as in 1 shown. Accordingly, no information about the current yaw angle can be obtained from the acceleration measurements. To overcome this problem, a magnetometer can be used to support the yaw motion.

This sensor uses the earth's magnetic field as a reference to calculate the current orientation in relation to the earth's magnetic field. However, one problem is the many metal components of the mobile work machine, in this case the excavator, which have a significant influence on the sensor technology.

entspricht dem tatsächlichen, vom erdfesten Navigationssystem mittels homogener Koordinatentransformation $C_n^b$

in das sensoreigene Koordinatensystem transformierten Magnetfeld mⁿ, welches Verzerrungen aufgrund naher metallischer Objekte (Soft-Iron, ausgedrückt in der Matrix K) sowie magnetischer Störfelder o^b (Hard-Iron) unterliegt. Aufgrund der Abhängigkeit von inkonsistenten Magnetfeldern im nahen Umkreis der mobilen Arbeitsmaschine ist ein Magnetometer zur Stützung nur bedingt geeignet.The magnetic field measured by the sensor $${\widehat{m}}^b=K{C}_n^b{m}^n+O^b$$

corresponds to the actual coordinates determined by the earth-based navigation system using homogeneous coordinate transformation $C_n^b$

magnetic field m ⁿ transformed into the sensor's own coordinate system, which is subject to distortions due to nearby metallic objects (soft iron, expressed in the matrix K) as well as magnetic interference fields o ^b (hard iron). Due to the dependence on inconsistent magnetic fields in the immediate vicinity of the mobile work machine, a magnetometer is only partially suitable for support.

$${\dot{r}}_{k+1}^n={\dot{r}}_k^n+T{\ddot{r}}_k^n$$

$$q_{k+1}^n=q_k^{nb}\square \text{ exp}\left(\frac{T}{2}{\omega }_{nb,k}^b\right)$$

In order to support the offset-affected yaw rate sensor, the upper carriage rotation is now detected using the camera mentioned and fed into a fusion algorithm, e.g. a Kalman filter. The visual recording of the current rotation of the upper carriage can be carried out using special image processing algorithms. In this regard, an overall model consisting of an acceleration sensor, yaw rate sensor and camera can be created: $$r_{k+1}^n=r_k^n+T{\dot{r}}_k^n+\frac{T^2}{2}{\ddot{r}}_k^n$$

$${\dot{r}}_{k+1}^n={\dot{r}}_k^n+T{\ddot{r}}_k^n$$

$$q_{k+1}^n=q_k^{nb}\square \text{ex}\left(\frac{T}{2}{\omega }_{nb,k}^b\right)$$

$${\omega }_{nb,k}^b=z_{\omega ,k}-R_k^{bn}{\omega }_{ie}^n-b_{\omega ,k}^b-{\eta }_{\omega ,k}^b,$$

ausgedrückt im n-Frame bzw. Navigationskoordinatensystem, setzt sich aus der gemessenen Beschleunigung des Beschleunigungssensors z_a, dem Offset $b_a^b,$

weißem gaußverteiltem Rauschen η^b mit Mittelwert 0 sowie der Erdbeschleunigung gⁿ zusammen. $2{\omega }_{ie}^n\times {\dot{r}}_k^n$

beschreibt die Coriolis-Beschleunigung und ${\omega }_{ie}^n\times \left({\omega }_{ie}^n\times r_k^n\right)$

die Zentripetalbeschleunigung bedingt durch die Erdrotation ${\omega }_{ie}^n.$

Here, q ^nb denotes the unit quaternion, which describes the orientation in the navigation coordinate system (so-called n-frame) relative to the sensor's own coordinate system (so-called b-frame). r ⁿ and ṙ ⁿ describe the position and speed of the upper carriage in the n-frame. The sampling rate is specified with the sampling interval T. ⌷ corresponds to the quaternion multiplication. The resulting acceleration $${\ddot{r}}_k^n=R_k^{nb}\left(z_{a,k}-b_{a,k}^b-{\eta }_{a,k}^b\right)-2{\omega }_{ie}^n\times {\dot{r}}_k^n-{\omega }_{ie}^n\times \left({\omega }_{ie}^n\times r_k^n\right)+G^n$$

$${\omega }_{nb,k}^b=z_{\omega ,k}-R_k^{bn}{\omega }_{ie}^n-b_{\omega ,k}^b-{\eta }_{\omega ,k}^b,$$

expressed in n-frame or navigation coordinate system, consists of the measured acceleration of the acceleration sensor z _a , the offset $b_a^b,$

white Gaussian distributed noise η ^b with mean value 0 and the acceleration due to gravity g ⁿ . $2{\omega }_{ie}^n\times {\dot{r}}_k^n$

describes the Coriolis acceleration and ${\omega }_{ie}^n\times \left({\omega }_{ie}^n\times r_k^n\right)$

the centripetal acceleration caused by the Earth’s rotation ${\omega }_{ie}^n.$

bei der $p_k^l$

das umgewandelte 2D-Feature und $p_k^c$

die zugehörige 3D-Position der Kamera im Kamera-Koordinatensystem zusammen mit weißem gaußschem Rauschen $e_k^l$

entspricht. Die Projektion ℘ beschreibt die Berechnung der Pose mit Hilfe des optischen Flusses. Dafür wird in einer Bildsequenz der optische Fluss berechnet, d.h. es wird ein Ortspunkt auf eine Abbildungsebene abgebildet, somit lässt sich die Bewegung des Abbildes mit einem Flussvektor darstellen. Die Segmentierung kann nach Farben oder Texturen erfolgen. Dieser Flussvektor entspricht einem Geschwindigkeitsvektor und dient als Basis zur Ermittlung der Oberwagendrehung.The camera pose can be described as projection ℘ according to $$p_k^l=\mathcal{R}\left(p_k^c\right)+e_k^l,$$

at the $p_k^l$

the converted 2D feature and $p_k^c$

the corresponding 3D position of the camera in the camera coordinate system together with white Gaussian noise $e_k^l$

The projection ℘ describes the calculation of the pose using optical flow. For this, the optical flow is calculated in an image sequence, ie a location point is mapped onto an image plane, so that the movement of the image can be represented with a flow vector. Segmentation can be done according to colors or textures. This flow vector corresponds to a speed vector and serves as the basis for determining the rotation of the upper carriage.

The fusion can be done using a Kalman filter. The function of the Kalman filter is divided into a prediction step and a correction step. The overall model described above is summarized in a vector x̂ _k , which contains the current pose (ie position r, speed ṙ and orientation q) and the sensor offsets b: $${\widehat{x}}_k=\left(r^n,{\dot{r}}^n,q^{nb},b_a^b,b_{\omega }^b\right).$$

In 2 A sequence of such a fusion is shown as part of a method according to the invention in a preferred embodiment. In a projection step 200, the measured rotation rate u _k is considered as input to the model. The rotation rate thus predicts the direction in which the orientation changes.

Dies erfolgt mit hoher Dynamik, ist jedoch Offsetbehaftet und macht einen Korrekturschritt 220 mit einer absoluten Größe z_k notwendig. In Nick- und Rollrichtung ist der Beschleunigungssensor zur Stützung ausreichend, jedoch ist aus den zuvor genannten Gründen die Stützung der Gierbewegung nicht möglich. Dementsprechend erfolgt eine Stützung der Gierbewegung durch die aus dem optischen Fluss berechnete Drehrate, welche ebenfalls im Vektor z_k zusammengefasst wird, um einen fusionierten Zustand x̂_k zu erhalten. Nach dem Korrekturschritt erfolgt eine Berechnung der Kovarianz in Schritt 230, was wiederum in den Prädiktionsschritt 200 mündet.This is incorporated in step 210 into a calculation of the Kalman gain, here with the input variables ${\widehat{x}}_0^{-},P_0^{-}.$

This is done with high dynamics, but is subject to offset and requires a correction step 220 with an absolute value z _k . In the pitch and roll directions, the acceleration sensor is sufficient for support, but for the reasons mentioned above, support of the yaw movement is not possible. Accordingly, the yaw movement is supported by the rotation rate calculated from the optical flow, which is also summarized in the vector z _k to obtain a fused state x̂ _k . After the correction step, the covariance is calculated in step 230, which in turn leads to the prediction step 200.

This method can be used in two situations. It enables drift-free support of the upper carriage rotation when the mobile work machine is at a standstill, as well as support of the joint angle calculation in the case of superimposed accelerations caused by a (translational) movement of the entire mobile work machine. A separation of the two cases should be ensured, since in the case of a combined movement (i.e. upper carriage rotation and translational movement) it is usually not possible to separate the velocity vectors in the optical flow. In order to cover this case as well, control inputs (e.g. from a joystick) could also be included in the fusion.

In 3 A sequence of a method according to the invention is now shown schematically in a further preferred embodiment, as has already been described. In a step 300, as mentioned, a position of the manipulator relative to the component or the upper carriage of the excavator can first be determined using the sensors arranged on the manipulator.

In a step 310, using the yaw rate sensor, a rotation D of the upper carriage about the vertical axis H (see also 1 ). Here, in a step 315, the rotation D around the vertical axis H can also be determined using the camera mentioned. In a step 320, the position of the manipulator within the excavator can then be determined based on the position from step 300 and the rotation from steps 310 and 315. In a step 330, the position of the excavator in space, which is determined, for example, using radio or satellite support and/or using the camera, can also be included so that the position of the manipulator in space can be determined.

In 4 A mobile working machine 100' according to the invention is shown schematically in a further preferred embodiment, here also in the form of an excavator. The excavator 100' corresponds to the excavator 100 from 1 , but with the difference that two cameras 126 and 127 are arranged here.

The two cameras 126 and 127 are arranged here on the underside of the upper carriage 120. In total, four cameras directed at the respective corners of the undercarriage 130 can be arranged in this way. These are advantageously designed as fisheye cameras and can therefore capture both the surroundings and the undercarriage over a large area. By capturing the undercarriage, it is possible to directly determine the relative speed between the upper carriage and the undercarriage and thus separate it from the overall vehicle speed. This can significantly improve the support of the yaw rate signals of the inertial sensor or yaw rate sensor on the upper carriage, since no distinction has to be made as to whether the speed measured by the camera comes from the rotation of the upper carriage or the translational movement of the excavator.

Claims (11)

Method for operating a mobile work machine (100, 100') which has a manipulator (110) which is movably arranged on a component (120) of the mobile work machine (100, 100'), wherein a position of the manipulator (110) relative to the component (120) is determined using at least one sensor (111, 112, 113, 114, 114) arranged on the manipulator (110), wherein a rotation (D) of the component (120) about a vertical axis (H) of the mobile work machine (100, 100') is detected using a rotation rate sensor (121) arranged stationary with respect to the component (120), and wherein a position of the manipulator (110) relative to the component (120) and the rotation (D) of the component (120) about the vertical axis (H) are used to determine a position of the manipulator (110) is determined within the mobile work machine (100, 100'), characterized in that the rotation (D) of the component (120) about the vertical axis (H) is further detected using at least one camera (125, 126, 127) arranged on the mobile work machine (100, 100'). Procedure according to Claim 1 , wherein a translational movement of the mobile work machine (100, 100') is further detected using the at least one camera (125, 126, 127) arranged on the mobile work machine (100, 100'). Procedure according to Claim 1 or 2 , wherein a position of the mobile work machine (100, 100') in space is determined by radio or satellite support, in particular using GPS. Procedure according to Claim 2 and 3 , wherein the determined position of the mobile work machine (100, 100') in space is improved based on the detected translational movement. Method according to one of the Claims 3 or 4 , wherein a position of the manipulator (110) in space is determined based on the position of the manipulator (110) within the mobile work machine (100, 100') and the position of the mobile work machine (100, 100') in space. Method according to one of the preceding claims, wherein the at least one sensor (111, 112, 113, 114, 114) arranged on the manipulator (110) is selected from displacement sensors, angle sensors and inertial sensors, in particular rotation rate sensors and acceleration sensors. Method according to one of the preceding claims, wherein the mobile work machine (100, 100') comprises an excavator in which the component (120) on which the manipulator (110) designed as an excavator arm is arranged is part of a superstructure, or wherein the mobile work machine comprises a telehandler, a truck with a loading crane or a forestry machine. Computing unit (140) configured to carry out a method according to one of the preceding claims. Mobile work machine (100, 100') with a manipulator (110) which is arranged on a component (120) of the mobile work machine, at least one sensor (111, 112, 113, 114, 114) arranged on the manipulator (110), a rotation rate sensor (121) arranged in a fixed position with respect to the component (120), at least one camera (125, 126, 127) and a computing unit (140) according to Claim 8 . Computer program which causes a computing unit (140) to carry out a method according to one of the Claims 1 until 7 when executed on the computing unit (140). Machine-readable storage medium with a computer program stored thereon in accordance with Claim 10 .

Images