GB2571004A

GB2571004A - Method for operating a mobile working machine and mobile working machine

Info

Publication number: GB2571004A
Application number: GB1900064.5A
Authority: GB
Inventors: Neyer Daniel; Steker Nils; nagel Philip; Rose Steffen
Original assignee: Robert Bosch GmbH
Current assignee: Robert Bosch GmbH
Priority date: 2018-01-04
Filing date: 2019-01-03
Publication date: 2019-08-14
Anticipated expiration: 2039-01-03
Also published as: DE102018200060A1; CN110027001A; DE102018200060B4; GB2571004B

Abstract

Mobile machine 100 has a manipulator arm 110 mounted movably on a component 120 such as the body of the excavator 100 or a loading crane of a lorry. The position of the manipulator arm 110 relative to the body 120 is determined using at least one sensor 111, 112, 113, 114 mounted on the manipulator, such as displacement sensors, angle sensors, inertial sensors, rotation rate sensors and acceleration sensors. Rotation D of the body 120 about a vertical axis H of the mobile working machine 100 is detected using a rotation rate sensor 121. Over-all position of the manipulator arm 110 is determined using manipulator arm 110 position relative to the body 120 and rotational position of body 120 about the vertical axis H. The rotation position D of the body 120 is redundantly detected using a redundant camera 125 such as the vehicles rear-view manoeuvring camera 125. This uses image analysis to confirm position relative to external elements such as comparison of ground features. Alternatively, two or more cameras (Fig 4: 126, 127) mounted under body 120 can analyse relative position of excavator substructure 130 or wheels. Redundant image is used to compensate for vibration or noise reduction.

Description

Method for operating a mobile working machine and mobile working machine

Description

This invention relates to a method for operating a mobile working machine having a manipulator, as well as a computer unit for implementing the method and such a mobile working machine.

Prior art

Kinematic systems respectively manipulators of mobile working machines, such as a working arm of an excavator for example, can be activated purely hydraulically but there is a trend towards the electrification of hydraulic systems of mobile working machines, particularly in the case of construction machines and especially in the case of excavators. Furthermore, operation can be improved by means of assistance systems whereby, for example, a driver is shown on a display exactly how he must excavate or whereby lines or contours which the driver may not exceed are predefined. Advanced systems are also able to intervene in controlling the hydraulic systems if the driver exceeds a boundary line or if they are carrying out partial tasks automatically, for example.

-2The principle behind such functions is the possibility of determining the status of the kinematic system respectively manipulator and especially the so-called tool center point (TCP), a reference point for the tool on the manipulator, for example on the excavator arm, exactly in relation to the mobile working machine, in other words to the superstructure of an excavator for example, and absolutely on the globe.

The position of a superstructure of an excavator in the form of a mobile working machine on the globe is usually determined via a GPS-based system. The position of the tool center point relative to the GPS reference point on the superstructure therefore also has to be determined in order to determine the tool center point on the globe. There are various ways of doing this, such as calculating the tool center point using the kinematic system of the excavator arm and displacement sensors fitted on the cylinders, angle sensors installed on the joints or also inertial sensors on the individual parts of the excavator arm, for example. Inertial sensors and rotary encoders or rotation rate sensors in particular are widely used.

A mobile working machine on which positions are determined by means of inclination sensors and rotation rate sensors is known from DE 10 2009 018 070 Al, for

-3example. A method for operating a mobile working machine whereby a magnetic compass is used to determine more accurate orientations of the mobile working machine respectively a manipulator of this mobile working machine is known from DE 10 2012 102 291 Al, for example.

Disclosure of the invention

Based on the invention, a method for operating a mobile working machine, a computer unit for the implementation thereof and such a mobile working machine having the features defined in the independent claims are proposed. Advantageous embodiments constitute the subject matter of the dependent claims and the following description.

The invention is based on a method for operating a mobile working machine having a manipulator which is mounted so as to be movable on a component of the mobile working machine. In the case where the mobile working machine is an excavator, such a component might be a superstructure of the excavator. In the case of a loading crane of a lorry, it may also be an articulation device on the lorry to which the

-4manipulator respectively crane jib is attached, for example .

Accordingly, a position of the manipulator relative to the component is determined using at least one sensor mounted on the manipulator. Such sensors might be displacement sensors, angle sensors and inertial sensors in particular, and thus in turn rotation rate sensors and/or acceleration sensors in particular. A rotation of the component about a vertical axis of the mobile working machine is then detected using a rotation rate sensor that is mounted stationary with respect to the component. A position of the manipulator within the mobile working machine is then determined using the position of the manipulator relative to the component and the rotation of the component about the vertical axis.

The method will be described and explained in more detail below based on the example of an excavator as a mobile working machine having an excavator arm respectively working arm as a manipulator. However, it should be noted that such a method may also be used accordingly for other mobile working machines having a manipulator .

-5In the case of an excavator, the aforementioned sensors may then be mounted on the individual members of the working arm and the rotation rate sensor is then mounted on the superstructure or a part thereof as the aforementioned component. In terms of sensors, these may be inertial sensors in particular, which are able to measure both accelerations and rotation rates about specific, preferably about all three, spatial axes. When the excavator and its kinematic system respectively its manipulator are stationary, a resultant measured acceleration can be interpreted as the earth's acceleration, from which an angle of the corresponding inertial sensor absolutely relative to the earth's gravitational field can then be derived.

Since this signal is very noisy as a rule and accelerations caused by movement when the superstructure is moving can be very severely distorted, the angle signal resulting from the acceleration signals may be merged with a rotation rate measured in the direction of the rotation axis of the superstructure respectively the vertical axis of the excavator respectively the integral thereof using a Kalman filter. Knowing all the absolute angles of the individual members of the working arm, the relative angles between the respective members can be determined so that (taking account of the other measurements) the

-6position of the manipulator respectively the tool center point (TCP) can be derived.

However, as a rule (or at least in most situations), since the rotation axis respectively vertical axis is always oriented parallel with the earth's gravitational field, only the corresponding rotation rate measured using the rotation rate sensor on the superstructure can be used to determine the position of the superstructure and hence also to determine the position of the tool center point in space. Support for this signal cannot be obtained using an acceleration sensor relative to the earth's gravitational field. Since the rotation rate sensor usually has an unknown offset and/or a noise, any error due to the integration of the rotation rate needed for determining position is repeatedly cumulated and thus magnified (resulting in a so-called drift) .

The invention therefore proposes that the rotation of the component about the vertical axis also (i.e. in addition) be detected using at least one camera mounted on the mobile working machine. The rotation rate signal (in other words the signal of the rotation rate sensor) can therefore be supported, i.e. improved respectively merged accordingly with another signal, namely the

-7signal of the camera, in order to determine the position.

Since excavators or other mobile working machines are often already equipped with rearview cameras, for example, and there is also a trend for detecting the surrounding area, future excavators or other mobile working machines will increasingly be equipped with cameras in any case. To enable the superstructure rotation rate to be supported using a signal of such a camera respectively to improve the rotation detected respectively determined using the rotation rate sensor, the speed of the superstructure absolutely relative to the surrounding area and/or relative to the undercarriage can be determined and forwarded to an already existing fusion algorithm (e.g. to a Kalman filter), for example.

In this respect, it is also preferable to also detect a translating movement of the mobile working machine using the at least one camera mounted on the mobile working machine.

As mentioned, a position of the mobile working machine in space may preferably be determined with radioassistance and/or satellite-assistance, in particular using GPS and/or mobile radio triangulation. The determined position of the mobile working machine in space can then be improved using the detected translating movement. This may be done by both a satellite-assisted determination of the position and a position determined by any other means, optionally also only relative.

Using the position of the manipulator within the mobile working machine and the position of the mobile working machine in space, it is then particularly preferable to determine a position of the manipulator (respectively the tool center point) in space.

It is particularly preferable if the mobile working machine comprises an excavator wherein the component on which the manipulator in the form of the excavator arm is mounted is part of a superstructure. Alternatively, however, it is likewise preferable if the mobile working machine comprises a telescopic handler, a lorry having a loading crane or a forestry machine, each of which also has a manipulator respectively working arm.

In the case of such mobile working machines, it is then also of practical effect if, as already mentioned, using the determined position of the manipulator, a

-9driver of the mobile working machine is guided, for example by displaying the position of the manipulator or tool center point relative to predefined working areas, represented by means of lines for example, on a display means (respectively a display). This simplifies work using the mobile working machine and increases safety. In this respect, automatic and active intervention in the movement of the manipulator would also be conceivable in order to keep to the predefined working area. Due to the proposed use of the camera, the advantages of such a guided movement are further enhanced because the position of the manipulator can be much better respectively more accurately determined.

A computer unit proposed by the invention, e.g. a control device of a mobile working machine, is set up, in particular by programming, to implement a method proposed by the invention.

Another object of the invention is a mobile working machine having a manipulator which is mounted on a component of the mobile working machine, at least one sensor mounted on the manipulator, a rotation rate sensor that is mounted stationary with respect to the component, at least one camera and a computer unit as proposed by the invention, namely a control unit or a

-10control device, by means of which the specified steps of the method can be implemented.

Implementation of the method in the form of a computer programme is also of advantage because it incurs particularly low costs, especially if an implementing control device is also used for other tasks and is provided in any case. Suitable data carriers for providing the computer programme in particular are magnetic, optical and electric storage devices, such as hard disks, flash memories, EEPROMs, DVDs, amongst others, for example. A download of a programme via computer networks (Internet, Intranet, etc.) is also possible .

Other advantages and embodiments of the invention may be found in the description and appended drawings.

It goes without saying that the features mentioned above and those to be explained below can be used not only in the respective specified combination but also in other combinations or on their own, without departing from the scope of this invention.

-11The invention is schematically illustrated in the drawings based on an example of an embodiment and described in detail below with reference to the drawings .

Description of the drawings

Figure 1 schematically illustrates a mobile working machine proposed by the invention based on preferred embodiment by means of which a method proposed by the invention can be implemented accordingly.

Figure 2

Figure 3 schematically illustrates a sequence of a part of a method proposed by the invention based on a preferred embodiment.

schematically illustrates a sequence of a method proposed by the invention based on another preferred embodiment.

Fiqure 4 schematically illustrates a mobile working machine proposed by the

-12invention based on another preferred embodiment.

More detailed description of the drawings

Figure 1 schematically illustrates a mobile working machine 100 proposed by the invention based on a preferred embodiment, in this instance in the form of an excavator, by means of which a method proposed by the invention can be implemented accordingly.

The excavator 100 has a manipulator 110 in the form of an excavator arm respectively working arm, which is mounted on respectively secured to a part of a superstructure 120 constituting a component of the mobile working machine so as to be movable. The superstructure 120 is in turn connected to an undercarriage 130 so as to be rotatable about a vertical axis H of the excavator.

In this example, five sensors 111, 112, 113, 114 and 115 are mounted on the manipulator 110, each on a member of the manipulator 110. These sensors may be inertial sensors, in particular acceleration and rotation rate sensors. Mounted on the superstructure

-13120 is a rotation rate sensor 121 by means of which a rotation rate respectively a rotation D (and then the rotation rate by means of integration) about the vertical axis H can be detected. Also mounted on the superstructure 120 is a camera 125, in this instance in the form of a rearview camera.

In this context, the rearview camera 125 is provided as a means of recording the area around the rear of the excavator 100. To this end, it is directed towards the ground. By an appropriate evaluation of the image signals, information about both the vehicle speed and the rotation speed (namely the speed of rotation about the vertical axis) of the excavator respectively superstructure can be derived. These signals can be applied in an appropriate manner to support the measurement signals obtained from the inertial sensors with a view to improving the estimate of the position of the tool center point, as will be explained in more detail below. Accordingly, such a method may be implemented by means of a computer unit 140 in particular, for example in the form of a control device of the excavator 100.

Evaluation of the optical flow may be seen as one possible method for enabling the vehicle speed or superstructure rotation to be derived based on a

-14camera. There are feature-based methods (e.g. matching method) or identity-based methods (differential techniques, for example by means of gradient or structure tensor, correlation techniques or filterbased techniques) available for this purpose. Since real time capability must be guaranteed, reference may be made to the matching method in particular, such as described in Handbuch Fahrerassistenzsysteme: Grundlagen, Komponenten und Systeme fur aktive Sicherheit und Komfort, Springer-Verlag, 2015, by Hermann Winner, Stephan Hakuli, Felix Lotz and Christina Singer, for example.

In this context, matches are sought only within the image position predefined by the image grid. The match is obtained by the most similar region within the meaning of a specific mass or object. Accordingly, the search area may be restricted to distinctive image structures (e.g. that are distinctive, always having recorded points of the excavator and/or its surrounding area). The speed can be derived from the image sequences and the resultant distance of the same features of consecutive images evaluated by the algorithm.

In principle, the superstructure rotation D can be determined by an integration of the rotation rate about

-15the z-axis, in this instance the vertical axis H. The disadvantage of this, however, is that rotation rate sensors, which are based on a MEMS technology for example, have an offset in addition to the measured rotation rate. During the integration, this would lead to a large drift of the calculated angle after only a short time and would vary greatly from the actual superstructure rotation.

This being the case, support until now has been implemented using acceleration sensors to enable the drift to be compensated. The disadvantage of this, however, is that there is no information available to support the yaw movement, i.e. the rotation about the vertical axis H, as already explained above. Assuming the excavator is in a level position, the yaw axis (zaxis) coincides with the direction of the earth's acceleration g, as illustrated in Figure 1. This being the case, no information can be obtained from the acceleration measurement values about the current yaw angle. To get round this problem, a magnetometer can be used to support the yaw movement, for example.

This sensor takes the earth's magnetic field as a reference for calculating the current orientation with respect to the earth's magnetic field. However, the many metal components of the mobile working machine, in

-16this instance the excavator, which have a significant effect on the sensor system, pose a problem.

The magnetic field measured by the sensor m^b = KC^bmⁿ + o^b corresponds to the actual magnetic field mⁿ transformed by the earthbound navigation system by means of homogeneous co-ordinate transformation C^b into the sensor's own coordinate system, which is subject to distortions due to close metal objects (soft iron, expressed in the matrix K) as well as magnetic interference fields o^b (hard iron). Due to the dependency of inconsistent magnetic fields in the close vicinity of the mobile working machine, a magnetometer is only suitable as a means of support in certain circumstances .

To enable support to be provided for the offsetaffected rotation rate sensor, the superstructure rotation is now detected with the aid of the aforementioned camera and forwarded to a fusion algorithm, e.g. a Kalman filter. The visual detection of the current rotation of the superstructure may be achieved by means of special image processing algorithms. In this respect, an overall model

-17consisting of acceleration sensor, rotation rate sensor and camera may be set up:

T² γ,η __ ~.n i τ~τι I__v.n 'fc+1 ^— 'fc Γ ¹ 'k Γ 2 '^fc

^.n _ ^,n i τν,η ' k+1 — 'k Γ ¹ 'k

Vk+i =

In the above, q^nb denotes the unit quaternion which describes the natural orientation in the navigation coordinate system (so-called n-frame) relative to the sensor's co-ordinate system (so-called b-frame). rⁿ and rⁿ describe the position and speed of the superstructure in the n-frame. The scanning rate is specified by the sampling interval T. □ corresponds to the quaternion multiplication.

The resultant acceleration

^nb,k

Πω,/c' expressed in the n-frame respectively navigation coordinate system is made up of the measured acceleration of the acceleration sensor z_a, the offset white

Gaussian distributed noise η^ϋ with mean value 0 and the

-18earth's acceleration gⁿ. Ζω^ΧΓ^¹ describes the Coriolis acceleration and ω£, X (ω£, X r/¹) the centripetal acceleration caused by the earth's rotation ωβ,.

The camera pose may be described as a projection p in accordance with

Pk = PiPk) + ^ek <

where p^l _k corresponds to the converted 2-D feature and p_kthe associated 3-D position of the camera in the camera co-ordinate system together with white Gaussian noise e_k. The projection p describes the calculation of the pose with the aid of the optical flow. To this end, the optical flow in an image sequence is calculated, i.e. a location point is imaged onto an image plane, thereby enabling the movement of the image to be represented by a flow vector. The segmentation may be based on colours or textures. This flow vector corresponds to a speed vector and is used as a basis for determining the superstructure rotation.

The fusion may be run with the aid of a Kalman filter. The function of the Kalman filter is divided into a prediction step and a correction step. The overall model described above is incorporated in a vector x_kwhich contains the current pose (i.e. position r, speed r and orientation q) and the sensor offsets b:

-19x_k = (.rⁿ,rⁿ,q^nb,b^b,b^b) .

In this context, Figure 2 illustrates a sequence of such a fusion as part of a method proposed by the invention based on a preferred embodiment. In a projection step 200, the measured rotation rate Uk is regarded as an input to the model. The rotation rate therefore predicts the direction in which the orientation changes.

In step 210, this flows into a calculation of the Kalman gain, in this instance with the input variables Xo,Pq. This takes place on a highly dynamic basis but is affected by offset and makes a correction step 220 with an absolute variable z_k necessary. In the pitch and roll direction, the acceleration sensor provides sufficient support but is not able to provide support for the yaw movement for the reasons outlined above. This being the case, support for the yaw movement is obtained by the rotation rate calculated from the optical flow, which is also incorporated in the vector Zk in order to obtain a merged state x_k. After the correction step, the covariance is calculated in step 230, which in turn opens into the prediction step 200.

-20This method can be used in two situations. It enables drift-free support for the superstructure rotation to be obtained when the mobile working machine is stationary as well as support for the joint angle calculation in the event of superimposed accelerations due to a (translating) movement of the entire mobile working machine. A separation of the two situations should be guaranteed because in the case of a combined movement (i.e. superstructure rotation and translating movement) no separation of the speed vectors in the optical flow can be achieved as a rule. To enable this situation to be covered as well, however, control inputs (for example by means of a joystick) could also be incorporated in the fusion for example.

Figure 3 now schematically illustrates a sequence of a method proposed by the invention based on another preferred embodiment, as already described above. In a step 300, a position of the manipulator relative to the component respectively the superstructure of the excavator can be determined first of all, as mentioned, using the sensors mounted on the manipulator.

In a step 310, a rotation D of the superstructure about the vertical axis H (see also Figure 1) can then be determined using the rotation rate sensor. Additionally in this instance, in a step 315, the rotation D about

-21the vertical axis H can also be determined using the aforementioned camera. Then, in a step 320, based on the position from step 300 and the rotation from steps 310 and 315, the position of the manipulator within the excavator can be determined. In a step 330, the position of the excavator in space determined with radio respectively satellite assistance and/or using the camera, for example, can then be incorporated so that the position of the manipulator in space can be determined.

Figure 4 schematically illustrates a mobile working machine 100' proposed by the invention based on another preferred embodiment, in this instance likewise in the form of an excavator. The excavator 100' corresponds to the excavator 100 in Figure 1 but with the difference that two cameras 126 and 127 are provided in this instance .

The two cameras 126 and 127 in this instance are mounted on the underside of the superstructure 120. All in all, four cameras may be provided in particular in such a manner, directed towards the respective corners of the undercarriage 130. These are advantageously provided in the form of fisheye cameras and are therefore capable of detecting both the surrounding area and the undercarriage in a large area. By

-22detecting the undercarriage, it is possible to determine the relative speed between the superstructure and the undercarriage directly and thus separate it from the overall vehicle speed. This significantly improves support for the rotation rate signals of the inertial sensor respectively rotation rate sensor on the superstructure because no distinction has to be made as to whether the speed measured by the camera is caused by the rotation of the superstructure or the translating movement of the excavator.

Claims

1. Method for operating a mobile working machine (100, 100') having a manipulator (110) which is mounted so as to be movable on a component (120) of the mobile working machine (100, 100'), whereby a position of the manipulator (110) relative to the component (120) is determined using at least one sensor (111, 112, 113, 114, 114) mounted on the manipulator (110), whereby a rotation (D) of the component (120) about a vertical axis (H) of the mobile working machine (100, 100') is detected using a rotation rate sensor (121) that is mounted stationary with respect to the component (120), and whereby a position of the manipulator (110) within the mobile working machine (100, 100') is determined

using the posit ion of the manipulator ( 110) re lative the component ( 120 ) and the rotation (D ) of th e component (120) abi out the vertical axis (H) , character! sed in that the rotation (D) of the component (120) abi out the vertical axis (H) is also detected using at least one camera (125 , 126 127)

mounted on the mobile working machine (100, 100').

2. Method as claimed in claim 1, wherein a translating movement of the mobile working machine (100, 100') is also detected using the at least one camera (125, 126, 127) mounted on the mobile working machine (100, 100').

3. Method as claimed in claim 1 or 2, wherein a position of the mobile working machine (100, 100') in space is determined with radio or satellite assistance, in particular using GPS.

4. Method as claimed in claim 2 and 3, wherein the determined position of the mobile working machine (100, 100') in space is improved using the detected translating movement.

5. Method as claimed in one of claims 3 or 4, wherein a position of the manipulator (110) in space is determined using the position of the manipulator (110) within the mobile working machine (100, 100') and the position of the mobile working machine (100, 100') in space .

6. Method as claimed in one of the preceding claims, wherein the at least one sensor (111, 112, 113, 114,

-25114) mounted on the manipulator (110) is selected from displacement sensors, angle sensors and inertial sensors, in particular rotation rate sensors and acceleration sensors.

7. Method as claimed in one of the preceding claims, wherein the mobile working machine (100, 100') comprises an excavator in which the component (120) on which the manipulator (110) in the form of an excavator arm is mounted is part of a superstructure or wherein the mobile working machine comprises a telescopic handler, a lorry having a loading crane or a forestry machine .

8. Computer unit (140) which is set up to implement a method as claimed in one of the preceding claims.

9. Mobile working machine (100, 100') having a manipulator (110) mounted on a component (120) of the mobile working machine, at least one sensor (111, 112, 113, 114, 114) mounted on the manipulator (110), a rotation rate sensor (121) that is mounted stationary with respect to the component (120), at least one camera (125,126, 127) and a computer unit (140) as claimed in claim 8.

10. Computer programme which enables a computer unit (140) to implement a method as claimed in one of claims 1 to 7 when run on the computer unit (140) .

5

11. Machine-readable memory medium having a computer programme as claimed in claim 10 stored thereon.