WO2022161748A1

WO2022161748A1 - Operational safety of a agricultural implement

Info

Publication number: WO2022161748A1
Application number: PCT/EP2022/050174
Authority: WO
Inventors: Srivatsan BALAJI; Sridhar ETHIRAJ; Surya JAYARAMAN; Hari Prasanth SUNDRAVALLI MUTHUMANICKAM
Original assignee: Zf Cv Systems Global Gmbh
Priority date: 2021-01-27
Filing date: 2022-01-06
Publication date: 2022-08-04

Abstract

The invention relates to a method (100) for improving operational safety of an agricultural machine (2). According to the invention said method comprises the steps: monitoring (202) a region of interest (8) around a movable component (4) of said agricultural machine (2), determining (203) a danger zone (9) inside said region of interest (8) and around the movable component (4) of said agricultural machine (2), determining (204) the presence of an object (12) in said region of interest (8), inhibiting (206) an operation of said movable component (4) of said agricultural machine (2) when the presence of an object (12) in said danger zone (9) is determined.

Description

OPERATIONAL SAFETY OF A AGRICULTURAL IMPLEMENT

The invention relates to a method for improving operational safety of an agricultural machine.

Agricultural machines are, for example, utilized for land preparation farming operations. For such farming operations, agricultural machines typically comprise different kinds of implements that are releasably coupled to a tractor vehicle. Examples of such implements are so called Rotovators, power harrows and other devices comprising rotating parts or even blades. Besides such implements, also the tractor vehicles itself comprise moving parts such as so called rotating power take offs for providing power to the implements and lifting devices for adjusting the height or orientation of the same.

The coupling of such agricultural implements with the tractor vehicle is often conducted manually. In case, for example, the power take-off is unwantedly engaged when an operator in the direct vicinity of a moving part of such agricultural machine, a severe health risk may evolve. All moving parts have in common that they may be harmful to humans and animals in the vicinity thereof. Due to the size of such agricultural machines, a driver may not determine these situations appropriately based on his limited field of view.

In the prior art, detection devices for agricultural machines are known. EP 2 756 748B1 , for example, discloses a detection device for detecting a foreign object in the vicinity of an agricultural harvesting machine. The core concept is directed towards preventing damage from the harvesting machine itself. From US 9563204 B2, an object detection method for a robotic lawn mower is known. According to this reference, the kind of object detected is considered when a decision regarding the path of the mower is made. These known system, however, fail to address safety issues arising for humans and animals in the vicinity of moving parts of such machines.

It was therefore an object of the invention to increase the operational safety of an agricultural machine.

To solve this object, the present invention proposes a method comprising the following steps: Monitoring a region of interest around a movable component of said agricultural machine, determining a danger zone inside said region of interest and around the movable component of said agricultural machine, determining the presence of an object in said danger zone, and inhibiting an operation of said movable component of said agricultural machine when the presence of an object in said danger zone is determined.

The invention is based on the finding that by determining the presence of an object in said region of interest and danger zone around a moveable part of said agricultural machine, and by inhibiting an operation of said moveable part when the presence of an object in said danger zone is determined, the operational safety of the agricultural machine can be increased significantly. In case an object is present close to the moveable part, the moveable part can either be stopped or prevented from being actuated or activated. With the help of this, an operator involved with tasks to be carried out at or in the vicinity of movable machine parts is protected from unsafe conditions. The object to be detected may preferably be a human but may alternatively also be an animal. When the object is present in the region of interest but not in said danger zone, the movable component may be operated safely.

According to a preferred embodiment, the method further comprises the step of inhibiting a longitudinal movement of said agricultural machine when the presence of an object in said danger zone is determined. In other words, not only the activation of rotating parts related to the agricultural machine or the implement is stopped, but also the movement of the agricultural machine itself. This furthermore increases the overall operational safety. The longitudinal movement may, for example, be inhibited by controlling the gear and/or throttle of said machine.

According to another preferred embodiment, the method furthermore comprises the step of presenting a warning message to an operator of said agricultural machine when the presence of an object in said danger zone is determined. The operator may, for example, be the driver of said agricultural machine. The warning message may be an audio message, a visual message or an indication on a human machine interface (HMI) of the driver. With the help of such warning message, the operator of the agricultural machine is informed about the presence of the object in said danger zone.

Preferably, said moveable part is configured as at least one of the following: power take off (PTO) clutch and/or shaft, implement lifting device, agricultural implement being coupled to said agricultural machine. The mentioned moveable parts are examples of potentially dangerous parts commonly present at agricultural machines.

Preferably, a size of said danger zone is determined based on said moveable part. In other words, may the dimensions of said danger zone be adjusted based on the type of movable component being present. For example, a larger agricultural implement may require greater danger zone dimensions than a rather compact agricultural implement.

According to yet another preferred embodiment the step of determining the presence of an object comprises determining the estimated distance of the object from a reference. With the help of this, it is not only possible to detect whether or not an object is present in said region of interest or danger zone but also to exactly locate the position of said object inside said region of interest or danger zone. Thereby, the overall detection quality and accuracy of said object determination is improved. Said reference may be the position of a sensor used to contact the monitoring of the region of interest or any alternative reference. Preferably, the object detection and/or the object distance are determined based on stereo images. With the help of such stereo images the entire surroundings of a moving part, such as an implement, may be covered while object detection and object distance may be determined from such stereo images.

According to yet another preferred embodiment, at least one of the following sensors is additionally utilized to determine the presence of an object in said danger zone: ultrasonic sensor, radar sensor. These optional sensors may be used to increase the sensor coverage or the monitoring accuracy.

Preferably, the step of determining the presence of an object comprises pre-processing stereo images utilizing a calibration information, conducting an object detection based on the pre-processed image, and generating an object list. By pre-processing said stereo images intrinsic (focal length, distortion coefficients) and extrinsic parameters may be determined. Also, noise in said stereo images may be reduced by utilizing smoothing filters. Moreover, left and right stereo images may be aligned using stereo rectification. Features in a left image of such stereo images are matched to corresponding features in a right image of said stereo images. The difference in pixel coordinates between matched features (so called stereo disparity) and known relations between a left and right stereo camera are used to get depth images. Based on the object detection an object list is generated.

According to another preferred embodiment, the step of determining the object distance additionally comprises generating a depth map utilizing the stereo images and the calibration information, estimating the distance of an object utilizing the depth map and said object list. With the help of this, the object may not only be classified but also exactly described with regard to its position and distance from a reference, for example a sensor. According to another aspect, the invention relates to an agricultural machine. The invention solves the above mentioned object with regard to the agricultural machine in that said machine comprises at least one movable component, a sensor arrangement for monitoring a region of interest around said at least one movable component, and a safety device configured to determining the presence of an object in said region of interest based on said sensor arrangement, wherein said safety device is further configured to conducting the steps of the method according to the previous embodiments. Said agricultural machine takes advantage of the same benefits and preferred embodiments as the method according to the invention. In this regard, reference is made to the above explanations and their content is included herein.

According to another preferred embodiment, said movable component comprises at least one of the following: power take off (PTO) clutch and/or shaft, implement lifting device, agricultural implement being coupled to said agricultural machine.

According to yet another preferred embodiment, said sensor arrangement comprises at least one stereo camera. Preferably, said sensor arrangement comprises two stereo cameras. Preferably, said sensor arrangement additionally comprises at least one of the following sensors: ultrasonic sensor, radar sensor.

According to yet another preferred embodiment, said sensor arrangement comprises a servo motor-gimbal arrangement for aligning said sensor arrangement towards said movable component. Such servo motor-gimbal arrangement may be utilized to ensure that although different movable components are utilized and monitored, it is ensured, that the moving part will be in the major focus of a sensors perspective. In other words, with regard to the stereo camera, the serval motor-gimbal arrangement may be utilized to ensure that the implement to be monitored will be the major focus in the cameras perspective.

The invention furthermore relates to a computer program product. The computer program product attains the object of the invention in that, when the computer program is executed by a computer, causes the computer to carry out the steps of the method according to the previous embodiments. The computer program product takes advantage of the same benefits and preferred embodiments as the method and the agricultural machine according to the invention. In this regard, reference is made to the above explanations and the content is included herein.

For a more complete understanding of the invention, the invention will now be described in detail with reference to the accompanying drawings. The detailed description will illustrate and describe what is considered as preferred embodiments of the invention. It should of course be understood that various modifications and changes in form or detail could readily be made without departing from the spirit of the invention. It is therefore intended that the invention may not be limited to the exact form and detail shown and described herein, nor to anything less than the whole of the invention disclosed herein and as claimed herein after. Further, the features described in the description, the drawings and the claims disclosing the invention may be essential for the invention considered alone or in combination. In particular, any reference signs in the claims shall not be construed as limiting the scope of the invention. The wording “comprising” or “including” does not exclude other elements or steps. The word “a” or “an” does not exclude the plurality. The wording “a number of” items comprising also the number 1 , i.e. a single item, and further numbers like 2, 3, 4 and so forth.

In the accompanying drawings:

Fig. 1 : shows an agricultural machine according to the concept of the invention in a schematic view;

Fig. 2: shows a sensor arrangement according to the concept of the invention in a schematic view;

Fig. 3: shows a method according to the concept of the invention in a block diagram; Fig. 4-7: show additional block diagrams according to the concept of the invention;

Fig. 8: shows a computer program product according to the concept of the invention.

Fig. 1 shows an agricultural machine 2. The agricultural machine 2 comprises a tractor vehicle 3 and an agricultural implement 18. Said agricultural implement 18 is coupled to the tractor vehicle 3. The tractor vehicle 3 and the agricultural implement 18 comprise movable components 4. Such movable components 4 are said agricultural implement 18, a power take off clutch and shaft 14 and an implement lifting device 16.

The agricultural machine 2 furthermore comprises a sensor arrangement 6. The sensor arrangement 6 is configured for monitoring a region of interest 8 around said movable components 4. Inside the region of interest 8 a danger zone 9 is arranged. The danger zone 9 is arranged around said movable components 4. Said danger zone 9 has a size 30. The agricultural machine 2 furthermore comprises a safety device 10. The safety device 10 is configured to determining the presence of an object 12 in said region of interest 8 and said danger zone 9 based on said sensor arrangement 6. The sensor arrangement 6 comprises two stereo cameras 20.

Fig. 2 shows a sensor arrangement 6. The sensor arrangement 6 comprises a stereo camera 20, an ultrasonic sensor 22 and a radar sensor 24. The sensor arrangement 6 furthermore comprises a servo motorgimbal arrangement 26 for aligning said sensor arrangement 6 towards said movable component 4.

Fig. 3 shows a method 100 for improving operational safety of an agricultural machine 2. The method 100 comprises monitoring 202 a region of interest 8 around a movable component of said agricultural machine 2. Said method 100 furthermore comprises the step of determining 203 a danger zone 9 inside said region of interest 8 and around the movable component 4 of said agricultural machine 2.

The method furthermore comprises determining 204 the presence of an object 12 in said region of interest 8, and the step of inhibiting 206 an operation of said movable component 4 of said agriculture machine 2 when the presence of an object 12 in said danger zone 9 is determined. The method 100 furthermore comprises the step of inhibiting 208 a longitudinal movement 28 of said agricultural machine 2 when the presence of an object 12 in said danger zone 9 is determined. Additionally, said method 100 comprises the step of presenting 210 a warning message to an operator of said agricultural machine 2 when the presence of an object 12 in said danger zone 9 is determined.

The step of determining 204 the presence of an object 12 comprises the step of determining 212 the estimated distance of the object 12 from a reference. Such reference may be the sensor arrangement 6 or the stereo camera 20, for example. The step of determining 204 the presence of an object 12 comprises pre-processing 216 stereo images utilizing a calibration information, conducting 218 an object detection based on the pre-processed image, and generating 220 an object list. The step of determining 212 the object distance additionally comprises generating 220 a depth map utilizing the stereo images and the calibration information, and estimating 222 the distance of an object utilizing the depth map and said object list.

Fig. 4 shows a system architecture of said system comprising the safety device 10. The sensor arrangement 6 comprises the stereo camera 20 and, optionally, the ultrasonic sensor 22 and the radar sensor 24. The sensor arrangement 6 is connected to the safety device 10. The safety device 10 is furthermore bidirectionally connected to the movable component 4. The movable component 4 comprises the power take off (PTO) clutch - shaft 14 and the implement lifting device 16. The movable component 4 optionally comprises an automated manual transmission (AMT) 36 and an engine control 38. The safety device 10 is configured to control the actuation of said components of said movable component 4. The safety device 10 is furthermore connected to a warning system 24. The warning system 24 comprises a speaker 32 and, optionally, a human machine interface (HMI) 34. The warning system 24 outputs a warning to an operator 40.

Fig. 5 shows the architecture of said safety device 10. The safety device 10 comprises a system monitor 2 45 and a system monitor 1 62. The system monitor 1 62 comprises an operating system 60, a base software 58 and a software API 56. The system monitor 2 45 comprises an image pre-processor 46, an object detector 48, a distance estimator 52 and a safety algorithm 53 being connected to a communication API 54. The safety device 10 is connected to an HMI 34 and a stereo camera 20. This connection can be a wireless, wired or Ethernet connection. Furthermore, the safety device 10 is connected to the movable component 4 comprising said power take off (PTO) clutch I shaft 14 and such implement lifting device 16. The movable component 4 optionally comprises an engine control 38. In general, said safety device 10 may act upon a mechanical actuator 64 and receive data from or send data to the movable component 4 while a bus system 66.

Fig. 6 shows a perception software architecture. An image pre-processor 46 receives data from the stereo camera 20 and a calibration file 44. The data from the stereo camera 20 may be left and right raw RGB image data. In the image pre-processor 46 an undistorted RGB image pair is generated and provided to the object detector 48 and the depth map generator 50. The object detector 48 and the depth map generator 50 furthermore receive camera intrinsic and extrinsic parameters from the calibration file 44. The object detector 48 generates a list of objects that are provided to the distance estimator 52. The depth map generator 50 generates a depth map that is also provided to the distance estimator 52. The distance estimator 52 outputs the objects and their 3D positions with regard to a sensor or camera position. In a safety decision logic 70 a decision is made to whether it is safe to operate said movable component 4 or not. A user input 68 in the form of a lever actuation is additionally considered within the safety decision logic 70. The safety decision logic 70 outputs whether it is safe to operate the movable component 4, in particular the power take off (PTO) clutch I shaft 14, the implement lifting device 16 or the implement 18 itself.

Fig. 7 shows a process flow chart illustrating the decision process in said safety decision logic 70. After starting the safety decision logic 70 a sensor and system check is conducted. In case the sensor check and system check are okay, it is estimated whether an operator engages a PTO drive or implements a lift. Furthermore, an object detection and distance estimation is conducted. A stereo image pair is used for this object detection and distance estimation. In case an object is detected within said danger zone, the PTO clutch is not engaged and the implement not lifted or lowered. Furthermore, a warning to the user is generated. In case no object has been detected within the danger zone, the PTO clutch may be engaged and the implement lifted or lowered. After the system has been switched off, the process is stopped.

Fig. 8 shows a computer program product 300. Said computer program 300 comprises instructions, which, when the program is executed by a computer, cause the computer to carry out the steps of said method 100.

List of references

2 agricultural machine

3 tractor vehicle

4 movable component

6 sensor arrangement

8 region of interest

9 danger zone

10 safety device

12 object

14 power take off (PTO) clutch I shaft

16 implement lifting device

18 agricultural implement

20 stereo camera

22 ultrasonic sensor

24 radar sensor

26 servo motor-gimbal arrangement

28 longitudinal movement

30 danger zone size

32 speaker

34 HMI

36 automated manual transmission (AMT)

38 engine control

40 operator

42 warning system

44 calibration file

45 system monitor 2

46 image pre-processor

48 object detector

50 depth map generator

52 distance estimator 53 safety algorithm

54 communication API

56 software API

58 base software

60 operating system

62 system monitor 1

64 mechanical actuator

66 bus system

68 user input

70 safety decision logic

100 method

202 monitoring a region of interest

203 determining a danger zone

204 determining the presence of an object in said region of interest

206 inhibiting the operation of said movable component

208 inhibiting a longitudinal movement of the agricultural machine

210 presenting warning message to an operator

212 determining the estimated distance of the object from a reference

216 pre-processing stereo images utilizing a calibration information

218 conducting an object detection based on the pre-processed image

220 generating an object list

220 generating a depth map

222 estimating the distance of an object utilizing the depth map and said object list

300 computer program product

Claims

1 . A method (100) for improving operational safety of an agricultural machine (2), said method (100) comprising:

- monitoring (202) a region of interest (8) around a movable component (4) of said agricultural machine (2),

- determining (203) a danger zone (9) inside said region of interest (8) and around the movable component (4) of said agricultural machine (2),

- determining (204) the presence of an object (12) in said danger zone (9),

- inhibiting (206) an operation of said movable component (4) of said agricultural machine (2) when the presence of an object (12) in said danger zone (9) is determined.

2. The method (100) according to claim 1 , further comprising:

- inhibiting (208) a longitudinal movement (28) of said agricultural machine (2) when the presence of an object (12) in said danger zone (9) is determined.

3. The method (100) according to any of the preceding claims, further comprising:

- presenting (210) a warning message to an operator (40) of said agricultural machine (2) when the presence of an object (12) in said danger zone (9) is determined.

4. The method (100) according to any of the preceding claims, wherein said movable component (4) is configured as at least one of the following:

- Power take off (PTO) clutch and/or shaft (14),

- Implement lifting device (16),

- Agricultural implement (18) being coupled to said agricultural machine (2).

5. The method (100) according to claim 4, wherein a size (30) of said danger zone (8) is determined based on said movable component (4).

6. The method (100) according to any of the preceding claims, wherein determining (204) the presence of an object (12) comprises determining (212) the estimated distance of the object (12) from a reference.

7. The method (100) according to any of the preceding claims, wherein the object detection (204) and/or the object distance are determined based stereo images.

8. The method (100) according to claim 6, wherein at least one of the following sensors (22, 24) is additionally utilized to determine the presence of an object (12) in the danger zone (8):

- Ultrasonic sensor (22),

- Radar sensor (24).

9. The method (100) according to any of the preceding claims, wherein the step of determining (204) the presence of an object (12) comprises:

- pre-processing (216) stereo images utilizing a calibration information,

- conducting (218) an object detection based on the pre-processed image,

- generating (220) an object list.

10. The method (100) according to claim 9, wherein the step of determining (212) the object distance additionally comprises:

- generating (220) a depth map utilizing the stereo images and the calibration information, - 15 -

- estimating (222) the distance of an object utilizing the depth map and said object list.

11. An agricultural machine (2) comprising:

- at least one movable component (4),

- a sensor arrangement (6) for monitoring a region of interest (8) around said at least one movable component (4),

- a safety device (10) configured to determining the presence of an object (12) in said region of interest (8) based on said sensor arrangement (6), wherein said safety device (10) is further configured to conducting the steps of the method (100) according to claims 1 -9.

12. The agricultural machine (2) according to claim 11 , wherein said movable component (4) comprises at least one of the following:

- Power take off (PTO) clutch and/or shaft (14),

- Implement lifting device (16),

- Agricultural implement (18) being coupled to said agricultural machine (2).

13. The agricultural machine (2) according to claim 11 or 12, wherein said sensor arrangement (6) comprises at least one stereo camera (20).

14. The agricultural machine (2) according to claim 13, wherein said sensor arrangement (6) additionally comprises at least one of the following sensors:

- Ultrasonic sensor (22),

- Radar sensor (24). - 16 -

15. The agricultural machine (2) according to any of claims 11 -14, wherein said sensor arrangement (6) comprises a servo motor-gimbal arrangement (26) for aligning said sensor arrangement (6) towards said movable component (4).

16. A computer program product (300) comprising instructions, which, when the program is executed by a computer, cause the computer to carry out the steps of the method (100) of claims 1 -10.