WO2020160863A1

WO2020160863A1 - Device for route planning for an agricultural machine on the basis of sensor data and image segmentation

Info

Publication number: WO2020160863A1
Application number: PCT/EP2020/050345
Authority: WO
Inventors: Zhihu Chen; Marcus Hiemer
Original assignee: Zf Friedrichshafen Ag
Priority date: 2019-02-08
Filing date: 2020-01-09
Publication date: 2020-08-13
Also published as: DE102019201632A1

Abstract

The invention relates to a device (10) for performing assisted route planning for an agricultural machine (50), comprising an input interface (12) for inputting image data relating to a piece of agricultural land (60) to be worked by the agricultural machine (50), and for inputting predefined map information of the piece of agricultural land (60), an updating unit (14) for updating the input map information on the basis of segmentation of the input image data, a route planning unit (16) for producing a route of the agricultural machine (50) on the basis of the updated map information, and an output interface (18) for outputting the produced route for controlling the agricultural machine (50).

Description

Device for route planning for an agricultural machine based on sensor data and Bildseqmentierunq

The present invention relates to a device for route planning for an agricultural machine according to claim 1. The invention further relates to the use of such a device in an agricultural machine according to claim 13. The invention also relates to a method for route planning for an agricultural machine according to claim 14. Finally, the invention relates to a computer program product according to claim 15.

In agricultural areas, the cultivation of agricultural land plays a central role. For example, when harvesting grass and grain, but also field crops, the harvested crop must be deposited or swathed in a certain form (for example in a row). For such purposes, agricultural machines are used that drive along a certain route on the agricultural area and work on the latter at the same time.

There is an increasing need to at least partially automate such agricultural processing operations. This allows the raking of the agricultural area to be carried out without a driver. In this way, on the one hand, the efficiency of these processes due to the extended operating time of the agricultural machinery, but on the other hand, the safety of these processes can also be increased.

Solutions are known from the prior art for making agricultural machining operations autonomous. Thus, from WO 2018/087546 a robot-based fruit harvesting system comprising an autonomous robot is known.

However, the systems known from the prior art are limited in their degree of automation, so that many processes continue to be carried out by people or drivers of the agricultural machinery. These known systems are therefore disadvantageous in terms of the efficiency and safety of the agricultural processing operations. The known systems are also predominantly based on the Global Positioning System (GPS), which is costly in terms of license fees for obtaining satellite data.

The object of the present invention is therefore to improve the known systems for processing agricultural land in order to increase the degree of automation of the processing operations while reducing costs.

The object is achieved by a device for route planning for an agricultural machine having the features of claim 1. Furthermore, the object is achieved by the use of such a device in an agricultural machine with the features of claim 13. In addition, the object is achieved by a method for route planning for an agricultural machine with the features of claim 14.

Finally, the object is achieved by a computer program product with the features of claim 15.

If the agricultural machine is used to process the agricultural land, the route of the agricultural machine must first be determined in order to ensure the processing quality. For this purpose, the predefined map information of the usable area to be processed is fed in via the input interface of the route planning device. The map information is information relating to the geographic conditions and / or agricultural usage data of the usable area to be processed.

In addition, at least one sensor is provided for generating the image data to be entered. The sensor can be a camera, a radar sensor, a light detection and ranging sensor (lidar sensor) and / or an ultrasonic sensor. The sensor can be attached to the agricultural machine, for example, in order to detect a close range of the agricultural machine. Alternatively, it can also be a separate, central sensor system from the agricultural machinery for monitoring the agricultural machinery to be processed. The image data to be processed thereby generated Usable area or the immediate area of the agricultural machine are also fed in via the input interface of the route planning device.

The update unit processes the entered map information and image data in order to update the map information based on a segmentation of the image data. The segmentation is preferably a semantic segmentation. The segmentation can be carried out by a segmentation unit integrated in the route planning device or, alternatively, by an external segmentation unit. Pixel and / or voxel data are classified from the image data in order to recognize different areas in the pixel or voxel data on the basis of predefined classification criteria and to distinguish them from one another. Object detection preferably takes place in this case in order, for example, to highlight those areas on which the crop to be swathed is located from remaining areas (for example areas that have already been worked or that are vacant).

The update unit uses the result of the segmentation to modify / supplement the previously defined map information. Information relating to objects recognized in the image data that does not emerge from the predefined map information is preferably included in the map information. Here, the position data of the objects are read from the update unit, the object information being integrated into the map information in accordance with the position data read.

The route planning unit uses the map information updated in this way to create a route for the agricultural machine. In this case, the objects that were not previously obtained from the predefined map information but were detected by the sensor are also taken into account, for example in order to avoid a collision with a boundary structure and / or another vehicle on the usable area and / or to identify the most efficient route. The route created in this way is passed on from the route planning unit to the output interface and output via this to a control unit in order to control the agricultural machine in its driving movement accordingly.

The route planning of the agricultural machine according to the invention is therefore based on an update of predefined map information of the agricultural usable area to be processed, the update in turn being based on an image segmentation of the image data generated by the sensor. Compared to the systems known from the prior art, the current location of the usable area with regard to the usage situation and geographical conditions are better taken into account. In addition, sensors such as cameras are used to capture a close range of the agricultural machine. Route planning can thus be accomplished with increased accuracy and reliability and at the same time reduced costs. Complete automation of route finding, in particular autonomous guidance of the agricultural machine, can also be achieved in this way.

Advantageous refinements and developments are given in the subclaims.

In a preferred embodiment, the entered maps contain information a predefined raster map of the agricultural area, the update unit being designed to update the predefined raster map based on the segmentation of the entered image data.

A grid map (English: "grid map") is a two-dimensional map on which a grid or grid shape consisting of square grid or grid units is visually "placed". In this way, the usable area to be processed is divided into surface units that correspond to the grid or grid units. This pictorial subdivision of the usable area increases the compatibility of the map information with the sensor image data in terms of visualization and is therefore advantageous for updating the map information. In a further preferred embodiment, the predefined grid map contains a plurality of grid units to which one or more attributes of the agricultural area are assigned.

The attribute or the multiple attributes can include the position data of the respective grid units in an image coordinate system of the sensor and / or in a global world coordinate system that is independent of the sensor. Alternatively or additionally, the attribute or the multiple attributes can relate to the usage situation, in particular the plant name, the name and / or amount of the chemical / biological substances used, the composition of the soil of the respective grid units of the usable area. The attribute or the multiple attributes can also relate to the agricultural processing operations that have already been carried out and / or planned on this grid unit of the usable area. This measure improves the automation of route planning.

In a further preferred embodiment, the update unit is designed to update the attribute or the attributes and / or a dimension of the plurality of raster units based on the segmentation of the input image data.

In this way it can be avoided that the route planning is based on data about the usable area that has already expired and increases the reliability of the automated route planning. At the same time, the dimensions of the grid units are variable depending on the desired degree of resolution of the grid map.

In a further preferred embodiment, the update unit is designed to update the map information after entering new image data and / or after covering a new route section.

The route planning device thus reacts automatically to new image data and / or newly traveled route sections in order to always determine reliable routes. The update unit is preferably designed to within a predefined period after entering the new image data. Furthermore, the map information is preferably updated as long as the agricultural machine is moving.

In a further preferred embodiment, the device further comprises a task creation unit for creating one or more tasks relating to the processing of the agricultural area based on the updated map information.

For example, the task creation unit can be designed in such a way as to assign one or more agricultural tasks to each of the several grid units. The assignment is preferably based on the update of the map information, in particular the attributes of the grid units. For example, before the update, the attribute of a grid unit can be "Mow grass". In the event that the grass has already been mowed before the area unit of the usable area corresponding to this grid unit was recorded, the map information or the grid map is updated accordingly by the update unit. In this way, a new task is created as an updated attribute, for example "sprinkle seeds" or "continue driving". With this measure, the tasks to be assigned can be adapted to the current conditions of the usable area. The cultivation of the usable area by means of the agricultural machine is therefore more reliable. The task creation unit is preferably designed to assign a new task to at least one of the grid units when the agricultural machine changes direction.

In a further preferred embodiment, the route planning unit is designed to take into account a route covered or planned by the agricultural machine on the agricultural area in the created route.

It is thus possible to register the route that has already been traveled in order to call it up again if necessary. At the same time, based on the updated map information, a route that is still to be driven can be created so that the agricultural machine can carry out its driving task on the usable area independently or autonomously. In a further preferred embodiment, the device further comprises a localization unit for localizing the agricultural machine based on the segmentation of the input image data.

The localization unit can, for example, compare the image content captured by the sensor, which can be analyzed using the segmentation and the associated object classification, with the predefined map information and, based on this, determine the grid unit of the usable area in which the agricultural machine is currently located. This increases the accuracy of the localization of the agricultural machine.

In a further preferred embodiment, the device further comprises a segmentation unit for segmenting the input image data.

In this way, the route planning device according to the invention is able to independently segment the input image data, which makes route planning more efficient.

In a further preferred embodiment, the segmentation unit is designed to recognize a vehicle located on the agricultural area, in particular an additional agricultural machine, and / or a boundary line of the agricultural area in the input image data.

By recognizing the other vehicle or the other agricultural machine, the vehicle or the agricultural machine can be taken into account when creating the route in order to avoid a collision. By recognizing the boundary line of the usable area, the route can be created in such a way that it runs at least partially along the recognized boundary line. This results in reliable route planning.

In a further preferred embodiment, the route planning unit is designed to have a predefined minimum distance between the agricultural machine and to consider a boundary line of the agricultural usable area recognized by the segmentation of the image data in the created route.

This measure advantageously means that the predefined minimum distance from the boundary line is maintained by the agricultural machine when driving along the route created. Any collisions with objects outside the boundary line can be reduced or even avoided. In this way, the security of the agricultural working process is increased. This measure also increases the accuracy of the localization of the agricultural machine in the longitudinal direction (along the route) and / or in the lateral direction (across the route).

In a further preferred embodiment, the device comprises at least one sensor for generating the image data to be input.

In this way, the sensor is integrated in the route planning device according to the invention, so that an external sensor for generating the image data is not necessary. This increases the simplicity of route planning.

The computer program product according to the invention is designed to be loaded into a memory of a computer and comprises software code sections with which the method steps of the method according to the invention for route planning for an agricultural machine are executed when the computer program product is running on the computer.

A program is part of the software of a data processing system, for example an evaluation device or a computer. Software is a collective term for programs and associated data. The complement to software is hardware. Hardware describes the mechanical and electronic alignment of a data processing system. A computer is an evaluation device.

Computer program products usually comprise a sequence of instructions which, when the program is loaded, cause the hardware to carry out a certain process. ren that leads to a certain result. When the program in question is used on a computer, the computer program product causes a technical effect, namely to improve the known systems for processing agricultural land in order to increase the degree of automation of the processing operations while simultaneously reducing costs.

The computer program product according to the invention is platform independent. That means it can run on any computing platform. The computer program product is preferably executed on a device according to the invention for route planning for an agricultural machine.

The software code sections are written in any programming language, for example in Python.

The invention is illustrated by way of example in the figures. Show it:

Fig. 1 is a schematic representation of a device according to the invention for rout planning for an agricultural machine according to an embodiment;

2 shows a schematic representation of a method according to the invention for planning routes for an agricultural machine according to an exemplary embodiment;

3 shows a schematic representation of the route planning according to the invention;

4 shows a further schematic representation of the route planning according to the invention;

5 shows a further schematic representation of the route planning according to the invention;

6 shows a further schematic representation of the route planning according to the invention; 7 shows a further schematic representation of the route planning according to the invention;

8 shows a further schematic representation of the route planning according to the invention;

9 shows a further schematic representation of the route planning according to the invention; and

10 shows a further schematic representation of the route planning according to the invention.

1 shows a schematic representation of a device 10 according to the invention for route planning for an agricultural machine according to an exemplary embodiment.

The device 10 comprises an input interface 12a for inputting image data that have been generated by a sensor 22. The image data relate to an agricultural area 60 (not shown in FIG. 1) to be processed by an agricultural machine 50 (not shown in FIG. 1). Another input interface 12b is used to input predefined map information of the agricultural area 60. The map information preferably comprises a grid map with several grid units, each of the grid units preferably being assigned one or more attributes. The attributes can contain information such as the circumstances of the part of the usable area corresponding to the associated grid unit with regard to the geographical nature, the usage situation, in particular the plant name, the name and / or quantity of the chemical / biological substances used, the composition of the soil of the respective grid units Affect usable area.

The device 10 further comprises an update unit 14 for updating the entered map information based on a segmentation of the entered image data. The segmentation can be from a not shown in Fig. 1, in the device 10 integrated segmentation unit can be performed. Alternatively, segmentation results can be obtained from an external unit, such as a cloud server or a storage medium. The segmentation is preferably a semantic segmentation.

The device 10 also includes a route planning unit 16 for creating a travel route for the agricultural machine 50 based on the updated map information. The route planning unit 16 compares, for example, the predefined map information with the segmented image data in order to identify a discrepancy between the two data sets. The route planning unit 16 supplements the predefined map information with additional information from the segmented image data. The grid units of the grid map, in particular the attributes of the grid units, are thereby preferably at least partially updated. The driving route is created on the basis of the map information updated as a result, in particular the updated raster map.

The device 10 finally comprises an output interface 18 for outputting the created travel route for controlling the agricultural machine 50. The agricultural machine 50 is thus able to carry out the created travel route.

2 shows a schematic representation of a method according to the invention for route planning for an agricultural machine according to an exemplary embodiment. In a first step S1, image data relating to an agricultural area 60 to be worked by the agricultural machine 50 as well as predefined map information of the agricultural area 60 are input. In a second step S2, the map information entered is updated based on a segmentation of the image data entered. In a third step S3, a travel route of the agricultural machine 50 is created based on the updated map information.

In a fourth step S4, the created route for controlling the Landma machine 50 is output. In the figures, the same reference symbols relate to the same or functionally similar reference parts. The relevant reference parts are identified in the individual figures.

FIG. 3 shows a schematic illustration of an agricultural area 60, the schematic illustration being provided with a grid map. The grid map has a plurality of square grid units which are distributed over the entire usable area 60. In addition, the grid units also extend over an edge region surrounding the usable area 60. In this illustration, the agricultural machine 50 is located on the first grid unit at the bottom right of the usable area 60. The grid map forms at least part of the map information that the device 10 according to the invention receives via its input interface 12.

The device 10 is preferably integrated in the agricultural machine 50.

With the aid of an environment sensor 20, not shown in FIG. 1, preferably integrated in the agricultural machine 50, image data relating to a close-up area on the usable area 50 in the immediate vicinity of the agricultural machine 50 are generated. This image data is used to update the predefined raster map. A driving route for the agricultural machine 50 is created on the basis of the updated grid maps.

One or more attributes are preferably assigned to the individual grid units. The attribute or the plurality of attributes can include the position data of the respective grid units in an image coordinate system of the sensor 20 and / or a global world coordinate system that is independent of the sensor. Alternatively or additionally, the attribute or the several attributes can relate to the usage situation, in particular the plant name, the name and / or amount of chemical / biological substances used, the composition of the soil of the respective grid units of the usable area. The attribute or the multiple attributes can also relate to the agricultural processing operations that have already been carried out and / or planned on this grid unit of the usable area. These processing operations can be, for example, “mow the grass”, “scatter the seeds” and / or “drive on”. The image data are segmented in order to detect any objects located in the near area of the usable area 60 detected by the sensor 20, such as other vehicles, people, buildings, plantings, on the usable area. In FIG. 3, a boundary line 62 can be seen which marks a right edge of the usable area 60. This boundary line 62 is recognized by means of segmentation, in particular semantic segmentation, of the image data generated by sensor 20. The route planning unit 16 of the device 10 according to the invention is preferably designed to create the driving route in such a way that the agricultural machine 50 moves along the boundary line 62. A travel route 62 'created in this way is also shown in FIG. 3.

Fig. 4 shows a further schematic representation of the usable area 60. There the agricultural machine 50 is located on the route 62 'shown in Fig. 3 and partially shown in Fig. 4. The section 51 of the route 62 'covered by the agricultural machine 50 is also shown.

Fig. 5 shows a further schematic representation of the usable area 60. Fig. 5 shows specifically an intermediate result of the agricultural processing of the usable area 60 by the agricultural machine 50 with the help of the device 10 according to the invention machining part 66 shown.

In Fig. 5, the agricultural machine 50 is located at one end of the covered by the Landma machine 50 section 51 of the route 62 ', which runs along a wide Ren boundary line 62. In a direction essentially perpendicular to the boundary line 62, a further boundary line 64 is recognized by means of the segmentation of the sensor-based image data. On the basis of the further boundary line 64, the driving route 62 ′ is created in such a way that it contains a curve adapted to the course of the two boundary lines 62, 64. Thus, the device 10 according to the invention is generally designed, in particular independently of the specific usable area 60 shown in FIG. 5, to create the driving route in such a way that it is adapted to the objects recognized by means of the segmentation of the image data. 6 shows a further function of the device 10 according to the invention, namely the optimization of the created driving route. On the basis of the segmentation of the sensor-based image data, the device 10 recognizes four potential driving routes 52, 54, 56, 58 to continue the driving route 51 already covered for further processing of the usable area 60. Each of these four potential driving routes 52, 54, 56, 58 follows a straight section 53, 55, 57, 59 which runs along a number of the Rastereinhei th. From these four potential driving routes 52, 54, 56, 58, the most favorable driving route is identified on the basis of a cost function calculation in which, for example, the time required for driving the driving route, the route length or the fuel consumption is minimized. The identified driving route is output to the control unit 32 of the agricultural machine 50 so that it can carry out the corresponding driving order.

7 shows a further schematic representation of the usable area 60. In particular, the route planning unit 16 is designed to take into account another vehicle 61 that is located on the usable area when creating the driving route. This serves to avoid a collision with the further vehicle 61.

8 shows a further schematic representation of the usable area 60. A further boundary line 67, which was recognized when driving on the usable area 60, in particular in the part 66 of the usable area 60 to be machined, is shown here. The device 10 according to the invention is thus able to update the map information as long as the agricultural machine 50 is moving on the usable area 60 and new image data is generated by the sensor and fed into the device 10.

9 shows a further schematic representation of the usable area 60. The driving route 51 shown there by way of example has a serpentine course, due to which the part 66 of the usable area 60 to be processed is divided into several parts 66a, 66b, 66c, 66d. Here, too, the device 10 according to the invention is able to calculate several potential driving routes 69, 71, 73 and to select the best driving route from these, as described above, and to generate a corresponding control signal which is output to the control unit of the agricultural machine 50. Fig. 10 shows three further schematic representations of the usable area. In FIG. 10A, a first agricultural machine 50a is moving along a first travel route 51a and performs the task of "mowing the grass". This creates a mowed field surface 72a. In Fig. 10B, a second agricultural machine 50b performs the task of "swathing the mown grass" along a second travel route 51b. This creates a plurality of swaths 72b. In FIG. 10C, a third agricultural machine 50c performs the task of “collecting the swaths” along a third driving route 50c.

The tasks are defined, as described above, preferably by updating the individual relevant grid units of the grid map. In particular, on the basis of the segmentation of the image data generated by the sensor 20, the attributes assigned to the individual grid units are updated in accordance with the predefined map information. Alternatively or additionally, new attributes can also be reassigned to the raster units on the basis of the image segmentation. The attributes can relate, for example, to one or more tasks that are to be carried out on the parts of the usable area 60 belonging to the respective grid units of the agricultural machine.

Reference route planning device

, 12a, 12b input interface

Update unit

Route planning unit

Output interface

sensor

Control unit of an agricultural machine

, 50a, 50b, 50c agricultural machinery

, 52, 53, 54, 55, 56, 57, 58, 59, 62 ', 69, 71, 73

Driving route

Usable area

, 64 border line

, 66a, 66b, 66c part of the usable area to be worked part of the usable area a1 mowed grass

b swaths

to S4 process steps

Claims

A device (10) for route planning for an agricultural machine (50), comprising: an input interface (12) for inputting image data relating to an agricultural area (60) to be processed by the agricultural machine (50) and for inputting predefined agricultural map information Usable area (60);

an update unit (14) for updating the entered map information based on a segmentation of the entered image data;

a route planning unit (16) for creating a travel route of the agricultural machine (50) based on the updated map information; and

an output interface (18) for outputting the created route for controlling the agricultural machine (50).

2. Device (10) according to claim 1, wherein the entered map information contains a predefined grid map of the agricultural area (60), where in the update unit (14) is designed to update the predefined grid map based on the segmentation of the input image data.

3. Device (10) according to claim 2, wherein the predefined grid map contains a plurality of grid units to which one or more attributes of the agricultural area are assigned.

4. The device (10) according to claim 3, wherein the update unit (14) is formed to update the attribute or the attributes and / or a dimension of the plurality of grid units based on the segmentation of the input image data.

5. Device (10) according to one of claims 1 to 4, wherein the updater unit (14) is designed to update the map information after entering new image data and / or after covering a new route section.

6. Device (10) according to one of claims 1 to 5, further comprising a task benerstellungseinheit for creating one or more tasks relating to the processing of the agricultural area (60) based on the updated map information.

7. Device (10) according to one of claims 1 to 6, wherein the route planning unit (16) is designed to determine the driving route for a distance covered by the agricultural machine (50) on the agricultural area (60) and / or for one of the agricultural machine (50) on the agricultural area (60) still to be traveled route to create.

8. Device (10) according to one of claims 1 to 7, further comprising a localization unit (15) for localizing the agricultural machine (50) based on the segmentation of the inputted image data.

9. Device (10) according to one of claims 1 to 8, further comprising a segmentation unit for segmenting the input image data.

10. The device (10) according to claim 9, wherein the segmentation unit is formed from, in the input image data, a vehicle located on the agricultural surface (60), in particular an additional agricultural machine, and / or a boundary line (62) of the agricultural surface (60) to be recognized.

1 1. Device (10) according to one of claims 1 to 10, wherein the route planning unit (16) is designed to set a predefined minimum distance between the agricultural machine (50) and a boundary line (62) of the agricultural area (60) recognized by the segmentation of the image data ) must be taken into account in the created route.

12. Device (10) according to one of claims 1 to 1 1, further comprising at least one sensor (22) for generating the image data to be entered.

13. Use of a device (10) according to one of claims 1 to 12 in an agricultural machine (50).

14. A method for route planning for an agricultural machine (50), comprising:

a first step (S1) in which image data relating to an agricultural area (60) to be processed by the agricultural machine (50) and predefined map information of the agricultural area (60) are entered; a second step (S2) in which the entered map information is updated based on a segmentation of the entered image data; a third step (S3) in which a travel route of the agricultural machine (50) is created based on the updated map information; and

a fourth step (S4) in which the created travel route is output for controlling the agricultural machine (50).

15. Computer program product which is stored in a data carrier and is designed to carry out the method according to claim 14 when the computer program product is executed by a computer.