DE102019218186A1 - Method of working crops in a field - Google Patents

Abstract

A method (100) for processing plants in a field on which a specific crop is grown comprises the following steps: selecting (S102) a processing tool for processing plants; Capturing (S104) an image of the field, the image (12) being correlated with positional information; Determining (S106) a position of a plant to be tilled in the field using a neural network (10) into which the captured image (12) is input, the neural network (10) having several specific classes (14 to 17) and one general class (22), the useful plant belonging to one of the specific classes (eg 14) and plants that do not correspond to the useful plant both to one of the special classes (eg 16) and to the general class (22), or at least belong to general class (22); Guiding (S108) the processing tool to the position of the plant; and processing (S110) the plant with the processing tool.

Description

Technical area

The present invention relates to a method for working plants in a field.

Of the diverse tasks in agriculture, weed regulation plays a central role in the success of the yield. The cost of pesticides is significant and its impact on the environment is problematic. For this reason, autonomously working systems are increasingly being used to process plants, i.e. useful plants and weeds. The processing can be done mechanically, e.g. by a milling machine, but also by a targeted application of pesticides, e.g. by a controlled sprayer. In this way, the use of pesticides can be avoided or at least reduced, thereby reducing the impact on the environment and reducing costs.

For the selective (the plant to be worked is differentiated from other plants and the soil) plant cultivation in a field, it is necessary to exactly recognize the position of a plant to be cultivated in a field. This can be achieved by different object recognition methods, with optical image recognition using a classifier being used in particular. A semantic segmentation of a captured image, but also a classification of the image can be carried out.

In practice, vehicles or devices are used for processing plants in fields on which a certain type of crop is grown and a large number of different weeds are growing.

For the classification of plants in a field, it is an obvious approach to use a classifier that enables at least one differentiation between the cultivated crop (e.g. sugar beet) and the weeds growing in the field. However, if the classifier is to be used on a field on which another crop type (e.g. maize) is grown, this procedure has the disadvantage that annotated training data must be available for this field or newly recorded. Then the classifier must be retrained (e.g. a sugar beet is a weed on the corn field) or another classifier must be added that is trained with the new training data. This classification procedure is referred to as the one-versus-other procedure, since a particular crop species is differentiated from a large number of weeds.

If a device is to be used in different fields on which different crops are grown, it is more efficient for this reason to define the individual weed and crop plant species as a separate class so that, for example, a species-specific classification of plants is possible. The classifier has a large number of classes so that the plants in a field can consequently be recognized in a species-specific manner. This approach is known as the all-versus-all approach. With this procedure it is consequently possible that the classifier can be used on several fields on which different crops are grown, since a separate class is defined for each crop species.

However, during a processing operation it can happen that unknown plants or plants that are not defined in the classifier or are not recognized by it are present, or that a classification result that is output by the classifier is due to varying environmental conditions (e.g. other Soil types, different seeds with different genetics, nutrient-related growth differences, etc.) is not sufficiently precise. These variations or unknown plants that are not present in the training data should, however, be retrained directly in the field with as little effort as possible, so that an exact differentiation between the plants to be worked and the plants not to be worked is still possible.

An obvious solution would be to acquire images of the unknown plant, annotate these images with Groud-Truth in order to generate new training data, and then train the classifier with the newly generated training data. However, this procedure is associated with a high level of effort and can therefore not be carried out efficiently in practice. In addition, in fields with a high degree of vegetation, it is almost impossible to specify each plant exactly in order to annotate it in the new training data.

It is therefore the object of the present invention to provide a method for working plants in a field, which method can be easily adapted in the field.

Figure list

• 1 shows a flow chart of the method according to the invention;
• 2 shows a neural network for determining a position of a plant in a field.

Description of the embodiments

Embodiments of the present invention will be described below with reference to the accompanying figures.

A vehicle, on which a device for processing the plants is attached, travels a field along a predetermined route and the objects or plants to be processed are obtained by executing the method according to the invention 100 processed one after the other. The vehicle drives through the field autonomously, but can also drive through the field according to a control by an operator.

A field can be understood to be a delimited area of land for the cultivation of useful plants or a part of such a field. A useful plant is understood to be an agriculturally used plant that is used itself or its fruit, e.g. as food, feed or as an energy plant. The seeds and consequently the plants are primarily arranged in rows, it being possible for objects to be present between the rows and between the individual plants within a row. However, the objects are undesirable because they reduce the yield of the plants or represent a disruptive influence during cultivation and / or harvest. An object can be understood to mean any plant that is different from the useful plant, or any object. Objects can in particular be weeds, woods and stones.

For this purpose, the device for processing plants has at least the following elements: a processing tool, an image acquisition means, various sensor elements (e.g. a position sensor, a speed sensor, an inclination sensor, a distance sensor, etc.), a memory unit and a computing unit.

The device for processing plants is installed on a vehicle provided for this purpose, which is operated by a battery, but can also be operated by another energy source, such as an internal combustion engine. The device can also be attached to an agricultural vehicle or a trailer for the agricultural vehicle. The device is operated by an energy source of the vehicle, but can also be operated by a separate energy source provided for this purpose.

The processing tool is a mechanical tool which is attached to a movable device so that it can be guided towards or away from a plant to be processed and is designed in such a way that a plant is processed with it. The movable device is, for example, an articulated arm that is moved by electric motors or hydraulics. The processing tool is e.g. a milling cutter that cuts off the plant, i.e. in this case a weed, in the area of the roots. However, the processing tool can also be a sprayer with which a pesticide is sprayed in the direction of a plant to be processed. It should be noted that the sprayer can also be used to apply a crop protection agent or fertilizer to a crop. In addition, other processing tools, such as an electrical processing tool, a laser, microwaves, hot water or oil, are also conceivable. The processing tool installed on the vehicle has a specific spatial accuracy. The spatial accuracy of a milling cutter depends on the movable device and the mechanical design (e.g. the diameter) of the milling cutter itself. The spatial accuracy of a sprayer depends on a nozzle angle of the sprayer. The spatial accuracy of a sprayer is many times less than that of a milling machine. In addition, it is also possible that several processing tools are attached to a device for processing plants, which can be operated simultaneously. Different types of processing tools can also be attached to the same device for processing plants.

The image acquisition means is a camera, such as a CCD camera, a CMOS camera, etc., which acquires an image in the visible area and provides it as RGB values or as values in another color space. The image acquisition means can, however, also be a camera that acquires an image in the infrared range. An image in the infrared range is particularly suitable for capturing plants, as the reflection of the plants is significantly increased in this frequency range. The image acquisition means can also be, for example, a mono, RGB, multispectral, hyperspectral camera. The image acquisition means can also provide a depth measurement, e.g. by a stereo camera, a time-of-flight camera, etc. It is possible for a plurality of image acquisition means to be present and for the images from the different image acquisition means and the data from the various sensor elements to be acquired essentially synchronously.

For the operation of the device for processing plants, further data are required, which are recorded using various sensor elements. The sensor elements can be a position sensor, for example GPS, high-precision GPS, etc., a speed sensor, an inclination sensor, a distance sensor, but also other sensors such as a weather sensor, etc. include.

The storage unit is a non-volatile physical storage medium, such as a semiconductor memory, in which data can be stored for a longer period of time. The data remain stored in the storage unit even when there is no operating voltage on the storage unit. The memory unit stores a program for carrying out the method according to the invention and the operating data required for this. In addition, the images captured by the image capturing means and the data captured by the sensor elements are stored on the storage unit. However, other data and information can also be stored in the memory unit.

The program stored in the memory unit contains instructions in the form of program code, which is written in any programming language, which are executed in sequence, so that the method according to the invention 100 to process the plants in the field. The program can also be divided into several files that have a predefined relationship to one another.

The computing unit is an arithmetic-logic unit that is implemented in the form of a processor (e.g. CPU, GPU, TPU). The computing unit is able to read data from the storage unit and output instructions in accordance with the program in order to control the image acquisition means, the sensor elements and actuators, such as the processing tool, all of which are communicatively (wired or wireless) connected to the computing unit.

• Eingangs wird in Schritt S102 das Bearbeitungswerkzeug ausgewählt, mit dem die Pflanzen bzw. Objekte auf einem Feld bearbeitet werden sollen. Die räumliche Genauigkeit, mit der die Pflanzen durch das Bearbeitungswerkzeug bearbeitet werden, hängt dabei, wie oben beschrieben, von der Art des Bearbeitungswerkzeuges ab. Das Bearbeitungswerkzeug kann vor einem Start des Abfahrens des Felds für die gesamte Dauer des Abfahrens festgelegt werden. Das Bearbeitungswerkzeug kann aber auch während eines Abfahrens gewechselt werden.

The individual process steps S102 to S110 of the method according to the invention 100 , as in 1 shown, executed in sequence. The individual steps are described in detail below:

• The input is in step S102 the processing tool selected with which the plants or objects in a field are to be processed. The spatial accuracy with which the plants are processed by the processing tool depends, as described above, on the type of processing tool. The processing tool can be specified for the entire duration of the process before the start of the process of running through the field. However, the processing tool can also be changed during a run.

Then in step S104 a picture 12th of the field in which the plants are growing, captured by the image capturing means. The image capturing means is attached to the vehicle such that an image sensor is substantially parallel to a ground surface of the field. It also becomes essentially synchronous with capturing the image 12th acquires positional information on the position where the image 12th is captured in the field. The position information obtained by the position sensor is with the image 12th correlated so that actual positions of pixels of the image 12th can be determined on the field taking into account the position information, the angle of view of the image capturing means used and the distance of the image capturing means from the ground. The image capturing means can, however, also be attached in such a way that the image sensor is inclined in any direction in order to capture a larger area of the field. In this case, the angle of inclination must be taken into account when determining the position of a pixel on the field.

In the next step S106 becomes the captured image 12th processed to determine a position of the plant to be processed in the field. The positions of the plants to be processed are individually determined by the pixels of the captured image 12th information about the displayed content is assigned. Since the position of the individual pixels in the field is known, the respective positions of the plants to be processed can be determined. The position of a plant in a field is preferably done by means of semantic segmentation of the captured image 12th correlated with the position information is determined. The semantic segmentation in which each pixel of an image 12th is classified individually, is obtained by using a so-called Fully Convolutional DenseNet. A semantic segmentation can, however, also be obtained by a fully convolutional neural network or another suitable neural network. Methods for the pixel-by-pixel semantic segmentation of images are known in the prior art from the following documents: Long, J., Shelhamer, E., & Darrell, T. (2015). "Fully convolutional networks for semantic segmentation". In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431-3440) . Jegou, S., Drozdzal, M., Vazquez, D., Romero, A., & Bengio, Y. (2017). "The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation". In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 11-19) . It should also be noted that areas, so-called super pixels, are also in the image 12th can be semantically segmented. Furthermore, the position of the plant to be processed can be determined by classifying the image 12th or another known method for object recognition using a neural network can be determined. The following is the semantic segmentation of the pixels and superpixels as well as the classification of the image, simply referred to as classification.

As already mentioned, a respective position of the plants to be processed in the field is determined using a neural network 10 certainly.

The problem of differentiating a specific plant to be worked (in most cases the useful plant growing in the field) from other plants (mostly weeds) can be handled by the all-versus-all approach. This approach has the advantage that several specific classes 14th to 17th To be defined. In addition, different classes can be defined for useful plants, eg 14 for maize and 15 for sugar beet, whereby only one of these classes is defined as a useful plant according to the field on which the cultivation takes place. The neural network 10 can therefore be used flexibly on fields with different crops.

In addition, any class, e.g. 16, of the neural network can be used 10 can be specified as the plant species to be processed. Of course, several plant classes, for example 16 and 17, can also be defined as the plant species to be processed. When applying fertilizer by the sprayer, for example, the class of the useful plant, for example 14 on a corn field, is determined as the plant to be worked. If a herbicide effective for dicotyledonous plants is to be applied, only dicotyledonous weeds are specified as the plants to be processed. In the mechanical processing by the milling machine, all plants with the exception of the useful plant are defined as the plants to be processed.

As already explained at the beginning, when using the all-versus-all approach, however, the case may arise that an unknown plant (meaning a weed, since the useful plant growing in the field should be known) through the neural network 10 is not classifiable. So that unknown plants can still be classified when using the all-versus-all procedure, they are placed in a general class 22nd summarized so that a combination with the one-versus-other approach is generated. The neural network 10 thus points in addition to the specific classes 14th to 17th at least one general class 22nd , which usually includes a higher hierarchy level. The general class 22nd can be the very general class weeds, but is not limited to it, and can also include, for example, dicotyledonous or monocotyledonous plants. In addition, several more general classes can be defined (e.g. both dicotyledonous and monocotyledonous plants). In 2 is the reference sign of the general class as an example 22nd assigned. It should be noted that the classes of the neural network are not limited to this number of classes.

In the specific case, the 2 shown is the general class 22nd for weeds, i.e. all plants that do not correspond to the useful plant. The neural network according to the invention 10 is thus able to distinguish the useful plant not only from the individual weed species for which a separate class is defined, but also from all other weeds for which no class is defined and which is not annotated in the ground truth with which the neural network 10 was initially trained to distinguish. The neural network according to the invention 10 consequently offers the advantage that unknown weeds can also be classified and then processed if necessary. Retraining the neural network 10 is described in a later section.

In addition, the neural network according to the invention 10 designed as a multi-label classifier so that plants can belong to more than one class. A plant can consequently both belong to one of the specific classes 14th to 17th as well as to the general class 22nd belong. This multiple classification (e.g. to the classes 16 and 22nd ) is, however, when using the neural network according to the invention 10 to be taken into account and can be resolved by classifying a plant as belonging to the class (e.g. class 16 ) is classified accordingly, for which the maximum confidence is determined.

If it is determined that the processing of the plants to be processed in the field is only inadequately carried out because many plants in the field are unknown or there is a strong variation, the neural network must 10 be adapted to the changed conditions. Since the neural network according to the invention 10 the general class 22nd retraining can be done in a simple manner.

For this purpose, new training data are generated for the relevant field by capturing images in an area of the field in which no useful plants are growing, so that only weeds are present in the images. For this purpose, an area at the edge of the field can be recorded. If the individual rows of the field in which the crops are sown are known, images can also be captured in an area between the rows. Subsequently, in these images all plants are segmented from the ground and classified as belonging to the general class 22nd appropriately annotated. Provides the neural network 10 a reliable classification result for the useful plant, images can also be taken of an area in which useful plants are growing. In this case, the recognized crops are annotated accordingly in the images all other plants are automatically considered to be in the general class 22nd appropriately annotated. As described below, this data can be used to retrain the neural network 10 respectively.

A method for training the neural network according to the invention is described below 10 described. Usually, when training neural networks, a cost function of the errors that are caused in the classification is minimized. During the initial training of the neural network according to the invention 10 however, the cost function is adjusted in such a way that no costs are generated in the event that a plant which does not correspond to the useful plant and for which a special class (e.g. 16) is defined as belonging to the general class 22nd is classified accordingly. In addition, the training data with which the neural network 10 initially trained, adapted so that weeds are also always considered to be in the general class 22nd be annotated accordingly. Consequently, weeds have several annotations in the initial training data (ie type of weeds, eg 16, and general class of weeds 22nd ) on.

If the neural network 10 is now to be retrained in the field, the training data, as described above, are only for the general class 22nd appropriately annotated. In this case the cost function is adjusted so that no error costs are generated when a plant belongs to the general class 22nd belongs as one of the special classes 15th to 17th is classified accordingly. The neural network according to the invention 10 thus offers the advantage that retraining with new training data can be carried out in a simple manner. In addition, new training data can be generated without exact and species-specific annotation of the unknown plants, since they belong to the general class 22nd are annotated accordingly. As a result, the neural network has to be retrained 10 possible in the field with little effort.

The neural network according to the invention 10 thus offers the advantage that it can be retrained quickly and efficiently if an insufficiently precise classification is determined during processing in the field. In addition, the neural network according to the invention 10 can be used for the classification of plants in fields with different crops, as each has its own specific class defined.

After the position of the plant to be worked in the field in step S106 is determined using the neural network according to the invention, the selected processing tool in step S108 to the position of the plant and the corresponding processing can be carried out for the individual plant. A mechanical tool can be guided exactly to the position of the plant or the sprayer can be brought up to a predetermined distance to the weeds or the useful plant to apply the pesticide, plant protection agent or fertilizer and pointed at it. In order to enable an exact control of the movable device, it may be necessary to convert the position of the plant determined by the image into the coordinate system of the movable device. In addition, a speed at which the vehicle is moving forward must be taken into account when the machining tool is introduced.

Then the plant is in step S110 edited with the editing tool. The plant is removed, chopped up or destroyed using the mechanical tool or sprayed with the pesticide, plant protection agent or fertilizer. As a result of the mechanical processing of the plants or the targeted application of chemical substances, the amount of chemical substances applied in conventional methods can consequently be significantly reduced, so that costs and the impact on the environment are reduced.

The intended area of application of the method according to the invention relates to autonomous field robots or intelligent attachments for soil cultivation and crop protection in vegetable, horticultural and arable farming. In principle, the neural networks described above can also be used in other areas in which unknown objects may be present, but which have to be classified by a classifier.

QUOTES INCLUDED IN THE DESCRIPTION

This list of the documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Non-patent literature cited

• Long, J., Shelhamer, E., & Darrell, T. (2015). "Fully convolutional networks for semantic segmentation". In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 3431-3440) [0023]
• Jegou, S., Drozdzal, M., Vazquez, D., Romero, A., & Bengio, Y. (2017). "The one hundred layers tiramisu: Fully convolutional densenets for semantic segmentation". In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops (pp. 11-19) [0023]

Claims (10)

Method (100) for processing plants in a field on which a specific crop is grown, by means of a processing tool, with the following steps: Capturing (S104) an image of the field, the image (12) being correlated with positional information; Determining (S106) a position of a plant to be tilled in the field using a neural network (10) into which the captured image (12) is input, the neural network (10) having several specific classes (14 to 17) and one general class (22), the useful plant belonging to one of the specific classes (eg 14) and plants that do not correspond to the useful plant both to one of the special classes (eg 16) and to the general class (22), or at least belong to general class (22); Guiding (S108) the processing tool to the position of the plant; and Processing (S110) the plant with the processing tool. Method (100) according to Claim 1 , with plants that do not correspond to the useful plant being annotated in training data for the initial training of the neural network (10) both with the special class (eg 16) and with the general class (22). Method (100) according to Claim 2 , wherein a cost function is adapted during an initial training with the initial training data that no costs are generated in the event that a plant that does not correspond to the useful plant and for which a special class (e.g. 16) is defined as belonging to the is classified as belonging to general class (22). Method (100) according to one of the preceding claims, wherein plants which do not correspond to the useful plant are annotated in training data for retraining the neural network (10) with the general class (22). Method (100) according to Claim 4 , the training data for retraining the neural network (10) being recorded in a region of the field in which there are no useful plants. Method (100) according to Claim 4 or 5 , wherein a cost function is adapted for retraining with the training data for retraining so that no costs are generated in the event that a plant belonging to the general class (22) as belonging to one of the special classes (e.g. 16) is classified. Method (100) according to one of the preceding claims, wherein the plants to be processed are useful plants. Method (100) according to Claim 1 , wherein the plants to be processed are either plants from a special class (eg 16) or plants from several special classes (eg 16 and 17) or plants from the general class (22). Control unit for controlling a cultivating tool for cultivating plants in a field, the control unit being set up to carry out the following steps: Receiving a captured image of the field, the image (12) being correlated with position information; Determining (S106) a position of a plant to be tilled in the field using a neural network (10) into which the captured image (12) is input, the neural network (10) having several specific classes (14 to 17) and one general class (22), the useful plant belonging to one of the specific classes (eg 14) and plants that do not correspond to the useful plant both to one of the special classes (eg 16) and to the general class (22), or at least belong to general class (22); Outputting a control signal for controlling the processing tool in order to process the plant. Agricultural working machine with a processing tool for processing plants in a field, and a control unit according to Claim 9 .

Images