DE102022206224A1 - Method for automatically labeling input images, training images, method for training or retraining a weed analyzer, analysis device, computer program and machine-readable storage medium - Google Patents

Abstract

For the selective spraying of the crop protection product, the weeds must be detected in real time while an appropriate agricultural vehicle drives over the field. It is known that such weed detection can be made possible using cameras and image processing algorithms. A method for automatically labeling input images 1 with a weed type of a weed and for generating labeled output images 2 is proposed, with a weed determiner 3, the weed determiner 3 is based on a neural network and wherein the weed determiner 3 determines the weed type of the weed in the input images 1, the input images 1 being labeled at least with the weed type and the labeled output images 2 including a localization of the weed and the weed type as labels L1, L2.

Description

State of the art

The invention relates to a method for automatically labeling input images with a weed type of weed and for generating labeled output images. The invention further relates to training images which are labeled by the method, a method for training or retraining a weed analyzer based on the training images, an analysis device with the weed analyzer, a respective computer program and a machine-readable storage medium with the computer program.

Row crops are crops whose distances in the sowing row are 40 to 70 cm. Examples include sugar beet and corn. The latest developments enable selective spraying of crop protection products on weeds in row crops, thus saving crop protection products and resulting in a reduction in environmental pollution. For the selective spraying of the crop protection product, the weeds must be detected in real time while an appropriate agricultural vehicle drives over the field. It is known that such weed detection can be made possible using cameras and image processing algorithms.

For example, the publication discloses EP 38155 2 9 A1 , which represents the closest prior art, a computer system comprising image receiving logic configured to receive image data indicating an image of a field, ground identification logic configured to identify a first image portion of the image representing the ground in the field, image segmentation logic configured to identify a remaining image portion that omits the first image portion from the image, and crop classification logic configured to apply a crop classifier to the remaining image portion and a second image portion of the Identify image depicting locations of crop plants in the field. The computer system also includes weed identification logic configured to identify locations of weed plants in the field based on the identification of the first and second image portions, and control signal generation logic configured to generate a machine control signal based on the identified locations of the weed plants .

Disclosure of the invention

The invention relates to a method for automatically labeling input partial images with the features of claim 1, training images with the features of claim 10, a method for training or retraining a weed analyzer with the features of claim 11, an analysis device with the features of claim 12 Computer program with the features of claim 13, and a machine-readable storage medium with the features of claim 14. Preferred or advantageous embodiments of the invention result from the subclaims, the following description and the attached figures.

The invention relates to a method for automatically labeling input partial images. The input partial images are preferably designed as color images; alternatively and/or additionally, these include black-and-white images, grayscale images or IR images. In particular, the input partial images are designed as sections of input images.

Labeling refers in particular to annotating additional information in the input partial image. The input sub-images are labeled with respect to a weed type of a weed contained in the input sub-image to thereby generate labeled output sub-images, the output sub-images having the respective label. In particular, the respective label is selectively and/or individually assigned to the respective input partial image. The output field and the input field have the same and/or unchanged image content.

A weed type or a specific weed type is understood to mean, in particular, a weed species and/or a weed genus. The weed type refers in particular to a type of weed or a subspecies of the respective weed. In particular, the different weed types differ in terms of their appearance on the input image, e.g. in terms of leaf shape, flower shape, outline, leaf surfaces/texture, color, number of leaves, leaf veins, leaf top versus leaf underside, leaf blade, leaf sheath, leaf hairiness, leaf base, leaf position (hanging,. ..), stem or culm shape, stem or culm cross section, nodes,..., flowers, inflorescence, texture, color,..., fruits, seeds, texture, color, ... etc. Can optionally be added The weed types differ in terms of a growth state, such as young plants, flowering plants, etc. In particular, it is intended that the weed type is determined so precisely that plants can be planted selectively and/or individually protective agents can be selected in relation to this type of weed.

The method uses a weed determiner, the weed determiner being based on a neural network. For example, the input partial image is provided to an input layer of the neural network. The neural network is designed in particular as a CNN (Convolutional Neural Network) or as a folded neural network. The weed determiner determines the weed type of the weed in the input sub-images.

The input sub-images are labeled at least with the weed type, with the labeled output sub-images comprising as a label a localization of the weed and the weed type of the weed. The labeled output subimages thus provide all the data that is needed to control the spraying of crop protection products in a location-selective and weed-selective manner. The labels “location/position” and “weed type” can also be designed as a common label or as a set of labels.

The label for locating the weed contains in particular information about the position of the weed and/or the input partial image and/or output partial image in an original image from which the input partial image is taken, in particular as a section. The label for locating the weed in the initial partial image can include a center of gravity of the weed, a bounding box around the weed or other image position information. Optionally, the label for localization can include a data link, such as a link to the original image. This makes it possible to link the labeled initial partial image with the original image in terms of data technology and to transfer the label of the localization and the weed type to the original image so that it is present as a labeled original image. In this way, a plurality of labels or sets of labels can be annotated in the original image, so that, for example, several weeds are labeled with the location and weed type.

It is an idea of the invention that in the future the crop protection agent should not only be applied to the field in a location-selective manner in order to achieve maximum protection of the crops with a minimal amount of crop protection agent. Rather, the corresponding agricultural equipment should have a plurality of plant protection products and corresponding spray devices, so that the plant protection product can be adapted to the respective weed type not only in a location-selective manner, but also in the type of plant protection product and/or in the amount of the plant protection product and can therefore be applied in a weed-type-selective manner . For example, a weed type A may require a different plant protection product than a weed type B. The method for automatically labeling input partial images generates training images - as will be explained below - which are used for training or retraining by a weed analyzer to localize the weeds and classify the weeds in relation to a weed type can be used in one. The process is therefore used for technical preparation for the use of the weed analyzer in agriculture.

In a preferred embodiment of the invention, the input partial images are at least partially designed as unlabeled partial images. In particular, these are unlabeled with regard to localization of the weed and/or weed type. The weed determiner is designed to determine the localization of the weed in the input partial image in addition to the weed type. The input subimage is thus labeled with the weed type and the location of the weed to generate the labeled output subimages.

In a preferred development of the invention, the labeled initial partial images are stored and/or collected in a weed database. In the event that the input sub-images are taken from the weed database, it is preferred that only the labels are transferred to the already existing input sub-images and/or existing labels are updated in order to generate and/or collect the labeled output sub-images in the weed database . In particular, the weed database only has partial images. It is particularly preferred that only one weed is labeled with the weed type in each partial image.

The weed database therefore includes training images for training and/or retraining the weed analyzer.

In a preferred alternative or development of the invention, the input partial images are partly designed as pre-labeled partial images from one or the weed database. The localization of a recognized but non-specific weed is labeled in the pre-labeled input sub-images. It is therefore possible with the method to use pre-labeled input partial images which have already been used or were created from the method for the location-selective spraying of plant protection products, but without the weed-type-specific spraying. This means that it is not necessary to procure a large number of new, unlabeled input subimages. Rather, on the one hand, existing input parts can be used images are used and, on the other hand, the localization of the recognized, non-specific weeds and thus the pre-labeling of the input partial images can be used advantageously, since the weed determiner can concentrate on positions of localization of the already recognized, non-specific weeds. The weed determiner determines the weed type as a specification of the non-specific weed based on and/or using the localization of the identified non-specific weed. The output sub-images have the localization of the weed as a label, in particular the localization of the weeds of the pre-labeled input sub-images, and the specific weed type.

In a preferred development of the invention, the weed determiner is trained with a subset of expert images, comprising expert sub-images, the expert images being labeled by experts optionally with respect to the weed type and optionally with respect to the localization of the weed. In this way it is ensured that the weed detector receives basic training, with the manually labeled expert images being used for the basic training. It is therefore possible to first train the weed determiner with the subset of expert images, then to generate the labeled output sub-images from the input sub-images and then to train and/or retrain the weed analyzer on the basis of the labeled output sub-images, in particular from the weed database.

The expert partial images can be limited to the label of the weed type and in particular without the label for the localization/position of the weed. In this case these are only used as a sample for the weed determiner. In the event that these also have the label for the localization/position of the weed, they can be merged with the expert images from which the expert partial images come, so that labeled expert images are available that can be used as training images for the weed analyzer.

In the architecture of the neural network, which is designed as or includes a folded neural network, in particular a convolutional neural network CNN, in particular a fully convolutional CNN/fully convolutional CNN, it has proven to be particularly advantageous that the folded neural network has at least one Pooling layer, wherein the at least one pooling layer is designed as a generalized mean pooling layer. The folded neural network preferably has a plurality of pooling layers, with some, a majority or all of the pooling layers being designed as a generalized mean pooling layer. The generalized mean pooling layer has trainable weights in the feature maps, where the trainable weights are trained using backpropagation.

For example, AlexNet, VGG, or ResNet are used as CNN, with their fully connected layer being discarded. Given the input subimage, the output is a 3D tensor X with dimensions W × H × K, where K is the number of feature maps in the last layer. Let Xk be the set of W xH activations for the feature map k ∈ {1 ... K}. The network output consists of K such activation sets or 2D feature maps. Preferably, the very last layer is designed as a Rectified Linear Unit (ReLU), so that X is non-negative. The Generalized Mean Pooling layer takes X as input and produces a vector f as the output of the pooling process.

In the case of the conventional global max pooling/MAC vector, this vector f is given by $${\text{f}}^{\left(m\right)}={\left[\begin{array}{ccc}{\text{f}}_1{}^{\left(m\right)}\dots & {\text{f}}_k{}^{\left(m\right)}\dots & {\text{f}}_K{}^{\left(m\right)}\end{array}\right]}^{\text{T}},{\text{f}}_k{}^{\left(m\right)}=\underset{x\in X}{\text{Max}}x,$$

While this is given for a mean value pooling (SPoC vector), by $${\text{f}}^{\left(a\right)}={\left[\begin{array}{ccc}{\text{f}}_1{}^{\left(a\right)}\dots & {\text{f}}_k{}^{\left(a\right)}\dots & {\text{f}}_K{}^{\left(a\right)}\end{array}\right]}^{\text{T}},{\text{f}}_k{}^{\left(a\right)}=\frac{1}{\left|X_k\right|}{\displaystyle \sum _{x\in X_k}x.}$$

Instead, a Generalized Mean Pooling (GeM) is proposed, given by $${\text{f}}^{\left(G\right)}={\left[\begin{array}{ccc}{\text{f}}_1{}^{\left(G\right)}\dots & {\text{f}}_k{}^{\left(G\right)}\dots & f_K{}^{\left(G\right)}\end{array}\right]}^{\text{T}},{\text{f}}_k{}^{\left(G\right)}={\left(\frac{1}{\left|X_k\right|}{\displaystyle \sum _{x\in X_k}{x}^{p_k}}\right)}^{\frac{1}{p_k}}.$$

The pooling methods (1) and (2) are special cases of GeM pooling according to (3), i.e. global max pooling when pk approaches infinity 1 and mean pooling for pk = 1. The feature vector ultimately consists of a single value per feature map, i.e. the generalized mean activation, and its dimensionality is equal to K. The pooling parameter pk can be set manually or learned since this operation is differentiable. The pooling parameter pk is preferably learned through backpropagation. The generalized mean pooling layer makes the weed analyzer particularly powerful. For further details, please refer to the publication Fine-tuning CNN Image Retrieval with No Human Annotation by Filip Radenovic, Giorgos, Tolias Ondrej Chum, which is available at https://arxiv.org/pdf/1711.02512v2.pdf and the disclosure of which is taken over entirely via referencing.

In the architecture of the neural network, which is designed as or includes a folded neural network, in particular a convolutional neural network CNN, in particular a fully convolutional CNN/fully convolutional CNN, it is further preferred that the features x _i after normalization of the center and the respective feature are mathematically distributed on a hypersphere, with the center describing the position in the hypersphere of the learned feature. It is envisaged that after the transformation in the hypersphere, a geodesic distance between the sample/feature xi and the center is added to the feature x _i as an Additive Angular Margin Loss to generate a margin penalty. After the inverse transformation, the feature xi is rescaled and further processed to prepare a loss function by a softmax algorithm, in particular an angle softmax loss. By implementing the Additive Angular Margin Loss, the features x _i are closer to the assigned center and at the same time further away from the next center. Details on the implementation of the Additive Angular Margin Loss can be found in the publication: ArcFace: Additive Angular Margin Loss for Deep Face Recognition by Jiankang Deng, Jia Guo, Niannan Xue, available at https://arxiv.org/pdf/1801.07698.pdf . Details about the angular softmax loss can be found in the publication: SphereFace: Deep Hypersphere Embedding for Face Recognition by Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, Le Song, available at https://arxiv.org /pdf/1704.08063.pdf.

In a possible development of the invention, the weed determiner and/or the weed database are connected to an image database with a plurality of input images as original images. The input partial images form a section of the input images. It is envisaged that the labeled output sub-images, which are based on the input sub-images, are data-technically connected to the associated input images. The data connection ensures that labeled source images are generated, with the labeled source images having the location of the weed and the specific weed type as a label.

The data connection can be implemented by the label for localization, since this - as explained above - can be, for example, a position of the output partial image in the input image. Optionally, the label can also have a path to the input image.

In other words, a section of an input image is first generated, the section representing one of the input partial images. This input partial image is subsequently labeled using the previously described method in order to generate the labeled output partial image. Subsequently, the label of the labeled output partial image is provided via the data connection of the original input image in order to generate the labeled output image in this way.

A further subject of the invention forms a plurality of training images, the training images being designed as the labeled initial partial images and/or the labeled initial images, which are generated by the method according to one of the preceding claims. Optionally, the training images also include the labeled expert images or the expert partial images with the label of the weed type.

A further subject of the invention relates to a method for training or retraining a weed analyzer, which is based on a neural network, which is designed to localize weeds and/or to classify the localized weeds in relation to a weed type in a working image, wherein the weed analyzer is trained and/or retrained based on the training images.

Another subject of the invention is an analysis device for analyzing a working image. Crops and weeds can be shown in the working image. The device has the weed analyzer or the weed determiner, which is designed as or includes a neural network, in particular as a CNN. The weed analyzer and/or the weed determiner is trained according to one of the preceding methods. The analysis device is designed to be mounted on an agricultural vehicle or device, the analysis device having an output interface for outputting data on the location of the weed and the weed type. The agricultural vehicle or device is designed to implement site-selective and weed-type-specific spraying of plant protection products based on the data. For this purpose, for example, it has a plant protection product distribution device which, depending on the location of the weed and the type of weed, selects or mixes plant protection products from a plurality of plant protection products in a weed-type-selective or weed-dependent manner and applies them locally, depending on the location of the weed. In this way, plant protection products can be saved because they are only sprayed onto the weeds in a location-selective and weed-type-selective manner.

An optional item is formed by the agricultural vehicle or device with the analysis device. In particular, the agricultural vehicle or device has field sprayers as part of the crop protection agent distribution device, whereby it is possible to select from various crop protection agents carried, for example at the respective nozzles, and to mix them and/or dilute them with water in order to apply them in a weed-specific manner.

A further subject of the invention is a computer program, wherein the computer program is designed and/or set up to carry out, apply and/or implement the method according to the invention and/or the steps of the method when it is executed.

A further subject of the invention is a machine-readable storage medium, the computer program being stored on the storage medium.

• 1 ein Blockdiagramm zur Illustration eines Ausführungsbeispiels des erfindungsgemäßen Verfahrens;
• 2 ein Blockdiagramm einer Analysevorrichtung mit einem Unkrautanalysator ausgebildet als ein gefaltetes neuronales Netzwerk, welches mit Trainingsbildern aus dem Verfahren gemäß 1 trainiert ist;
• 3 eine schematische Darstellung des gefalteten neuronalen Netzwerks des Unkrautbestimmers;
• 4 eine schematische Darstellung von einem Teil des gefalteten neuronalen Netzwerks des Unkrautbestimmers.

Further advantages, effects and/or refinements result from the attached figures and their description. Show:

• 1 a block diagram to illustrate an exemplary embodiment of the method according to the invention;
• 2 a block diagram of an analysis device with a weed analyzer designed as a folded neural network, which uses training images from the method according to 1 is trained;
• 3 a schematic representation of the convolutional neural network of the weed determiner;
• 4 a schematic representation of part of the convolutional neural network of the weed determiner.

The 1 shows in a schematic block diagram a method for automatically labeling input sub-images 1. The input sub-images 1 are provided with a label L1 for locating a weed and a label L2 for a weed type of the weed and form labeled output sub-images 2. The weed type describes the type of weed and/or or the weed genus, i.e. the specific name of the weed. In the labeled initial partial image 2, the location of the weed and the weed type are known with the labels L1 and L2. The labels L1 and L2 can also be designed as a common label or a set of labels. The L1 label describes the location and the L2 label describes the object class (e.g. weed or crop). Ie there is a label set with several parameters: the position of the bounding box in the image (x,y), the width (w) and height (h) of the bounding box as label L1 and the class of the object (cls), which is the weed type describes, as label L2. The label L1 may also include the position of the input field and/or the output field in the input image as well as a path or a link to the input image.

The method uses a weed determiner 3, wherein the weed determiner 3 is designed as or includes a folded neural network (CNN). The input partial images 1 are provided to an input layer of the weed determiner 3.

The input partial images 1 are at least partially designed as unlabeled partial images 4, which at least do not carry any labels L1, L2 with regard to the localization and the weed type. Alternatively, the input images 1 can be designed as pre-labeled images 5, with at least the label L1 being provided for locating a weed in the partial image 5.

In the event that the input partial images 1 are designed as unlabeled partial images 5, the weed determiner 3 is designed to determine both the localization of the weed and to determine the weed type and to annotate the labels L1 and L2. In the event that the input partial images 1 are designed as pre-labeled partial images 5, the label L1 of the localization of the weed can form input information for the weed determiner 3, whereby the weed determiner 3 only determines the weed type for the label L2.

In both cases, the output subimage 2 has the label L1 and the label L2. It can also be provided that a first weed determiner 3 is designed for unlabeled partial images 4 and a second weed determiner 3 for the pre-labeled partial images 5. The input partial images 1 can come, for example, as excerpts from a collection of input images 17 or from a camera.

Alternatively, these come from a weed database 6, the weed database 6 comprising a large number of input partial images 1 or partial images 5. The input partial images 1 can be designed both as unlabeled partial images 4 and as pre-labeled partial images 5. On the one hand, the weed determiner 3 can be used to process an input partial image 1 and to output the output partial image 2 directly. However, it can also be provided that the weed determiner 3 takes an input partial image 1 from the weed database 6, which is designed as an unlabeled partial image 4 or as a pre-labeled partial image 5, which supplements the label, so that the labeled output Partial image 2 is formed and the labeled initial partial image 2 is entered back into the weed database 6. With this procedure it is possible to supplement an already existing weed database 6 with regard to the label L1 of the weed type.

The weed determiner 3 is trained in particular by expert images 7 comprising expert sub-images, which have been manually labeled, in particular with regard to the localization of the weed and the weed type.

With the structure shown, it is not only possible for the weed database 6 to be further filled and improved. It is also possible for input images 17 from another image database or image collection with already pre-labeled input partial images 5 to be post-processed via the weed determiner 3 and converted into labeled output images 18. For this purpose, there is a data connection between input image 17, the pre-labeled input partial image 5 and the labeled output partial image 2.

In one variant, it is possible for the input image 17 to be re-labeled with the pre-labeled input partial image 5 by the weed determiner 3 by first generating the labeled output partial image 2 and assigning the labels L1 and L2 to the input image 17 in order to create the labeled output image 18 generate. Input image 17 and output image 18 have the same image content and only differ in the labels L1 and L2.

In another variant, a pre-labeled input partial image 5 is re-labeled in the weed database 6 in order to generate the labeled output partial image 2. Through a data connection to the original input image 17, the labels L1 and L2 are annotated in the original input image 17 in order to generate the labeled output image 18.

The 2 shows an analysis device 8 in a schematic block diagram, the analysis device 8 being designed for mounting on an agricultural vehicle or device. The analysis device 8 can be connected via an input interface 9 to a camera 10, which records and provides working images as input images 1 online and/or in real time. The analysis device 8 has a weed analyzer 11, which is designed as or includes a neural network, in particular a CNN.

The weed analyzer 11 locates any weeds in the input image 1 and determines the weed type of the located weeds. The analysis device 8 has an output interface 12 for outputting data on the localization of the weed and the weed type in the input image 1. The data is forwarded to a plant protection product distribution device 13, whereby the plant protection product distribution device 13 can issue plant protection products on the one hand in a location-selective manner and on the other hand in a weed-type-selective manner.

The plant protection agent distribution device 13 can, on the one hand, use one or more spray nozzles to direct the plant protection agent specifically to the localized weeds (location-selective) and, on the other hand, to selectively select or mix a plant protection agent that is suitable for the weed type (weed-type-selective). In particular, the plant protection agent distribution device 13 has more than two, preferably more than four, different plant protection agents.

It is envisaged that the weed analyzer 11 is trained by output partial images 2 of the weed determiner 3 and/or by the output images 18 labeled via the weed determiner 3. The initial partial images 2 can be taken over directly by the weed determiner 3 or made available via the weed database 6. The initial images 18 labeled via the weed determiner 3 can be taken from the image database or the image collection. As an alternative to the weed analyzer 11, the weed determiner 3 can also be used in the analysis device 8.

The 3 shows a schematic diagram of a folded neural network (CNN) 14 as an exemplary embodiment of the weed determiner 3. Roughly described, the input partial image 1 is transferred to an input layer of the network 14. The network 14 has a plurality of convolution layers 15, followed by at least one pooling layer, in particular as an output layer. The pooling layer is designed as a generalized mean pooling layer 16, as already described previously. The generalized mean pooling layer 16 has trainable weights in the feature maps, the trainable weights being trained by backpropagation. By training the generalized mean pooling layer 16, the weed determiner 3 can be adapted particularly well.

In the 4 is the section of the folded neural network 14 of the weed determiner 3 after which a generalized mean pooling layer 16 is shown. Based on the feature xi and normalizing the weight W, the cosθj (logit) for each class is obtained as W ^T _j x _i . The arccos(cosθ _yi ) is then calculated and the angle between the feature xi and the is obtained Ground truth weight W _yi . W _j forms a kind of center for each class. Subsequently, an angle penalty or edge penalty m is added to the angle θ _yi . Then cos(θ _yi + m) is calculated and all logits are multiplied by the feature scale s. The logits then go through a softmax function and contribute to the cross-entropy loss. By adding the marginal penalty m, classes can be particularly well delimited by the weed determiner 3.

Particular advantages arise in particular from a reduction in effort or a reduction in the expertise required for labeling images of crops with and without a wide variety of weed species/weed genera. A method for automatically labeling images with regard to specific weed species or weed genera is presented. The key component is a so-called “weed hunter”, the weed determiner 3, - a CNN with a trainable generalized mean pooling layer 16, which with angular softmax loss and additive angular margin loss based on a limited number of images labeled by experts (expert images 7) with regard to one (or several) specific weed species or weed genus (e.g. “Conyza Sample Image”). The trained “Weed Hunter” can automatically re-label an existing image database as a weed database 6 with partial images of crop plantings with and without a wide variety of weed species / weed genera with regard to specific weed species / weed genera and subsequently structure the image database in this regard or label current partial images, which are then stored in the Image database can be saved.

The images (groups) found via a similarity search are the basis for training or retraining the object detector or classifier for weed detection (weed analyzer) on the sprayer. The advantage is a significant reduction in the effort involved in labeling image data of crops with and without a wide variety of weed species/weed genera by the necessary plant/weed experts. The “Weed Hunter” essentially contains a CNN with a trainable GeM (Generalized Mean) pooling layer (16). The similarity search (and at the end the features/distances found = “training”) is carried out with angular softmax loss for CNNs (e.g. A-softmax loss) and additive angular margin loss using a limited number of images labeled by experts ( Expert images 7) regarding one (or several) specific weed species or weed genus (“Conyza Sample Image). Now the “trained weed hunter” process can be used in two different ways, separately or in parallel.

Version 1:

Excerpts from currently recorded and transmitted input images 17, for example from the sprayer's cameras as unlabeled input partial images 5, are checked for similarity using the "Weed Hunter" (image retrieval) and, if similar, are labeled according to the recognized weed species or weed genus(s). stored in the weed database 6. This means that the weed database 6 is filled with images labeled with respect to one (or several) specific weed species or weed genus(s).

Variant 2:

In the weed database there are partial images with and without weeds, but they are not labeled as a weed type with more specificity regarding specific weed species or weed genus. This corresponds to the current procedure, whereby all plants that do not correspond to the specific crop are initially labeled as weeds in a very general and non-specific manner. A limited number of partial images from or outside the weed database are now subsequently labeled by experts regarding one (or several) specific weed species or weed genus (“Conyza Sample Image”) as weed type L2 and the “Weed Hunter “ trained with these expert images 7, the trained “Weed Hunter” can be used to examine all weed images in the database for similarity and subsequently re-identify one (or more) specific weed species or weed genus as a weed type be labeled.

With the partial images labeled by the experts and the “Weed Hunter”, object detectors or classifiers can be trained or more specifically retrained as weed analyzers 11 for specific weed detection and localization. This saves the effort involved in labeling image data as well as time and costs for plant experts. Plant protection products can be selected more selectively and thereby protect the environment and nature. Since the training of the object detectors or classifiers to detect weeds is carried out offline, the “weed hunter” and the weed database are preferably implemented in the back end of a service provider (cloud, server, PC) of the method according to the invention. The current images of crops with/without weeds can be transmitted wirelessly or collected/stored on the sprayer and read out using a data storage medium and transported to the service provider. Particularly noteworthy is that when new weed species or weed genus partial images are available, old/former weed species or weed genus partial images can be relabeled. Or when available With the help of experts or new knowledge for certain weed species or weed genera, labeling can be subsequently improved (reranking). The process can also be applied or extended to plant diseases.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• EP 3815529 A1 [0003]

Claims (14)

Method for automatically labeling input sub-images (1) with a weed type of a weed and for generating labeled output sub-images (2), with a weed determiner (3), the weed determiner (3) being based on a neural network and the weed determiner (3) determining the weed type of the weed in the input sub-images (1), wherein the input partial images (1) are labeled at least with the weed type and the labeled output partial images (2) include a localization of the weed and the weed type as a label (L1, L2). Procedure according to Claim 1 , characterized in that the input partial images (1) are at least partially designed as unlabeled partial images (4), the weed determiner (3) being designed to determine the localization of the weed in addition to the weed type and the input partial images (1) in addition to the weed type can be labeled with the localization of the weed in order to generate the labeled initial partial images (2). Procedure according to Claim 1 or 2 , characterized in that the labeled output partial images (2) are stored and/or collected in a weed database (6) and/or that the unlabeled input partial images (4) are taken from one or the weed database (6). Method according to one of the preceding, characterized in that the input partial images (1) are at least partially designed as pre-labeled partial images (5) from one or the weed database (6), the localization of the weeds being labeled in the pre-labeled input partial images (5), wherein the weed determiner (3) determines the weed type and the labeled initial partial images (2) have the location of the weed and the weed type as a label (L1, L2). Method according to one of the preceding claims, characterized in that the weed determiner (3) is trained with expert partial images (7), the expert partial images (7) being labeled by experts with regard to the weed type and optionally additionally with regard to the localization of the weed. Method according to one of the preceding claims, characterized in that the neural network is designed as a convolutional neural network (CNN). Procedure according to Claim 6 , characterized in that the convolutional neural network has at least one generalized mean pooling layer (16). Procedure according to Claim 6 or 7 , characterized in that the folded neural network implements an additive angular margin loss and a softmax algorithm after the output layer and/or the last generalized mean pooling layer (16). Method according to one of the preceding claims, characterized in that the weed determiner (3) and/or the weed database (6) is connected to an image database with input images, the input partial images forming a section of the input images, the weed determiner (3) and/or the weed database (6) is designed to connect the labeled output sub-images (2) with the input sub-images in order to generate labeled output images, the labeled output images having the location of the weed and the weed type as a label (L1, L2). Training images, wherein the training images are labeled by the method according to one of the preceding claims, wherein the training images are designed as the labeled initial partial images (2) or are the labeled initial images (2). Method for training or retraining a weed analyzer (11) for locating weeds and classifying weeds in relation to a weed type in a working image, the weed analyzer (11) being based on a neural network, characterized in that the weed analyzer (11). Based on the training images Claim 10 is trained or retrained. Analysis device (8) for mounting on an agricultural vehicle or device with a weed analyzer (11), the weed analyzer (11) using the method according to Claim 11 is trained or retrained, or is trained as the weed determiner (3) who uses the method according to one of the Claims 1 until 9 is trained, wherein the analysis device (8) is designed to localize the weed and to classify the weed in relation to a weed type in a working image, wherein the analysis device (8) has an output interface (12) for outputting data on the localization and the weed type of the weeds. Computer program, wherein the computer program is designed and / or set up to execute the method according to one of the Claims 1 until 9 or after Claim 10 to carry out, apply and/or implement. Machine-readable storage medium, with the computer program on the storage medium Claim 13 is stored.

Images