DE102022207830A1 - Weed detection with polar masks - Google Patents

Abstract

A detector unit (14) for detecting weeds (4) in image data (8) has an input (22) for the image data (8), an output (20), a frame detector (24) which is set up for this purpose in the image data (8) to recognize a respective weed (4) and to provide a respective selection frame (26) which frames the respective weed (4), and a polar detector (28) which is set up to detect a respective weed in the image data (8). (4) and to provide a respective polar mask (30) which follows a contour of the respective weed (4). A device (2) for spraying weeds (4) has a camera (6) which is set up to provide image data (8) from an area (10) that potentially includes or contains weeds (4), the detector unit (14) and a spray module (16) that is spatially correlated with the camera (6) and is set up to detect the weeds (4). Based on the variables provided by the detector unit (14) with a spray agent (18).

Description

State of the art

From the DE 10 2017 124 934 A1 a system for semi-automatic and/or automatic weed removal is known, with at least one system for the automatic detection of weeds, which has at least one optical sensor and a central server unit, with at least one communication network, with at least one control and/or regulating device and with at least a weed removal device, wherein the optical sensor, the server unit and the control and/or regulation device are in data connection by means of the communication network, wherein image data of at least one weed can be generated by means of the optical sensor and can be transmitted to the server unit via the communication network, which corresponds to the transmitted image data evaluates that the weed can be determined, and wherein by means of the server unit, in the event of a clear determination of the weed, weed confirmation data can be transmitted via the communication network to the control and / or regulation device, which controls and / or regulates the weed removal device in response to this weed confirmation data, so that the weeds detected by the optical sensor can be removed.

Disclosure of the invention

The subject of the invention is a detector unit according to claim 1. Preferred or advantageous embodiments of the invention and other categories of invention result from the further claims, the following description and the attached figures.

The detector unit serves or is set up to detect weeds in image data. The detector unit has an input for the image data and an output.

The image data is intended to be an image of an area, e.g. a field from a bird's eye view. The field is used to grow crops (including cultivated plants). There are crops in the field that - after they have sprouted - are shown in particular in the image data. Weeds are also potentially present in the field; this is then also reflected in the image data. One or more identical or different weeds can be present within the image data, in particular a single image. The invention therefore assumes at least that one or more crops and/or one or more weeds are potentially depicted in image data or a single image.

The detector unit has a frame detector. This is set up to recognize a respective weed in the image data. The frame detector assigns a selection frame to each of the individual weeds detected and makes this available for further use (within the detector unit or outside, then at the exit). Provision at the output is therefore optional. Provision at the output therefore takes place depending on the case if the corresponding output variable is desired outside the detector unit for further processing. Otherwise, the provision serves in particular for further processing within the detector unit.

The selection frame is one that (in the image data) frames the weed or its image precisely (within the capabilities of the detector). In particular, the image data/image is rectangular, the selection frame then follows the four sides of the image in parallel and therefore also represents a rectangle.

The detector unit has a polar detector. This is set up to recognize the respective weeds in the image data (like the frame detector) and to provide a respective polar mask (like the frame detector). However, the polar mask is one that follows a contour of the weed, i.e. its outline shape, in particular the outline lines of the individual leaves in the image data/image. Again, this is within the accuracy/capability of the detector. The polar mask is a continuous series of straight lines. The corners of the mask (common end of two straight lines) lie on rays that emanate from a center point and enclose equal angles between them (e.g. 10 rays of 36° each).

Strictly speaking, the “frame detector” could also be understood as a special case of a “polar detector” if the latter worked with four polar points. Conversely, a special “polar detector” also represents a “frame detector”. A “selection frame” (bounding box) can therefore be a special case of a “polar mask” with 4 points. However, in order to clearly separate the “selection frame” from the “polar mask”, the different terms “frame detector” and “polar detector” were chosen here.

The above statements (creation of selection frames / polar masks) apply in the event that weeds are actually depicted in the image data, which can actually be detected by the detectors.

The two detectors already work in particular with regard to the detection of Weeds independently of each other, without using each other's recognition information. In particular, both work in parallel.

The “provision” of selection frames/polar masks means the provision of data/values that represent these variables, e.g. coordinates, size, etc.

The invention offers the advantage that, on the one hand, the weeds identified in an image or image data are each framed or limited by a selection frame. On the other hand, a polar mask is generated using the polar detector, which follows the outline shape and thus the leaf contours of the weed. The leaf contours can thus be recognized. With the help of the detector unit or the size selection frame and polar mask provided in particular at the output, a wide range of options are available for further processing in the context of weed localization and control. In particular, the control of a flexible spray module of a device for spraying weeds, which can spray individual weeds precisely and individually.

The detection of weeds is therefore carried out in that (assuming successful detection of a weed that is actually present or imaged) that image section of the image data in which the weed is imaged is limited by a selection frame and, on the other hand, the polar mask of the contour at the location of the weed of the weeds follows and therefore the weeds are also located there.

In any case, the polar mask and selection frame show a location of the weed in question in the image data, in the image. The invention assumes that the location of the image in the field is known through knowledge of the cultivation position, orientation and characteristics of the respective camera and the kinematics of the sprayer (spray module/device for spraying weeds) and thus also the location of the weeds the field. This means that the weeds can then be sprayed at a precise location.

The invention assumes that the polar mask is located in particular within the selection frame or at least there is an intersection between the selection frame and the polar mask.

In a preferred embodiment, the detector unit has a classifier. This is set up to provide a first class for the respective weed, especially at the exit, based on the selection frame and the polar mask (see above, if necessary for internal use, etc., this also applies below). The first class is one that classifies the weeds. A corresponding classification indicates, for example, the probability that there is actually a weed to be classified/detected in the selection frame or the polar mask, e.g. as a numerical value between zero and one.

In a preferred embodiment, the detector unit has a convolutional neural network (CNN). This is set up to generate or output a feature vector (at least part of it, see below) for the weeds based on the selection image data determined by the selection frame and/or the polar mask and, in particular, to provide it at the output. The selection image data is therefore that section or subset of the image data that lies within the selection frame and/or the polar mask or is delimited by them, and should therefore represent or contain the image of the weed.

The detector unit also has a class module, in particular a fully connected neural network (FCNN, all layers are fully connected / fully connected layers) and / or SVM (Support Vector Machine) and / or a random forest module. Because a feature vector (feature map) is created here, the choice is free. This is set up to generate or output a second class classifying the weeds based on the feature vector and to provide it, in particular at the output of the detector unit. The explanations above for the first class apply to the second class. The first and/or second class also has, for example, the name of the detected weed if the detector unit is set up to detect and differentiate between different weeds. The second class serves to increase the detection probability of the detector unit, as it particularly provides an opportunity for cross-checking with the first class.

In contrast to the first class, the class determination here takes place in particular exclusively on the basis of part of the image data, namely the selection image data, without explicit use of the selection frame or the polar mask. The selection frame (bounding box) ultimately determines the selection image. In this respect, an independent review of the weed or its classification can be made possible.

In a preferred variant of this embodiment, the feature vector additionally has the selection frame and/or the polar mask. The entire feature vector is then generated from the outputs of the CNN and selection frames and/or the polar mask, e.g. indirectly the coverage rate det. The class module, e.g. FCNN, works not only on the basis of the components in the feature vector (output of the CNN) determined based on pure image data, but also on the basis of selection frames / polar masks. This increases the capabilities of the class module / FCNN to determine the second class.

In a preferred embodiment, the detector unit has a plausibility module. This is designed to determine and provide a third class classifying the weed, especially at the exit. However, this only happens if the first and second classes are plausible to each other based on a check by the plausibility module. Otherwise, the plausibility module outputs, for example, an error message or a message that a clear classification of the weed was not possible and a third class is not output. “Plausible” means, for example, agreement between weed type and probability within certain tolerance limits. In particular, only the third class is output at the output of the detector unit, but without outputting the first and second classes. Because these are then implicitly contained in the third class or processed into and into it. Additional other outputs/sizes (selection frame, polar mask, ...) at the output are possible. A particularly reliable classification of weeds is possible using the third class.

In a preferred embodiment, the detector unit has a center point module. This is set up to determine a center point of the polar mask and to provide it, especially at the output. The invention assumes that the center of the polar mask is the center of the leaf contour of the weed. The invention is based on the assumption that, on the one hand, the weed has been completely and correctly recognized in terms of its contour and that a shoot axis and, as an extension, a root base are located in the corresponding center. By providing the location of the center point, it is possible, for example, to specifically spray weeds at the location of the root base. “The center point” are in particular image coordinates in the image data or in the corresponding image.

In a preferred embodiment, the detector unit has a leaf surface module. This is set up to use the polar mask to determine a leaf area of the weed as an area delimited by the polar mask and to provide it, especially at the exit. The determined leaf area of the weed can also be determined in particular as an intersection with a virtual spray cone. The leaf area of the weed that would or will lie within a spray cone is then issued or provided. With such a size, for example, the amount of spray that should be applied to the weeds can be controlled.

In a preferred embodiment, the detector unit has a coverage rate module. This is set up to determine a coverage rate of the weeds based on, in particular in terms of shape, the ratio of the areas of the selection frame and the polar mask. This coverage rate is also provided, especially at the exit. Alternatively or additionally, it is provided to the classifier and/or the CNN and/or the class module/FCNN. The coverage rate is therefore in particular the ratio of the area of the polar mask to the area of the selection frame and therefore the leaf area of the weed to its rectangular outline area in the form of the selection frame. Such a size is particularly useful for the classifiers, since weeds differ in particular by their average leaf width, which is well represented by the coverage rate.

In a preferred embodiment, the detector unit has a coverage rate module. This is set up to recognize a crop or its image or its surface area in the selection frame/polar mask at least within the polar mask and/or the selection frame. A coverage rate of the weed by a crop is then determined and provided, especially at the exit. This provides information that states how high - in the viewing direction of the camera that records the image data - the (area) proportion of weeds that is covered or covered by a crop is. This information can be used in particular not to spray crops with sprays or not to spray them unnecessarily, or to set up spraying in such a way that the crop can be avoided or the weeds can still be reached.

Another subject of the invention is a device according to claim 10 for spraying weeds. The device has a camera. This is set up to generate the above-mentioned image data from an area of the field to be sprayed (e.g. field from a bird's eye view) that potentially has weeds. The device also has the detector unit according to the invention. The device also has a spray module, which is spatially correlated with the camera and which is set up to spray the weeds with a spray agent based on the sizes provided by the detector unit. “Spatially correlated” means that the spraying device is able to localize a specific location in the image data on the area and spray specifically. The “sizes” are at least one of the above-mentioned sizes provided by the detector unit, i.e. - depending on what is provided at the output - selection frame, polar mask, first to third class, center point, ... The device enables particularly targeted and selective spraying of weeds on the area possible.

The device and at least some of its possible embodiments as well as the respective advantages have already been explained in connection with the detector unit according to the invention.

A further aspect of the invention is a method according to claim 11 for detecting weeds in the above-mentioned image data using the detector unit or device according to the invention. In the method (if weeds are present and detection is successful) the frame detector recognizes a respective weed in the image data and provides a respective selection frame that frames the respective weed. The polar detector recognizes (see above) a respective weed in the image data and provides a respective polar mask that follows a contour of the respective weed.

The invention is based on the following findings, observations and considerations and also has the following preferred embodiments. To simplify matters, these embodiments are also called “the invention”. The embodiments can also contain parts or combinations of the above-mentioned embodiments or correspond to them and/or possibly also include previously unmentioned embodiments.

• Als Reihenkultur werden Feldfrüchte bezeichnet, deren Abstände in der Säreihe 40 bis 70 cm betragen. Als Beispiele sind Zuckerrüben und Mais zu nennen. Da bspw. die Einzelkorn - Sämaschinen mittels GNSS (Global Navigation Satellite System) basierter Feldkarten und Lenksysteme arbeiten, ergeben sich in der Regel Abstände der Nutzpflanzen in der Reihe mit nahezu äquidistanten Abständen.

The invention is based on the following observations from practice:

• Row crops are crops whose distances in the sowing row are 40 to 70 cm. Examples include sugar beet and corn. Since, for example, the single grain seed drills work using GNSS (Global Navigation Satellite System) based field maps and steering systems, the spacing of the crops in the row is usually almost equidistant.

It is conceivable to equip field sprayers with weed detection so that cameras and image processing algorithms can be used to enable selective spraying using individual nozzle control or segment-by-segment control, thereby saving on pesticides and reducing environmental pollution. In order to drive in the same lanes as the seed drills and to document the application of crop protection products, among other things, they should also have GNSS-based field maps and steering systems. It is also conceivable to select and mix and/or dilute with water from various plant protection products carried, for example at the respective nozzles.

Field sprayers are known, for example, from “Wikipedia: Plant protection device”, Internet entry, https://de. wikipedia. org/ wiki/ Plant protection device, accessed on May 24, 2022.

• Ein Mustererkennungsprozess lässt sich in mehrere Teilschritte zerlegen, bei denen am Anfang die Erfassung und am Ende eine ermittelte Klasseneinteilung steht. Bei der Erfassung werden Daten oder Signale mittels Sensoren aufgenommen und digitalisiert. Aus den meist analogen Signalen werden Muster gewonnen, die sich mathematisch in Vektoren, sogenannten Merkmalsvektoren, und Matrizen darstellen lassen. Zur Datenreduktion und zur Verbesserung der Qualität findet eine Vorverarbeitung statt. Durch Extraktion von Merkmalen werden die Muster bei Merkmalsgewinnung anschließend in einen Merkmalsraum transformiert. Die Dimension des Merkmalsraums, in dem die Muster nun als Punkte repräsentiert sind, wird bei der Merkmalsreduktion auf die wesentlichen Merkmale beschränkt. Der abschließende Kernschritt ist die Klassifikation durch einen Klassifikator, der die Merkmale verschiedenen Klassen zuordnet. Das Klassifikationsverfahren kann auf einem Lernvorgang mit Hilfe einer Stichprobe / Trainingsdaten basieren. Dazu existieren verschiedene Klassifikationsverfahren, z.B. bekannt aus „Wikipedia: Klassifikationsverfahren“, Interneteintrag, https:// de. wikipedia. org/ wiki/ Klassifikationsverfahren, Abruf am 24.05.2022.

The invention is further based on the following considerations for pattern recognition:

• A pattern recognition process can be broken down into several sub-steps, at the beginning there is the recording and at the end there is a determined classification. During capture, data or signals are recorded and digitized using sensors. Patterns are obtained from the mostly analog signals, which can be represented mathematically in vectors, so-called feature vectors, and matrices. Pre-processing takes place to reduce data and improve quality. By extracting features, the patterns are then transformed into a feature space during feature extraction. The dimension of the feature space, in which the patterns are now represented as points, is limited to the essential features during feature reduction. The final core step is classification by a classifier, which assigns the features to different classes. The classification process can be based on a learning process using a sample/training data. There are various classification methods for this purpose, for example known from “Wikipedia: Classification Methods”, Internet entry, https://de. wikipedia. org/ wiki/ Classification procedure, accessed on May 24, 2022.

• Maschinelles Lernen ist ein Oberbegriff für die „künstliche“ Generierung von Wissen aus Erfahrung und z.B. bekannt aus „Wikipedia: Maschinelles Lernen“, Interneteintrag, https:// de. wikipedia. org/ wiki/ Maschinelles_ Lernen, Abruf am 24.05.2022: Ein künstliches System lernt aus Beispielen und kann diese nach Beendigung der Lernphase verallgemeinern. Dazu bauen Algorithmen beim maschinellen Lernen ein statistisches Modell auf, das auf Trainingsdaten beruht. Das heißt, es werden nicht einfach die Beispiele auswendig gelernt, sondern Muster und Gesetzmäßigkeiten in den Lerndaten erkannt. So kann das System auch unbekannte Daten beurteilen (Lerntransfer) ... Aus dem weiten Spektrum möglicher Anwendungen seien hier genannt: automatisierte Diagnoseverfahren, ..., Sprach- und Texterkennung sowie autonome Systeme. Allerdings werden, wie bei statistischen Verfahren üblich, nur Korrelationen keine Kausalitäten identifiziert, was auch zu falschen Schlüssen führen kann. Ebenso müssen die Trainingsdaten sehr sorgfältig ausgewählt werden, damit sie ohne Bias sind und das komplette Szenario abdecken.

The invention is based on the following machine learning considerations:

• Machine learning is a generic term for the “artificial” generation of knowledge from experience and is known, for example, from “Wikipedia: Machine Learning”, Internet entry, https://de. wikipedia. org/ wiki/ Machine learning, accessed on May 24, 2022: An artificial system learns from examples and can generalize them after the learning phase has ended. To do this, machine learning algorithms build a statistical model that is based on training data. This means that the examples are not simply memorized, but patterns and regularities are recognized in the learning data. This means the system can also assess unknown data (learning transfer)... From the white The spectrum of possible applications should be mentioned here: automated diagnostic procedures, ..., speech and text recognition as well as autonomous systems. However, as is usual with statistical methods, only correlations and no causalities are identified, which can also lead to incorrect conclusions. Likewise, the training data must be selected very carefully so that it is without bias and covers the complete scenario.

The invention is based on the following considerations regarding artificial neural networks. These are known, for example, from “Wikipedia: Artificial neural network”, Internet entry, https:// de. wikipedia. org/ wiki/ Artificial_ neural_ network, accessed on May 24, 2022. It has been shown that learning makes it possible to recognize complex patterns, even without first abstracting the rules.

• Die Erkennung von Objekten mittels eines Kamerasystems gewinnt in vielen Bereichen zunehmend an Bedeutung. Z.B. ist es bekannt aus „ATZ/MTZ Fachbuch Handbuch Fahrerassistenzsysteme Springer Vieweg 2015“:
  1. S. 387 Abschnitt 21.5.1 Objektdetektion:
     • "Erscheinungsbasierte Verfahren erfassen ... die Charakteristik eines Objekttyps ... auf Basis eines Trainingsdatensatzes. Dieser Datensatz beinhaltet unterschiedliche Ausprägungen eines einzelnen Objekttyps und drückt die unterschiedlichen Ausprägungen eines einzelnen Objektes aus. Offline in der Entwicklungs- und Applikationsphase werden in einem Trainingsschritt aus den Trainingsbildern charakteristische Merkmale erzeugt. Eine Sammlung von solchen Merkmalen wird dann zusammengefasst zu einer Gesamtbeschreibung des Objekttyps. Um einen Merkmalsvektor eindeutig einem Objekttyp zuordnen zu können, muss ... ein Klassifikator trainiert ... werden."
  2. S. 388 Abschnitt 21.5.1 Verifikation:
     • "Aufgabe der Verifikation ist es, die fehlerhaften und ungenauen Objekthypothesen aus dem ersten Schritt (Objektdetektion) zu plausibilisieren und durch zeitliche Integration ... zu verbessern.

The invention is based on the following considerations regarding cameras and object recognition:

• The detection of objects using a camera system is becoming increasingly important in many areas. For example, it is known from “ATZ/MTZ technical book manual driver assistance systems Springer Vieweg 2015”:
  1. P. 387 Section 21.5.1 Object detection:
     • "Appearance-based methods record ... the characteristics of an object type ... on the basis of a training data set. This data set contains different characteristics of a single object type and expresses the different characteristics of a single object. Offline in the development and application phase, the "Characteristic features are generated from training images. A collection of such features is then combined to form an overall description of the object type. In order to be able to clearly assign a feature vector to an object type, a classifier must be trained."
  2. P. 388 Section 21.5.1 Verification:
     • "The task of verification is to make the incorrect and inaccurate object hypotheses from the first step (object detection) plausible and to improve them through temporal integration.

In order to recognize a desired object type... a large number of image examples are required for this type, which represent a wide variety of characteristics of the object in ideally all existing environmental conditions. In order for the characteristic image-based properties of the desired object to be determined using machine learning, a large number of counterexamples are also required.

For object recognition, the classifier's (feature) numbers describing the object are converted using many nested and automatically generated weights into a number that indicates the probability of the object to be classified in the current search window. In the previous approach (including with trained neural networks, DNNs), all object examples (labels) are trained. This then results in a rigid structure of preprocessing/object detection based on fixed filter masks and weights."

The invention is based on the following considerations about professional herbs (Conyza), see e.g. “Wikipedia: Professional herbs”, Internet entry, https://de. wikipedia. org/ wiki/ Berufskräuter, accessed on May 24, 2022.

The professional herbs (Conyza) are a genus of plants within the Asteraceae family. Like the closely related genus Erigeron, the German common name is “professional herbs”. From a systematic point of view, Conyza has often been included in the genus Erigeron; To distinguish the German names, Conyza is also called Katzenpfad; The name “fine jet” can also be used for Erigeron. The genus Conyza has 25 to 155 species in temperate and warmer areas, especially the New World.

There are many species of the genus Conyza, which differ significantly in their visible characteristics as well as in their appearance in certain environments (climate, soil, crop culture, date, etc.).

• - Blattform / Umriss, Blattoberflächen / Textur, Farbe, Blattanzahl, Blattrippen, Blattoberseite versus Blattunterseite, Blattspreite, Blattscheide, Blattbehaarung, Blattansatz, Blattstellung (hängend, ...) ...
• - Stengel- bzw. Halmform, Stengel- bzw. Halmquerschnitt, Knoten, ...
• - Blüten, Blütenstand, Textur, Farbe, ...
• - Früchte, Samen, Textur, Farbe, ...

The characteristics visible to humans can be examples:

• - Leaf shape / outline, leaf surfaces / texture, color, number of leaves, leaf veins, top of leaf versus underside of leaf, leaf blade, leaf sheath, leaf hair, leaf base, leaf position (hanging, ...) ...
• - Stem or stalk shape, stem or stalk cross section, nodes, ...
• - flowers, inflorescence, texture, color, ...
• - Fruits, seeds, texture, color, ...

• - Klima, Wetter, Datum / Jahreszeit
• - Nutzpflanzkultur (Nutzpflanze und Unkraut treten i.d.R. gemeinsam auf/ gehören zusammen)
• - Boden, Nährstoffe

In addition, there are visible correlations between the occurrence of the type and appearance of weeds and the environment, for example:

• - Climate, weather, date/season
• - Crop culture (crop and weeds usually occur together/belong together)
• - soil, nutrients

In addition, there are geometric, structural and morphological correlations in the characteristics and patterns of plants themselves that we as humans hardly or often only notice unconsciously.

• Die im Rahmen der Erfindung genannte „Polarmaske“ ist insbesondere eine solche „PolarMask“, wie sie in folgender Veröffentlichung erläutert ist: „PolarMask: Single Shot Instance Segmentation with Polar Representation“, Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Ding Liang, Chunhua Shen, Ping Luo, The University of Hong Kong, Sensetime Group Ltd, Xi'an Jiaotong University, Nanjing University, The University of Adelaide, E-mail: xieenze@hku.hk, Download von „https://arxiv.org/pdf/1909.13226.pdf“ am 24.05.2022.

The invention is based on the following considerations regarding polar masks:

• The “polar mask” mentioned in the context of the invention is in particular such a “PolarMask” as explained in the following publication: “PolarMask: Single Shot Instance Segmentation with Polar Representation”, Enze Xie, Peize Sun, Xiaoge Song, Wenhai Wang, Ding Liang, Chunhua Shen, Ping Luo, The University of Hong Kong, Sensetime Group Ltd, Xi'an Jiaotong University, Nanjing University, The University of Adelaide, E-mail: xieenze@hku.hk, Download from “https://arxiv .org/pdf/1909.13226.pdf” on May 24, 2022.

So the present “PolarMask” is a single-shot anchor box-free instance segmentation method. Unlike previous works that typically solve mask prediction as a binary classification in a spatial layout, PolarMask proposes to represent a mask by its contour and model the contour by a center and rays emitted from the center to the contour in polar coordinates ( Instance segmentation as instance center classification and dense distance regression in a polar coordinate).

PolarMask is designed to be almost as simple and structured as single-stage object detectors, resulting in negligible computational overhead. A single-stage detection strategy means that the images to be analyzed only need to be read once.

The model takes an input image and predicts the distance from a sampled positive position (candidate from the instance center) to the instance contour at any angle and outputs the final mask after assembly. PolarMask's entire pipeline is almost as simple and structured as FCOS (Fully Convolutional One-Stage-Detector). It results in negligible computational effort. In addition, PolarMask can be viewed as a generalization of FCOS. In other words, FCOS is a special case of PolarMask because bounding boxes can be viewed as the simplest mask with only 4 directions. Therefore, it is suggested to use PolarMask over FCOS. To maximize the benefits of polar representation, polar center and polar IoU (Intersection over Union) loss are proposed to sample high-quality center samples and optimize for dense distance regression, respectively.

• Die aus einer Vogelperspektive erkennbare Kontur von Pflanzenteilen insbesondere von Blättern und mittels der umgebenden viereckigen Bounding Box (Auswahlrahmen) bestimmbaren Bedeckungsrate ist ein geeignetes Kriterium zur Bestimmung von Pflanzenarten und deren Zuständen (Wachstumsstatus, SOH (State of Health = Gesundheitszustand), ...), die in der Praxis derart nicht bestimmt werden.

The invention is based on the following problem:

• The contour of plant parts, especially leaves, that can be seen from a bird's eye view and the coverage rate that can be determined using the surrounding square bounding box (selection frame) is a suitable criterion for determining plant species and their states (growth status, SOH (State of Health), ...) , which are not determined in this way in practice.

In addition, methods for instance segmentation are carried out in two stages, i.e. serially, which costs computer resources and in particular running time, but in an online execution such as the passage of a sprayer with cameras for plant recognition over a plant population for selective spraying of weeds, for example.

An object of the invention is in particular an improved detection of plants, in particular weeds, as well as a determination of further plant characteristics and plant states derived therefrom for adapted corresponding plant protection measures, for example of the sprayer.

• - Erkennung der Blattkonturen, der umgebenden Bounding Box (Auswahlrahmen: Bildposition, Höhe, Breite) und Bedeckungsrate (Verhältnis Konturfläche (Fläche, die von der Polarmaske eingeschlossen ist) zu Bounding Box Fläche) mittels PolarMask (erste Klasse)
• - Mit dem zugehörigen Unkrautbild werden Merkmals- Vektoren / - Karte (Feature Map) in einem CNN trainiert.
• - Mit der Feature Map und den Daten aus der PolarMask (Polarmaske) und dem Auswahlrahmen, die ebenfalls in die Feature Map gehen, wird ein Klassifikator (Klassenmodul), z.B. FCNN (Fully Connected Neuronal Network) zur Unkrauterkennung trainiert.
• - Das erkannte Unkraut (Art und Wahrscheinlichkeit, zweite Klasse), zugehörige Bedeckungsrate und Bounding Box (erste Klasse) werden miteinander plausibilisiert, d.h. ein bestimmtes Unkraut hat normalerweise eine in einem bestimmten Bereich sich befindende Bedeckungsrate und Bounding Box (Vergleiche über applizierbare Wertebereiche, PolarMask steckt implizit in der Bedeckungsrate),
• - sind die Informationen plausibel zueinander so wird das Unkraut behandelt, ansonsten nicht.
• - Für das (gleichzeitige) Training von CNN und FCOS wird nicht ein ausgeschnittenes Unkraut mit Kontur und Bounding Box verwendet (erste Klasse), sondern mittels separatem (Offline) Tool gelabelte Bilder einer Nutzpflanzung mit Unkräutern verwendet. Nur für die zur Laufzeit ausgeführte Perception mit den trainierten CNN und FCOS existiert die dargestellte Kopplung.

One aspect of the invention is in particular:

• - Detection of the leaf contours, the surrounding bounding box (selection frame: image position, height, width) and coverage rate (ratio of contour area (area enclosed by the polar mask) to bounding box area) using PolarMask (first class)
• - Feature vectors/maps are trained in a CNN using the associated weed image.
• - With the feature map and the data from the PolarMask and the selection frame, which also go into the feature map, a classifier (class module), e.g. FCNN (Fully Connected Neural Network) is trained for weed detection.
• - The identified weed (type and probability, second class), associated coverage rate and bounding box (first class) are checked for plausibility with each other, i.e. a specific weed usually has a coverage rate and bounding box (comparisons across applicable value ranges, PolarMask) located in a specific area is implicit in the coverage rate),
• - If the information is plausible, the weeds will be treated, otherwise not.
• - For the (simultaneous) training of CNN and FCOS, not a cut out weed with a contour and bounding box is used (first class), but rather labeled images of a crop with weeds using a separate (offline) tool. The coupling shown only exists for the perception executed at runtime with the trained CNN and FCOS.

The advantage lies in a weed detection process, which further reduces positive and negative false weed detections and increases the probability of weed detection.

According to the invention, the determined plant data from PolarMask such as center of mass (plant center/style, center point) and leaf area (from BoundingBox and coverage rate) are additionally used to carry out spray alignment (activation of certain spray valves/nozzles as well as spray conditioning, e.g. droplet sizes). The concealment and the degree of concealment (coverage rate) of recognized weeds by recognized crop plants can also be recognized in order to align and condition the spray (spraying of the weeds) depending on this.

• PolarMask wird als Instanzsegmentierung für Einzelunkräuter und Bounding Boxes für Einzelunkräuter (im Bild unten rote Rechtecke) als Instanzsegmentierungsmethode verwendet.

A preferred aspect of the invention is:

• PolarMask is used as the instance segmentation for individual weeds and Bounding Boxes for individual weeds (red rectangles in the image below) as the instance segmentation method.

The methods and information are used to derive further parameters according to the invention and use them according to the invention. Since during an application the farmer knows what type of crop he is driving his sprayer into, the classifier can be trained accordingly, i.e. with regard to PolarMask, the number of points / rays of the Polarmask is already assigned to the type of plant to be recognized or the leaf shapes to be expected Training of models / neural networks selected. When used for different crops, the most complex contour can determine the selection.

• Der Kamerafokus ist beim Sprayer senkrecht von oben auf die Pflanzen gerichtet, d.h. die Sprossachsen der Pflanzen (die Wurzel und Blatt verbindet) ist kaum sichtbar. Der mittels PolarMask bestimmte Ursprungspunkt der Polarkoordinate kann als Mittelpunkt des Objekts angesehen werden (Massenzentrum). Dadurch kann die Lage der Sprossachse und in Verlängerung der Wurzelansatz geschätzt werden. Somit können insbesondere bei Pflanzenschutzmitteln, die über die Blätter wirken oder an die Wurzel gelangen sollen, ein Pflanzen bzw. Blattzentriertes gezieltes Sprühen erfolgen bspw. bei großen Nutzpflanzen

The invention is based on the following considerations regarding the center of the plant/style or stalk/shoot axis/root base:

• The sprayer's camera focus is directed vertically from above onto the plants, meaning that the shoot axes of the plants (which connect the root and leaf) are barely visible. The origin point of the polar coordinate determined using PolarMask can be viewed as the center of the object (center of mass). This allows the position of the shoot axis and, as an extension, the root base to be estimated. This means that planting or leaf-centered targeted spraying can be carried out, for example in the case of large crops, particularly in the case of plant protection products that work via the leaves or are intended to reach the roots

• Mittels der über PolarMask bestimmten Kontur kann eine Blätterfläche (Fläche vieler Blätter) eines Unkrauts bzw. die in einem Spraykegel einer Düse sich befindlichen Blätterflächen vieler Unkräuter berechnet werden (Blätterfläche über Bodenfläche der Spraykegelfläche). Somit können insbesondere bei Pflanzenschutzmitteln, die über die Blätter wirken sollen, eine notwendige Pflanzenschutzmenge bestimmt werden (Applikation über Kennlinien / Kennfelder).

The invention is based on the following considerations regarding leaf contours/leaf surfaces:

• Using the contour determined via PolarMask, a leaf area (area of many leaves) of a weed or the leaf areas of many weeds located in a spray cone of a nozzle can be calculated (leaf area over the bottom area of the spray cone area). This means that the necessary amount of plant protection can be determined, particularly for plant protection products that are intended to act on the leaves (application via characteristic curves/characteristic fields).

• Bestimmte Unkräuter haben in bestimmten Nutzpflanzungen grundsätzlich schmalere (alternativ breitere) Blätter als die Nutzpflanzen. Mittels Verhältnisbildung der Fläche der Kontur von Polarmask zur Fläche der umgebenden Bounding Box einer Pflanze kann die Unkauterkennung verbessert werden, da Unkräuter betragsmäßig kleinere (größere) Verhältnisse haben als Nutzpflanzen. Weiterführend kann die Bedeckungsrate und Bounding Box (implizit damit dann auch die Konturfläche) zusätzlich in den Eingang des Klassifikators zur Unkrauterkennung geben werden.

The invention is based on the following considerations regarding bounding box coverage rate/filling rate:

• In certain crops, certain weeds generally have narrower (alternatively wider) leaves than the crop plants. By calculating the ratio of the area of the polar mask contour to the area of the surrounding bounding box of a plant, weed detection can be improved, since weeds have smaller (larger) ratios than crops. Furthermore, the coverage rate and bounding box (implicitly also the contour area) can also be entered into the input of the classifier for weed detection.

As an example, consider a covering of plants (leaves), whereby, despite the covering, the bounding box and PolarMask/contour of the hidden plant are recognized or predicted quite well with the PolarMask method. A cover of plants can be seen, although despite the cover, the bounding box and PolarMask with the contour of the covered plant are recognized quite well or predicted from learned characteristics. If, for example, a weed hidden by a crop is to be sprayed, a drop size of the spray can be selected depending on the degree of concealment (how much weed leaf area out of the total weed leaf area) or the overall concealment of weeds by crops in a spray cone of a nozzle (Pressure and/or nozzle selection). In the case of greater coverage, the spray can be changed to produce finer drops (e.g. by higher spray pressure or switching to a different nozzle head). the spray aerosol gets under the canopy of the crops to the weeds. Especially when crops and weeds are from the same family, the crops also suffer from accidental spraying of weeds, which is reduced by the process

The number of stages of the neural network is not the subject of the present invention.

• 1 eine Einrichtung zum Besprühen von Unkraut mit einer Detektoreinheit gemäß der Erfindung in einem Blockdiagramm.

Further features, effects and advantages of the invention result from the following description of a preferred exemplary embodiment of the invention and the attached figures. Show, each in a schematic principle sketch:

• 1 a device for spraying weeds with a detector unit according to the invention in a block diagram.

1 shows a device 2 for spraying weeds 4. The device 2 has a camera 6, which in the example is arranged in the direction of travel at the front of a field sprayer, not shown. The camera 6 is aimed at a surface 10, here the ground, in order to generate image data 8 from the surface 10 that potentially includes or contains the weeds 4, here a plant field for crops 12. In an exemplary image recorded by the camera 6 in the form of the image data 8, a number of useful plants 12 and several weeds 4 are therefore shown.

The device 2 also has a detector unit 14 and a spray module 16. The spray module 16 here is a spray arm with spray nozzles for issuing a spray agent 18 (symbolized by an arrow in the figure) in order to spray the weeds 4 with the spray agent 18. The spray module 16 is spatially correlated with the camera 6, that is, it has a known relative position to the camera 6. In the present case, this is accomplished in that the spray module 16 is also attached to the field sprayer against the direction of travel. By driving over the area 10 with the field sprayer, camera 6 and spray module 16 are moved synchronously. The spray module 16 is set up to spray the weeds 4 with the spray agent 18 based on the sizes provided or output by the detector unit 14 at an output 20.

The detector unit 14 is used to detect and localize the weeds 4 in the image data 8. The detector unit 14 has an input 22 for the image data 8. The detector unit 14 has a frame detector 24. This is set up to detect a respective weed 4, i.e. individual weed plants, in the image data 8 and to provide a selection frame 26 at the output 20 for each identified weed 4. Each of the selection frames 26 accurately frames the respective weed 4 or its image (within the scope of detection accuracy). The selection frame here is a rectangle that runs parallel to the edges of the image data 8. Optionally, provision is made at output 20 (this also applies to all subsequent sizes provided accordingly).

The detector unit 14 also has a polar detector 28. This is set up to also detect the weeds 4 in the image data 8 and to provide a polar mask 30 for each of the detected weeds 4. The polar mask 30 follows a contour of the weed 4 and consists of straight lines strung together. For one of the detected weeds 4 in 1 This is shown enlarged as an example as an image section of the image data 8. It can be seen how both a selection frame 26 and a polar mask 30 - in this respect a precise fit - are described to the recognized weed 4.

The detector unit 14 also has a classifier 32. This assigns a first class 34 to the detected weed 4 in a conventional manner and provides this. In the example, the first class 34 is a weed name (weed 4) and a number between zero and one that indicates the probability (0=0%, 1=100%) that the detected element in the selection frame 26 is actually a weed Weed 4 is about. In the example, the value of class 34 for the weed shown enlarged is 4 “0.98”.

The detector unit 14 also has a convolutional neural network (CNN) 40. This is based on a part of the image data 8 for each of the detected weeds 4, which is limited by the selection frame 26 and/or the polar mask 30 and thus represents selection image data 42. The CNN 40 is set up to use the selection image data 42 to generate part of a feature vector 44 for the viewed image content of the selection image data 42 (potential weed 4) and to make it available for further use.

The detector unit 14 also has a class module 46, here a fully connected neural network (FCNN). This in turn is set up to use the feature vector 44 to determine a second class 48 that classifies the (potential) weed 4 and to make it available for further use.

In addition to the outputs of the CNN 40, the feature vector 44 in the example also has the selection frame 26 and the polar mask 30 (its coordinate data / size / course, etc.), so that these data are also evaluated in the class module 46 / FCNN.

The detector unit 14 also has a plausibility module 50. This is set up to check the plausibility of the first class 34 and the second class 48 with respect to one another. In the case of such a positively determined plausibility, the plausibility module 50 determines a third or final class 52 and provides this (at the output 20).

The detector unit 14 also has a center point module 54. This is set up to determine a center point 56 (coordinates in the image or the image data 8, symbolized as a point in the figure) of the polar mask 30 and to make it available for further use.

The detector unit 14 also has a leaf area module 58. This is set up to use the polar mask 30 to determine the leaf area 60 (area, indicated by hatching in the figure) of the weed 4 as an area enclosed by the polar mask 30 and to make it available for further use.

The detector unit 14 also has a coverage rate module 62. This is set up to determine the ratio of the areas of the selection frame 26 and the polar mask 30 (areas indicated by hatching) as a coverage rate 64 of the weed 4 and to make it available for further use.

In particular, the coverage rate 64 is provided to the class module 46 / FCNN with the feature map.

Finally, the detector unit 14 also has a coverage rate module 66. This is set up to detect a crop in the image data 8, especially within the selection frame 26 / the polar mask 30, and to determine and provide a coverage rate of the weed 4 by the crop 12. The coverage rate 68 thus indicates how much of the area of the weed 4 in the selection frame 26 / the polar mask 30 is covered by the crop 12.

The sizes optionally provided at the output 20 are at least one of: selection frame 26, polar mask 30, first class 34, second class 48, third class 52, center point 56, leaf area 60, coverage rate 64, coverage rate 68. Depending on the capabilities of the spray module 16, i.e. depending on which of the variables it can and should process, the variables are provided at the output 20 or possibly not. Due to the determination and processing of the corresponding variables in the spray module 16 or the device 2, it is therefore possible for the spray module 16 to spray the weeds 4 precisely with the precisely metered amount of spray agent 18 at the correct target location, and for the crops 12 to benefit from a corresponding spraying if possible left out.

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102017124934 A1 [0001]

Claims (14)

Detector unit (14) for detecting weeds (4) in image data (8), with - an input (22) for the image data (8), - a frame detector (24), which is set up to recognize a respective weed (4) in the image data (8) and to provide a respective selection frame (26) which frames the respective weed (4), and - a polar detector (28), which is set up to recognize a respective weed (4) in the image data (8) and to provide a respective polar mask (30) which follows a contour of the respective weed (4). Detector unit (14). Claim 1 , characterized in that the detector unit (14) has a classifier (32) which is set up to provide a first class (34) classifying the weeds (4) based on the selection frame (26) and the polar mask (30). Detector unit (14) according to one of the preceding claims, characterized in that the detector unit (14) - has a convolutional neural network (40) which is set up to, based on the selection frame (26) and / or the polar mask (30) to generate and provide certain selection image data (42) at least part of a feature vector (44) for the weed (4), - has a class module (46) which is set up to classify the weed (4) based on the feature vector (44). to create and provide second class (48). Detector unit (14). Claim 3 , characterized in that the feature vector (44) additionally includes or includes the selection frame (26) and / or the polar mask (30). Detector unit (14) according to one of the preceding claims, characterized in that the detector unit (14) has a plausibility module (50) which is set up to determine and provide a third class (52) classifying the weeds (4) if the first (34) and second class (48) are plausible to each other. Detector unit (14) according to one of the preceding claims, characterized in that the detector unit (14) has a center point module (54) which is designed to determine and provide a center point (56) of the polar mask (30). Detector unit (14) according to one of the preceding claims, characterized in that the detector unit (14) has a leaf surface module (58) which is set up to use the polar mask (30) to identify a leaf surface (60) of the weed (4) as being from the Polar mask (30) to determine and provide the delimited area Detector unit (14) according to one of the preceding claims, characterized in that the detector unit (14) has a coverage rate module (62) which is set up to determine a coverage rate (64) based on the ratio of the areas of the selection frame (26) and the polar mask (30). ) of the weeds (4) to be determined and provided. Detector unit (14) according to one of the preceding claims, characterized in that the detector unit (14) has a coverage rate module (66) which is designed to detect a crop (12) at least within the polar mask (30) and/or the selection frame (26). ) and to determine and provide a coverage rate (68) of the weeds (4) by the crop (12). Detector unit (14) according to one of the preceding claims, characterized in that the detector unit (14) has an output (20) for providing at least one variable which is selected from the group consisting of: selection frame 26, polar mask 30, first class 34, second Class 48, third class 52, center 56, leaf area 60, coverage rate 64, coverage rate 68. Device (2) for spraying weeds (4), - with a camera (6) which is set up to generate image data (8) from an area (10) that potentially includes or contains weeds (4), - with the detector unit (14) according to one of the preceding claims, - with a spray module (16), which is spatially correlated with the camera (6), and which is set up to spray the weeds (4) with a spray agent (18) based on the sizes provided by the detector unit (14). Method for detecting weeds (4) in image data (8), in particular by means of a detector unit (14) according to one of Claims 1 until 10 or an institution Claim 11 , characterized by the steps: - recognizing a respective weed (4) in the image data (8) and providing a respective selection frame (26), which frames the respective weed (4), by means of a frame detector (24) and - detecting a respective weeds (4) in the image data (8) and providing a respective polar mask (30) that corresponds to a contour of the respective one Weeds (4) follows, using a polar detector (28). Computer program that is set up to follow the steps of a procedure Claim 12 to perform and/or control when the computer program is executed on a computer. Machine-readable storage medium on which the computer program can be written Claim 13 is stored.

Images