DE102018120756A1 - Mobile analysis and processing device, method and carrier system - Google Patents

Abstract

The invention relates to a mobile analysis and processing device (14) for agriculture for cultivating the soil and / or for manipulating flora and fauna, comprising a visual detection unit (62) which comprises: a camera (64) with which images are captured a segmentation and data reduction device (66), with which, inter alia, an image recorded by the camera (64) is generated from a plurality of pixels in an RGB (Red, Green, Blue) color model and from pixel fields from the pixels of the Intermediate image is generated. In addition, a classifier (68) is provided which, on the basis of the intermediate image generated by the segmentation and data reduction device (66), takes over the classification of several pixel fields consisting of pixels, only pixel fields which have at least one pixel that corresponds to the binary value “being classified” being classified. 1 ”is assigned.

Description

The invention relates to a mobile analysis and processing device for agriculture for cultivating the soil and / or for manipulating flora and fauna according to claim 1, a method according to claim 8 and a carrier system according to claim 17.

Weed control in agriculture is a very labor-intensive task, especially in organic farming, which prohibits or restricts the use of chemicals. Depending on the cultivated crop, weed control is necessary in the immediate vicinity of the crop. This regulation generally takes place in the early stage of growth. Both the useful plant on one side and the weeds or weeds on the other are still very small and close together. To avoid damaging the crop, selective methods should be used. This takes place in organic cultivation, for example for carrots, through labor-intensive, body-damaging manual work with so-called "weeding planes". Seasonal workers lie on their bumps on a platform and remove the weeds.

For special crops with larger plant spacings, such as sugar beets or lettuce, tractor attachments are known which are able to recognize individual crops and to control appropriate tools so that the area of the crop remains unprocessed. Selectivity is not required for this task. This means that these systems do not check the areas to be reworked, but rather control the tool "on blind" based on the known crop position. In general, the distance to the crop defines the accuracy requirements.

From the DE 40 39 797 A1 A device for weed control is known, an actuator for destroying the weeds running continuously and only being interrupted briefly when a sensor detects a crop.

From the DE 10 2015 209 879 A1 a device for damaging weeds is known, which has a processing tool. This editing tool is used to damage the weeds. In addition, a classification unit is provided, which either has the position data of the weed or recognizes the weed and determines the position data. A localization unit determines a relative position between the processing tool and the weed. A manipulator unit in the form of a carriage positions the machining tool accordingly in accordance with the determined relative positions.

A corresponding device with a pressure delivery unit and a liquid delivery unit is from the DE 10 2015 209 891 A1 known. In this embodiment, the weeds are destroyed by spraying liquid under pressure.

Agricultural robots and harvesting machines that provide automated, technical equipment to support agriculture are currently treading new ground. In many cases, technical principles and knowledge from space travel, remote sensing and robotics can be adopted for agricultural issues, but these must be adapted to agricultural tasks and require new devices and processes.

A problem with the automated treatment of the fields is also the determination of the data, where the weed is and what, on the other hand, is to be treated as a useful plant.

From the DE 10 2005 050 302 A1 is a method for contactless determination of the current nutritional status of a plant stand and for processing this information taking into account further parameters such as fruit type and / or variety and / or development stage and / or yield target in a fertilization recommendation, in which at least one digital picture of part of the plant stand recorded in at least two spectral channels by means of an image recording system, the current nutritional status is determined from the image by means of an image analysis and the fertilizer recommendation is derived from the latter. This known prior art has the disadvantage that the data processing takes a relatively long time and is not suitable for an in situ system.

In the DE 10 2006 009 753 B3 Furthermore, a method for the contactless determination of the biomass and morphological parameters of plant stands is described, in which the plants of the stand acted upon during the crossing with a sound field emitted by an ultrasound source attached to a mobile carrier, the sound echoes reflected by the plants and the soil by a Receiver attached to the carrier is detected and, after conversion into digital signals, is passed on to an evaluation and signal processing unit which currently evaluates the signals, stores them on a data carrier and displays them on a monitor, optionally with the signals for control commands in an electronically controlled application unit Application of product funds are processed. Although this prior art makes it possible to directly determine morphological parameters such as the number of leaf days, the leaf position and the vertical distribution of the biomass, however, it is not possible to determine the weed.

The invention is based on the object of a mobile analysis and processing device for agriculture for cultivating the soil and / or for manipulating flora and fauna, and a method which is a real-time controlled, qualified removal of the detected flora and / or fauna and also enables a parallel analysis of flora and fauna. In this context, real-time means that in-situ analysis and processing is possible.

This object is achieved for the mobile analysis and processing device by the features of claim 1, for the method by the features of claim 8 and for the carrier system by the features of claim 17.

The subclaims form advantageous developments of the invention.

The invention is based on the finding that by designing a mobile analysis and processing device with a visual detection unit, which enables the data to be evaluated to be reduced to the necessary areas, the analysis and evaluation time can be shortened considerably, and analysis and processing are thereby possible , for example of agricultural land, can be done in just one operation.

The invention therefore relates to a mobile analysis and processing device for agriculture for cultivating the soil and / or for manipulating flora and fauna. This mobile device is provided with a visual detection unit which comprises a camera with which images are captured. In addition, a segmentation and data reduction device is provided with which an image recorded by the camera is generated from a plurality of pixels in an RGB (Red, Green, Blue) color model. Then each pixel is based on the RGB color model HSV (hue, saturation, value) color model transferred. Each pixel based on the HSV color model is then rated for color saturation based on a threshold, and if the value of color saturation exceeds a threshold, the pixel is assigned the binary value " 1 "Is assigned, and if the value of the color saturation falls below a threshold value, the pixel corresponds to the binary value" O "Is assigned. In parallel, each pixel is evaluated based on the HSV color model with regard to the color angle (hue) using a predetermined range, the pixel having the binary value “if the color angle is in the predetermined range“ 1 "And if the color angle is outside the range, the pixel is assigned the binary value" O "Is assigned. From the binary information of the color angle and the color saturation, a first intermediate image is created which contains considerably less data than the image generated by the camera. Then pixel fields are generated from the intermediate image. Furthermore, a classifier is provided which, on the basis of the intermediate image generated by the segmentation and data reduction device, takes over the classification of several pixel fields consisting of pixels, wherein only pixel fields are classified which have at least one pixel that corresponds to the binary value “ 1 " assigned. This results in a not inconsiderable data reduction and thus acceleration of the analysis on site. There are also other optimization options, as can be seen from the following.

It has proven to be advantageous if the segmentation and data reduction device forms a pixel field by a square number of pixels, in particular in this case has 100, 81, 77, 36, 25, 16, 9 or 4 pixels. The pixel field is preferably formed square.

According to one embodiment of the invention, the mobile device comprises at least one tool unit with at least one motor-driven tool, an actuator for moving at least the tool of the tool unit, a motor for driving the tool unit and / or the actuator, a database, a first communication unit with interface and a first computer for controlling the motor, the visual detection unit, the tool unit and / or the actuator on the basis of generated control commands. The data recorded via the visual detection unit is continuously compared with the data stored in the database in order to generate corresponding control signals for the motor, the visual detection unit, the tool unit, the actuator and / or an assigned carrier. This creates mobility and flexibility, according to which the device forms a unit with which all data are processed in real time, control signals are generated for the tool unit and the tool unit can immediately work in accordance with the control signals. This results in possibilities of combining, for example, with different carriers, which move the device over the field if necessary.

In this context, real-time means the possibility of being able to carry out analysis and processing operations in situ in one work step.

In order to be able to connect the device to a carrier that moves the device as required, appropriate means are provided for connecting to the carrier.

The tool unit preferably has at least one feed unit and one rotation unit which interacts with the motor. The range of use of the tool is thereby expanded in a simple manner without the device having to be moved.

The rotation unit is preferably provided at one distal end with at least one tool, in particular with a milling cutter or with a knife unit. By rotating the knife unit, for example, small insects or weeds can be selectively destroyed.

• - zeitlich kontinuierliche Aufnahme von datentechnisch definierter Voxel und/oder Pixel und/oder Bildern durch die visuelle Detektionseinheit;
• - Übermittlung der Aufnahmedaten an die Datenbank;
• - Speicherung der Aufnahmedaten in der Datenbank;
• - qualitativer Datenabgleich der Aufnahmedaten mit den in der Datenbank hinterlegten Daten, vorzugsweise mit Durchführung einer Segmentierung, einer Datenreduktion, einer und/oder Verifikation der Daten durch den Rechner;
• - Bewertung der abgeglichenen Daten mit bestehenden definierten Daten und Datensätzen in der Datenbank durch einen Klassifikator in Verbindung mit dem Rechner;
• - Verarbeitung und Umsetzung der Bewertung durch den Rechner in regelungs- und/oder steuerungstechnische Daten für den Motor, den Aktor, die Werkzeugeinheit und/oder einen zugeordneten Träger.

The above-mentioned object is also achieved by a method for operating the analysis and processing device according to the type just mentioned, the method comprising the following steps:

• - Continuous recording of voxels and / or pixels and / or images defined in terms of data technology by the visual detection unit;
• - transmission of the recording data to the database;
• - Storage of the recording data in the database;
• - Qualitative data comparison of the recording data with the data stored in the database, preferably by performing a segmentation, data reduction, and / or verification of the data by the computer;
• - Evaluation of the matched data with existing defined data and data records in the database using a classifier in connection with the computer;
• - Processing and implementation of the evaluation by the computer in regulation and / or control engineering data for the motor, the actuator, the tool unit and / or an assigned carrier.

After the regulatory and / or control-related data are available, the motor, the actuator, the tool unit and / or the associated carrier are preferably started up for working the soil and for manipulating flora and fauna.

In order to accelerate the analysis of the recording data, the recording data are transmitted to the segmentation and data reduction device which generates intermediate images during the data comparison. The intermediate images have significantly reduced data.

In particular after the intermediate images have been generated, they are transmitted to the classifier, which evaluates the generated intermediate images on the basis of existing defined images and data records in the database in cooperation with the computer.

• - Jedes übermittelte Bild aus mehreren Pixeln wird in das RGB-(Red, Green, Blue)-Farbmodell überführt;
• - jedes Pixel des übermittelten Bildes basierend auf dem RGB-Farbmodell wird in ein HSV (hue, saturation, value)-Farbmodell transferiert;
• - jedes Pixel basierend auf dem HSV-Farbmodell wird im Hinblick auf die Farbsättigung (saturation) anhand eines Schwellenwerteses bewertet, wobei, wenn der Wert der Farbsättigung einen Schwellenwert übersteigt, das Pixel dem binären Wert „1" zugeordnet wird, und, wenn der Wert der Farbsättigung einen Schwellenwert unterschreitet, das Pixel dem binären Wert „O“ zugeordnet wird;
• - vorzugsweise parallel mit dem eben genannten Schritt wird jedes Pixel basierend auf dem HSV-Farbmodell im Hinblick auf den Farbwinkel (hue) anhand eines vorbestimmten Bereiches bewertet, wobei, wenn der Farbwinkel in dem vorbestimmten Bereich liegt, das Pixel dem binären Wert „1“ zugeordnet wird, und, wenn der Farbwinkel außerhalb des Bereichs liegt, das Pixel dem binären Wert „O“ zugeordnet wird;
• - aus den binären Informationen des Farbwinkels und der Farbsättigung entsteht das erste Zwischenbild, welches erheblich weniger Daten als das von der Kamera erzeugte Bild enthält.

According to one embodiment of the invention, the following steps are carried out in succession in the segmentation and data reduction device for generating the intermediate images:

• - Each transmitted image from several pixels is converted into the RGB (Red, Green, Blue) color model;
• - Each pixel of the transmitted image based on the RGB color model is transferred to an HSV (hue, saturation, value) color model;
• - Each pixel based on the HSV color model is evaluated with regard to the color saturation using a threshold value, wherein if the value of the color saturation exceeds a threshold value, the pixel corresponds to the binary value “ 1 "is assigned, and if the value of the color saturation falls below a threshold value, the pixel corresponds to the binary value" O “Is assigned;
• - preferably in parallel with the step just mentioned, each pixel is evaluated based on the HSV color model with regard to the color angle (hue) using a predetermined range, the pixel having the binary value “if the color angle is in the predetermined range“ 1 "And if the color angle is outside the range, the pixel is assigned the binary value" O “Is assigned;
• - From the binary information of the color angle and the color saturation, the first intermediate image is created, which contains considerably less data than the image generated by the camera.

Preferably, pixel fields are generated from the pixels of the intermediate image in the segmentation and data reduction device in order to further accelerate and optimize the analysis.

The segmentation and data reduction device provides the pixels with position coordinates so that the results from the images determined by the evaluation can subsequently be converted into regulation and / or control engineering data for the actor and / or the wearer.

According to one embodiment of the invention, the classifier carries out the classification using an artificial neural network. The artificial neural network is preferably connected by a convolutional neural network ( CNN ) educated.

The invention also relates to a carrier system with a carrier and a mobile device for working the soil and / or for manipulating the flora and fauna in the manner mentioned above, in particular for carrying out the method just mentioned. The carrier is provided with a drive for moving the carrier system, with a control device, with an energy source which interacts with the drive and provides the necessary voltage for operation, with a receptacle for receiving and connecting to the mobile device, with a communication unit , with a first communication interface interacting with the communication unit for data exchange with the mobile device, which has an assigned interface, and with a second communication interface interacting with the communication unit. At least control data are received via the second communication interface and passed on to the control device for predetermined movement of the carrier by means of the communication unit. This creates a flexibility of the carrier system, which enables a wide range of applications, but also meets different needs in agriculture, in that mobile devices can be connected to a uniform carrier as required.

The carrier can preferably be designed as an unmanned aerial vehicle (drone), unmanned land vehicle, unmanned watercraft, or unmanned underwater vehicle, for example for aquaculture. Depending on the area of application, one or the other carrier is used.

According to one embodiment of the invention, the second communication interface is an antenna which is designed for data exchange with a central, preferably fixed, computer separate from the carrier. This enables devices, for example for telemetry and or the evaluation of the data determined, to be processed and evaluated independently of the carrier. The carrier can thus be made slightly more weighty.

However, in order to enable data exchange with the mobile device, the second communication interface is designed for this.

For example, the carrier requires a voltage source to operate the drive. In order to provide such a voltage source only once and thus save weight in the carrier system, the carrier and the mobile device each have a voltage connection via which the mobile device can also be supplied with voltage. Thus, only one energy source is provided for the carrier and the mobile device.

In order to ensure the interchangeability of the carrier with different mobile devices, the carrier and the mobile device have means for connecting to one another as required.

According to one embodiment of the invention, the carrier has a GPS unit for continuously recording the position coordinates of the carrier. In this way, the position coordinates can be assigned to the data determined by the sensor. This simplifies evaluation.

Further advantages, features and possible uses of the present invention result from the following description in connection with the exemplary embodiments shown in the drawings.

• 1 eine schematische Darstellung eines Tragsystems mit räumlich getrennten Gehäusen einer mobilen Vorrichtung gemäß einer ersten Ausführungsform der Erfindung;
• 2 eine schematische Darstellung eines Tragsystems mit räumlich getrennten Gehäusen einer mobilen Vorrichtung, die über eine Steckverbindung miteinander verbunden sind, gemäß einer zweiten Ausführungsform der Erfindung;
• 3 eine Seitenansicht auf das Tragsystem gemäß der ersten Ausführungsform der Erfindung mit Anbindung der mobilen Vorrichtung an eine Flugdrohne;
• 4 ein Flussdiagramm, welches die Schritte eines Verfahrens mit dem Tragsystem veranschaulicht;
• 5 ein Flussdiagramm, welches die Schritte des Verfahrens zur Ermittlung der notwendigen Maßnahmen darstellt;
• 6 ein durch die visuelle Detektionseinheit aufgenommenes Bild;
• 7 ein schematischer Aufbau eines Convolution Neuronalen Netzwerkes ausgehend von dem Bild von 6;
• 8 ein Flussdiagramm, welches ein Verfahren der Segmentierungs- und Datenreduktionseinheit zeigt;
• 9 ein aus der Segmentierungs- und Datenreduktionseinheit erstelltes Zwischenbild;
• 10 einen Teilausschnitt des Zwischenbildes mit drei unterschiedlichen Fällen für den Klassifikator;
• 11 zwei prinzipielle Darstellungen von weiteren Pixelfeldern für die Bewertung durch den Klassifikator;
• 12 zwei prinzipielle Darstellungen von weiteren Pixelfeldern für die Bewertung durch den Klassifikator;
• 13 ein durch den Klassifikator erstelltes und bewertetes Bild, und
• 14 die prinzipielle Darstellungen der Arbeitsweise des Klassifikators

In the description, in the claims and in the drawing, the terms and associated reference symbols used in the list of reference symbols given below are used. In the drawing means:

• 1 a schematic representation of a support system with spatially separate housings of a mobile device according to a first embodiment of the invention;
• 2 a schematic representation of a support system with spatially separate housings of a mobile device, which are connected to one another via a plug connection, according to a second embodiment of the invention;
• 3 a side view of the support system according to the first embodiment of the invention with connection of the mobile device to a drone;
• 4 a flowchart illustrating the steps of a method with the support system;
• 5 a flow diagram illustrating the steps of the method for determining the necessary measures;
• 6 an image captured by the visual detection unit;
• 7 a schematic structure of a convolutional neural network based on the image of 6 ;
• 8th a flowchart showing a method of the segmentation and data reduction unit;
• 9 an intermediate image created from the segmentation and data reduction unit;
• 10 a partial section of the intermediate image with three different cases for the classifier;
• 11 two basic representations of further pixel fields for the evaluation by the classifier;
• 12 two basic representations of further pixel fields for the evaluation by the classifier;
• 13 an image created and rated by the classifier, and
• 14 the basic representations of how the classifier works

In 1 is a schematic of a carrier system 10 shown, which consists of a carrier in the form of a drone 12 and a mobile device 14 for cultivating the soil and manipulating flora and fauna in agriculture. The drone 12 includes a drive 16 , the four electric motors 18 and about this powered propeller 20 includes, see 3 , In addition, the flight drone 12 With 4 feet 22 below the electric motors 18 Mistake.

According to the first embodiment of the invention, the flight drone comprises 12 an energy source in the form of batteries 24 which is the energy supply for the drive 16 as well as for the other units of the drone 12 as well as the mobile device 14 supplies. There is a voltage interface for this 26a on the drone 12 and one to this voltage interface 26a corresponding voltage interface 26b on the mobile device 14 provided which via a detachable connector 28 are interconnected. There is also a communication unit 30 with an antenna 32 and a GPS unit 34 provided which continuously the location of the drone 12 determined this location data of the flight drone 12 to the mobile device 14 for mapping to those from the mobile device 14 recorded data and transmitted to a remote central processing unit, which is not shown here. With the help of GPS unit 34 , the communication unit 30 and the mobile device 14 can be done telemetry. There is also a control device 12b provided which the drive 16 controls.

The communication unit 30 the drone 12 includes next to the antenna 32 another interface 36a which an assigned interface 36b the mobile device 14 is assigned and via a detachable connector 38 are connected to each other for data exchange.

The mobile device 14 consists of two units 14a . 14b , namely a first unit 14a with a first housing 40 and a second unit 14b with a second housing 42 , The first housing 40 and the second housing 42 are detachable via a plug connection 44 with each other to a the mobile device 14 forming unit connected. There is a set of different first units 14a on one side and a set of different second units 14b on the other hand, which can be individually configured and adapted to needs by simply connecting them together.

In the first case 40 is a first computer 46 , an actuator in the form of a movable, motor-driven arm 48 , one with the arm 48 interacting motor 50 , one on the arm 48 arranged tool unit 52 which is a feed unit 54 and a rotating unit 56 having. At the distal end of the rotation unit 56 is a milling machine 58 provided as a tool. The motor 50 drives both the arm 48 as well as the feed unit 54 , the rotation unit 56 and with it the milling machine 58 on. The arm 48 can be formed in several parts and have different joints, which are not shown in detail, since such motor-operated kinematics are known. Over the arm 48 becomes the tool unit 52 relative to the drone 12 moved to their area of application so that the tool unit 52 with the feed unit 54 and the rotation unit 56 the milling machine 58 can be used for cultivating the plants, for example removing weeds, and / or cultivating the soil.

Furthermore is in the first unit 14a a communication unit 60 and a visual detection unit 62 intended. The visual detection unit 62 includes a camera 64 , with which images are captured, a segmentation and data reduction device 66 , a classifier 68 , which is based on one of the segmentation and data reduction device 66 generated intermediate image or intermediate data takes over the classification of several pixel fields consisting of pixels, as will be described in more detail below. The visual detection unit 62 is with the communication unit 60 connected.

The first unit 14a has an interface 70a on what an interface 70b the second unit 14b assigned. Via a communication link 72 is the communication unit 60 over the interface 70a with the interface 70b and above with a communication unit 74 in the second unit 14b connected. The communication unit 74 the second unit 14b is over the interface 36b via the plug connection 38 with the interface 36a and the communication unit 30 the drone 12 connected.

In the second unit 14b are also a second computer 76 and a database 78 intended.

In the 2 is another embodiment of the carrier system 10 shown with the drone 12 is identical to the first embodiment. Only the mobile device 14 differs by a plug connection 80 between the first unit 14a and the second unit 14b , which is also the communication unit 60 the first unit 14a with the communication unit 74 releasably connects. By simply plugging together different first units 14a with different second units 14b combined and into a mobile unit 14 be put together.

In 3 is the flight drone 12 shown in side view, whereby only two of four electric motors 18 with assigned propellers 20 are visible. Below the electric motors 18 are the feet 22 arranged. Between your feet 22 are two gripper arms 82a . 82b provided which the mobile device 14 If necessary, grab and lift and put down and put down again. The mobile device 14 consists of the two units 14a and 14b which via the connector 80 are releasably connected to each other. In the first session 14a is the camera 64 as part of the visual detection unit 62 as well as the milling machine 58 at the distal end of the rotation unit 56 recognizable.

The mobile device 14 can also be used with several different tool units 52 be equipped with a common arm 48 and, for example, a tool turret is provided which has the required tool unit 52 in the activation position. However, it is also conceivable that the different tool units each have their own actuator.

In 4 are shown in a flowchart the steps which are carried out successively in order to use the carrier system 10 cultivating the soil and manipulating flora and fauna in agriculture.

In a first step 84 are first with the carrier system 10 the necessary measures are determined on the assigned agricultural area. For example, the carrier system for this 10 brought to an agricultural area to be worked on, for example to an agricultural field, or flown there directly from a central location. The drone 12 with the mobile device 14 starts and flies off the agricultural field. The carrier system is received via a fixed central processing unit 10 the necessary data about the agricultural field to be assessed. The central processing unit can also be a smartphone. Via the visual detection unit 62 with the camera 64 the mobile device 14 the agricultural field is captured in pictures. The images are evaluated and compared in the database 78 the necessary measures for this agricultural field are finally determined.

In a next step 86 is based on the determined measures for the agricultural field or for parts of the agricultural field from a set of first units 14a and a set of different second units 14b the mobile unit suitable for the necessary measure 14 put together and the two units 14a . 14b connected with each other.

In a next step 88 becomes the mobile device 14 with the drone 12 over the gripper arm 82a and the gripper arm 82b gripped laterally and through this towards the drone 12 up into a shot 12a the drone 12 method. The voltage interfaces 26a . 26b via the plug connection 28 and the interfaces 36a . 36b via the plug connection 38 connected with each other. This will make the mobile device 14 with voltage from the battery 24 the drone 12 supplied and a data exchange via the antenna 32 the communication unit 30 the drone 12 with the communication units 60 and 74 the mobile device 14 on the one hand and a central processing unit on the other. As stated above, the central processing unit, which is from the carrier system 10 is independent, also be a smartphone.

In a next step 90 the measures determined with the carrier system 10 performed on the agricultural field. For example, the drone flies 12 to the area of the agricultural field to be worked. The arm 48 with the tool unit 52 drives to the weed to be removed. The feed unit 54 the milling machine moves 58 so to the weed that this with activation of the rotation unit 56 is milled away.

In a fifth step 92 then flies the flight drone 12 back, and swaps the mobile device 14 against another mobile device 14 which is optimized for another measure, for example with a spreading device for pesticides or fertilizers.

Alternatively, the steps 86 and 88 also omitted when the flight drone 12 is already ready for the measure to be carried out.

Based on 5 is now the determination of the necessary measures by the carrier system 10 , especially through the mobile device 14 spelled out in detail.

In a first step 94 the visual detection unit continuously records data of technically defined voxels and / or pixels and / or images 62 the mobile device 14 , The voxels, pixels and images form recording data, which are continuously sent to the database 78 transmitted - second step 96 ,

In a third step 98 the recording data is saved.

In a fourth step 100 there is a qualitative data comparison of the recording data with that in the database 78 stored data. The segmentation and data reduction device carries out a segmentation and data reduction of the recording data 66 , In particular, verification of the recording data by the second computer can also be carried out 76 respectively.

In a fifth step 102 the evaluation is carried out by the classifier 68 in connection with the second computer 76 by supporting artificial intelligence, as will be explained in detail below.

In a sixth step 104 Finally, the processing and implementation of the evaluation is carried out by the first computer 46 in open-loop and open-loop control data for the engine 50 , the arm 48 , the tool unit 52 and the flight drone 12 ,

In a seventh step 106 the engine is finally started 50 , of the arm 48 and the tool unit 52 for working the soil or manipulating flora and fauna.

Insofar as this application speaks of artificial intelligence, it involves, among other things, the use of a classic convolutional neural network - CNN - consisting of one or more convolutional layers, followed by a pooling layer. In principle, this sequence of convolutional layer and pooling layer can be repeated as often as required. As a rule, the input is a two- or three-dimensional matrix, e.g. B. the pixels of a grayscale or color image. Accordingly, the neurons are arranged in the convolutional layer.

The activity of each neuron is calculated using a discrete convolutional convolution. Intuitively, a comparatively small convolution matrix (filter kernel) is gradually moved over the input. The input of a neuron in the convolutional layer is calculated as the inner product of the filter kernel with the current image section. Accordingly, neighboring neurons in the convolutional layer react to overlapping areas.

A neuron in this layer only responds to stimuli in a local environment of the previous layer. This follows the biological model of the receptive field. In addition, the weights are identical for all neurons of a convolutional layer (shared weights). The result of this is that, for example, each neuron in the first convolutional layer encodes the intensity at which an edge is present in a specific local area of the input. Edge detection as the first step in image recognition has high biological plausibility. It immediately follows from the shared weights that translation invariance is an inherent property of CNNs.

The input of each neuron, determined by means of discrete convolution, is now converted into the output by an activation function, usually CNLs Rectified Linear Unit, or ReLu (f (x) = max (0, x)), which models the relative fire frequency of a real neuron Since back propagation requires the computation of the gradients, a differentiable approximation of ReLu is used in practice: f (x) = In (1 + e ^x ). Analogously to the visual cortex, the size of the receptive fields increases in lower convolutional layers as well as the complexity of the identified features.

In the next step, pooling, superfluous information is discarded. For example, for object recognition in images, the exact position of an edge in the image is of negligible interest - the approximate localization of a feature is sufficient. There are different types of pooling. Max pooling is by far the most widespread, whereby from every 2 × 2 square of convolutional layer neurons only the activity of the most active (hence “ Max “) Neurons are retained for the further calculation steps; the activity of the other neurons is discarded. Despite the data reduction (in the example 75%), the performance of the network is not usually reduced by pooling.

The use of the convolutional neural network and the segmentation and data reduction device 66 is based on the pictures below 6 to 14 explained in more detail.

For classification by the classifier 68 Different approaches exist for all objects in a picture. Many approaches begin with finding the individual objects in the image and then classifying them. However, this is not always the case possible. As an example, here is the classification of plants 108 of a field. An example picture 108 shows 6 ,

In 6 are different plants 108 mapped and are intended to be represented by the classifier 68 all are classified and in real time. In this case, the camera act of 10 Frames per second. Because, as in this example, it is not easy to distinguish where exactly a plant is 110 ends, a different approach must be used, since the computing time is not sufficient, only the plants 110 to distinguish themselves and then classify them.

The picture 108 of 6 consists of pixels and each pixel can logically only contain exactly one class. Therefore, the whole picture would be a trivial approach 108 to be classified pixel by pixel. This means that every pixel is assigned to a class one after the other.

However, since a single pixel does not contain the necessary information to make a statement about the class membership, a surrounding area must be used for the classification. This area can then be connected to a convolutional neural network ( CNN ) as described above. The network can have a consequence like 7 exhibit.

The entrance picture 110 here is the picture of 6 , On this entrance picture 110 the elements of the CNN applied. In this example it would be convolution 112 with the features, then pooling 114 , another convolution with further features, a new pooling and a summary in the Dense layer 116 , The output of the network then gives the class of the middle pixel of the input image 110 or a pixel of the image 110 of 6 ,

Then a new image section, generally an image section that is shifted by one pixel, is selected and again by means of CNN classified. The result of this procedure is that the calculations that the convolution neural network needs must be repeated for the number of pixels to be classified. This is time consuming. The picture 110 of 6 has a resolution of 2000 x 1000 pixels. The CNN would have to be calculated two million times. The initial problem, however, is only the classification of the plants 108 per se. On average, such an image contains about 5% plant pixels, which corresponds to about 100,000 pixels.

By means of simple segmentation and data reduction by the segmentation and data reduction unit 66 can be found out whether a pixel represents part of a plant 108 or around background 118 is. This segmentation is not as complex as a calculation CNN and therefore faster. The segmentation and data reduction by the segmentation and data reduction unit 66 is carried out analogously to 8th , In 8th the individual steps are shown.

In a first step 120 each is going to the database 78 transferred image from several pixels into the RGB (Red, Green, Blue) color model.

In a next step 122 each pixel of the transmitted image based on the RGB color model HSV (hue, saturation, value) color model transferred.

In a next step 124 this HSV color model is evaluated.

Each pixel based on the HSV color model is rated for color saturation based on a threshold, and if the value of color saturation exceeds a threshold, that pixel is the binary value 1 is assigned, and if the value of the color saturation falls below a threshold, the pixel the binary value 0 is assigned.

In parallel, each pixel is evaluated based on the HSV color model with regard to the color angle (hue) based on a predetermined range, and if the color angle is in the predetermined range, the pixel is the binary value 1 is assigned, and if the color angle is out of range, the pixel will be the binary value 0 is assigned.

In a next step 126 an intermediate image is generated from the binary information of the color angle and the color saturation, which has considerably less data than the image generated by the camera 108 contains.

From the segmentation 8th the following formula results, which must be applied to each pixel. The segmented image S (x, y) is about the division of the RGB image ψ (x, y) divided into its three components red, green and blue. A pixel of the segmented image is then set to 1 if the minimum value of red, green or blue of a pixel divided by the green pixel is less than or equal to a threshold value (THs). The threshold in the 8th Bit space of the image is done via scaling using 255. If the threshold is not reached, as in Equation 1 of the pixels of the segmented image 0 set.

This results in the first optimization: Before the entire picture 108 is broken down into two million images, the segmentation, according to 8th , applied. That means the whole picture 108 is run through and it is decided using the formula given above whether it is a plant pixel or not. Firstly, the picture 108 segmented, that is the background 118 will be on black ( 0 ) set like this in 9 is shown. On the other hand, if it is a plant pixel, its coordinates are written in a list. Then only the coordinates in the CNN given that are also in this list. The unnecessary pixels of the earth, i.e. the background 118 , omitted. So that will be CNN in approximately 20 - called less times.

By segmenting the background 118 on the value 0 set. Even the picture elements that CNN viewed, now have segmented images. The feature calculation would normally be applied to every pixel of the picture element on a convolution layer. However, there are three cases 128 . 130 . 132 for the calculation the in 10 are shown, each for a feature 134 size 5x5.

The red case 128 shows a feature calculation in which the feature is completely on the background 118 lies. Here each element is multiplied by 0, which means that the entire calculation 0 results, or the bias value. The result of this calculation is therefore known before the calculation. Even if the background 118 would not be zero, i.e. would have soil, this calculation would not contain any information about the plant 110 include, so the result can simply be a constant fictitious value.

In the yellow case 130 the average feature value is not on a plant 110 , This means that a part is also a multiplication by zero. This case consumes the plant 110 in the border area and makes this size in the feature map.

In the blue case 132 at least the middle pixel of the feature lies on a plant.

After considering these three cases 128 . 130 . 132 just have the yellow and blue case 130 and 132 be calculated. So the cases 130 . 132 in which the feature has at least one non-zero input value. Results from all other feature calculations are known before the calculation, they are zero or only have the bias value. The coordinates in which the blue case 132 occurs are known. These are the coordinates that are saved during segmentation. The yellow case 130 again requires a calculation as to whether this has occurred.

This requires a check of each plant pixel found in the segmentation. Because this control is too much effort and the yellow case 130 only in the edge area of a plant 110 this case should be ignored.

Therefore, the calculation can be optimized so that the feature calculation and all other elements of the CNN should only be applied to the plant pixels found.

In 11 is shown schematically how two neighboring plant pixels 136 . 138 differ. A plant pixel on the left 136 and to the right of the neighboring plant pixel 138 , The blue / purple area 140 . 142 could be different plants that need to be classified. The red / purple area 144 . 142 represents the picture element that the CNN viewed to the orange pixel 146 to classify.

On closer inspection, it can be seen that the area under consideration (red / purple) 144 . 142 overlaps strongly. This in turn means that in both picture elements 136 . 138 mostly the same values. If so CNN If the feature was calculated in the convolution layer, the same values would also be calculated from the feature calculation.

In 12 is schematically a feature 148 sketched in green with the size of 5 x 5 pixels. This feature 148 stands at the same coordinates within the whole picture, but it is shifted within the from CNN image element to be considered (red / purple) 144 . 142 , However, since the location is the same in the whole picture, the calculation for the middle black box would be 150 same value in both the left picture 136 as well in the right picture 138 result. This knowledge can be applied to all elements of one CNN apply. As a result, if the border area is ignored, the individual feature calculation can first be applied to the entire image. Theoretically, the decomposition of the input image plays 108 only with the dense layer 116 a crucial role. The dense layer 116 can be just like a convolution 112 be calculated. The feature size results from the interaction of the input image size and the existing pooling layers in the network. This allows the classification to be further optimized; CNN Elements only on the found plant pixels applied. The feature map, which is calculated from the last convolution, shows the classification result, as shown in 13 is shown. Here are all carrot plants in green 152 and all the weeds in red 154 Classified pixel by pixel.

However, these optimizations also result in changes to the result of the classification. The pooling layers have the greatest influence here. With each pooling, information is removed from the network. However, due to the fact that the picture elements no longer exist individually, pooling loses a local reference. The 14 illustrates the problem.

In the 14 is a picture element 156 as a red frame 158 shown. Before the optimization, each picture element would be separated by the CNN run to classify its middle pixel. The right picture element 160 is shifted one pixel to the right. The four colors: purple, blue, yellow and green indicate the individual applications of pooling. As can be seen, they can produce different results, since pooling always starts at the edge and continues around a pooling element (here two fields). This results in two neighboring picture elements 156 . 160 two different pooling elements. As a result, if this is to be taken into account in the optimization, two new branches would result for the further calculation with each pooling. Since pooling would have to be applied to the entire image once with the starting point at the top left and another pooling with the starting point at the top left plus one pixel. In the further calculation, both pooling results would then have to be processed separately. A further second pooling would result in two new paths, so that four separate results would have to be calculated. The result is then composed of the four results rotating pixel by pixel through the results. If only one path is considered after pooling, the initial image would be smaller after pooling twice. The length and width of the output image would then only be ¼ as large as the input image. When considering all paths, the input image size would come out.

Another difference is the missing border areas of the plants. Since the features are not applied to all elements by having any overlap with the plant, there are calculation differences here. This can also change the classification result compared to the conventional calculation.

The lack of calculation of the feature values outside the plant can lead to other values, since the result is shown with zero, which is actually the bias value.

Although these three factors influence the results, it shows that that CNN is very robust and therefore the results still meet a very high accuracy value.

The next step would be to train the network directly with these modifications so that the network can adjust itself even better to its new calculation and thus compensate for any errors in the calculation directly.

The segmentation and data reduction device provides the weed pixels 154 with position coordinates.

LIST OF REFERENCE NUMBERS

10

carrier system

12

drone

12a

Recording, recording room of the flight drone 12

12b

Control device of the flight drone

14

mobile device

14a

first unit

14b

second unit

14c

Holder for gripper arm on the mobile device 14

16

drive

18

electric motor

20

propeller

22

feet

24

battery

26a

Voltage interface on the drone 12

26b

Voltage interface on the mobile device 14

28

connector

30

communication unit

32

antenna

34

GPS unit

36a

Interface on the flight drone 12

36b

Interface on the mobile device 14

38

connector

40

first housing of the first unit 14a

42

second housing of the second unit 14b

44

connector

46

first computer

48

Arm as an actuator

50

engine

52

tool unit

54

feed unit

56

rotation unit

58

milling machine

60

first communication unit of the first unit 14a

62

visual detection unit

64

camera

66

Segmentation and data reduction facility

68

classifier

70a

Interface of the first unit 14a

70b

Interface of the second unit 14b

72

communication link

74

second communication unit of the second unit 14b

76

second computer

78

Database

80

connector

82a

Gripper arm, left

82b

Gripper arm, right

84

first step: identify the necessary measures

86

second step: selecting the mobile device 14 from the available mobile devices 14

88

third step: connecting the device to the flight drone 12

90

fourth step: implementation of the identified measures

92

fifth step: replace the mobile device 14 against another mobile device 14 and take another action

94

first step: continuous recording

96

second step: transmission of the data

98

third step: storage of the data

100

fourth step: data comparison

102

fifth step: evaluation by the classifier 68

104

sixth step: implementation in regulation and control engineering data

106

seventh step: starting the units

108

Example picture, input picture

110

plant

112

convolution

114

pooling

116

Summary in a dense layer

118

background

120

first step: convert to an RGB color model

122

second step: transfer to an HSV color model

124

third step: evaluation of the HSV image

126

fourth step: generation of an intermediate image

128

first case, red

130

second case, yellow

132

third case, blue

134

feature

136

Plant pixels, left

138

Plant pixel, right

140

blue area

142

purple area

144

red area

146

orange area

148

Feature, green

150

medium black box

152

carrot plant

154

Weeds, weeds

156

Image element, left

158

red frame

160

Image element, right

QUOTES INCLUDE IN THE DESCRIPTION

This list of documents listed by the applicant has been generated automatically and is only included for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent literature cited

• DE 4039797 A1 [0004]
• DE 102015209879 A1 [0005]
• DE 102015209891 A1 [0006]
• DE 102005050302 A1 [0009]
• DE 102006009753 B3 [0010]

Claims (23)

Mobile analysis and processing device (14) for agriculture for cultivating the soil and / or for manipulating flora and fauna, with a visual detection unit (62), which comprises: a camera (64) with which images are captured, a segmentation and data reduction device (66), a. with which an image recorded by the camera (64) is generated from a plurality of pixels in an RGB (Red, Green, Blue) color model; b. each pixel is converted to an HSV (hue, saturation, value) color model based on the RGB color model; c. each pixel is rated based on the HSV color model for saturation based on a threshold, where if the value of color saturation exceeds a threshold, the pixel is assigned the binary value "1" and if the value of Color saturation falls below a threshold value, which is assigned to the binary value "0"; d. each pixel is evaluated in parallel based on the HSV color model with regard to the color angle (hue) using a predetermined range, the pixel being assigned the binary value “1” if the color angle is in the predetermined range, and if the Color angle is outside the range, the pixel is assigned the binary value "0"; e. from the binary information of the color angle and the color saturation of the individual pixels, a first intermediate image is created which contains considerably less data than the image generated by the camera; f. Generating pixel fields from the pixels of the intermediate image; a classifier (68) which, on the basis of the intermediate image generated by the segmentation and data reduction device (66), takes over the classification of a plurality of pixel fields consisting of pixels, only pixel fields which have at least one pixel assigned to the binary value “1” being classified is. Analysis and processing device after Claim 1 , characterized in that the segmentation and data reduction device (66) forms a pixel field by a square number of pixels, in particular has 100, 81, 77, 36, 25, 16, 9 or 4 pixels. Analysis and processing device after Claim 1 or 2 , characterized in that the pixel field is square. Analysis and processing device according to one of the preceding claims, characterized by a tool unit (52) with at least one motor-driven tool (58), an actuator (48) for moving at least the tool (58) of the tool unit (52), a motor (50 ) for driving the tool unit (52) and / or the actuator (48), a database (78), a first communication unit (60, 74) with interface (70a, 70b; 36a, 36b) and a first computer (46, 76 ) for controlling the motor (50), the visual detection unit (62), the tool unit (52) and / or the actuator (48) on the basis of generated control commands, the data recorded via the visual detection unit (62) with the data in the database ( 78) stored data are continuously compared in order to generate corresponding control signals for the motor (50), the visual detection unit (62), the tool unit (52), the actuator (48) and / or an assigned carrier (12). Analysis and processing device according to one of the preceding claims, characterized in that means (12a, 38, 72, 82a, 82b) are provided for connecting, if required, to a carrier (12) which moves the device (14). Analysis and processing device according to one of the preceding claims, characterized in that the tool unit (52) has at least one feed unit (54) and one rotation unit (56) which interacts with the motor (50). Analysis and processing device after Claim 6 , characterized in that the rotation unit (56) has at least the tool (58) at a distal end, in particular a milling cutter or a knife unit. Method for operating the analysis and processing device according to one of the preceding claims, characterized by the following steps: a. continuous recording of data-technically defined voxels and / or pixels and / or images by the visual detection unit (62); b. Transmission of the recording data to the database (78); c. Storage of the (recording data in the database (78); d. Qualitative data comparison of the recording data with the recording data stored in the database (78), preferably by performing a segmentation, data reduction, and / or verification of the recording data images by the computer (46 , 76); e. Evaluation of the matched recording data with existing defined recording data and data records in the database (78) by a classifier (68) in connection with the computer (46, 76); f. Processing and implementation of the evaluation by the computer ( 46, 76) in regulation and / or Control data for the motor (50), the actuator (48), the tool unit (52) and / or an associated carrier (12). Procedure according to Claim 8 , characterized in that after the regulation and / or control-related data are available, the motor (50), the actuator (48), the tool unit (52) and / or the associated carrier (12) are started up for working the soil and / or manipulation of flora and fauna occurs. Procedure according to Claim 8 or 9 , characterized in that during the data comparison, the recording data are transmitted to the segmentation and data reduction device (66) which generates intermediate images. Procedure according to Claim 10 , characterized in that after the intermediate images have been generated, they are transmitted to the classifier (68), which evaluates the generated intermediate images on the basis of existing defined images and data records in the database (78) in cooperation with the computer (46, 76). Procedure according to Claim 10 or 11 , characterized in that the following steps are carried out in succession in the segmentation and data reduction device (66) for generating the intermediate images: a. Each transmitted image from several pixels is converted into the RGB (Red, Green, Blue) color model; b. each pixel of the transmitted image based on the RGB color model is transferred to an HSV (hue, saturation, value) color model; c. each pixel based on the HSV color model is rated for saturation based on a threshold, where if the value of color saturation exceeds a threshold, the pixel is assigned the binary value "1" and if the value of Color saturation falls below a threshold value, which is assigned to the binary value “0”; d. preferably in parallel with the step just mentioned, each pixel is evaluated based on the HSV color model with regard to the color angle (hue) using a predetermined range, the pixel being assigned the binary value “1” if the color angle is in the predetermined range and if the color angle is out of range. the pixel is assigned the binary value "0"; e. From the binary information of the color angle and the color saturation, the first intermediate image is created, which contains considerably less data than the image generated by the camera (64). Procedure according to one of the Claims 8 to 12 , characterized in that in the segmentation and data reduction device (66), pixel fields are generated from the pixels of the intermediate image. Procedure according to one of the Claims 8 to 13 , characterized in that the segmentation and data reduction device (66) provides the pixels with position coordinates. Procedure according to one of the Claims 8 to 14 , characterized in that the classifier (68) uses an artificial neural network to carry out the classification. Procedure according to Claim 15 , characterized in that the artificial neural network is formed by a convolutional neural network (CNN). Carrier system (10) with a carrier (12) and a mobile device (14) for working the soil and / or for manipulating the flora and fauna according to one of the Claims 1 to 7 , in particular to carry out the method according to one of the Claims 8 to 16 , wherein the carrier (12) a drive (16) for moving the carrier system (10), a control device (12b), an energy source (24) which interacts with the drive (16) and provides the necessary voltage for operation , a receptacle (12a) for receiving and connecting to the mobile device (14), a communication unit (30), a first communication interface (36a) interacting with the communication unit (30) for data exchange with the mobile device (14), which has an associated Interface (36b), and has a second communication interface (32) interacting with the communication unit (30), which receives at least control data and forwards it to the control device (12b) for predetermined movement of the carrier (12). Carrier system after Claim 17 , characterized in that the carrier (12) is designed as an unmanned aerial vehicle (drone), unmanned land vehicle, unmanned watercraft, or unmanned underwater vehicle. Carrier system after Claim 17 or 18 , characterized in that the second communication interface is an antenna (32) for data exchange with a central, preferably fixed, computer separate from the carrier (12). Carrier system after Claim 19 , characterized in that the second communication interface (32) is also designed for data exchange with the mobile device (14) and the central computer. Carrier system according to one of the Claims 17 to 20 , characterized in that the carrier (12) and the mobile device (14) have a voltage connection (26a, 26b) via which the mobile device (14) can be supplied with voltage. Carrier system according to one of the Claims 17 to 21 , characterized in that the carrier (12) and the mobile device (14) have means (12a, 82a, 82b) for connecting to one another as required. Carrier system according to one of the Claims 17 to 22 , characterized in that the carrier (12) has a GPS unit (34) for detecting the position coordinates of the carrier (12).

Images