WO2019063255A1

WO2019063255A1 - Method for recognizing a leaf edge, method for the targeted treatment of plants by means of a leaf treatment agent, and use of an event-based image sensor for the recognition of a leaf edge

Info

Publication number: WO2019063255A1
Application number: PCT/EP2018/073932
Authority: WO
Inventors: Gregor Blott; Damian Koesters
Original assignee: Robert Bosch Gmbh
Priority date: 2017-09-28
Filing date: 2018-09-06
Publication date: 2019-04-04
Also published as: DE102017217275A1; EP3688660A1; BR112020002571A2

Abstract

The invention relates to a method (100) for recognizing a leaf edge, comprising the following steps: sensing (110) events of an image of a ground covered with plants at a travel speed relative to the ground, a ground resolution of the image being between 0.5 and 1 mm per pixel, a bandpass filter of a first frequency range being arranged before a first pixel of the image and a bandpass filter of a second frequency range being arranged before a second pixel of the image; and determining (120) that a leaf edge has been recognized if the first pixel senses at least one event of a first polarity at a first time point t1 and the second pixel senses at least one event of a second polarity at a second time point t2, the first polarity and the second polarity being opposite, and the at least one event of a first polarity of the first pixel and the at least one event of a second polarity of the second pixel corresponding to the same location of the ground covered with plants, or if the detected events were detected either only at the first pixel or only at the second pixel. The invention further relates to a use of an event-based image sensor for the recognition of a leaf edge, a computer program, a machine-readable storage medium and an electronic control device.

Description

description

title

Method for detecting a sheet edge, method for targeted treatment of plants with a foliar treatment agent and use of an event-based image sensor for detecting a sheet edge

The present invention relates to a method for detecting a sheet edge, a method for the targeted treatment of plants with a

Sheet handling means, a use of an event-based image sensor for detecting a sheet edge, a computer program, a machine-readable storage medium and an electronic control unit.

State of the art

The present technical field relates to the detection of leaves or plants on a field soil by image sensors in a

Travel speed of up to 20 km / h of a vehicle traveling over the field.

It is known in the art that the information available in the scene is usefully reduced only to the sheet edge curve, which can be detected with a bottom resolution between 0.5 and 1 mm per pixel, the so-called sheet-edge resolution. This is due to the fact that the leaf texture can be added as the next higher information level only at least ten times the ground resolution, which is not practicable with respect to data quantities, dynamic range of the sensor and costs for sensor / optics with conventional image sensors.

Furthermore, the dynamic range is in a single image with direct

Sunlight and existing drop shadow already at the leaf margin resolution of clods or plants very high. In the limited Dynamic range of about 60 to 80 dB usable CMOS cameras remain at the same time overexposed and underexposed image areas, with which no meaningful Normalized Difference Vegetation Index (NDVI) can be calculated. The NDVI is a

Difference index from near infrared and red channel to plant / soil

Discrimination. The NDVI is calculated as the difference in near infrared reflectance and red visible range divided by the sum of these reflectance values. The NDVI is the most commonly used vegetation index and has historically been defined for remote sensing tasks.

Event-based image sensors or cameras, in particular DVS cameras (DVS = Dynamic Vision Sensor), are known in the prior art. DVS cameras provide a change-sensitive form of a camera, whose

Functionality was adapted from the biology of the human eye out. The Dynamic Vision sensor does not take like classic image sensors

(e.g., CMOS technology) images at equidistant time intervals, but measures intensity differences, so-called events, at individual pixel locations and only instantaneously sends them with microsecond accuracy and millisecond latency. If the intensity of a pixel does not change or only slightly changes, no event will be triggered and no data will be sent for that pixel. Such a

Sensor is also called change sensor.

It can be adjusted from which intensity difference the

Change sensor detects an event, for example, a change by a certain percentage or by a certain amount. Other definitions of change are possible. The change sensor is configured to determine intensity changes in a pixel from a time tnl to a time tn2 as intensity data. In particular, tn2 - tnl = dt2, where dt2 <10 ^{"is 4} seconds Preferably, the time interval dt2 results from the time points of an intensity change detected by the change sensor The times tnl and tn2 and the time interval dt2 are preferably variable and / or not necessarily constant for different intensity changes For example, the time interval dt2 is variable and results from the detected intensity changes in a cell and / or from the transmission rate of the change sensor. Disclosure of the invention

The sheet edge detection method comprises, as a first step, acquiring events of an image of a plant or vegetation

overgrown soil at a ground speed relative to the ground.

Such an image of a vegetated soil usually has mainly two types of areas. On the one hand, this is an area in which the color green predominates, which comes from the chlorophyll of the plants.

On the other hand, this is an area where a brownish color from the ground predominates. A plant is to be understood as meaning a plant with leaves whose leaves contain chlorophyll.

In the following, the reflection coefficients of vegetation and dry soil are described, provided that in the red spectral range and in the near infrared equally intense illumination is present and that the sensors used have the same spectral sensitivity.

Vegetation has a very low visible level

Reflection coefficients below 25%, however, the reflection coefficient steeply increases from the red region to the NI R region where it is nearly 50%. The reflection coefficient of dry soil changes less strongly from the red to the NI R range. The corresponding value of the reflection coefficient is about 25% in the red region and about 30 to 35% in the NI R region. Throughout the NI R range, the reflection coefficient of the vegetation is significantly higher than the reflection coefficient of the dry soil. In the red spectral range, the reflection coefficient of the dry soil is usually larger than that

Reflection coefficient of vegetation, however, the reflection coefficient of the vegetation in the direction of the NI R range is greater than the reflection coefficient of the dry soil.

Under these conditions, when the camera moves relative to the scene, eg to the right, events are generated at the brightness gradients. These are positively correlated for ground-to-ground and ground-to-stone transitions because of the nearly constant reflectivity between R and NI R filtered event current. For vegetation, the reflectivity difference of both channels is significantly higher, so that the sign of the gradient strength in the case of a bottom of medium reflectivity is always opposite between the two channels. This reverses the ON / OFF polarity of the events accordingly.

If the soil is dark, you get a faint gradient in the red channel at the plant edges and strong gradients in the NI R channel. If the soil is light, there are strong gradients in the R channel at the plant edges and weak gradients in the NI R channel.

The method uses a bottom resolution between 0.5 and 1 mm per pixel for the determination of a sheet edge. The advantage of this ground resolution is that it allows the so-called leaf edge curve to be detected by conventional means. In addition, this type of detection is very fast and allows a high dynamic range. This ensures a reliable analysis of the sheet edge curve in real time.

Furthermore, a bandpass filter of a first frequency range is arranged in front of a first pixel of the image sensor and a bandpass filter of a second frequency range is arranged in front of a second pixel adjacent to the first pixel. By such an adjacent arrangement of bandpass filters, a different behavior in the corresponding spectral regions of different materials can be detected.

In one embodiment of the method, the first frequency range is equal to the second frequency range. This allows the use of a filter that uses machine learning for false positive removal.

However, the first frequency range is preferably the near infrared and the second frequency range is a red spectral range. By such

adjacent array of bandpass filters in the NI R and in the red

Spectral range, a sheet edge can be reliably detected.

According to the method, it is determined that a leaf margin has been detected if the first pixel at a first time tl at least one event of a first Polarity and the second pixel detected at a time t2 at least one event of a second polarity, wherein the first and second polarity are opposite, and that the at least one event of a first polarity of the first pixel and the at least one event of a second polarity of the second pixel the same Place of plant-covered soil corresponds. In this case, the term "the same place is to be understood as meaning the plant-covered ground", in particular adjacent pixels of a single image, or alternatively pixels with a spacing of up to three pixels can be understood Plant-covered soil both in the first and in the second

Frequency range has an equally intense lighting.

Alternatively, according to the method, it is determined that a leaf margin has been detected if the detected events were detected at only the first or the second pixel. This can be achieved in particular by suitable selection of threshold values of the object contrast. Here, the thresholds for on and off events can be set separately. The

Thresholds may also be used for the different channels, i. the R channel and the NI R channel are set separately. This case occurs, among other things, if the ground is dark or light.

In particular, by suitable selection of the threshold values, it can be achieved for any brightness conditions that either the above-mentioned first case, according to which adjacent pixels have an opposite polarity, or the above-mentioned second case, after which only one is present at adjacent pixels

Pixel events occurs.

Alternatively, for any brightness ratios, the same information can also be deduced from the brightness histogram of the frame. Here, the method of scene reconstruction is suitable, which is e.g. among others in the scientific publication entitled "Real-Time 3D Reconstruction and 6-DoF Tracking with an Event Camera" by Hanme Kim, Stefan

Leutenegger, Andrew J. Davison in European Conference on Computer Vision, ECCV 2016: Computer Vision - ECCV 2016, pp 349-364, is published. See there in particular Figure 4. Here, the brightness histogram is in at least one of the two

Bimodal color channels, in the case of the average ground brightness even in both, thereby allowing automatic definition of the contrast threshold in the event generation, e.g. by Otsu method.

As already explained above, intensity changes at the individual pixel positions represent events. A detected event is sent immediately to a computing unit with microsecond accuracy and millisecond latency. If only the two adjacent pixels are considered in the time period, the event-based image sensor provides a time-dependent function for each pixel, which indicates for each time point whether an event was detected or not. If an event has been detected, the function additionally gives the

Intensity change with.

When events are considered to be simultaneous depends on a time window. The size of the time window depends on ground resolution and speed. If e.g. If two pixels represent one millimeter and the speed is 1 meter per second, then events with a time difference of approximately half a millisecond are to be regarded as simultaneous. As described elsewhere, one then detects a leaf margin with opposite polarity of the events.

Using the appropriate evaluation software, the method described can be used to extract leaf margin information and to determine the associated plant species in real time. The method generates a low data rate through the use of event-based image sensors, which has an advantageous effect on subsequent data processing and enables efficient local calculations on the camera and field spray modules. Particularly advantageous is the high robustness of the

Process with a dynamic range of 130 dB. Thus, a trouble-free sequence of sheet edge detection is ensured even with fast moving cameras and daylight conditions. The detection of events by the event-based image sensor advantageously makes it possible to find and classify regions of equal semantics in the imaged region as such.

In particular, the duration between the two times t1 and t2 for a given direction of travel depends on a first direction, which extends from a center point of the first pixel to a center point of the second pixel, and a second direction, which corresponds to a tangential direction of the page edge.

Preferably, the events are detected with an event-based image sensor. The event-based image sensor is preferably a DVS sensor. Since the detected events usually require much less memory than the

Storing each pixel for each frame, the amount of data delivered by a DVS camera is much less than that of a regular image sensor. Furthermore, the processing speed of a DVS sensor is much faster than with ordinary cameras. Since a DVS sensor looks at individual pixels independently, a much higher dynamic range up to 130 d B is possible. This high dynamic range prevents overexposed image areas, which in turn allows a more reliable image analysis and thus a more accurate detection of the sheet edge.

According to a preferred embodiment, the bottom resolution is between 0.1 and 1 mm per pixel. As a result, finer structures can advantageously be resolved in the recorded images. This helps to differentiate between different plant species more reliably.

In accordance with a further embodiment, a color filter array (C FA) of band pass filters in the NI R and R spectral range is arranged in front of the image sensor. Here, the R-spectral range stands for the red spectral range.

The two different bandpass filters can advantageously be arranged in a checkerboard pattern. This advantageously achieves that next to each pixel with NI R bandpass filter is a pixel with bandpass filter in the red spectral range. This in turn has the advantage that a sheet edge detection can be performed for each pixel. According to another embodiment, the two are different

Bandpass filter arranged in strips. In this case, the strips are preferably arranged transversely to the direction of travel. This has the advantage that with the help of this arrangement each sheet edge transition can be detected. According to yet another embodiment, the color filter array is NDVI optimized. An NDVI-optimized color filter array is understood to mean that the color filter array optimally distinguishes between the NI R and the red spectral region so that the NDVI index has high expressivity. This is especially the case when the NI R band pass filter of the color filter array is centered around 850 nm and the R band pass filter of the color filter array is centered around 660 nm.

The respective bandpass filters preferably have a width of 20 nm. Such a color filter array is an optimal distinction between

Vegetation and soil achieved according to the NDVI. Preferably, a bandpass filter of a third frequency range is arranged in front of a third pixel of the image sensor. According to such an embodiment, the color filter array has a third bandpass filter, which is preferably in the green color range. As a result, advantageously, the accuracy of the detection of the sheet edge can be further increased. For example, in addition to the NDVI index, other indexes such as e.g. Excessive Green, to be used. Preferably, the color filter array is optimized for other filters, for example for the Excessive Green Index.

According to a further preferred embodiment, the soil, which is overgrown with plants, has equally intense illumination both in the first and in the second frequency range. If this is not the case, alternatively the intensity in the first frequency range and in the second frequency range is determined. This can also be done pixel by pixel. Preferably, a new image is calculated in which the intensity in the first and second frequency ranges is the same or identical. This can also be done pixel by pixel. According to another embodiment, a passing object is measured several times. A passing object is an object that passes through a detection area of the event-based image sensor. Since the vehicle speed and direction of travel and the orientation and position of the event-based image sensor relative to the vehicle are known, an object detected by the image sensor can be tracked. If the object does not rotate relative to the image sensor during acquisition, all points of the object are recorded multiple times. Preferably, the passing object is continuously measured. Due to the multiple or continuous measurement of the object, a time averaging is possible, which in turn allows an increase in the detection accuracy and thus a more accurate detection of the sheet edge. According to a further embodiment, a plant height of a plant is determined from at least one image of the soil covered with plants. Here, an image is understood as an image taken with an event-based image sensor. Preferably, the stature height is determined from a variety of images of the vegetated soil with soil. The prerequisite for this is that an interpretable flow field can be determined from the at least one image. Known in the art is the technique "Structure from motion (SfM)." This technique is photogrammetric imaging technique for three-dimensional structures based on two-dimensional image sequences, and is used in the fields of "computer vision" and visual

Perception examined. Humans and other living things can reconstruct three-dimensional structures based on the projected two-dimensional motion of a moving object or scene in the brain. The determination of the stature height has the advantage that subsequent spraying operations can be dosed more accurately and thus can be flexibly adapted to the particular circumstances.

When taking pictures of a vegetated soil with a moving camera, the speed of which is known, areas of the image closer to the camera require less time to pass through than more distant areas of the image. Thus, that is through the Camera recorded flow field an interpretable flow field. For metric accuracy of the results, only intrinsic parameters of the optics such as the focal length (opening angle) and driving speed need to be known.

Comparative statements without metric accuracy are possible even without this calibration.

Due to the continuous recording of events, occlusions of small, ground-level weeds in high-growing crops with broad leaves (for example maize) rarely lead to missing detections than discrete-time full-frame images.

Preferably, the method comprises a machine-based learning algorithm for recognizing a classification of the plant. With its help, the

Determination of the plant species continuously improved and the

Overall process results more accurate.

The method for the targeted treatment of plants with a

Foliar treatment agent recognizes a sheet edge in a first step of the method according to the method set out above. In a second step of the method, a leaf shape of the plant is detected based on the recognized leaf margin of the plant. For a still image of a field soil with a leaf, a leaf margin is detected for each pixel pair in which only one pixel predominantly detects events. Accordingly, any location of the sheet edge can be recognized for the entire still image. If each location of the leaf margin is known, a leaf shape can be determined or recognized. For a moving image, a leaf shape can be determined for each time you see the leaf.

In a third step of the method, a classification of the plant is determined on the basis of the specific leaf shape of the plant. This has the advantage that the thus classified plant can be specifically treated with a suitable agent for this plant.

In a fourth step of the procedure, the plant is treated with one of the

Classification of the plant appropriate means treated. This advantageously ensures that certain plants are selective or targeted can be treated. For example, it is possible for weeds to be specifically treated with a weed control or herbicide. Alternatively or additionally, other recognized plants may be treated by other means. Such agent is especially a pesticide, a pesticide, an insecticide, a fungicide, a herbicide, a biocide, a bactericide

Virucide, an acaricide, an avicide, a molluscicide, a nematicide, an ovicide or a rodenticide. The targeted treatment of the plant with a suitable agent for this plant has the advantage that no resources are wasted, in particular no plants are damaged because no plant is treated with an inappropriate agent.

The method for the targeted treatment of plants enables a rapid differentiation between the different plant species and, based on this, a targeted treatment of the respective plant species with the appropriate agent.

Furthermore, the invention relates to a use of an event-based image sensor for detecting a sheet edge using images recorded by an event-based image sensor. This use has already been disclosed in the sheet edge detection method described above.

But this also results in the benefits associated with the use.

The computer program is set up to perform each step of the method, especially when running on an electronic controller or computing device. This refers to both the method for the detection of a leaf margin and the method for the targeted treatment of plants with a foliar treatment agent. This allows the implementation of the method in a conventional control unit, without constructive

To make changes. For this purpose, the computer program is stored on a machine-readable storage medium. By loading the computer program on a conventional electronic control unit, the electronic control unit is obtained, which is set up to recognize a leaf margin or to treat a plant specifically. Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

It is understood that the features mentioned above and those yet to be explained below can be used not only in the particular combination indicated, but also in other combinations or in isolation, without departing from the scope of the present invention.

Short description of the drawing

An embodiment of the invention is illustrated in the drawing and will be explained in more detail in the following description.

FIG. 1 shows a flowchart of a method for sheet edge detection according to an exemplary embodiment of the invention.

Embodiment of the invention

FIG. 1 shows a flow chart of a method 100 for sheet edge detection. In a first step 110 of the method 100, events of an image of a vegetated soil at a ground speed relative to the ground are detected using an event-based image sensor. The ground resolution used here is 0.5 mm per pixel and in front of the image sensor, an NDVI-optimized color filter array (CFA) of bandpass filters in the NI R and R spectral range is arranged. Here, the NI R band pass filter is centered around 850 nm and the R band pass filter centered at 660 nm.

In a second step 120 of the method 100, it is determined that a sheet edge was detected if the detected events of two adjacent pixels were detected predominantly only at one of the two adjacent pixels. The

Step 120 is performed for all adjacent pixels of the captured events of the image. This also means that the method 100 repeatedly measures a passing object. Thus, one obtains the leaf margin for all recorded images of the plant, from which one can determine a leaf shape of the plant. In step 130 of the method 100, a plant height of a plant is determined from all images showing the same plant of the overgrown soil.

In the subsequent step 140 of the method 100, a machine-based learning algorithm is used to classify the plant based on the images of the leaf margin taken by the plant.

In the next step 150, method 100, the classified plant is specifically treated with a leaf treatment agent corresponding to the classification of the plant.

Claims

claims

A method (100) for detecting a sheet edge with the following

steps:

Capturing (110) events of an image of a plant

overgrown soil at a groundspeed relative to ground, with a bottom resolution of the image between 0.5 and 1 mm per pixel; in which

in front of a first pixel of the image a bandpass filter of a first

Frequency range and before a second pixel of the image

Bandpass filter of a second frequency range is arranged; and

Determining (120) that a sheet edge has been detected

if the first pixel detects at least one event of a first polarity at a first time t 1 and the second pixel detects at least one event of a second polarity at a time t 2,

wherein the first and second polarities are opposite, and that the at least one event of a first polarity of the first pixel and the at least one event of a second polarity of the second pixel correspond to the same location of the vegetated soil or

if the detected events were detected either only at the first or only the second pixel.

2. The method according to claim 1, characterized in that with

Plant-covered soil in both the first and in the second frequency range has an equally intense lighting.

3. The method according to any one of the preceding claims, characterized

characterized in that the events are detected with an event-based image sensor. Method according to one of the preceding claims, characterized in that the bottom resolution is between 0.1 and 1 mm per pixel.

Method according to one of the preceding claims, characterized in that the first frequency range is the near infrared and the second frequency range is a red spectral range.

Method according to one of the preceding claims, characterized in that a bandpass filter of a third frequency range is arranged in front of a third pixel of the image sensor.

Method according to one of the preceding claims, characterized in that in front of the image sensor from a color filter array

Bandpass filters in the NI R and R spectral range is arranged.

Method according to the preceding claim, characterized

characterized in that the color filter array is NDVI optimized.

Method according to the preceding claim, characterized

characterized in that the NI R bandpass filter is centered at 850 nm and the R bandpass filter at 660 nm.

Method according to one of the preceding claims,

characterized by a multiple measurement of a passing object.

Method according to one of the preceding claims, characterized in that a stature height of a plant from a short temporal sequence of images of the plant-covered soil are determined (130).

Method according to one of the preceding claims,

characterized by a machine-based learning algorithm for detecting a classification of the plant (140).

13. A method for the targeted treatment of plants with a

Foliar treatment agent comprising the following steps:

Detecting a sheet edge according to a method according to one of claims 1 to 12:

Recognizing a leaf shape of the plant based on a recognized leaf margin of the plant;

Determining (140) a classification of the plant;

Treatment (150) of the plant with an agent corresponding to the classification of the plant.

14. Using an event-based image sensor to detect a sheet edge when using an event-based image sensor

recorded pictures.

15. Computer program, which is set up every step of a

Method according to one of claims 1 to 13 perform.

16. Machine-readable storage medium on which a

Computer program according to the preceding claim is stored.

17. Electronic control device, which is adapted to detect a sheet edge by means of a method according to one of claims 1 to 12 or to treat by means of a method according to claim 13, a plant targeted.