DE102021212020A1 - Method for detecting soiling on a lens unit of a camera of an agricultural working machine - Google Patents

Abstract

The invention relates to a method for identifying soiling on a lens unit of an optical detection unit, in particular a camera, preferably an agricultural working machine, with the steps:
- Receiving images (16a), in particular of field sections (18) of an agricultural area, which were recorded by means of the optical recording unit, in particular the camera of the agricultural working machine;
- Segmentation of image cells (22a), which fall below a defined gray threshold value, of the received images (16a) by means of a computing unit in order to identify potential contamination cells (24a) in the received images (16a); and
- determining the matching contamination cells, which are the same in the received images (16), by means of the computing unit in order to identify them as contamination on the lens unit.

Description

State of the art

The invention is based on a method for identifying soiling on a lens unit of an optical detection unit, in particular a camera, preferably an agricultural working machine and a corresponding computing unit, an optical detection unit and an agricultural working machine according to the species of the independent claims. The subject matter of the present invention is also a computer program and a machine-readable storage medium

Agricultural machines are in a harsh environment with a lot of dirt. Dirt on the camera lens or on the cover glass of the camera lens blocks the incoming light. This is imaged as a fuzzy point on the sensor, resulting in a dark spot in the image. However, the scene is always adequately illuminated by the sun and the LEDs. If a camera lens becomes soiled by clods of earth, the camera must detect and report this, since a dirty camera lens is partially or fully blind and therefore cannot see objects.

In order to detect dirt on the camera lens, difference images are often generated over a series of images. Where the image content changes, the difference becomes large and where it is dirty it becomes small. This indicates contamination. Alternatively, the same scene can be recorded with different f-numbers in order to evaluate relative brightness fluctuations in the images.

From the DE 10 2016 118 335 A1 a method for detecting dirt on a lens of a camera of a motor vehicle is known.

The DE 10 2018 209 020 A1 describes a device which is designed to detect soiling of a light-permeable cover of at least one transmission window and/or one reception window of an optical sensor.

Disclosure of Invention

• - Empfangen von Bildern, insbesondere von Feldabschnitten einer landwirtschaftlichen Fläche, welche mittels der optischen Erfassungseinheit, insbesondere Kamera der landwirtschaftlichen Arbeitsmaschine, erfasst wurden;
• - Segmentieren von Bildzellen, welche einen definierten Grauschwellenwert unterschreiten, der empfangenen Bilder, mittels einer Recheneinheit, um potentielle Verschmutzungszellen in den empfangenen Bildern zu identifizieren; und
• - Ermitteln der übereinstimmenden Verschmutzungszellen, welche in den empfangenen Bildern gleich sind, mittels der Recheneinheit, um diese als Verschmutzung auf der Linseneinheit zu identifizieren.

The subject matter of the present invention is a method for identifying soiling on a lens unit of an optical detection unit, in particular a camera, preferably an agricultural working machine, with the steps:

• - Receiving images, in particular of field sections of an agricultural area, which were recorded by means of the optical detection unit, in particular the camera of the agricultural working machine;
• - Segmentation of image cells, which fall below a defined gray threshold value, of the received images by means of a computing unit in order to identify potential contamination cells in the received images; and
• - determining the matching contamination cells, which are the same in the received images, by means of the computing unit, in order to identify them as contamination on the lens unit.

The subject matter of the present invention is also a computing unit which is set up to carry out and/or control the steps of a previously described method.

The subject matter of the present invention is also an optical detection unit, in particular a camera for detecting a field section of an agricultural area in order to capture images of field sections, with a lens unit and a computing unit described above.

The subject matter of the present invention is also an agricultural working machine, in particular a field sprayer, with an agricultural working tool, in particular a spraying device, and an optical detection unit as described above.

The present invention also relates to a computer program that is set up to carry out and/or control the steps of a method described above when the computer program is executed on a computer, and a machine-readable storage medium on which the computer program is stored.

The method according to the invention now makes it possible to carry out soiling on a lens unit of an optical detection unit or camera in a very simple manner with very little computing power. Furthermore, the method can be carried out using a series of images, so that no still image is necessary.

The optical detection unit is preferably a camera. In this case, the lens unit comprises the lens and, if necessary, the cover glass or protective glass.

The received images are preferably a defined selection of an image series captured directly one after the other by the optical capture unit. The defined selection can include all images of the series of images. However, the defined selection can also only include a defined number of images of the image series captured immediately one after the other. Depending on how quickly you have to react to the contamination and, if necessary, in order to save computing time, the defined number can also include only every xth image, for example every 5th image of the image series recorded.

The images can be images of different field sections or of the same field section (when the optical detection unit is stationary).

The image cells are preferably picture elements or pixels. However, the image cells can also have a higher resolution and thus comprise two or more pixels.

In the segmentation step, the image cells or pixels that fall below a defined gray threshold value are segmented in order to identify potential contamination cells in the received images. The defined gray threshold value can, for example, be less than or equal to 100 out of 12 bits, i.e. out of 4096 gray values.

In one embodiment, the segmented image cells are themselves the potential contamination cells.

In a further embodiment, in the segmentation step, the images are overlaid with a grid whose grid cells comprise at least two, in particular a large number of, image cells, the segmented image cells being linked to the grid and grid cells having a coverage greater than or equal to a defined coverage threshold value the pollution cells are identified. The defined coverage threshold can be 50%, for example, so that grid cells that are at least 50% covered with segmented image cells are identified as the contamination cells.

Since in this embodiment only "rough" grid cells are compared across the received images, further computing time can be saved since the computing effort is very low, and the method can also be carried out while stationary.

The steps of receiving the images and segmenting the image cells are preferably carried out successively and repeatedly. In other words, for example, an image is received and the image cells in it are segmented and then the steps are repeatedly carried out one after the other for all other received images.

In the determination step, the matching contamination cells, which are the same in the received images, are determined using the processing unit in order to identify them (using the processing unit) as contamination on the lens unit. That is, in other words, the identified potential contamination cells are compared across the received images, and the potential contamination cells identified in all images are identified as actual contamination on the lens unit. In other words, the segmented image cells that have not changed or have remained segmented in the received images or the image series are identified as contamination.

In this case, the matching contamination cells are preferably identified as contamination only above a defined contamination cell threshold value (by means of the computing unit). The defined pollution cell threshold may be, for example, 5000 pollution cells.

An agricultural area can be understood to mean an area used for agriculture, an area under cultivation for plants or also a parcel of such an area or area under cultivation. The agricultural area can thus be arable land, grassland or pasture.

The field section can be a detection section or a detected image section of the optical detection unit.

The method can include a step of capturing the field sections of the agricultural area by means of the optical capturing unit. The step of detecting can be carried out during a crossing or a flight of the agricultural working machine. At least one further step of the method, in particular all steps of the method, can be carried out during a crossing or a flight of the agricultural working machine.

In this case, the agricultural working machine can comprise a mobile unit or be arranged on a mobile unit, with the mobile unit being able to be designed in particular as a land vehicle and/or aircraft and/or trailer. The mobile unit can also be a self-propelled or autonomous robot. The agricultural working machine is preferably a weed regulating machine, in particular a field sprayer. The agricultural working tool is preferably a spray device, but can also be a mechanical tool for weed control.
Furthermore, it is advantageous if the method a step of outputting a signal, in particular a control signal for activating a cleaning device, in particular the optical detection unit, and/or an information signal, depending on the identified contamination on the lens unit. The information signal can in particular be an optical and/or acoustic signal, for example to signal the contamination to a driver or operator of the agricultural working machine.

The arithmetic unit is or the arithmetic units are designed or set up for image processing, so that it can carry out calculation steps or image processing steps for carrying out the method according to the invention. Accordingly, each computing unit has corresponding image processing software. The arithmetic unit can be, for example, a signal processor, a microcontroller or the like, and the storage unit can be a flash memory, an EPROM or a magnetic storage unit. The communication interface can be designed to read in or output data wirelessly and/or by wire, with a communication interface that can read in or output wire-bound data reading this data, for example electrically or optically, from a corresponding data transmission line or outputting it into a corresponding data transmission line.

Accordingly, the method according to the invention can be implemented, for example, in software or hardware or in a mixed form of software and hardware in the processing unit or a control unit.

The processing unit can be arranged completely or partially on the optical detection unit or the agricultural working machine or integrated into it. However, the computing unit can also be completely or partially external, for example integrated in a cloud. The arithmetic unit can thus also be divided among different units, for example mobile and stationary units.

A computer program product or computer program with program code, which can be stored on a machine-readable carrier or storage medium such as a semiconductor memory, a hard disk memory or an optical memory and for carrying out, implementing and/or controlling the steps of the method according to one of the embodiments described above, is also advantageous used, especially when the program product or program is run on a computer or device.

• 1 eine als verschmutzten Kamera ausgeführte optische Erfassungseinheit einer landwirtschaftlichen Arbeitsmaschine;
• 2 ein mittels der verschmutzten Kamera aus 1 erfasstes Bild;
• 3a-c eine Bilderserie von empfangenen Bildern mit identifizierten potentiellen Verschmutzungszellen gemäß einer Ausführungsform;
• 4 übereinstimmende Verschmutzungszellen der Bilder aus 3a-c;
• 5 das Bild aus 2 mit einem überlagerten Gitter;
• 6a-c eine Bilderserie von empfangenen Bildern mit identifizierten potentiellen Verschmutzungszellen gemäß einer weitere Ausführungsform;
• 7 übereinstimmende Verschmutzungszellen der Bilder aus 6a-c; und
• 8 ein Ablaufdiagramm eines Verfahrens gemäß einem Ausführungsbeispiel.

Embodiments of the invention are shown in the drawings and explained in more detail in the following description. It shows:

• 1 an optical detection unit, designed as a dirty camera, of an agricultural working machine;
• 2 on by means of the dirty camera off 1 captured image;
• 3a-c an image series of received images with identified potential contamination cells according to an embodiment;
• 4 matching pollution cells of the images 3a-c ;
• 5 the picture off 2 with a superimposed grid;
• 6a-c an image series of received images with identified potential contamination cells according to another embodiment;
• 7 matching pollution cells of the images 6a-c ; and
• 8th a flow chart of a method according to an embodiment.

1 FIG. 1 shows an optical detection unit 10 of an agricultural working machine (not shown). The optical detection unit 10 is designed as a camera 10 . The optical detection unit 10 has a lens unit 12 . Dirt 14 adheres to or on the lens unit 12.

In 2 is on by means of the optical detection unit 10 or camera 10 1 captured image 16 of a field portion 18 of an agricultural area is shown. In this case, the image 16 has a dark section 20 or spot 20 due to the dirt 14 on the lens unit 12 .

In 3a-c an image series of received images 16a-c is shown. How out 3a-c As can be seen, image cells 22a-c or image pixels 22a-c which fall below a defined gray threshold value are segmented in the received images 16a-c by means of a computing unit (not shown). In the exemplary embodiment shown, the segmented image cells 22a-c are each identified as potential contamination cells 24a-c in the received images 16a-c of the image series.

4 FIG. 12 now shows the corresponding contamination cells 26, which are the same in all images 16a-c. The dirt cells 26 are then identified as dirt 14 on the lens unit 12 by means of the computing unit.

Alternatively, the matching contamination cells 26 can also only be identified as contamination 14 from a defined contamination cell threshold value, for example from 5000 contamination cells 26, which is the case here.

In 5 until 7 the sequence of a further embodiment of the method is shown. show here 5 the captured image 16 out 2 and the 6a-c the series of images from images 16a-c 3a-c In contrast to the previously described embodiment, these are additionally superimposed with a grid 28 in the segmentation step, the grid cells 30 of which each comprise 100 image cells or image pixels. In this case, the grid 28 is linked to the respective segmented image cells 22a-c, grid cells 30 with a degree of coverage of greater than or equal to 50% of segmented image cells 22a-c being identified as potential contamination cells 24a-c. This leads to a lower resolution, which, however, can save computing time.

7 now shows analogously to 4 the corresponding contamination cells 26, which are the same in all images 16a-c. The dirt cells 26 are then identified as dirt 14 on the lens unit 12 by means of the computing unit.

Alternatively, the matching contamination cells 26 can also only be identified as contamination 14 from a defined contamination cell threshold value, for example from 5 contamination cells 26, which is the case here.

8th shows a flowchart of an exemplary embodiment of the approach presented here as a method 100 for identifying soiling 14 on a lens unit 12 of an optical detection unit 10, in particular a camera 10, preferably an agricultural working machine. The method 100 comprises a step of receiving 102 images 16, 16a-c, in particular of field sections 18 of an agricultural area, which were recorded by means of the optical detection unit 10, in particular camera 10, preferably the agricultural working machine. The method 100 also includes a step of segmenting 104 image cells 22a-c, which fall below a defined gray threshold value, of the received images 16, 16a-c by means of a computing unit in order to identify potential contamination cells 24a-c in the received images 16, 16a-c to identify. The method 100 also includes a step of determining 106 the matching contamination cells 26, which are the same in the received images 16, 16a-c, by means of the computing unit in order to identify them as contamination 14 on the lens unit 12.

Furthermore, the method 100 includes an optional step of outputting 108 a signal, in particular a control signal for controlling a cleaning device, in particular the optical detection unit 10, and/or an information signal, depending on the identified contamination 14 on the lens unit 12.

QUOTES INCLUDED IN DESCRIPTION

This list of documents cited by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Patent Literature Cited

• DE 102016118335 A1 [0004]
• DE 102018209020 A1 [0005]

Claims (13)

Method (100) for identifying soiling (14) on a lens unit (12) of an optical detection unit (10), in particular a camera (10), preferably an agricultural working machine, with the steps: - Receiving (102) of images (16, 16a-c), in particular of field sections (18) of an agricultural area, which were recorded by means of the optical recording unit (10), in particular camera (10) of the agricultural working machine; - Segmenting (104) of image cells (22a-c), which fall below a defined gray threshold value, of the received images (16, 16a-c) by means of a computing unit in order to detect potential contamination cells (24a-c) in the received images (16, 16a -c) identify; and - determining (106) the matching contamination cells (26) which are the same in the received images (16, 16a-c) by means of the computing unit in order to identify them as contamination (14) on the lens unit (12). Method (100) according to claim 1 , characterized in that the segmented image cells (22a-c) are the potential contamination cells (24a-c). Method (100) according to claim 1 , characterized in that in the step of segmentation (104) the images (16, 16a-c) are superimposed with a grid (28), the grid cells (30) of which comprise at least two, in particular a large number of, image cells (22a-c), wherein the segmented image cells (22a-c) are linked to the grid (30) and grid cells (30) with a degree of coverage greater than or equal to a defined coverage threshold value of segmented image cells (22a-c) are identified as the contamination cells (26). Method (100) according to one of the preceding claims, characterized in that in the step of recognition (106) the matching contamination cells (26) are identified as contamination (14) only above a defined contamination cell threshold value. Method (100) according to one of the preceding claims, characterized in that the received images (16, 16a-c) are a defined selection of an image series captured directly one after the other by the optical capture unit (10). Method (100) according to one of the preceding claims, characterized in that the defined selection - all images (16, 16a-c) of the image series, or - a defined number of images (16, 16a-c) of the image series captured immediately one after the other, or - includes every xth image (16, 16a-c) of the series of images. Method (100) according to one of the preceding claims, characterized by a step of outputting (108) a signal, in particular a control signal for controlling a cleaning device, in particular the optical detection unit (10), and/or an information signal, depending on the identified contamination (14) on the lens unit (12). Arithmetic unit which is set up to carry out and/or to control the steps of a method (100) according to one of the preceding claims. Optical detection unit (10), in particular camera (10) for detecting a field section (18) of an agricultural area in order to capture images of field sections, with a lens unit (12) and a computing unit claim 8 . Optical detection unit (10) after claim 9 , characterized by a cleaning device which can be controlled as a function of the identified contamination (14). Agricultural working machine, in particular a field sprayer, with an agricultural working tool, in particular a spraying device, and an optical detection unit (10). claim 9 or 10 . Computer program that is set up to carry out the steps of a method (100) according to one of Claims 1 until 7 perform and/or control when the computer program is run on a computer. Machine-readable storage medium on which the computer program claim 12 is saved.

Images