EP4088223A1

EP4088223A1 - Method for generating a plurality of annotated images

Info

Publication number: EP4088223A1
Application number: EP20811285.4A
Authority: EP
Inventors: Markus Hoeferlin
Original assignee: Robert Bosch GmbH
Current assignee: Robert Bosch GmbH
Priority date: 2019-11-25
Filing date: 2020-11-20
Publication date: 2022-11-16
Also published as: WO2021105019A1; DE102019218189A1

Abstract

A method (100) for generating a plurality of pixel-by-pixel annotated images of a field in which plants grow, has the following steps: capturing (S102) the plurality of images of the field using an image capture means, the images overlapping one another; substantially simultaneously requesting (S104) positional information about the position at which each of the images was captured in the field; arranging (S106) the plurality of images in a more global context as a stitched image, using the positional information; annotating (S108) pixels of the stitched image with annotation information containing at least classification information that indicates a class of the plant; and transferring (S110) the annotation information to the pixels of the plurality of images, using the positional information.

Description

Method for generating a large number of annotated images

description

Technical area

The present invention relates to a method for generating a plurality of annotated images, and in particular a method for generating a plurality of pixel-by-pixel annotated images of a field on which plants are growing.

background

The classification of Plants are essential in a field. An incorrect determination of the class to which an individual plant belongs would lead, for example, to the weeding of a useful plant or the counting, measurement and / or treatment of a weed. It should be noted that “class” is to be understood in the following as a class of a classification and not as a class in the sense of biosystematics.

For the training of a classifier for plant classification, supervised learning methods are predominantly used, such as in "Plant classifications system for crop / weed discrimination without segmentation"; Applications of Computer Vision (WACV), 2014 IEEE Winter Conference, 2014, 1142-1149; "Support Vector Machines for crop / weeds Identification in maize fields"; Expert Systems with Applications, 2012, 39, 11149-11155; and "Evaluation of Features for Leaf Classification in Challenging Conditions"; Applications of Computer Vision (WACV), 2015 IEEE Winter Conference, 2015, 797-804; disclosed.

For the monitored training of the classifier, large amounts of learning or training data are required in the form of images that are annotated or classified with annotation information. This annotation information can have different information. The annotation information preferably includes a classification that shows the pixels in the images that a plant to be classified, ie useful plant or weed, shows correct class too. The annotation information can also have further information, such as row information. The annotation information is also referred to as the ground truth.

All of today's methods for obtaining ground truth are very time-consuming. On the one hand, a large number of images have to be annotated individually (semi-) manually. On the other hand, the annotation information is required at the pixel level, which is required e.g. for semantic segmentation using neural networks. Tools or programs are available for annotating images with annotation information, but they do not solve the problem that many pixels in a large number of images have to be annotated in a time-consuming manner.

5 shows a tool 50 for annotating images at the pixel level with the annotation information, the annotation information in particular specifying classification information of a plant. The annotation information can be, for example, species, genus or simply a distinction between useful plants and weeds. Each captured image must be annotated separately with annotation information, which is shown by a controller 52. In the image section shown in FIG. 1, the plants shown are annotated with annotation information that distinguishes between two classes, useful plant (“bed”) 54 and weeds (“other”) 56. Despite semi-automatic annotation support, such as following the annotations over several images or frames of a video to be annotated, semi-automatic segmentation of the plants, etc., the effort is very high due to the large number of images to be annotated. The effort for annotating images also increases sharply in crops with smaller plant spacing due to obscurations.

In “Multimodal Image Stitching Algorithm for Weed Control Applications in Organinc Farming” by T. Holtdorf et.al., SAI Computing Conference 2016, July 13-15, 2016, London, it is proposed to merge several images into a panorama image and only that Annotate panoramic image. The images are put together using key points that are determined by an algorithm. As a result, an annotated panoramic image can be obtained with the one described.

It is therefore the object of the present invention to provide a method which reduces the effort involved in generating a large number of images annotated pixel by pixel. Brief description of the figures

The present invention will be clarified by the following description with reference to the accompanying figures. Show it:

1A shows a picture of the field on which arugula grows, in the visible area;

1B shows an image in the infrared range of a field on which arugula is growing;

2 shows a flow chart of the method according to the invention;

3 shows a stitched picture composed of a plurality of pictures;

4A shows a tool for annotating the deleted image;

4B shows an image annotated with row information; and

5 shows a tool for annotating plant data according to the prior art;

Detailed description of the invention

In agriculture, seeds are sown in a field from which crops grow. A field can be understood to be a delimited area of land for the cultivation of useful plants or a part of such a field. A useful plant is understood to be an agriculturally used plant which itself or its fruit is used, e.g. as food, feed or as an energy plant. The seeds and consequently the plants are primarily arranged in rows, it being possible for objects to be present between the rows and between the individual plants within a row. However, the objects are undesirable because they reduce the yield of the plants or represent a disruptive influence during cultivation and / or harvesting. An object can be understood to mean any plant that is different from the useful plant, or any object. Objects can in particular be weeds, woods and stones.

1A shows a picture of a field on which arugula is grown in rows, in the visible area. In the picture it can be clearly seen that the leaves of each Some of the rocket plants overlap strongly. As shown in FIG. 1B, the plants can be very robustly segmented from the subsurface by an additionally recorded image of the field in the infrared range and by using various segmentation methods (eg NDVI index, ExG index, ExG - ExR index, etc.) However, it is still necessary that areas between the individual plants have to be annotated manually in a laborious manner with information that indicates whether a pixel shows a groove plant, an object or the soil.

This information is referred to below as classification information. In addition, a pixel can be annotated with further information. In the following, the information with which a pixel is annotated is referred to as annotation information.

The method 100 according to the invention is intended to reduce this effort when annotating the multiplicity of images with the annotation information pixel by pixel. The individual steps of the method are shown in the flow chart in FIG. 2 and are described in detail.

In a first method step S102, the multiplicity of images of a surface of the field is acquired by an image acquisition means. The image capturing means is a camera, such as a CCD camera, a CMOS camera, etc., which captures an image in the visible range (see e.g. Fig. 1A) and provides it as RGB values or as values in another color space. The image acquisition means can, however, also be a camera which acquires an image in the infrared range (see e.g. Fig. 1B). An image in the infrared range is particularly suitable for capturing plants, as the reflection of the plants is significantly increased in this frequency range. The image acquisition means can also be, for example, a mono, RGB, multispectral, hyperspectral camera. In addition, other data can be acquired using sensors such as 3D sensors, etc. It is possible for several image acquisition means to be present on the vehicle and for several images to be acquired essentially synchronously by the different image acquisition means and data from different sensors.

The field on which the plants and objects are present is detected by the image capturing means during a drive over with a vehicle on which the image capturing means is attached. The image acquisition means can be attached to a vehicle specially provided for this purpose. However, the image acquisition means can also be attached to an agricultural vehicle, such as a tractor, a trailer, etc., or an aircraft, such as a drone, for example. The image capturing means is attached to the vehicle in such a way that an image sensor of the image capturing means is in the Is substantially parallel to a surface of the field. The image sensor of the image acquisition means can, however, also be inclined to the surface of the field.

The vehicle to which the image capturing means is attached drives or flies the field and the image capturing means captures the images at a predetermined time interval. The images are preferably captured in such a way that they overlap. For this reason, the image capturing means captures several images per second during the passage, as a result of which the images strongly overlap at a low passage speed. In this way, images of a single plant are recorded from several perspectives, so that consequently a large number of images with different views of the same plant are available as learning data. In this way, a recognition accuracy of the classifier is improved. The multitude of images can also be recorded as video. The vehicle can also drive off / fly off the field autonomously or remotely.

The images are then stored in a memory and are then available for further processing. For the purpose of processing, the images can be transferred to a separate computing unit that is not attached to the vehicle. However, the images can also be further processed in a computing unit that is mounted on the vehicle.

The large number of images should preferably show all growth phases of the useful plant and, if possible, all occurring weeds with all possible biological growth morphologies, since different circumstances such as water, heat, soil, nutrients, wind, etc., the growth and / or the appearance of the useful plants and the Can affect weeds. By using these different learning data, the accuracy of the results of a classifier for plant recognition can be increased further.

For this purpose, a crossing and acquisition of an additional multiplicity of images of the same field on which the same plants are growing can or should be repeated at different times in order to record the above-mentioned growth states of the individual useful plants and weeds. In this way, a larger number of learning data is available for the different stages of the plants, so that a recognition accuracy of the classifier can be improved. The subsequent step S104 is carried out essentially synchronously with step S102. In step S104, position information is acquired using position detection means. The position detection means detects the position information using high-precision GPS, but can alternatively also detect it using, for example, odometry, visual odometry, and other tracking methods. The position information is given as world coordinates, but can also be given as field coordinates, longitude + latitude, etc., for example. The position information is then correlated with the image recorded in step S102, so that the position in the field at which the image is recorded can be precisely determined.

For this purpose, the position information obtained is assigned to a point in the center of the image. The position information can also be assigned to another point in the image, e.g. a corner point. It should be noted that distances between the mounting position of the image capturing means and the mounting position of the position capturing means on the vehicle must be taken into account when correlating the position information with the captured image. The spatial extent of the image on the field in an X and a Y direction can then be determined using an image angle of the image capturing means and the distance of the image capturing means to the floor surface. If the image sensor of the image acquisition means is inclined to the surface of the field, this inclination must also be taken into account when calculating the spatial extent of the image on the field. In this way it is possible to determine the section of the field shown by the picture. Taking into account the resolution of the image, position information and consequently a position on the field can also be assigned to each of the pixels of the image. This procedure can also be applied to images that are acquired by another image acquisition means and to data that are acquired by different sensors.

In the following step S106, the images that are captured during the same passage and for which the position information was obtained in S104 are combined to form a stitched image using the position information in order to display a coordinate system that is more global than the pixel coordinate system on the image level Has resolution to create. A position in the field that is recorded due to the movement of the vehicle and the rapid repetition rate when recording the images in different images from different perspectives has the same position information in all images. A stitched image as obtained after performing step S106 is shown in FIG. 3. The stitched picture in Fig. 3 was under Using high-precision GPS put together from several hundred images.

Due to the overlap between the images, only a part of the individual images is shown in the stitched image. The theoretical position of the pictures only partially shown in the stitched picture can, however, be determined exactly on the basis of the available position information.

In step S108, the stitched image previously generated in S106 is annotated pixel by pixel with the annotation information, the amount of pixels to be annotated due to the overlap of the images and the representation as a stitched image compared to the case in which all images are to be annotated individually pixel by pixel, is significantly reduced. 4A shows a tool with which the panoramic image is annotated. The useful plants are identified by a marking 42. The pixels belonging to the useful plant around those with marking 42 can be automatically recognized and annotated by the tool and classification information can be added for the individual pixels. In addition to the classification information, as shown on the right in Fig. 4B, the panoramic image can be annotated with row information, i.e. information about the course of a row of plants 44. The annotation information of the pixels of a row of plants 44 thus has the row information in addition to the classification information.

In the following step S110, the annotation information is again transferred to the individual images using the position information, which were captured in step S102 and from which the stitched image was combined in step S106. In this way, from the individual “global” stitched image that was annotated with the annotation information, a multiplicity of pixel-by-pixel annotated images can be obtained that also have the annotation information. It should be noted that the annotation information can also be transferred to images that are only partially shown in the stitched image, since the pixels of the images only partially shown in the stitched image also have the position information. As a result, the amount of annotated image material can be increased compared to the representation as a stitched image.

Various segmentation methods can then be applied to the individual images in order to additionally increase the level of detail of the annotation information.

The described transmission of the annotated annotation information is not limited to the plurality of images captured by the same image capturing means during the same passage, but can be to the various other images to be annotated and Data collected on the same field can be applied. The prerequisite for this, however, is that these images and data contain the position information and that the images show the same plants during the same cultivation period.

On the one hand, using the position information, the annotation information can also be transferred to the multitude of images showing the same plants (or the same annotation object) at a different point in time with varying perspectives, plant growth stages, lighting conditions, weather conditions,

Shows wind conditions, plant conditions (sick, healthy, etc.), drought conditions, etc. As a result, a new creation of a panorama image and a new time-consuming annotation can be avoided. As a result, the outlay for generating a large number of images annotated pixel by pixel, which are recorded at a different point in time, is also reduced.

On the other hand, using the position information, the annotation information can also be transferred to a large number of images that are recorded by another image recording device during the same passage. The annotation information can, however, also be transferred to a large number of images that are recorded by the other image recording means during a passage at a different point in time. The prerequisite for transmitting the annotation information is again that the data to which the annotation information is to be transmitted has the position information. This procedure can also be applied to other data collected by sensors. As a result, the effort for generating a multiplicity of pixel-wise annotated images of a different type and of further data is also reduced. Another advantage of transmitting the annotation information using the position information is that the resolution and the angle of view of the other image capture means do not have to be taken into account.

Finally, it should be noted that the present invention uses one or more information sources for annotating images with annotation information pixel by pixel in order to have to annotate each plant only once manually in a larger context. By using the additional position information, the images of the same plant or a part of it, which are not shown in the larger context, can then be automatically annotated at the pixel level. With the aid of the method according to the invention, a large number of images that have annotation information at the pixel level can be obtained with very little effort. This large number of images, which have the annotation information at the pixel level, can then be used for a classifier, such as. For example, to train a neural network, etc., which is to be used for the classification of plants. In this way, the regulation of weeds, the counting and measuring of plants for phenotyping, the selective harvest, the selective application of herbicides, fungicides, pesticides and insecticides and much more. improved because a larger amount of images and / or data is available for training the classifier.

It should also be noted that the method according to the invention is not limited to images of a field on which plants are growing. The method can be used for various applications in which a large number of images that overlap one another are to be annotated with information on a pixel-by-pixel basis.

Claims

Expectations

1. A method (100) for generating a plurality of pixel-by-pixel annotated images of a field on which plants are growing, the method comprising the following steps:

Capturing (S102) the plurality of images of the field by means of image capturing means, the images each overlapping; essentially in sync with one

Obtaining (S104) positional information on the position at which the images are each captured on the field;

Arranging (S106) the plurality of images in a more global context using the position information as a stitched image;

Annotating (S108) pixels of the stitched image with annotation information including at least one classification information indicating a class of the plant; and

Transferring (S110) the annotation information to the pixels of the plurality of images using the position information in order to generate a plurality of images annotated pixel by pixel from a field.

2. The method (100) according to claim 1, wherein the annotation information can be transmitted using the position information to the pixels of a plurality of images of the field acquired at another point in time using the position information.

3. The method according to claim 1 or 2, wherein during the step of capturing (S102) a further plurality of images are captured by means of an image capturing means of a different type and / or further data by means of a sensor, and the annotation information on the pixels of the image capturing means of another type captured plurality of images and / or the data captured by the sensor can be transmitted using the position information.

4. The method according to any one of the preceding claims, wherein the annotation information additionally has row information.

5. The method according to any one of the preceding claims, wherein the plurality of images is captured as video.

6. The method according to any one of the preceding claims, wherein the plurality of pixel-wise annotated images is used to train a classifier.

7. Computer program which is designed to carry out the method according to one of the preceding claims.

8. Machine-readable storage medium on which the computer program according to claim 7 is stored.

9. Computing unit for generating a plurality of pixel-by-pixel annotated images of a field on which plants grow, the computing unit being designed to carry out the following steps:

Receiving the plurality of images of the field captured by image capture means, the images each overlapping; essentially in sync with one

Receiving position information obtained about the position at which the images have been captured in each case on the field;

Arranging the plurality of images in a more global context using the position information as the stitched image;

Annotating pixels of the stitched image with annotation information which has at least one classification information indicating a class of the plant; and

Transferring the annotation information to the pixels of the plurality of images using the position information in order to generate a plurality of pixel-by-pixel annotated images of a field.

10. Agricultural working machine with a computing unit according to claim 9.