EP3718045A1 - Method, device and computer program for determining a distance to an object - Google Patents

Method, device and computer program for determining a distance to an object

Info

Publication number
EP3718045A1
EP3718045A1 EP18807901.6A EP18807901A EP3718045A1 EP 3718045 A1 EP3718045 A1 EP 3718045A1 EP 18807901 A EP18807901 A EP 18807901A EP 3718045 A1 EP3718045 A1 EP 3718045A1
Authority
EP
European Patent Office
Prior art keywords
machine learning
learning system
images
distance
determining
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP18807901.6A
Other languages
German (de)
French (fr)
Inventor
Johannes Maximilian DOELLINGER
Joerg Wagner
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Robert Bosch GmbH
Original Assignee
Robert Bosch GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Robert Bosch GmbH filed Critical Robert Bosch GmbH
Publication of EP3718045A1 publication Critical patent/EP3718045A1/en
Withdrawn legal-status Critical Current

Links

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C3/00—Measuring distances in line of sight; Optical rangefinders
    • G01C3/02—Details
    • G01C3/06—Use of electric means to obtain final indication
    • G01C3/08—Use of electric radiation detectors
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/02—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness
    • G01B11/026—Measuring arrangements characterised by the use of optical techniques for measuring length, width or thickness by measuring distance between sensor and object
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C11/00—Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
    • G01C11/04—Interpretation of pictures
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/145—Illumination specially adapted for pattern recognition, e.g. using gratings
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Definitions

  • the invention relates to a method for determining a distance to an object. Likewise, the invention relates to a computer program and apparatus each adapted to carry out the method.
  • DE 102011081384 B4 discloses a method for distance determination with the following steps: Determining a time offset between a change in a radiation characteristic of a headlamp of a vehicle and an effect of the change of the radiation pattern on a Schmbe range of an image. Determining a distance to an object imaged by the image area in the environment of the vehicle based on the time offset.
  • DE 102011005368 A1 discloses a driver assistance system for maneuvering and / or parking a vehicle with a video camera.
  • a video image is detected with an object located in the vicinity of the vehicle, and this video image is combined with other determined information (e.g.
  • the method having the features of independent claim 1 and the device have the following advantages: A more reliable determination of a distance to an object from a detected image is achieved, wherein the determination is not adversely affected by a reflection. Reflecting objects on smooth or mirrored surfaces can cause distance measurements based on images to produce incorrect results. The method and apparatus therefore provide a simple and cost effective way of making a more robust and reliable range finding that is unaffected by reflections or deceived. A further advantage is that the machine learning system does not require the development of complex image processing algorithms which firstly detect an object in an image, secondly determine their associated distance or associated depth information from the image and are thereby robust to reflections.
  • the machine learning system independently develops training data provided by a methodology to determine the distance to an object and to recognize reflections in the captured images and to take into account accordingly. As a result, during the determination of the distance, the method and the device are not deceived by reflections and therefore determine reliable distances.
  • the invention relates to a method for determining a
  • the method comprises the following steps:
  • a polarization filter arranged in front of a camera is used.
  • Determining the distance based on a second of the at least two captured images by means of a machine learning system.
  • the machine learning system also uses the first of the at least two captured images during the determination of the distance to disregard reflection in one of the at least two captured images when determining the distance.
  • determining a distance to an object it can be understood that a distance, in particular a distance, between a predefinable Reference point and the object is determined.
  • the reference point preferably corresponds to the position at which the camera is located.
  • the reference point can also be in front of or behind the camera. It should be noted that the method is independent of the position of the reference point, since depending on the selected reference point, the machine learning system can be adapted appropriately.
  • a reflection can be understood as any type of mirror image of an object that occurs. By way of example, by means of a reflecting surface, the mirror image of an object or a distortion and / or a rotation of the mirror image can occur, which are referred to below by the term reflection.
  • the at least two recorded images can either simultaneously or immediately after each other, in particular at a predeterminable time, who recorded the.
  • the images can also be captured by means of several differently positioned cameras.
  • the advantage according to this method is that the machine learning system is provided with sufficient information by the at least two captured images, which are differently filtered, that the reflection does not affect the distance determination.
  • one of the captured images is unfiltered and the other image is filtered with a polarizing filter or both have been detected filtered with a polarizing filter, wherein the polarizing filter each have a different orientation of the polarization planes.
  • the machine learning system is taught in such a way that the machine learning system determines the distance on the basis of the second of the at least two captured images. Furthermore, in this embodiment, the machine learning system can be taught in such a way that the machine learning system does not consider the reflection from the first of the at least two captured images when determining the distance.
  • the advantage of this method is that by learning the maschi nelle learning system, this independently learns a distance determination. Therefore, no complex algorithms have to be developed for this purpose in order to solve this complex image processing task. Furthermore, this is the learned machine Learning system in operation more computationally efficient or faster than conventional image processing algorithms for distance measurement based on captured images. For learning the machine learning system is similar to optimizing the academic learning system in all respects, and further, machine learning systems, especially neural networks, can achieve higher performance by concatenating the computing operations in the machine learning system.
  • the machine learning system is additionally trained to determine an object classification on the basis of the second of the at least two captured images and not to consider the reflection based on the first of the at least two captured images when determining the object classification. It is also advantageous if the machine learning system determines the object classification when determining the distance.
  • the maschi nelle learning system is additionally trained to an optical flow to ermit stuffs and is trained to reflect the reflection when determining the optical flow. Furthermore, the machine learning system determines an optical flow on the basis of the images acquired at the predeterminable successive time points.
  • a quantity in particular a vector, can be understood, which characterizes a movement of a point in the image, for example a speed and / or a direction of this point relative to a selected reference point. It is advantageous if the reference point of the optical flow and the reference point of the determination of the distance are at an identical position. It is advantageous if the captured images are stored and the machine learning system is learned by means of the stored captured images. This has the advantage that the method for determining the distance with the captured images is further improved in order to achieve a higher accuracy of the Ab measurement.
  • the actuator may comprise an at least partially autonomous machine, such as e.g. be a robot or a vehicle. It is also advantageous if the machine learning system is a deep neural network, in particular a "Revolutionary Neural Network” or a "Recurrent Neural Network”.
  • the invention in another aspect, relates to a device adapted to carry out the method according to the first aspect of the invention.
  • the device comprises the following features: At least one camera for capturing the at least two images and at least one polarization filter.
  • the polarization filter is arranged in front of the camera and is used for detecting the first of the at least two images.
  • the device also includes the machine learning system.
  • the device also the actuator, in particular a at least teilautonomome machine such. a robot or a vehicle.
  • the machine learning system is a deep neural network, in particular a "convolutional neural network” or a "recurrent neutral network”.
  • the advantage of the device is that by means of the particular linear polarization filter, reflections which are unpolarized are filtered out so that the captured image is an at least partially reflection-free image.
  • reflections which are unpolarized are filtered out so that the captured image is an at least partially reflection-free image.
  • the maschi nelle learning system can be carried out on this a precise distance determination, which is not affected by the reflection.
  • the polarizing filter comprises a plurality of consecutively arranged polarizing filter, wherein one of the polarization planes of the successively arranged polarizing filter is not perpendicular to the respective polarization planes of the successively arranged polarizing filter.
  • the advantage is that the polarization-filtered image can contain more reduced reflections.
  • the polarization filter is a circular polarization filter. This has the advantage that circular polarizing filters have higher compatibility with commonly used digital cameras, e.g. have an autofocus or an automated exposure meter.
  • the polarization filter additionally comprises a color filter.
  • the advantage here is that polarizing filters typically filter out the blue portion of the light, as it is highly unpolarized by the scattering of the light in the atmosphere. Therefore, the color filter can counteract the increased reduction of the blue component and preserve the color neutrality of the images.
  • the invention relates to a computer program which is set up to carry out one of the above-mentioned methods, that is to say comprises instructions which cause a computer, one of the above-mentioned
  • the invention relates to a machine-readable storage element on which the computer program is stored.
  • Fig. 1 is a schematic representation of a device for determining a
  • Fig. 2 is a schematic representation of an embodiment of a method for determining a distance to an object.
  • FIG. 1 shows a schematic representation of an exemplary device (10) for a reliable and robust determination of a distance to an object on the basis of a captured image.
  • the determination of the distance is not impaired by unwanted reflections, in particular by mirror images.
  • the device (10) comprises a machine learning system (11) which determines a distance to an object on the basis of at least two captured images (12a, 12b).
  • the machine learning system (11) is preferably a "convolutional neural network".
  • At least one of the at least two captured images (12b) is detected by a polarization filter (13) arranged in front of a camera.
  • the polarizing filter (13) By means of the polarizing filter (13), the unwanted reflections on smooth or reflective surfaces, such as on windows or on water surfaces can be suppressed.
  • the captured images (12a, 12b) are filtered differently.
  • the different filtered images (12a, 12b) are then used with the below-mentioned method of the machine learning system (11) to determine the distance.
  • the machine learning system (11) recognizes a reflection on the basis of the two differently filtered images and does not take these into account when determining the Distance.
  • the term "distance to an object” can be understood to mean that the machine learning system (11) determines a distance between a reference point and the object.
  • the reference point may be, for example, the position of the camera. Alternatively or additionally, a plurality of differently placed cameras can be used to capture the images. For this, the reference point must be chosen accordingly. It is also conceivable that the reference point lies in front of or behind or to the side of the camera, e.g. When the camera is placed on the windshield of a vehicle, the reference point near the bumper can be selected.
  • the machine learning system (11) can determine the distance to the object.
  • the machine learning system (11) uses another captured image (12b) to disregard reflection in the images while determining the distance. Because through the additional image (12b), the machine learning system (11) can be provided with further information so that the ma chine learning system (11) can detect reflection in the images (12a, 12b) used. This has the advantageous effect that the reflection is not included in the determination of the distance and thus erroneous result of the distance determination can be avoided the.
  • the result of the machine learning system (11) can optionally be used by a control unit of the device (10) to determine a control variable (14) depending on this result.
  • the control variable (14) can be used to control an actuator (17) who the.
  • the actuator (17) may be an at least partially autonomous machine, in particular a robot or a vehicle.
  • a parking operation of the at least partially autonomous machine with the control variable (14) can be carried out.
  • the actuator (17) can also readjust alignment of a polarization plane of the Polari sationsfilters (13) depending on the control variable (14), so that the result of the machine learning system (11) can be checked with another differently filtered image.
  • the machine learning system (11) may also generate a distance image.
  • the distance image can be an image in which the determined information of the machine Learning system (11), for example, the distance to the object, superimposed on a section of ei nes of the captured images (12a, 12b) are output. For example, each pixel can be assigned one of the determined information.
  • the machine learning system (11) obtains a plurality of captured images (12a, 12b), the images all being detected by means of different polarization filters. That one polarization plane of the polarization filters is oriented differently to the polarization planes of the other polarization filters so that differently filtered images can be captured and used for distance determination.
  • the device (10) comprises a computing unit (15) and a memory element (16) on which a computer program is stored.
  • the computer program may include instructions causing the computer program to run at e.g. the computing unit (15) is carried out one of the embodiments of the below-mentioned method.
  • Figure 2 shows a schematic representation of an embodiment of a method (20) for determining a distance to an object based on the captured images (12a, 12b).
  • the method (20) begins with step 21.
  • the machine learning system (11) of the device (10) is provided with a training data set, for example, from a database.
  • the training data set may include a plurality of training images, each of which may be real captured images or images generated by a computer.
  • the training images can be with or without reflections.
  • each of multiple images represent a same scene, but these are each Weil filtered differently, preferably with the polarizing filter (13) of Figure 1.
  • the training data are labeled by means of a distance value and be preferred with a note whether a reflection in the Training image is present or not.
  • the distance values of the training images can be determined by means of a "ground truth" method.
  • the labels can be used to train the machine learning system (11) in a more targeted way for distance determination, neglecting the reflections.
  • the machine learning system (11) is taught such that the machine learning system determines a distance to an object in the acquired image as a function of a captured image and taking into account a further acquired image.
  • the further acquired image is in particular a filtered image by means of a polarization filter or by means of a differently oriented polarization plane of the polarization filter.
  • the machine learning system can also be taught in such a way that the machine learning system, on the basis of the further provided image, does not consider reflections in the images for determining the distance to an object.
  • a gradient descent method is used for determining the parameter values of the machine learning system (11).
  • the gradient descent method can be applied to a cost function.
  • the cost function may be dependent on the parameters of the machine learning system (11) and preferably on the lab of the training data used.
  • the machine learning system (11) can also be taught in step 21 that the machine learning system (11) can determine an object detection, in particular an object classification, on the basis of the provided images.
  • the training images are preferably additionally or alternatively labeled, which characterize the object classes.
  • the machine learning system (11) can be taught in such a way that the machine
  • step 22 follows.
  • step 22 at least two images are acquired, in particular at a predefinable time.
  • a first of the at least two images is detected by means of a polarizing filter, which is arranged in front of the camera, filtered.
  • This polarization filter preferably has the same orientation from the plane of polarization as the polarization filters used for detecting the images of the training data set. This has the advantageous effect that the captured images are detected similarly to the training images of the machine learning system (11), whereby a higher accuracy of the determination of the distance to an object can be achieved.
  • the machine learning system (11) determines the distance to the object from one of the at least two captured images.
  • the further captured image in particular the image filtered by the polarization filter, is used during the determination of the distance to an object by the machine learning system that reflections in one of the two images are not taken into account for determining the distance to an object.
  • an object detection can also be determined in step 22 on the basis of the captured images (12a, 12b) become.
  • the advantageous effect here is that reflections in the images can be determined by the different filtered captured images and are not taken into account in the determination of the object detection by means of the machine learning system (11).
  • a reliable object detection can be achieved, because, for example, a reflection of aêt in a shop window can lead to erroneous object detection or classification.
  • step 23 follows optionally.
  • a control variable (14) for controlling the actuator (17) can be determined.
  • the actuator (17) in particular a machine that is at least partly autonomous, such as a robot or a vehicle, can, for example, execute a movement or driving maneuver depending on the control variable.
  • the machine learning system (11) can also be taught in step 21 such that the machine learning system (11) can determine an optical flow based on a sequence of captured images at different predefinable successive times .
  • the machine learning system (11) for this must be trained as well that reflections in the images are not taken into account for the determination of opti's flow.
  • the training images for learning the machine learning system (11) are additionally or alternatively labeled with labels that characterize the optical flow.
  • this can be learned with this Machine learning system (11) in step 22 of a plurality of different captured images, which were detected at predetermined times, the optical flow of an object determine.
  • step 21 can also be repeated several times in succession until a predeterminable sufficiently high accuracy of the distance measurement is achieved.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Electromagnetism (AREA)
  • Artificial Intelligence (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Length Measuring Devices By Optical Means (AREA)
  • Measurement Of Optical Distance (AREA)
  • Image Analysis (AREA)

Abstract

The invention relates to a method (20) for determining a distance to an object, comprising the following steps: capturing at least two images (12a, 12b), a polarization filter (13) arranged in front of a camera being used for capturing at least a first of the at least two images (12b); determining the distance on the basis of a second of the at least two captured images (12b) by means of a machine learning system (11), the machine learning system (11) also using the first of the at least two captured images (12a) during the determination of the distance, in order to disregard a reflection in one of the at least two captured images (12a, 12b) in the determination of the distance. The invention further relates to a computer program and to a device for carrying out the method (20) and to a machine-readable storage element (16) on which the computer program is stored.

Description

Beschreibung  description
Titel title
Verfahren, Vorrichtung und Computerprogramm zum Ermiteln eines Abstandes zu einem Objekt  Method, apparatus and computer program for determining a distance to an object
Technisches Gebiet Technical area
Die Erfindung betrifft ein Verfahren zum Ermitteln eines Abstandes zu einem Ob jekt. Ebenso betrifft die Erfindung ein Computerprogramm und eine Vorrichtung, die jeweils eingerichtet sind, das Verfahren auszuführen. The invention relates to a method for determining a distance to an object. Likewise, the invention relates to a computer program and apparatus each adapted to carry out the method.
Stand der Technik State of the art
Die DE 102011081384 B4 offenbart ein Verfahren zur Abstandsbestimmung mit den folgenden Schritten: Ermitteln eines zeitlichen Versatzes zwischen einer Ver änderung einer Abstrahlcharakteristik eines Scheinwerfers eines Fahrzeugs und einer Auswirkung der Veränderung der Abstrahlcharakteristik auf einen Bildbe reich eines Bildes. Bestimmen eines Abstands zu einem durch den Bildbereich abgebildeten Objekt in dem Umfeld des Fahrzeugs, basierend auf dem zeitlichen Versatz. DE 102011081384 B4 discloses a method for distance determination with the following steps: Determining a time offset between a change in a radiation characteristic of a headlamp of a vehicle and an effect of the change of the radiation pattern on a Bildbe range of an image. Determining a distance to an object imaged by the image area in the environment of the vehicle based on the time offset.
Die DE 102011005368 Al offenbart ein Fahrerassistenzsystem zum Rangieren und/oder zum Einparken eines Fahrzeuges mit einer Videokamera. Es wird ein Videobild mit einem Objekt, das sich in der Umgebung des Fahrzeuges befindet, ermittelt und dieses Videobild wird mit weiteren ermittelten Informationen (z.B. DE 102011005368 A1 discloses a driver assistance system for maneuvering and / or parking a vehicle with a video camera. A video image is detected with an object located in the vicinity of the vehicle, and this video image is combined with other determined information (e.g.
ein Abstand zu dem Objekt) angereichert. a distance to the object) enriched.
Vorteile der Erfindung Advantages of the invention
Das Verfahren mit den Merkmalen des unabhängigen Anspruchs 1 und die Vorrichtung mit den Merkmalen des unabhängigen Anspruchs 6 haben demgegenüber die folgenden Vorteile: Es wird eine zuverlässigere Ermittlung einer Distanz zu einem Objekt anhand eines erfassten Bildes erzielt, wobei die Ermittlung nicht durch eine Reflexion negativ beeinflusst wird. Reflexionen von Objekten auf glatten bzw. spiegelnden Oberflächen können dazu führen, dass Entfernungsmessungen ausschließlich anhand von Bildern falsche Ergebnisse liefern. Das Verfahren und die Vorrichtung stellen daher eine einfache und kostengünstige Möglichkeit dar, eine robustere und zuverlässigere Entfernungsmessung durchzuführen, die nicht durch Reflexionen beeinträchtigt oder getäuscht wird. Ein weiterer Vorteil ist, dass durch das maschinelle Lernsystem keine komplexen Bildverarbeitungsalgorithmen entwickelt werden müssen, die als Erstes ein Objekt in einem Bild detektieren, als Zweites deren zugehörigen Abstand bzw. eine zugehörige Tiefeninformation aus dem Bild ermitteln und dabei Robust gegenüber Reflexionen sind. Das maschinelle Lernsystem entwickelt eigenständig durch bereitgestellte Trainingsdaten eine Methodik, um den Abstand zu einem Objekt zu ermitteln und auch Reflexionen in den erfassten Bildern zu erkennen und entsprechend zu berücksichtigen. Dies führt dazu, dass das Verfahren und die Vorrichtung während der Ermittlung des Abstandes nicht durch Reflexionen getäuscht werden und deshalb zuverlässiger Abstände ermitteln. The method having the features of independent claim 1 and the device By contrast, the features of independent claim 6 have the following advantages: A more reliable determination of a distance to an object from a detected image is achieved, wherein the determination is not adversely affected by a reflection. Reflecting objects on smooth or mirrored surfaces can cause distance measurements based on images to produce incorrect results. The method and apparatus therefore provide a simple and cost effective way of making a more robust and reliable range finding that is unaffected by reflections or deceived. A further advantage is that the machine learning system does not require the development of complex image processing algorithms which firstly detect an object in an image, secondly determine their associated distance or associated depth information from the image and are thereby robust to reflections. The machine learning system independently develops training data provided by a methodology to determine the distance to an object and to recognize reflections in the captured images and to take into account accordingly. As a result, during the determination of the distance, the method and the device are not deceived by reflections and therefore determine reliable distances.
Offenbarung der Erfindung Disclosure of the invention
In einem ersten Aspekt betrifft die Erfindung ein Verfahren zum Ermitteln eines In a first aspect, the invention relates to a method for determining a
Abstandes zu einem Objekt. Das Verfahren umfasst folgende Schritte: Distance to an object. The method comprises the following steps:
Erfassen von mindestens zwei Bildern. Zum Erfassen wenigstens eines ersten der mindestens zwei Bilder wird ein vor einer Kamera angeordneter Polarisationsfilter verwendet.  Capture at least two images. For detecting at least a first of the at least two images, a polarization filter arranged in front of a camera is used.
Ermitteln des Abstandes anhand eines zweiten der mindestens zwei erfassten Bilder mittels eines maschinellen Lernsystems. Das maschinelle Lernsystem verwendet während des Ermittelns des Abstandes auch das erste der mindestens zwei erfassten Bilder, um eine Reflexion in einem der mindestens zwei erfassten Bilder beim Ermitteln des Abstandes nicht zu berücksichtigen.  Determining the distance based on a second of the at least two captured images by means of a machine learning system. The machine learning system also uses the first of the at least two captured images during the determination of the distance to disregard reflection in one of the at least two captured images when determining the distance.
Unter„Ermitteln eines Abstandes zu einem Objekt“ kann verstanden werden, dass ein Abstand, insbesondere eine Entfernung, zwischen einem vorgebbaren Referenzpunkt und dem Objekt ermittelt wird. Vorzugsweise entspricht der Refe renzpunkt der Position, an welcher sich die Kamera befindet. Der Referenzpunkt kann sich aber auch vor bzw. hinter der Kamera befinden. Es sei angemerkt, dass das Verfahren unabhängig von der Position des Referenzpunktes ist, da je nach gewählten Referenzpunkt das maschinelle Lernsystem passend angelernt werden kann. Unter einer Reflexion kann jede Art von auftretenden Spiegelbilder eines Objektes verstanden werden. Beispielhaft kann durch eine reflektierende Oberfläche das Spiegelbild eines Objektes oder eine Verzerrung und/oder eine Verdrehung des Spiegelbilds auftreten, welche im Folgenden mit dem Begriff Re flexion bezeichnet werden. By "determining a distance to an object" it can be understood that a distance, in particular a distance, between a predefinable Reference point and the object is determined. The reference point preferably corresponds to the position at which the camera is located. The reference point can also be in front of or behind the camera. It should be noted that the method is independent of the position of the reference point, since depending on the selected reference point, the machine learning system can be adapted appropriately. A reflection can be understood as any type of mirror image of an object that occurs. By way of example, by means of a reflecting surface, the mirror image of an object or a distortion and / or a rotation of the mirror image can occur, which are referred to below by the term reflection.
Die mindestens zwei erfassten Bilder können entweder gleichzeitig oder unmittel bar nacheinander, insbesondere an einem vorgebbaren Zeitpunkt, erfasst wer den. Die Bilder können auch mittels mehren unterschiedlich positionierten Kame ras erfasst werden. The at least two recorded images can either simultaneously or immediately after each other, in particular at a predeterminable time, who recorded the. The images can also be captured by means of several differently positioned cameras.
Der Vorteil gemäß dieses Verfahrens ist, dass dem maschinellen Lernsystem durch die mindestens zwei erfassten Bilder, die unterschiedlich gefiltert sind, aus reichend Information zur Verfügung gestellt werden, dass die Reflexion die Ab standsermittlung nicht beeinträchtigt. Beispielsweise ist eines der erfassten Bilder ungefiltert und das andere Bild mit einem Polarisationsfilter gefiltert oder beide sind mit einem Polarisationsfilter gefiltert erfasst worden, wobei die Polarisations filter jeweils eine unterschiedliche Ausrichtung der Polarisationseben aufweisen. The advantage according to this method is that the machine learning system is provided with sufficient information by the at least two captured images, which are differently filtered, that the reflection does not affect the distance determination. For example, one of the captured images is unfiltered and the other image is filtered with a polarizing filter or both have been detected filtered with a polarizing filter, wherein the polarizing filter each have a different orientation of the polarization planes.
Besonders vorteilhaft ist, wenn das maschinelle Lernsystem derart angelernt wird, dass das maschinelle Lernsystem anhand des zweiten der mindestens zwei erfassten Bilder den Abstand ermittelt. Des Weiteren kann in dieser Ausführungs form das maschinelle Lernsystem derart angelernt werden, dass das maschinelle Lernsystem anhand des ersten der mindestens zwei erfassten Bilder die Refle xion beim Ermitteln des Abstandes nicht berücksichtigt. It is particularly advantageous if the machine learning system is taught in such a way that the machine learning system determines the distance on the basis of the second of the at least two captured images. Furthermore, in this embodiment, the machine learning system can be taught in such a way that the machine learning system does not consider the reflection from the first of the at least two captured images when determining the distance.
Der Vorteil gemäß dieses Verfahrens ist, dass durch das Anlernen des maschi nellen Lernsystems, dieses eigenständig eine Abstandsermittlung erlernt. Daher müssen hierfür keine komplexen Algorithmen entwickelt werden, um diese kom plexe Bildverarbeitungsaufgabe zu lösen. Ferner ist das angelernte maschinelle Lernsystem im Betrieb recheneffizienter bzw. schneller als herkömmliche Bildver arbeitungsalgorithmen zur Entfernungsmessung anhand erfasster Bilder. Denn das Anlernen des maschinellen Lernsystems gleicht einer Optimierung des ma schinellen Lernsystems in jeglicher Hinsicht und des Weiteren können maschi nelle Lernsysteme, insbesondere neuronale Netze, durch eine Verkettung der Rechenoperationen in dem maschinellen Lernsystem eine höhere Leistungsfä higkeit erreichen. The advantage of this method is that by learning the maschi nelle learning system, this independently learns a distance determination. Therefore, no complex algorithms have to be developed for this purpose in order to solve this complex image processing task. Furthermore, this is the learned machine Learning system in operation more computationally efficient or faster than conventional image processing algorithms for distance measurement based on captured images. For learning the machine learning system is similar to optimizing the academic learning system in all respects, and further, machine learning systems, especially neural networks, can achieve higher performance by concatenating the computing operations in the machine learning system.
Besonders vorteilhaft ist, wenn das maschinelle Lernsystem zusätzlich angelernt wird, eine Objektklassifikation anhand des zweiten der mindestens zwei erfass ten Bilder zu ermitteln und die Reflexion anhand des ersten der mindestens zwei erfassten Bilder beim Ermitteln der Objektklassifikation nicht zu berücksichtigen. Vorteilhaft ist ferner, wenn das maschinelle Lernsystem beim Ermitteln des Ab standes auch die Objektklassifikation ermittelt. It is particularly advantageous if the machine learning system is additionally trained to determine an object classification on the basis of the second of the at least two captured images and not to consider the reflection based on the first of the at least two captured images when determining the object classification. It is also advantageous if the machine learning system determines the object classification when determining the distance.
Dies hat den Vorteil, dass innerhalb eines Verarbeitungsschrittes des maschinel len Lernsystems neben der Entfernungsmessung auch die Objektklassifikation durchgeführt wird, wodurch aus den erfassten Bildern mehrere Informationen gleichzeitig extrahiert werden können. This has the advantage that, in addition to the distance measurement, the object classification is carried out within one processing step of the machine learning system, as a result of which a plurality of information can be extracted simultaneously from the acquired images.
Ebenfalls besonders vorteilhaft ist, wenn an vorgebbaren nacheinander folgen den Zeitpunkten jeweils mindestens zwei Bilder erfasst werden und das maschi nelle Lernsystem zusätzlich angelernt wird, um einen optischen Fluss zu ermit teln und auch angelernt wird, die Reflexion beim Ermitteln des optischen Flusses nicht zu berücksichtigen. Des Weiteren ermittelt das maschinelle Lernsystem ei nen optischen Fluss anhand von den an den vorgebbaren nacheinander folgen den Zeitpunkten erfassten Bildern. Likewise, it is particularly advantageous if at predetermined times successively at least two images are detected and the maschi nelle learning system is additionally trained to an optical flow to ermit stuffs and is trained to reflect the reflection when determining the optical flow. Furthermore, the machine learning system determines an optical flow on the basis of the images acquired at the predeterminable successive time points.
Unter dem optischen Fluss kann eine Größe, insbesondere ein Vektor, verstan den werden, die eine Bewegung eines Punktes in dem Bild charakterisiert, bei spielsweise eine Geschwindigkeit und/oder eine Richtung dieses Punktes relativ zu einem gewählten Bezugspunkt. Vorteilhaft ist, wenn der Bezugspunkt des op tischen Flusses und der Referenzpunkt der Ermittlung des Abstands auf einer identischen Position liegen. Vorteilhaft ist, wenn die erfassten Bilder gespeichert werden und das maschinelle Lernsystem mittels der gespeicherten erfassten Bildern nachgelernt wird. Dies weist den Vorteil auf, dass das Verfahren zum Ermitteln des Abstandes mit den erfassten Bildern weiter verbessert wird, um eine höhere Genauigkeit der Ab standsmessung zu erzielen. Under the optical flow, a quantity, in particular a vector, can be understood, which characterizes a movement of a point in the image, for example a speed and / or a direction of this point relative to a selected reference point. It is advantageous if the reference point of the optical flow and the reference point of the determination of the distance are at an identical position. It is advantageous if the captured images are stored and the machine learning system is learned by means of the stored captured images. This has the advantage that the method for determining the distance with the captured images is further improved in order to achieve a higher accuracy of the Ab measurement.
Vorteilhaft ist, wenn abhängig von dem Ergebnis der Ermittlung des Abstandes ein Aktor angesteuert wird. Der Aktor kann eine zumindest teilautonome Ma schine wie z.B. ein Roboter oder ein Fahrzeug sein. Ferner ist vorteilhaft, wenn das maschinelle Lernsystem ein tiefes neuronales Netz, insbesondere ein„Con- volutional Neural Network“ oder ein„Recurrent Neural Network“ ist. It is advantageous if, depending on the result of determining the distance, an actuator is actuated. The actuator may comprise an at least partially autonomous machine, such as e.g. be a robot or a vehicle. It is also advantageous if the machine learning system is a deep neural network, in particular a "Revolutionary Neural Network" or a "Recurrent Neural Network".
In einem weiteren Aspekt betrifft die Erfindung eine Vorrichtung, die eingerichtet ist, das Verfahren nach dem ersten Aspekt der Erfindung auszuführen. Die Vor richtung umfasst folgende Merkmale: Zumindest eine Kamera zum Erfassen der mindestens zwei Bilder und zumindest einen Polarisationsfilter. Der Polarisati onsfilter ist vor der Kamera angeordnet und wird zum Erfassen des ersten der mindestens zwei Bilder verwendet. Die Vorrichtung umfasst auch das maschi nelle Lernsystem. In another aspect, the invention relates to a device adapted to carry out the method according to the first aspect of the invention. The device comprises the following features: At least one camera for capturing the at least two images and at least one polarization filter. The polarization filter is arranged in front of the camera and is used for detecting the first of the at least two images. The device also includes the machine learning system.
Vorteilhaft ist, wenn die Vorrichtung auch den Aktor, insbesondere eine zumin dest teilautonome Maschine wie z.B. einen Roboter oder ein Fahrzeug, umfasst. Ferner ist vorteilhaft, wenn das maschinelle Lernsystem ein tiefes neuronales Netz, insbesondere ein„Convolutional Neural Network“ oder ein„Recurrent Neu ral Network“ ist. It is advantageous if the device also the actuator, in particular a at least teilautonomome machine such. a robot or a vehicle. Furthermore, it is advantageous if the machine learning system is a deep neural network, in particular a "convolutional neural network" or a "recurrent neutral network".
Der Vorteil der Vorrichtung ist, dass mittels des insbesondere linearen, Polarisati onsfilters Reflexionen, die unpolarisiert sind, herausgefiltert werden, sodass das erfasste Bild ein zumindest teilweise reflexionsfreies Bild ist. Mittels des maschi nellen Lernsystems kann hierauf eine genaue Abstandsermittlung durchgeführt werden, die nicht durch die Reflexion beeinträchtig wird. The advantage of the device is that by means of the particular linear polarization filter, reflections which are unpolarized are filtered out so that the captured image is an at least partially reflection-free image. By means of the maschi nelle learning system can be carried out on this a precise distance determination, which is not affected by the reflection.
Vorteilhaft ist, wenn eine Mehrzahl von unterschiedlichen Polarisationsfiltern ver wendet wird und die Kamera jeweils ein gefiltertes Bild mittels einer der unter- schiedlichen Polarisationsfilter erfasst. Vorteilig hieran ist, dass durch die Mehr zahl von unterschiedlich polarisationsgefilterten Bildern mit jeweils unterschiedli chen Polarisationsfiltern ein Bild vorhanden ist, welches mit einer geeigneten Ausrichtung des Polarisationsfilters aufgenommen wurde, sodass dieses Bild ein reflexionsfreies Bild ist bzw. ein Ausschnitt des Bildes reflexionsfrei ist. Dadurch stehen mehrere unterschiedlich gefilterte Bilder zur Verfügung, wodurch die Ge nauigkeit und Zuverlässigkeit der Abstandsermittlung zusätzlich erhöht werden kann. It is advantageous if a plurality of different polarization filters is used and the camera in each case uses a filtered image by means of one of the different polarization filters. recorded different polarization filter. This has the advantage that an image is present through the plurality of different polarization filtered images, each with unterschiedli chen polarization filters, which was taken with a suitable orientation of the polarizing filter, so that this image is a reflection-free image or a section of the image is free of reflection. As a result, several differently filtered images are available, whereby the Ge accuracy and reliability of the distance determination can be further increased.
Ebenso vorteilhaft ist, wenn eine Polarisationsebene einer der Polarisationsfilter zu den Polarisationsebenen der weiteren Polarisationsfilter nicht gleich ausge richtet ist. It is equally advantageous if a plane of polarization of one of the polarization filter is not aligned equal to the polarization planes of the other polarization filter.
Vorteilhaft ist ferner, wenn der Polarisationsfilter mehrere hintereinander ange ordnete Polarisationsfilter umfasst, wobei eine der Polarisationsebenen der hin tereinander angeordneten Polarisationsfilter nicht senkrecht zu den jeweiligen Polarisationsebenen der hintereinander angeordneten Polarisationsfilter liegt.It is also advantageous if the polarizing filter comprises a plurality of consecutively arranged polarizing filter, wherein one of the polarization planes of the successively arranged polarizing filter is not perpendicular to the respective polarization planes of the successively arranged polarizing filter.
Der Vorteil ist, dass das polarisationsgefilterte Bild stärker reduzierte Reflexionen enthalten kann. The advantage is that the polarization-filtered image can contain more reduced reflections.
In einer Weiterentwicklung der Vorrichtung ist der Polarisationsfilter ein zirkularer Polarisationsfilter. Dies weist den Vorteil auf, dass zirkulare Polarisationsfilter eine höhere Kompatibilität mit den üblicherweise verwendeten digitalen Kameras aufweisen, die z.B. einen Autofokus oder einen automatisierten Belichtungsmes ser haben. In a further development of the device, the polarization filter is a circular polarization filter. This has the advantage that circular polarizing filters have higher compatibility with commonly used digital cameras, e.g. have an autofocus or an automated exposure meter.
Besonders vorteilhaft ist, wenn der Polarisationsfilter zusätzlich einen Farbfilter umfasst. Der Vorteil hierbei ist, dass typischerweise Polarisationsfilter verstärkt den Blauanteil des Lichts herausfiltern, da dieser durch die Streuung des Lichts in der Atmosphäre stark unpolarisiert ist. Deshalb kann durch den Farbfilter der verstärkten Reduktion des Blauanteils entgegengewirkt und eine Farbneutralität der Bilder bewahrt werden. In einem weiteren Aspekt betrifft die Erfindung ein Computerprogramm, welches eingerichtet ist, eines der oben genannten Verfahren auszuführen, also Anwei sungen umfasst, die einen Computer veranlassen, eines der oben genannten It is particularly advantageous if the polarization filter additionally comprises a color filter. The advantage here is that polarizing filters typically filter out the blue portion of the light, as it is highly unpolarized by the scattering of the light in the atmosphere. Therefore, the color filter can counteract the increased reduction of the blue component and preserve the color neutrality of the images. In a further aspect, the invention relates to a computer program which is set up to carry out one of the above-mentioned methods, that is to say comprises instructions which cause a computer, one of the above-mentioned
Verfahren mit all seinen Schritten auszuführen, wenn das Computerprogramm auf diesem Computer abläuft. Ferner betrifft die Erfindung ein maschinenlesba res Speicherelement, auf welchem das Computerprogramm gespeichert ist. To perform the procedure with all its steps when the computer program runs on this computer. Furthermore, the invention relates to a machine-readable storage element on which the computer program is stored.
Ausführungsbeispiele der vorliegenden Erfindung sind in den beiliegenden Zeich nungen dargestellt und in der nachfolgenden Beschreibung näher erläutert. Da bei zeigen: Embodiments of the present invention are illustrated in the accompanying drawing calculations and explained in more detail in the following description. As shown in:
Kurze Beschreibung der Zeichnungen Brief description of the drawings
Fig. 1 eine schematische Darstellung einer Vorrichtung zum Ermitteln eines Fig. 1 is a schematic representation of a device for determining a
Abstandes zu einem Objekt; und  Distance to an object; and
Fig. 2 eine schematische Darstellung einer Ausführungsform eines Verfahren zum Ermitteln eines Abstandes zu einem Objekt.  Fig. 2 is a schematic representation of an embodiment of a method for determining a distance to an object.
Figur 1 zeigt eine schematische Darstellung einer beispielhaften Vorrichtung (10) für eine zuverlässige und robuste Ermittlung eines Abstandes zu einem Objekt anhand ei nes erfassten Bildes. Die Ermittlung des Abstandes wird durch unerwünschte Reflexio nen, insbesondere durch Spiegelbilder, nicht beeinträchtigt. FIG. 1 shows a schematic representation of an exemplary device (10) for a reliable and robust determination of a distance to an object on the basis of a captured image. The determination of the distance is not impaired by unwanted reflections, in particular by mirror images.
Die Vorrichtung (10) umfasst ein maschinelles Lernsystem (11), welches anhand von mindestens zwei erfassten Bildern (12a, 12b) einen Abstand zu einem Objekt ermittelt. Das maschinelle Lernsystem (11) ist bevorzugt ein„Convolutional Neural Network“. Wenigstens eines der mindestens zwei erfassten Bilder (12b) wird mit einem vor einer Kamera angeordneten Polarisationsfilter (13) erfasst. Mittels des Polarisationsfilters (13) können die unerwünschten Reflexionen an glatten oder spiegelnden Oberflächen, wie zum Beispiel an Fenstern oder an Wasseroberflächen unterdrückt werden. Daraus resultiert, dass die erfassten Bilder (12a, 12b) unterschiedlich gefiltert sind. Die unter schiedlichen gefilterten Bilder (12a, 12b) werden anschließend mit dem unten genann ten Verfahren von dem maschinellen Lernsystem (11) zur Ermittlung des Abstandes verwendet. Das maschinelle Lernsystem (11) erkennt anhand der zwei unterschiedlich gefilterten Bilder eine Reflexion und berücksichtigt diese nicht bei der Ermittlung des Abstandes. The device (10) comprises a machine learning system (11) which determines a distance to an object on the basis of at least two captured images (12a, 12b). The machine learning system (11) is preferably a "convolutional neural network". At least one of the at least two captured images (12b) is detected by a polarization filter (13) arranged in front of a camera. By means of the polarizing filter (13), the unwanted reflections on smooth or reflective surfaces, such as on windows or on water surfaces can be suppressed. As a result, the captured images (12a, 12b) are filtered differently. The different filtered images (12a, 12b) are then used with the below-mentioned method of the machine learning system (11) to determine the distance. The machine learning system (11) recognizes a reflection on the basis of the two differently filtered images and does not take these into account when determining the Distance.
Unter der Bezeichnung„Abstand zu einem Objekt“ kann verstanden werden, dass das maschinelle Lernsystem (11) einen Abstand zwischen einem Referenzpunkt und dem Objekt ermittelt. Der Referenzpunkt kann zum Beispiel die Position der Kamera sein. Alternativ oder zusätzlich können mehrere unterschiedlich platzierte Kameras zur Er fassung der Bilder verwendet werden. Hierfür muss der Referenzpunkt dementspre chend gewählt werden. Denkbar ist auch, dass der Referenzpunkt vor bzw. hinter oder seitlich der Kamera liegt, z.B. wenn die Kamera an der Windschutzscheibe eines Fahr zeugs platziert ist, kann der Referenzpunkt nahe der Stoßstange gewählt werden. The term "distance to an object" can be understood to mean that the machine learning system (11) determines a distance between a reference point and the object. The reference point may be, for example, the position of the camera. Alternatively or additionally, a plurality of differently placed cameras can be used to capture the images. For this, the reference point must be chosen accordingly. It is also conceivable that the reference point lies in front of or behind or to the side of the camera, e.g. When the camera is placed on the windshield of a vehicle, the reference point near the bumper can be selected.
Anhand des ersten erfassten Bildes (12a), kann das maschinelle Lernsystem (11) den Abstand zu dem Objekt ermitteln. Das maschinelle Lernsystem (11) verwendet ein wei teres erfasstes Bild (12b), um eine Reflexion in den Bildern während der Ermittlung des Abstandes nicht zu berücksichtigen. Denn durch das weitere Bild (12b) können dem maschinellen Lernsystem (11) weitere Informationen gestellt werden, sodass das ma schinelle Lernsystem (11) eine Reflexion in den verwendeten Bildern (12a, 12b) erken nen kann. Dies hat den vorteilhaften Effekt, dass die Reflexion nicht in das Ermitteln des Abstandes einfließt und damit fehlerhafte Ergebnis der Abstandsermittlung vermie den werden können. Based on the first captured image (12a), the machine learning system (11) can determine the distance to the object. The machine learning system (11) uses another captured image (12b) to disregard reflection in the images while determining the distance. Because through the additional image (12b), the machine learning system (11) can be provided with further information so that the ma chine learning system (11) can detect reflection in the images (12a, 12b) used. This has the advantageous effect that the reflection is not included in the determination of the distance and thus erroneous result of the distance determination can be avoided the.
Nachdem das maschinelle Lernsystem (11) die Distanz ermittelt hat, kann optional das Ergebnis des maschinellen Lernsystems (11) von einer Steuerungseinheit der Vorrich tung (10) verwendet werden, um abhängig von diesem Ergebnis eine Steuergröße (14) zu ermitteln. Die Steuergröße (14) kann zum Steuern eines Aktors (17) verwendet wer den. Beispielsweise kann der Aktor (17) eine zumindest teilautonome Maschine, insbe sondere ein Roboter oder ein Fahrzeug, sein. Beispielhaft kann ein Einparkvorgang der zumindest teilautonomen Maschine mit der Steuergröße (14) durchgeführt werden. Al ternativ kann der Aktor (17) auch eine Ausrichtung einer Polarisationsebene des Polari sationsfilters (13) abhängig von der Steuergröße (14) passend nachjustieren, sodass das Ergebnis des maschinellen Lernsystems (11) mit einem weiteren unterschiedlich gefilterten Bild überprüft werden kann. After the machine learning system (11) has determined the distance, the result of the machine learning system (11) can optionally be used by a control unit of the device (10) to determine a control variable (14) depending on this result. The control variable (14) can be used to control an actuator (17) who the. For example, the actuator (17) may be an at least partially autonomous machine, in particular a robot or a vehicle. By way of example, a parking operation of the at least partially autonomous machine with the control variable (14) can be carried out. Al ternatively, the actuator (17) can also readjust alignment of a polarization plane of the Polari sationsfilters (13) depending on the control variable (14), so that the result of the machine learning system (11) can be checked with another differently filtered image.
Optional kann das maschinelle Lernsystem (11) auch ein Distanzbild erzeugen. Das Distanzbild kann ein Bild sein, in dem die ermittelten Informationen des maschinellen Lernsystems (11), z.B. der Abstand zu dem Objekt, überlagert auf einem Ausschnitt ei nes der erfassten Bilder (12a, 12b) ausgegeben werden. Beispielsweise kann hierbei jedem Pixel jeweils eine der ermittelten Informationen zugeordnet werden. Optionally, the machine learning system (11) may also generate a distance image. The distance image can be an image in which the determined information of the machine Learning system (11), for example, the distance to the object, superimposed on a section of ei nes of the captured images (12a, 12b) are output. For example, each pixel can be assigned one of the determined information.
In einer alternativen Ausführungsform der Vorrichtung (10) erhält das maschinelle Lernsystem (11) eine Mehrzahl von erfassten Bilder (12a, 12b), wobei die Bilder alle mittels unterschiedlichen Polarisationsfiltern erfasst wurden. D.h. eine Polarisationse bene der Polarisationsfilter ist zu den Polarisationsebenen der anderen Polarisationsfil ter unterschiedlich ausgerichtet, sodass unterschiedlich gefilterte Bilder erfasst werden und zur Abstandsermittlung verwendet werden können. In an alternative embodiment of the device (10), the machine learning system (11) obtains a plurality of captured images (12a, 12b), the images all being detected by means of different polarization filters. That one polarization plane of the polarization filters is oriented differently to the polarization planes of the other polarization filters so that differently filtered images can be captured and used for distance determination.
Ferner umfasst die Vorrichtung (10) eine Recheneinheit (15) und ein Speicherelement (16), auf dem ein Computerprogram gespeichert ist. Das Computerprogramm kann Be fehle umfasst, die bewirken, dass beim Ausführen des Computerprogramms auf z.B. der Recheneinheit (15) eine der Ausführungsformen des unten genannten Verfahrens ausgeführt wird. Furthermore, the device (10) comprises a computing unit (15) and a memory element (16) on which a computer program is stored. The computer program may include instructions causing the computer program to run at e.g. the computing unit (15) is carried out one of the embodiments of the below-mentioned method.
Figur 2 zeigt eine schematische Darstellung einer Ausführungsform eines Verfahrens (20) zur Ermittlung einer Distanz zu einem Objekt anhand der erfassten Bilder (12a, 12b). Figure 2 shows a schematic representation of an embodiment of a method (20) for determining a distance to an object based on the captured images (12a, 12b).
Das Verfahren (20) beginnt mit Schritt 21. In Schritt 21 wird dem maschinellen Lernsys tem (11) der Vorrichtung (10) ein Trainingsdatensatz z.B. aus einer Datenbank bereit gestellt. Der Trainingsdatensatz kann beispielsweise eine Mehrzahl von Trainingsbil dern enthalten, die jeweils reale erfasste Bilder oder von einem Computer erzeugte Bil der sein können. Die Trainingsbilder können mit bzw. ohne Reflexionen sein. Vorzugs weise können jeweils mehrere Bilder eine gleiche Szene darstellen, diese sind aber je weils unterschiedlich gefiltert, bevorzugt mit dem Polarisationsfilter (13) aus Figur 1. Vorzugsweise sind die Trainingsdaten mittels eines Abstandswertes gelabelt und be vorzugt mit einem Vermerk, ob eine Reflexion in dem Trainingsbild vorhanden ist, oder nicht. Die Abstandswerte der Trainingsbilder können mittels einer„ground truth“ Me thode bestimmt werden. Die Label können im nachfolgenden genutzt werden, um das maschinelle Lernsystem (11) gezielter zur Abstandsermittlung unter Vernachlässigung der Reflexionen anzulernen. Nachdem der Trainingsdatensatz dem maschinellen Lernsystem (11) bereitgestellt wurde, wird das maschinelle Lernsystem (11) derart angelernt, dass das maschinelle Lernsystem abhängig von einem erfassten Bild und unter Berücksichtigung eines wei teren erfassten Bildes eine Distanz zu einem Objekt in dem erfassten Bild ermittelt.The method (20) begins with step 21. In step 21, the machine learning system (11) of the device (10) is provided with a training data set, for example, from a database. For example, the training data set may include a plurality of training images, each of which may be real captured images or images generated by a computer. The training images can be with or without reflections. Preference, each of multiple images represent a same scene, but these are each Weil filtered differently, preferably with the polarizing filter (13) of Figure 1. Preferably, the training data are labeled by means of a distance value and be preferred with a note whether a reflection in the Training image is present or not. The distance values of the training images can be determined by means of a "ground truth" method. In the following, the labels can be used to train the machine learning system (11) in a more targeted way for distance determination, neglecting the reflections. After the training data set has been provided to the machine learning system (11), the machine learning system (11) is taught such that the machine learning system determines a distance to an object in the acquired image as a function of a captured image and taking into account a further acquired image.
Das weitere erfasste Bild ist insbesondere ein gefiltertes Bild mittels eines Polarisati onsfilters oder mittels einer unterschiedlich ausgerichteten Polarisationsebene des Po larisationsfilters. Dabei kann das maschinelle Lernsystem auch derart angelernt sein, dass das maschinelle Lernsystem anhand des Weiteren bereitgestellten Bildes, Refle xionen in den Bildern nicht zur Ermittlung der Distanz zu einem Objekt berücksichtigt. Vorzugsweise wird zum Anlernen des maschinellen Lernsystems (11) ein Gradienten abstiegsverfahren zum Bestimmen der Parameterwerte des maschinellen Lernsystems (11) verwendet. Das Gradientenabstiegsverfahren kann auf eine Kostenfunktion ange wendet werden. Die Kostenfunktion kann abhängig von den Parametern des maschi nellen Lernsystems (11) und vorzugsweise von den Labein der verwendeten Trainings daten sein. The further acquired image is in particular a filtered image by means of a polarization filter or by means of a differently oriented polarization plane of the polarization filter. In this case, the machine learning system can also be taught in such a way that the machine learning system, on the basis of the further provided image, does not consider reflections in the images for determining the distance to an object. Preferably, for learning the machine learning system (11), a gradient descent method is used for determining the parameter values of the machine learning system (11). The gradient descent method can be applied to a cost function. The cost function may be dependent on the parameters of the machine learning system (11) and preferably on the lab of the training data used.
Optional kann auch in Schritt 21 das maschinelle Lernsystem (11) des Weiteren angelernt werden, dass das maschinelle Lernsystem (11) eine Objektdetektion, insbesondere eine Objektklassifikation, anhand der bereitgestellten Bilder ermit teln kann. Bevorzugt sind hierfür die Trainingsbilder zusätzlich oder alternativ mit Label, die die Objektklassen charakterisieren, gelabelt. Ebenso kann hierbei das maschinelle Lernsystem (11) derart angelernt werden, dass das maschinelle Optionally, the machine learning system (11) can also be taught in step 21 that the machine learning system (11) can determine an object detection, in particular an object classification, on the basis of the provided images. For this purpose, the training images are preferably additionally or alternatively labeled, which characterize the object classes. Likewise, in this case, the machine learning system (11) can be taught in such a way that the machine
Lernsystem (11) unter Berücksichtigung des weiteren Bildes, insbesondere mit tels eines Polarisationsfilters gefilterten Bildes, Reflexionen in den Bildern nicht zur Ermittlung der Objektdetektion berücksichtigt. Learning system (11) taking into account the further image, in particular with means of a polarization filter filtered image, reflections in the images are not taken into account for determining the object detection.
Nachdem Schritt 21 abgeschlossen ist, folgt Schritt 22. In Schritt 22 werden mindes tens zwei Bilder erfasst, insbesondere an einem vorgebbaren Zeitpunkt. Ein erstes der mindestens zwei Bilder wird mittels eines Polarisationsfilters, der vor der Kamera ange ordnet ist, gefiltert erfasst. Bevorzugt weist dieser Polarisationsfilter die gleiche Aus richtung der Polarisationsebene auf, wie die verwendeten Polarisationsfilter zur Erfas sung der Bilder des Trainingsdatensatzes. Dies hat den vorteilhaften Effekt, dass die erfassten Bilder ähnlich wie die Trainingsbildern des maschinellen Lernsystems (11) erfasst werden, wodurch eine höhere Genauigkeit der Ermittlung des Abstandes zu ei nem Objekt erzielt werden kann. Anschließend ermitelt das maschinelle Lernsystem (11) anhand eines der mindestens zwei erfassten Bilder den Abstand zu dem Objekt. Das weitere erfasste Bild, insbeson dere das mitels des Polarisationsfilter gefilterte Bild, wird während der Ermitlung des Abstandes zu einem Objekt mitels des maschinellen Lernsystems verwendet, dass Reflexionen in einen der beiden Bildern nicht für die Ermitlung des Abstands zu einem Objekt berücksichtigt werden. After step 21 is completed, step 22 follows. In step 22, at least two images are acquired, in particular at a predefinable time. A first of the at least two images is detected by means of a polarizing filter, which is arranged in front of the camera, filtered. This polarization filter preferably has the same orientation from the plane of polarization as the polarization filters used for detecting the images of the training data set. This has the advantageous effect that the captured images are detected similarly to the training images of the machine learning system (11), whereby a higher accuracy of the determination of the distance to an object can be achieved. Subsequently, the machine learning system (11) determines the distance to the object from one of the at least two captured images. The further captured image, in particular the image filtered by the polarization filter, is used during the determination of the distance to an object by the machine learning system that reflections in one of the two images are not taken into account for determining the distance to an object.
In einer alternativen oder zusätzlichen Ausführungsform des Verfahrens (20), bei dem das maschinelle Lernsystem (11) auch für eine Objektdetektion, insbesondere eine Ob jektklassifikation, angelernt wurde, kann in Schrit 22 anhand der erfassten Bilder (12a, 12b) auch eine Objektdetektion ermitelt werden. Der vorteilhafte Effekt hierbei ist, dass durch die unterschiedlichen gefiltert erfassten Bilder, Reflexionen in den Bildern bestimmt werden können und mitels des maschinellen Lernsystems (11) nicht in der Ermitlung der Objektdetektion berücksichtigt werden. Dadurch kann eine zuverlässi gere Objektedetektion erzielt werden, denn beispielsweise eine Reflexion eines Pas santen in einem Schaufensterfenster kann zu einer fehlerhaften Objektdetektion bzw. Klassifikation führen. In an alternative or additional embodiment of the method (20), in which the machine learning system (11) has also been trained for object detection, in particular object classification, an object detection can also be determined in step 22 on the basis of the captured images (12a, 12b) become. The advantageous effect here is that reflections in the images can be determined by the different filtered captured images and are not taken into account in the determination of the object detection by means of the machine learning system (11). As a result, a reliable object detection can be achieved, because, for example, a reflection of a passant in a shop window can lead to erroneous object detection or classification.
Nachdem Schrit 22 beendet wurde, folgt optional Schrit 23. In Schrit 23 kann abhän gig von dem Ergebnis des maschinellen Lernsystems (11) eine Steuerungsgröße (14) zum Steuern des Aktors (17) ermitelt werden. Der Aktor (17), insbesondere eine zu mindest teilautonome Maschine wie ein Roboter oder ein Fahrzeug, kann beispielhaft eine Bewegung bzw. Fahrmanöver abhängig von der Steuerungsgröße ausführen. After step 22 has ended, step 23 follows optionally. In step 23, depending on the result of the machine learning system (11), a control variable (14) for controlling the actuator (17) can be determined. The actuator (17), in particular a machine that is at least partly autonomous, such as a robot or a vehicle, can, for example, execute a movement or driving maneuver depending on the control variable.
In einer weiteren alternativen Ausführungsform des Verfahrens (20) kann in Schrit 21 das maschinelle Lernsystem (11) auch derart angelernt werden, dass das maschinelle Lernsystem (11) anhand von einer Abfolge von erfassten Bildern an unterschiedlichen vorgebbaren nacheinander folgenden Zeitpunkten einen optischen Fluss ermiteln kann. Es sei angemerkt, dass das maschinelle Lernsystem (11) hierfür ebenso ange lernt werden muss, dass Reflexionen in den Bildern nicht für die Ermitlung des opti schen Flusses berücksichtigt werden. Vorzugsweise sind die Trainingsbilder zum An lernen des maschinellen Lernsystems (11) zusätzlich oder alternativ mit Label, die den optischen Fluss charakterisieren, gelabelt. Optional kann mitels dieses angelernten maschinelle Lernsystems (11) in Schritt 22 einer Mehrzahl von unterschiedlichen er fassten Bildern, die an vorgebbaren Zeitpunkten erfasst wurden, den optischen Fluss eines Objektes ermitteln. Optional kann Schritt 21 auch mehrfach hintereinander wiederholt werden, solange bis eine vorgebbare ausreichend hohe Genauigkeit der Abstandsmessung erreicht wird. In a further alternative embodiment of the method (20), the machine learning system (11) can also be taught in step 21 such that the machine learning system (11) can determine an optical flow based on a sequence of captured images at different predefinable successive times , It should be noted that the machine learning system (11) for this must be trained as well that reflections in the images are not taken into account for the determination of opti's flow. Preferably, the training images for learning the machine learning system (11) are additionally or alternatively labeled with labels that characterize the optical flow. Optionally, this can be learned with this Machine learning system (11) in step 22 of a plurality of different captured images, which were detected at predetermined times, the optical flow of an object determine. Optionally, step 21 can also be repeated several times in succession until a predeterminable sufficiently high accuracy of the distance measurement is achieved.
Damit endet das Verfahren (20). This ends the process (20).

Claims

Ansprüche claims
1. Verfahren (20) zum Ermiteln eines Abstandes zu einem Objekt, umfassend: A method (20) for determining a distance to an object, comprising:
Erfassen von mindestens zwei Bildern (12a, 12b),  Capturing at least two images (12a, 12b),
wobei zum Erfassen eines ersten der mindestens zwei Bilder (12b) ein vor einer Kamera angeordneter Polarisationsfilter (13) verwendet wird; und Ermiteln des Abstandes anhand eines zweiten der mindestens zwei erfass ten Bilder (12b) mitels eines maschinellen Lernsystems (11),  wherein for detecting a first of the at least two images (12b), a polarizing filter (13) arranged in front of a camera is used; and determining the distance from a second of the at least two captured images (12b) by means of a machine learning system (11),
wobei das maschinelle Lernsystem (11) während des Ermitelns des Abstan des auch das erste der mindestens zwei erfassten Bilder (12a) verwendet, um eine Reflexion in einem der mindestens zwei erfassten Bilder (12a, 12b) beim Ermiteln des Abstandes nicht zu berücksichtigen.  wherein the machine learning system (11) also uses the first of the at least two captured images (12a) during the determination of the distance to disregard reflection in one of the at least two captured images (12a, 12b) in determining the distance.
2. Verfahren nach Anspruch 1, wobei das maschinelle Lernsystem (11) derart angelernt wird, dass das maschinelle Lernsystem (11) anhand des zweiten der mindestens zwei erfassten Bilder (12b) den Abstand ermitelt, wobei das maschinelle Lernsystem (11) des Weiteren derart angelernt wird, dass das maschinelle Lernsystem (11) anhand des ersten der mindestens zwei erfassten (12b) Bilder die Reflexion beim Ermiteln des Abstandes nicht berücksichtigt. The method of claim 1, wherein the machine learning system (11) is taught such that the machine learning system (11) determines the distance from the second of the at least two captured images (12b), the machine learning system (11) further so is learned that the machine learning system (11) based on the first of the at least two captured (12b) images does not take into account the reflection in determining the distance.
3. Verfahren nach einem der vorherigen Ansprüche, wobei das maschinelle Lernsystem (11) zusätzlich angelernt wird, eine Objektklassifikation anhand des zweiten der mindestens zwei erfassten Bilder (12a) zu ermiteln und die Reflexion anhand des ersten der mindestens zwei erfassten Bilder (12b) beim Ermiteln der Objektklassifikation nicht zu berücksichtigen, 3. The method according to claim 1, wherein the machine learning system is additionally taught to determine an object classification based on the second of the at least two captured images and the reflection based on the first of the at least two acquired images Determining the object classification does not take into account
wobei das maschinelle Lernsystem (11) beim Ermiteln des Abstandes auch eine Objektklassifikation ermitelt. wherein the machine learning system (11) also determines an object classification when determining the distance.
4. Verfahren nach einem der vorherigen Ansprüche, wobei an vorgebbaren nacheinander folgenden Zeitpunkten jeweils mindestens zwei Bilder (12a, 12b) erfasst werden, 4. Method according to one of the preceding claims, wherein at predeterminable consecutive times each time at least two images (12a, 12b) are detected,
wobei das maschinelle Lernsystem (11) zusätzlich angelernt wird, um einen optischen Fluss zu ermitteln und auch die Reflexion beim Ermitteln des opti schen Flusses nicht zu berücksichtigen,  wherein the machine learning system (11) is additionally taught in order to determine an optical flow and also not to consider the reflection in determining the opti's flow,
wobei das maschinelle Lernsystem (11) einen optischen Fluss anhand von den an den vorgebbaren nacheinander folgenden Zeitpunkten erfassten Bil dern ermittelt.  wherein the machine learning system (11) determines an optical flow based on the images acquired at the predeterminable consecutive times.
5. Verfahren nach einem der vorherigen Ansprüche, wobei die erfassten Bilder (12a, 12b) gespeichert werden und das maschinelle Lernsystem mittels der gespeicherten erfassten Bildern (12a, 12b) nachgelernt wird. 5. The method according to any one of the preceding claims, wherein the captured images (12a, 12b) are stored and the machine learning system by means of the stored captured images (12a, 12b) is learned.
6. Vorrichtung (10), die eingerichtet ist, das Verfahren nach einem der Ansprü che 1 bis 5 auszuführen, umfassend: 6. Apparatus (10) adapted to carry out the method according to any one of claims 1 to 5, comprising:
Zumindest eine Kamera zum Erfassen der mindestens zwei Bilder (12a, 12b) und zumindest einen Polarisationsfilter (13),  At least one camera for capturing the at least two images (12a, 12b) and at least one polarization filter (13),
wobei der Polarisationsfilter (13) vor der Kamera angeordnet ist und zum Er fassen des ersten der mindestens zwei Bilder (12b) verwendet wird; und das maschinelle Lernsystem (11).  wherein the polarizing filter (13) is arranged in front of the camera and used to capture the first of the at least two images (12b); and the machine learning system (11).
7. Vorrichtung nach Anspruch 6, wobei eine Mehrzahl von unterschiedlichen Polarisationsfilter verwendet wird, 7. Apparatus according to claim 6, wherein a plurality of different polarizing filters are used,
wobei die Kamera jeweils ein Bild mittels einer der unterschiedlichen Polari sationsfilter erfasst.  wherein each camera captures an image by means of one of the different polarization filters.
8. Vorrichtung nach Anspruch 7, wobei eine Polarisationsebene einer der Pola risationsfilter zu den Polarisationsebenen der weiteren Polarisationsfilter nicht gleich ausgerichtet ist. 8. Device according to claim 7, wherein a polarization plane of one of the polarization filters is not aligned identically to the polarization planes of the further polarization filters.
9. Vorrichtung nach einem der Ansprüche 6 bis 8, wobei der Polarisationsfilter (13) mehrere hintereinander angeordnete Polarisationsfilter umfasst, wobei eine Polarisationsebene der hintereinander angeordneten Polarisati onsfilter nicht senkrecht zu den jeweiligen Polarisationsebenen der hinterei nander angeordneten Polarisationsfilter angeordnet ist. 10. Vorrichtung nach einem der Ansprüche 6 bis 8, wobei der Polarisationsfilter9. Device according to one of claims 6 to 8, wherein the polarization filter (13) comprises a plurality of successively arranged polarizing filter, wherein a polarization plane of the successively arranged Polarisati onsfilter is not arranged perpendicular to the respective polarization planes of the backerei nander arranged polarizing filter. 10. Device according to one of claims 6 to 8, wherein the polarizing filter
(13) ein zirkularer Polarisationsfilter ist. (13) is a circular polarizing filter.
11. Vorrichtung nach einem der Ansprüche 6 bis 10, wobei der Polarisationsfilter (13) zusätzlich einen Farbfilter umfasst. 11. Device according to one of claims 6 to 10, wherein the polarization filter (13) additionally comprises a color filter.
12. Computerprogramm, umfassend Befehle, die beim Ausführen des Compu terprogramms auf einem Computer diesen veranlassen, das Verfahren nach einem der Ansprüche 1 bis 5 auszuführen. 13. Maschinenlesbares Speicherelement (16), auf dem das Computerprogram nach Anspruch 12 gespeichert ist. A computer program comprising instructions for causing the computer program on a computer to execute the method according to any one of claims 1 to 5. 13. A machine-readable storage element (16) on which the computer program according to claim 12 is stored.
EP18807901.6A 2017-11-29 2018-11-14 Method, device and computer program for determining a distance to an object Withdrawn EP3718045A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102017221381.7A DE102017221381A1 (en) 2017-11-29 2017-11-29 Method, apparatus and computer program for determining a distance to an object
PCT/EP2018/081222 WO2019105737A1 (en) 2017-11-29 2018-11-14 Method, device and computer program for determining a distance to an object

Publications (1)

Publication Number Publication Date
EP3718045A1 true EP3718045A1 (en) 2020-10-07

Family

ID=64456944

Family Applications (1)

Application Number Title Priority Date Filing Date
EP18807901.6A Withdrawn EP3718045A1 (en) 2017-11-29 2018-11-14 Method, device and computer program for determining a distance to an object

Country Status (4)

Country Link
EP (1) EP3718045A1 (en)
CN (1) CN111373411A (en)
DE (1) DE102017221381A1 (en)
WO (1) WO2019105737A1 (en)

Families Citing this family (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102021202454A1 (en) 2021-03-15 2022-09-15 Zf Friedrichshafen Ag Environment modeling based on camera data
CN113256576B (en) * 2021-05-18 2022-10-28 福州大学 Automatic optical element detection system and method based on polarization imaging and machine learning

Family Cites Families (12)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102004053416A1 (en) * 2004-11-05 2006-05-11 Robert Bosch Gmbh Stereoscopic distance measurement system to determine distance of object from motor vehicle has splitter mirror element to deflect first part of virtual beam bundle from camera which is then overlapped by second part of beam bundle
DE102009026463A1 (en) * 2009-05-26 2010-12-09 Robert Bosch Gmbh Image acquisition method for acquiring multiple images by means of an automotive camera system and associated image capture device of the camera system
DE102011005368A1 (en) 2011-03-10 2012-09-13 Robert Bosch Gmbh Driver assistance system for vehicle, particularly designed as assistance system for shunters or for parking vehicle, has video camera, by which video image of surrounding area of vehicle is recorded with objects
DE102011051583A1 (en) * 2011-07-05 2013-01-10 Conti Temic Microelectronic Gmbh Image pickup device for a vehicle
DE102011081384B4 (en) 2011-08-23 2017-05-04 Robert Bosch Gmbh Method and device for distance determination for a vehicle
DE102012018121A1 (en) * 2012-09-13 2013-04-04 Daimler Ag Image detecting device for detecting images of environment of motor vehicle, has stereo camera unit which has optical unit for detecting images of environment and another optical unit for detecting images of environment
JP5893601B2 (en) * 2013-10-31 2016-03-23 富士重工業株式会社 Vehicle control system
DE102014224762B4 (en) * 2014-12-03 2016-10-27 Volkswagen Aktiengesellschaft Method and device for obtaining information about an object in a non-accessible, adjacent surrounding area of a motor vehicle
KR102534792B1 (en) * 2015-02-10 2023-05-19 모빌아이 비젼 테크놀로지스 엘티디. Sparse map for autonomous vehicle navigation
DE102016206493A1 (en) * 2015-06-23 2016-12-29 Robert Bosch Gmbh Method and camera system for determining the distance of objects to a vehicle
CN108028892B (en) * 2015-09-30 2021-02-09 索尼公司 Information acquisition apparatus and information acquisition method
US10055652B2 (en) * 2016-03-21 2018-08-21 Ford Global Technologies, Llc Pedestrian detection and motion prediction with rear-facing camera

Also Published As

Publication number Publication date
WO2019105737A1 (en) 2019-06-06
DE102017221381A1 (en) 2019-05-29
CN111373411A (en) 2020-07-03

Similar Documents

Publication Publication Date Title
DE102014209137B4 (en) Method and device for calibrating a camera system of a motor vehicle
DE69625423T2 (en) Device for recognizing an object in which the position of a flat object is estimated using a Hough transformation
DE102009046295A1 (en) Focus-height measurement i.e. isotropic focus-height measurement, correcting method, involves determining focus-height error correction value, and correcting focus-height measurement using value to provide corrected measurement
DE102006010735A1 (en) Vehicle environment monitoring device
EP2189349A2 (en) Method and device for compensating a roll angle
WO2022128014A1 (en) Correction of images from a panoramic-view camera system in the case of rain, incident light and contamination
EP3718045A1 (en) Method, device and computer program for determining a distance to an object
EP4264565A1 (en) Correction of images from a camera in case of rain, incident light and contamination
EP3938947A1 (en) Method and device for determining a danger source on a roadway
DE102012211961A1 (en) Method and apparatus for calculating a change in a magnification of an object
EP1098268A2 (en) Method for the threedimensional optical measurement of object surfaces
EP3659113B1 (en) Detection system, working method and training method for generating a 3d model with reference data
DE102010021221A1 (en) Camera direction determining method for use in vehicle, involves reconstructing vehicle coordination system from points, and determining camera direction from rotation of camera-coordination system relative to vehicle coordination system
DE102014009522A1 (en) Method for detecting an environment of a vehicle
WO2021073907A1 (en) Calibration of an active optical sensor system
EP3798969A1 (en) Depth image processing method, image processing apparatus and depth camera
DE102018207411A1 (en) Method for determining measurement information in an optical coordinate measuring machine
DE102019007344A1 (en) Method for determining a slip movement of a vehicle wheel
WO2015149970A1 (en) Method and device for adapting a three-dimensional projection surface for projecting a plurality of adjacent camera images
DE102020118723B4 (en) A method for determining a position of a first object in a positioning system, a positioning system and a computer program
DE102017217156B4 (en) Method and device for controlling a driver assistance system using a stereo camera system with a first and a second camera
DE102007021107A1 (en) Camera system main point determining system for driver assistance system, has comparing device comparing imaged position with calculated position of calibration mark and providing error measure of main point depending on comparison result
DE102009015500B4 (en) Method and device for determining a change in position of an object by means of a stereo camera
DE102022116317A1 (en) Method for detecting contamination on a camera of a vehicle, camera and driver assistance system
DE102016222548A1 (en) Method and control unit for determining a distortion characteristic of an optical medium during travel operation

Legal Events

Date Code Title Description
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: UNKNOWN

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE INTERNATIONAL PUBLICATION HAS BEEN MADE

PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: REQUEST FOR EXAMINATION WAS MADE

17P Request for examination filed

Effective date: 20200629

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

DAV Request for validation of the european patent (deleted)
DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20220329

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION HAS BEEN WITHDRAWN

18W Application withdrawn

Effective date: 20220509