EP2358601B1 - Method for aligning a container and computer program - Google Patents

Method for aligning a container and computer program Download PDF

Info

Publication number
EP2358601B1
EP2358601B1 EP09763831A EP09763831A EP2358601B1 EP 2358601 B1 EP2358601 B1 EP 2358601B1 EP 09763831 A EP09763831 A EP 09763831A EP 09763831 A EP09763831 A EP 09763831A EP 2358601 B1 EP2358601 B1 EP 2358601B1
Authority
EP
European Patent Office
Prior art keywords
container
image data
geometric
feature
search
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP09763831A
Other languages
German (de)
French (fr)
Other versions
EP2358601A1 (en
Inventor
Jürgen Hermann
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
KHS GmbH
Original Assignee
KHS GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by KHS GmbH filed Critical KHS GmbH
Priority to PL09763831T priority Critical patent/PL2358601T3/en
Publication of EP2358601A1 publication Critical patent/EP2358601A1/en
Application granted granted Critical
Publication of EP2358601B1 publication Critical patent/EP2358601B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • BPERFORMING OPERATIONS; TRANSPORTING
    • B65CONVEYING; PACKING; STORING; HANDLING THIN OR FILAMENTARY MATERIAL
    • B65CLABELLING OR TAGGING MACHINES, APPARATUS, OR PROCESSES
    • B65C9/00Details of labelling machines or apparatus
    • B65C9/06Devices for presenting articles in predetermined attitude or position at labelling station
    • B65C9/067Devices for presenting articles in predetermined attitude or position at labelling station for orienting articles having irregularities, e.g. holes, spots or markings, e.g. labels or imprints, the irregularities or markings being detected

Definitions

  • the present invention relates to a method for aligning a container. Furthermore, the present invention relates to a computer program for carrying out the method according to the invention. Such a procedure is over DE 20 2005 014456U1 or off IT M 120 072 267 A1 known.
  • the published patent application DE 10 2005 050 902 A1 a device for aligning containers with respect to at least one geometric container feature, comprising a conveyor with container receptacles for a respective container and along a transport path formed by the conveyor conveyor cameras of an image recognition system, wherein the image recognition system by comparison of the actual delivered by the cameras Image data with stored in an evaluation and control electronics target image data or characteristics causes the alignment of the container.
  • the containers are aligned by means of a geometric feature, for example by means of an embossment on the surface of the container.
  • an image is taken in the desired position of the bottle and then a respective distortion of this geometric feature is calculated for the cases in which the container is rotated by a certain angle from this desired position.
  • the images or image data thus generated are then stored as target image data and used.
  • an image of the container or of the bottle is then taken by means of a camera and the thus generated actual image is compared with the stored desired image data by means of an algorithm. It is determined with which image of the target image data is the highest match. Based on the rotation of the container or bottle deposited in the target image data for this image, it can then be rotated by the corresponding angle into the desired position.
  • a plurality of stages of such alignment processes may be provided in order to successively bring about the desired orientation or position of the container or the bottle.
  • a plurality of cameras at the same time take a picture of the bottle, which is then compared with the desired image data, wherein the alignment process is then carried out based on the best match.
  • a neural network is used, which has been taught beforehand in a learning process, what the geometric container feature used to align a container looks like in the different orientations of the container, and the one above it is able to recognize that in a search area the geometric container feature to be searched, the so-called search instance, does not exist.
  • an alignment process controlling control electronics is able to classify any other geometric container features or images in the search area to the effect that they do not represent the search instance by these so-called exclude instances are taught as such in advance.
  • control electronics in the context of the pattern identifier is able to discard certain recordings of the search area from the outset, since they do not contain the search instance.
  • pattern recognition takes place in that the control electronics classifies the actual image data on the basis of target image data as to whether they contain the geometric container feature.
  • the target image data in this case comprise the learned geometric container feature as a search entity and the further described above and also trained exclude instances.
  • the learning of the target image data takes place by first looking for the one located on a sample bottle geometric container feature, the search instance, recorded and stored.
  • the geometric container feature is then identified by a user, in particular based on its contour.
  • This training of the search entity can basically be done at the site of the procedure, ie. H. with the actually constructed filling and labeling, or off it, for example, in a laboratory.
  • the container or the bottle is rotated around itself in front of an image pickup device, with particular images of the container or the bottle being made in certain degree steps.
  • the multitude of degree steps depends in particular on how exactly the alignment should be performed during operation. If, for example, you want to align with 1 °, it is advisable to take the pictures in 1 ° increments, better in 0.5 ° increments.
  • the search instance is now cut out, if it is still found in the searched area, which can happen automatically, since the search instance has already been trained and the neural network is known.
  • the actual image data recorded by the image acquisition device is compared with the instances stored in the target image data and it is determined whether the search region contains an exclude entity or the search entity.
  • search area contains only one Exclude instance, this is displayed and the corresponding image is not used to calculate the actual position of the bottle.
  • the actual position of the container or its orientation is calculated on the basis of a perspective distortion of the geometric container feature, a known distance of the image recording device to the container during the recording and a known diameter of the container. From the difference between the actual position of the container to the desired desired position of the container, the degree and the direction in which the container is to be rotated can now be determined.
  • the present description also covers a computer program with program code to perform all the steps of a method according to the invention, when the program is executed on a computer or a corresponding computing unit.
  • the computer program may be stored on a computer-readable medium.
  • FIG. 1 shows a schematic view of an embodiment of a device according to the invention, on which the inventive method can be performed.
  • FIG. 1 shows a device 1, on which the method according to the invention can be carried out.
  • the device comprises a carousel 5, are transported into the container 2 via an inlet 3 and transported beyond a spout 4. From the labeling station 7, the containers 2 are aligned in the desired orientation for applying a label.
  • the first stage comprising the image pickup devices 8 and 9, the second stage the image pickup device 10, and the third stage the image pickup device 11.
  • Background elements 13, 14, 15 are provided for capturing high quality images. which can also be lit.
  • the alignment operation is controlled by an electronic control unit 12 which receives actual image data from the image pickup devices 8, 9, 10, 11 and can output control instructions to container receptacles 6 on which the containers 2 are arranged.
  • a learning process For commissioning and for the preparation of the alignment process, a learning process has to be carried out, in which first a search instance, ie a geometrical container feature used to align a container, is determined from a first image of a sample container identified a user and the search instance is so trained a neural network of the control electronics 12.
  • a search instance ie a geometrical container feature used to align a container
  • the corresponding target image data record can be called up by an operator. This is followed by a slow inlet of a first container or a first bottle, wherein the image recording devices 8, 9, 10, 11 are aligned so that they detect the possible positions of the search instance as accurately as possible.
  • a narrow optical evaluation range is a narrow optical evaluation range.

Landscapes

  • Image Analysis (AREA)
  • Image Processing (AREA)

Description

Technisches GebietTechnical area

Die vorliegende Erfindung betrifft ein Verfahren zum Ausrichten eines Behälters. Des weiteren betrifft die vorliegende Erfindung ein Computerprogramm zum Durchführen des erfindungsgemäßen Verfahrens. Ein derartiges Verfahren ist aus DE 20 2005 014456U1 oder aus IT M 120 072 267 A1 bekannt.The present invention relates to a method for aligning a container. Furthermore, the present invention relates to a computer program for carrying out the method according to the invention. Such a procedure is over DE 20 2005 014456U1 or off IT M 120 072 267 A1 known.

Beschreibung des Standes der TechnikDescription of the Prior Art

Verfahren und Vorrichtungen zum Ausrichten von Behältern, insbesondere von Flaschen und für den Einsatz in Abfüll- und Etikettieranlagen, sind im Stand der Technik bekannt.Methods and devices for aligning containers, in particular bottles and for use in filling and labeling, are known in the art.

So zeigt die Offenlegungsschrift DE 10 2005 050 902 A1 eine Vorrichtung zum Ausrichten von Behältern in Bezug auf wenigstens ein geometrisches Behältermerkmal, die einen Transporteur mit Behälteraufnahmen für jeweils einen Behälter und entlang einer von dem Transporteur gebildeten Transportstrecke angeordnete Kameras eines Bilderkennungssystem umfasst, wobei das Bilderkennungssystem durch einen Vergleich der von den Kameras gelieferten Ist-Bilddaten mit in einer Auswert- und Steuerelektronik abgelegten Soll-Bilddaten oder Kennwerten das Ausrichten der Behälter bewirkt.Thus, the published patent application DE 10 2005 050 902 A1 a device for aligning containers with respect to at least one geometric container feature, comprising a conveyor with container receptacles for a respective container and along a transport path formed by the conveyor conveyor cameras of an image recognition system, wherein the image recognition system by comparison of the actual delivered by the cameras Image data with stored in an evaluation and control electronics target image data or characteristics causes the alignment of the container.

In dieser bekannten Vorrichtung werden die Behälter anhand eines geometrischen Merkmals ausgerichtet, beispielsweise anhand einer Prägung auf der Oberfläche des Behälters.In this known device, the containers are aligned by means of a geometric feature, for example by means of an embossment on the surface of the container.

Von diesem geometrischen Merkmal wird gemäß dem Stand der Technik ein Bild in der gewünschten Position der Flasche aufgenommen und anschließend eine jeweilige Verzerrung dieses geometrischen Merkmals für die Fälle berechnet, in denen der Behälter jeweils um einen bestimmten Winkel von dieser gewünschten Position verdreht ist. Die so erzeugten Bilder bzw. Bilddaten werden dann als Soll-Bilddaten hinterlegt und verwendet. Im Betrieb der Vorrichtung wird dann mittels einer Kamera ein Bild von dem Behälter bzw. der Flasche aufgenommen und mittels eines Algorithmus das so generierte tatsächliche Ist-Bild mit den hinterlegten Soll-Bilddaten verglichen. Dabei wird bestimmt, mit welchem Bild der Soll-Bilddaten die höchste Übereinstimmung besteht. Anhand der zu diesem Bild in den Soll-Bilddaten hinterlegten Verdrehung des Behälters bzw. der Flasche kann diese dann um den entsprechenden Winkel in die gewünschte Position gedreht werden.From this geometric feature, according to the prior art, an image is taken in the desired position of the bottle and then a respective distortion of this geometric feature is calculated for the cases in which the container is rotated by a certain angle from this desired position. The images or image data thus generated are then stored as target image data and used. During operation of the device, an image of the container or of the bottle is then taken by means of a camera and the thus generated actual image is compared with the stored desired image data by means of an algorithm. It is determined with which image of the target image data is the highest match. Based on the rotation of the container or bottle deposited in the target image data for this image, it can then be rotated by the corresponding angle into the desired position.

Gemäß dem Stand der Technik können des weiteren mehrere Stufen solcher Ausrichtvorgänge vorgesehen sein, um sukzessive die gewünschte Ausrichtung bzw. Position des Behälters bzw. der Flasche herbeizuführen.Furthermore, according to the state of the art, a plurality of stages of such alignment processes may be provided in order to successively bring about the desired orientation or position of the container or the bottle.

Des weiteren kann innerhalb einer dieser Stufen vorgesehen sein, dass mehrere Kameras gleichzeitig ein Bild der Flasche aufnehmen, das dann mit den Soll-Bilddaten verglichen wird, wobei auch dann der Ausrichtvorgang basierend auf der besten Übereinstimmung vorgenommen wird.Furthermore, it may be provided within one of these stages that a plurality of cameras at the same time take a picture of the bottle, which is then compared with the desired image data, wherein the alignment process is then carried out based on the best match.

Bei einem derartigen Ausrichtvorgang wird jedoch stets eine beste Übereinstimmung des aufgenommenen Ist-Bilds mit den Soll-Bilddaten bestimmt und zum Ausrichten der Flasche verwendet, selbst wenn diese in absoluter Betrachtung nur eine sehr niedrige Übereinstimmung mit den Soll-Bilddaten darstellt. So wird auch in dem Fall, in dem eine der Kameras das geometrische Behältermerkmal nicht in ihrem Suchbereich erfaßt, etwa weil das geometrische Behältermerkmal so verdreht ist, dass es in einem rechten Winkel zur Kameraachse angeordnet ist, eine beste Übereinstimmung mit einem der Bilder aus den Soll-Bilddaten ermittelt und ein entsprechender Ausrichtvorgang eingeleitet.In such an alignment process, however, always a best match of the captured actual image is determined with the target image data and used to align the bottle, even if this in absolute terms, only a very low agreement with the Represents target image data. Thus, even in the case where one of the cameras does not detect the geometric bin feature in its search area, such as because the geometric bin feature is skewed to be at a right angle to the camera axis, it will best match one of the images from the binoculars Determined target image data and initiated a corresponding alignment process.

Auch wenn diese Problematik in der Regel lediglich in der ersten Stufe des Ausrichtvorgangs auftritt und in den darauffolgenden Stufen meist kompensiert werden kann, besteht generell ein Bedarf an einem Verfahren zum Ausrichten eines Behälters, das eine schnellere und genauere Ausrichtung des Behälters ermöglicht.Although this problem usually only occurs in the first stage of the alignment process and can usually be compensated in the subsequent stages, there is generally a need for a method for aligning a container that allows for faster and more accurate alignment of the container.

Zusammenfassung der ErfindungSummary of the invention

Daher wird erfindungsgemäß ein Verfahren zum Ausrichten eines Behälters entsprechend dem Patentanspruch 1 vorgeschlagen.Therefore, a method for aligning a container according to claim 1 is proposed according to the invention.

Zum Ausrichten der Behälter wird ein neuronales Netz eingesetzt, dem vorab in einem Anlernvorgang beigebracht worden ist, wie das zur Ausrichtung eines Behälters herangezogene geometrische Behältermerkmal in den verschiedenen Orientierungen des Behälters aussieht, und das darüber in der Lage ist, zu erkennen, dass in einem Suchbereich das zu suchende geometrische Behältermerkmal, die sogenannte Suchinstanz, nicht vorhanden ist. Des weiteren ist eine den Ausrichtvorgang steuernde Steuerelektronik dazu in der Lage, eventuelle andere geometrische Behältermerkmale oder Bilder im Suchbereich dahingehend zu klassifizieren, dass diese nicht die Suchinstanz darstellen, indem auch diese sogenannten Exclude-Instanzen als solche vorab angelernt werden.For aligning the containers, a neural network is used, which has been taught beforehand in a learning process, what the geometric container feature used to align a container looks like in the different orientations of the container, and the one above it is able to recognize that in a search area the geometric container feature to be searched, the so-called search instance, does not exist. Furthermore, an alignment process controlling control electronics is able to classify any other geometric container features or images in the search area to the effect that they do not represent the search instance by these so-called exclude instances are taught as such in advance.

Auf diese Weise ist die Steuerelektronik im Rahmen der Musterkennung dazu in der Lage, bestimmte Aufnahmen des Suchbereichs von vornherein zu verwerfen, da sie die Such-Instanz nicht enthalten.In this way, the control electronics in the context of the pattern identifier is able to discard certain recordings of the search area from the outset, since they do not contain the search instance.

In Ausrichtstufen, in denen mehr als eine Kamera Ist-Bilddaten erzeugt, besteht somit die Möglichkeit, die Aufnahmen der Kameras, die die Suchinstanz nicht enthalten, auszusortieren, so dass diese nicht in Konkurrenz zu der Aufnahme treten können, die die Such-Instanz enthält.In alignment stages, in which more than one camera generates actual image data, it is therefore possible to sort out the images of the cameras that do not contain the search instance so that they can not compete with the image containing the search entity ,

Auf diese Weise wird ein sehr viel schnelleres und genaueres Ausrichtverfahren bereitgestellt. Grundsätzlich erfolgt die Mustererkennung, indem die Steuerelektronik die Ist-Bilddaten anhand von Soll-Bilddaten dahingehend klassifiziert, ob sie das geometrische Behältermerkmal enthalten. Die Soll-Bilddaten umfassen dabei das angelernte geometrische Behältermerkmal als Such-Instanz und des weiteren voranstehend beschriebene und ebenso angelernte Exclude-Instanzen.In this way, a much faster and more accurate alignment procedure is provided. In principle, pattern recognition takes place in that the control electronics classifies the actual image data on the basis of target image data as to whether they contain the geometric container feature. The target image data in this case comprise the learned geometric container feature as a search entity and the further described above and also trained exclude instances.

Das Anlernen der Soll-Bilddaten erfolgt, indem zunächst das auf einer Musterflasche befindliche zu suchende geometrische Behältermerkmal, die Suchinstanz, aufgenommen und abgespeichert wird. Das geometrische Behältermerkmal wird daraufhin von einem Benutzer identifiziert, insbesondere anhand seiner Kontur.The learning of the target image data takes place by first looking for the one located on a sample bottle geometric container feature, the search instance, recorded and stored. The geometric container feature is then identified by a user, in particular based on its contour.

Dieses Anlernen der Such-Instanz kann grundsätzlich am Einsatzort des Verfahrens, d. h. mit der tatsächlich aufgebauten Abfüll- und Etikettieranlage, oder aber abseits davon erfolgen, beispielsweise in einem Labor.This training of the search entity can basically be done at the site of the procedure, ie. H. with the actually constructed filling and labeling, or off it, for example, in a laboratory.

Nachdem die Suchinstanz nun grundsätzlich der Steuerelektronik bekannt ist, wird der Behälter bzw. die Flasche vor einer Bildaufnahmevorrichtung um sich selbst gedreht, wobei in bestimmten Gradschritten jeweilige Bilder des Behälters bzw. der Flasche gemacht werden. Die Vielzahl der Gradschritte hängt insbesondere davon ab, wie genau die Ausrichtung im Betrieb erfolgen soll. Soll beispielsweise eine Ausrichtung auf 1° genau erfolgen, ist es angezeigt, die Bilder ebenfalls in 1°-Schritten, besser in 0,5°-Schritten zu aufzunehmen.After the search instance is now basically known to the control electronics, the container or the bottle is rotated around itself in front of an image pickup device, with particular images of the container or the bottle being made in certain degree steps. The multitude of degree steps depends in particular on how exactly the alignment should be performed during operation. If, for example, you want to align with 1 °, it is advisable to take the pictures in 1 ° increments, better in 0.5 ° increments.

Aus diesen Bildern wird die Suchinstanz nun ausgeschnitten, falls sie noch in dem gesuchten Bereich zu finden ist, was automatisch geschehen kann, da die Suchinstanz bereits vorher angelernt wurde und dem neuronalen Netz bekannt ist.From these images, the search instance is now cut out, if it is still found in the searched area, which can happen automatically, since the search instance has already been trained and the neural network is known.

Der verbleibende Rest des jeweiligen Bildes wird nun automatisch als Exclude-Instanz angelernt.The remainder of the respective image is now automatically taught in as an exclude instance.

Auf diese Weise werden also über den gesamten Umfang des Behälters bzw. der Flasche aufgenommene Bilder in dem Anlernvorgang klassifiziert.In this way, images recorded over the entire circumference of the container or of the bottle are thus classified in the teaching process.

Bei der Anwendung des Verfahrens werden die von der Bildaufnahmevorrichtung aufgenommenen Ist-Bilddaten mit den in den Soll-Bilddaten abgelegten Instanzen verglichen und festgestellt, ob der Suchbereich eine Exclude-Instanz oder die Suchinstanz enthält.In the application of the method, the actual image data recorded by the image acquisition device is compared with the instances stored in the target image data and it is determined whether the search region contains an exclude entity or the search entity.

Enthält der Suchbereich nur eine Exclude-Instanz, wird dies angezeigt und das entsprechende Bild nicht zur Berechnung der Ist-Position der Flasche verwendet.If the search area contains only one Exclude instance, this is displayed and the corresponding image is not used to calculate the actual position of the bottle.

Sollte keine der Bildaufnahmevorrichtungen einer Ausrichtungsstufe die Suchinstanz in ihrem Suchbereich aufgenommen haben, d.h. es wurden lediglich Übereinstimmungen mit Exclude-Instanzen festgestellt, kann vorgesehen sein, die Flasche um eine vorgegebene Gradzahl, beispielsweise um 120°, zu drehen, um die Wahrscheinlichkeit zu erhöhen, dass die Bildaufnahmevorrichtungen der nächsten Stufe das geometrische Behältermerkmal in ihrem Suchbereich erfassen können.Should none of the image pickup devices of an alignment level have picked up the search entity in its search area, i. if only matches have been found with exclude instances, it may be intended to rotate the bottle by a predetermined number of degrees, for example 120 °, in order to increase the likelihood that the next stage image pickup devices will be able to detect the geometric bin feature in their search area.

Wird festgestellt, dass die Suchinstanz aufgenommen wurde, wird anhand einer Perspektivenverzerrung des geometrischen Behältermerkmals, eines bekannten Abstands der Bildaufnahmevorrichtung zu dem Behälter während der Aufnahme und eines bekannten Durchmessers des Behälters die Ist-Position des Behälters bzw. seine Orientierung berechnet. Aus der Differenz der Ist-Position des Behälters zu der gewünschten Soll-Position des Behälters kann nun die Gradzahl und die Richtung bestimmt werden, in die der Behälter zu drehen ist.If it is determined that the search instance has been recorded, the actual position of the container or its orientation is calculated on the basis of a perspective distortion of the geometric container feature, a known distance of the image recording device to the container during the recording and a known diameter of the container. From the difference between the actual position of the container to the desired desired position of the container, the degree and the direction in which the container is to be rotated can now be determined.

Der genaue Algorithmus zur Bestimmung dieses Verdrehwinkels ist aus dem Stand der Technik bekannt, siehe beispielsweise die eingangs zitierte Offenlegungsschrift DE 10 2005 050 902 A1 .The exact algorithm for determining this twist angle is known in the art, see for example, the cited publication DE 10 2005 050 902 A1 ,

Des weiteren deckt die vorliegende Beschreibung auch ein Computerprogramm mit Programmcode ab, um alle Schritte eines erfindungsgemäßen Verfahrens durchzuführen, wenn das Programm auf einem Computer oder einer entsprechenden Recheneinheit ausgeführt wird. Das Computerprogramm kann auf einem computerlesbaren Datenträger gespeichert sein.Furthermore, the present description also covers a computer program with program code to perform all the steps of a method according to the invention, when the program is executed on a computer or a corresponding computing unit. The computer program may be stored on a computer-readable medium.

Weitere Vorteile und Ausgestaltungen der Erfindung ergeben sich aus der Beschreibung und der beiliegenden Zeichnung.Further advantages and embodiments of the invention will become apparent from the description and the accompanying drawings.

Es versteht sich, dass die voranstehend genannten und die nachstehend noch zu erläuternden Merkmale nicht nur in der jeweils angegebenen Kombination, sondern auch in anderen Kombinationen oder in Alleinstellung verwendbar sind, ohne den Rahmen der vorliegenden Erfindung zu verlassen.It is understood that the features mentioned above and those yet to be explained below can be used not only in the respectively specified combination but also in other combinations or alone, without departing from the scope of the present invention.

Die Erfindung ist anhand eines Ausführungsbeispieles in der Zeichnung schematisch dargestellt und wird im Folgenden unter Bezugnahme auf die Zeichnung ausführlich beschrieben.The invention is illustrated schematically with reference to an embodiment in the drawing and will be described in detail below with reference to the drawing.

Kurzbeschreibung der ZeichnungBrief description of the drawing

Figur 1 zeigt eine schematische Ansicht einer Ausführungsform einer erfindungsgemäßen Vorrichtung, auf der das erfindungsgemäße Verfahren ausgeführt werden kann. FIG. 1 shows a schematic view of an embodiment of a device according to the invention, on which the inventive method can be performed.

Ausführliche Beschreibung der ZeichnungDetailed description of the drawing

Figur 1 zeigt eine Vorrichtung 1, auf der das erfindungsgemäße Verfahren ausgeführt werden kann. FIG. 1 shows a device 1, on which the method according to the invention can be carried out.

Die Vorrichtung umfasst ein Karussell 5, in das Behälter 2 über einen Einlauf 3 hineintransportiert und über einen Auslauf 4 hinaustransportiert werden. Von der Etikettierstation 7 werden die Behälter 2 in die gewünschte Orientierung zum Aufbringen eines Etiketts ausgerichtet.The device comprises a carousel 5, are transported into the container 2 via an inlet 3 and transported beyond a spout 4. From the labeling station 7, the containers 2 are aligned in the desired orientation for applying a label.

In der dargestellten Ausführungsform sind drei Stufen zur Ausrichtung vorgesehen, wobei die erste Stufe die Bildaufnahmevorrichtungen 8 und 9 umfasst, die zweite Stufe die Bildaufnahmevorrichtung 10 und die dritte Stufe die Bildaufnahmevorrichtung 11. Zur Erfassung qualitativ hochwertiger Bilder sind Hintergrundelemente 13, 14, 15 vorgesehen, die auch beleuchtet sein können.In the illustrated embodiment, three stages are provided for alignment, the first stage comprising the image pickup devices 8 and 9, the second stage the image pickup device 10, and the third stage the image pickup device 11. Background elements 13, 14, 15 are provided for capturing high quality images. which can also be lit.

Der Ausrichtvorgang wird von einer Steuerelektronik 12 gesteuert, die Ist-Bilddaten von den Bildaufnahmevorrichtungen 8, 9, 10, 11 empfängt und Steuerungsanweisungen an Behälteraufnahmen 6 ausgeben kann, auf denen die Behälter 2 angeordnet sind.The alignment operation is controlled by an electronic control unit 12 which receives actual image data from the image pickup devices 8, 9, 10, 11 and can output control instructions to container receptacles 6 on which the containers 2 are arranged.

Zur Inbetriebnahme und zur Vorbereitung des Ausrichtverfahrens hat ein Anlernvorgang zu erfolgen, bei dem zunächst anhand einer ersten Aufnahme eines Musterbehälters eine Suchinstanz, d.h. ein zur Ausrichtung eines Behälters herangezogenes geometrisches Behältermerkmal, von einem Benutzer identifiziert und die Suchinstanz so einem neuronalen Netz der Steuerelektronik 12 angelernt wird.For commissioning and for the preparation of the alignment process, a learning process has to be carried out, in which first a search instance, ie a geometrical container feature used to align a container, is determined from a first image of a sample container identified a user and the search instance is so trained a neural network of the control electronics 12.

Nun folgt eine 360°-Drehung des Behälters vor der Bildaufnahmevorrichtung, wobei beispielhaft pro Grad eine Bildaufnahme getätigt wird. Die dem neuronalen Netz nunmehr bekannte Suchinstanz wird im Suchbereich der Bildaufnahmevorrichtung ausgeschnitten und der verbleibende Suchbereich als Exclude-Instanz automatisch angelernt. Der Anlernvorgang wird nach einer vollständigen 360°-Drehung des Behälters 2 abgeschlossen.Now follows a 360 ° rotation of the container in front of the image pickup device, wherein an image capture per example is made per degree. The search instance which is now known to the neural network is cut out in the search area of the image acquisition device and the remaining search area is automatically trained as an exclude entity. The learning process is completed after a complete 360 ° rotation of the container 2.

Ist die Vorrichtung 1 an ihrem Einsatzort aufgestellt, kann durch eine Bedienperson der entsprechende Soll-Bilddatensatz aufgerufen werden. Es folgt nun ein langsamer Einlauf eines ersten Behälters bzw. einer ersten Flasche, wobei die Bildaufnahmevorrichtungen 8, 9, 10, 11 so ausgerichtet werden, dass sie die möglichen Lagen der Suchinstanz möglichst genau erfassen. Gewünscht ist selbstverständlich ein möglichst enger optischer Auswertungsbereich.If the device 1 is set up at its place of use, the corresponding target image data record can be called up by an operator. This is followed by a slow inlet of a first container or a first bottle, wherein the image recording devices 8, 9, 10, 11 are aligned so that they detect the possible positions of the search instance as accurately as possible. Of course, what is desired is a narrow optical evaluation range.

Anschließend kann der Schnelllauf für die Produktion gestartet werden.Subsequently, the high-speed run for the production can be started.

Auf diese Weise kann mittels des erfindungsgemäßen Verfahrens und des erfindungsgemäßen Computerprogramms ein vereinfachtes und somit beschleunigtes sowie qualitativ verbessertes Ausrichten von Behältern, insbesondere Flaschen, bereitgestellt werden.In this way, by means of the method according to the invention and the computer program according to the invention a simplified and thus accelerated and qualitatively improved alignment of containers, in particular bottles, can be provided.

Claims (9)

  1. Method for aligning a container (2) in relation to at least one geometric container feature in a set position, wherein an alignment of the container (2) is carried out by an analysis of at least one actual image datum in a control circuit (12) taken by means of at least one image recording device (8, 9, 10, 11), wherein the actual image data are analysed in a search region by means of a pattern recognition which provides an actual position of the container as a result if the geometric container feature is recognised, and displays if the geometric container feature has not been recognised, wherein the pattern recognition takes place in that the control circuit (12) classifies the actual image data on the basis of nominal image data to check whether they contain the geometric feature, characterised in that the nominal image data comprise the geometric container feature as a search instance and also comprise exclude instances, and that the search instance and the exclude instances are stored in the nominal image data in a learning process.
  2. Method according to claim 1, in which the search instances are learned in that pattern image data of the geometric container feature are generated in which the geometric container feature is identified by a user.
  3. Method according to claim 2, in which the pattern image data are generated by means of the at least one image recording device (8, 9, 10, 11).
  4. Method according to claim 2, in which the pattern image data are generated by means of a separate image recording device (8, 9, 10, 11).
  5. Method according to claim 1, in which the learning of the exclude instances comprises the following steps:
    - rotation of the container through a specific number of degrees,
    - generation of pattern image data of the search region of the container by means of the image recording device (8, 9, 10, 11),
    - cutting of the search instance out of the search region,
    - automatic learning of the remaining search region as an exclude instance,
    - repetition of the preceding steps until the container has been rotated through 360°.
  6. Method according to any one of the preceding claims, in which on output of an actual position, the actual position of the container is calculated from a perspective distortion of the geometric feature, a diameter of the container and a distance of the container from the at least one image recording device (8, 9, 10, 11).
  7. Method according to any one of the preceding claims, in which the container is automatically rotated through a particular number of degrees if the geometric feature is not recognised by any of the at least one image recording devices (8, 9, 10, 11).
  8. Computer program with program code means to perform all steps of a method according to any one of claims 1 to 7 when the computer programme is executed on a computer or corresponding calculator unit.
  9. Computer program according to claim 8 which is stored on a computer-legible data carrier.
EP09763831A 2008-11-20 2009-11-11 Method for aligning a container and computer program Active EP2358601B1 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
PL09763831T PL2358601T3 (en) 2008-11-20 2009-11-11 Method for aligning a container and computer program

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102008059229A DE102008059229A1 (en) 2008-11-20 2008-11-20 Method for aligning a container
PCT/EP2009/008025 WO2010057592A1 (en) 2008-11-20 2009-11-11 Method for aligning a container

Publications (2)

Publication Number Publication Date
EP2358601A1 EP2358601A1 (en) 2011-08-24
EP2358601B1 true EP2358601B1 (en) 2012-09-05

Family

ID=41682749

Family Applications (1)

Application Number Title Priority Date Filing Date
EP09763831A Active EP2358601B1 (en) 2008-11-20 2009-11-11 Method for aligning a container and computer program

Country Status (6)

Country Link
EP (1) EP2358601B1 (en)
CN (1) CN102216161B (en)
BR (1) BRPI0916040B1 (en)
DE (1) DE102008059229A1 (en)
PL (1) PL2358601T3 (en)
WO (1) WO2010057592A1 (en)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8983167B2 (en) 2012-05-14 2015-03-17 Gauss Surgical System and method for estimating a quantity of a blood component in a fluid canister
DE102013109384A1 (en) * 2013-08-29 2015-03-05 Krones Ag Device and method for transporting containers in a beverage filling plant
CN107229291B (en) * 2017-06-14 2020-09-15 苏州西斯派克检测科技有限公司 Synchronization method of online vision positioning system
DE102020112191A1 (en) * 2020-05-06 2021-11-11 Krones Aktiengesellschaft Container treatment machine and method for aligning a container in a container receptacle of a container treatment machine
DE102021112479A1 (en) * 2021-05-12 2022-11-17 Espera-Werke Gmbh Procedure for operating a labeling system

Family Cites Families (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH0624424A (en) * 1992-07-02 1994-02-01 Kuwabara Yasunaga Facing/p0sitioning method for container
US6484478B1 (en) * 2000-01-24 2002-11-26 Illinois Tool Works Inc. System and method for packaging oriented containers
ITBO20020806A1 (en) * 2002-12-20 2004-06-21 Azionaria Costruzioni Acma Spa LABELING AND / OR MARKING MACHINE
DE102005041497A1 (en) * 2005-09-01 2007-03-08 Stratec Control-Systems Gmbh Container e.g. beverage bottle, rotational direction detecting method, involves imaging container during transport of container, and analyzing images of container to determine position of joints that are formed while molding container
DE202005014456U1 (en) * 2005-09-12 2007-02-01 Krones Ag Conveyor transporting container e.g. transparent bottle, aligning device, has camera for optical image at container detecting secondary characteristic that stands in range relation to container-specific characteristic
DE102005050902A1 (en) 2005-10-21 2007-05-03 Khs Ag Device for aligning containers and labeling machine with such a device
DE102006026618A1 (en) * 2006-09-02 2008-03-13 Khs Ag Method for the accurate application of labels and labeling machine
ITBO20060847A1 (en) * 2006-12-12 2008-06-13 Adriano Fusco SYSTEM TO DETECT AND OPTIMIZE THE ANGULAR POSITIONING OF CONTAINERS IN CONTINUOUS TRANSIT
ITMI20072267A1 (en) * 2007-12-03 2009-06-04 Sidel Holdings & Technology Sa DETECTION SYSTEM AND ANGULAR ORIENTATION OF CONTAINERS IN LABELING MACHINES

Also Published As

Publication number Publication date
WO2010057592A1 (en) 2010-05-27
CN102216161B (en) 2015-02-25
DE102008059229A1 (en) 2010-06-02
CN102216161A (en) 2011-10-12
PL2358601T3 (en) 2013-02-28
EP2358601A1 (en) 2011-08-24
BRPI0916040A2 (en) 2015-11-10
BRPI0916040B1 (en) 2019-10-22

Similar Documents

Publication Publication Date Title
EP1777163B1 (en) Device for orienting containers
DE102012110376B4 (en) Method and apparatus for ordering code candidates in an image for decoding attempts
EP1717573B1 (en) Method for automatic detection of faults in test parts by means of an X-ray test system
EP2358601B1 (en) Method for aligning a container and computer program
DE102018212531B4 (en) Article transfer device
WO2019007632A1 (en) Image analysis including targeted preprocessing
DE3212432A1 (en) INSPECTION AND SORTING DEVICE AND METHOD
DE102006017337A1 (en) Method for optically detecting moving objects and apparatus
DE112020003158T5 (en) Training method for a target acquisition model, method and device for characterizing the data
DE202005017180U1 (en) Adjusting system for bottles on turntable comprises cameras which detect orientation of bottles at different positions and control unit which compares data obtained with desired orientation and controls adjustment of bottles
WO2019238477A1 (en) Method for examining concordant test objects
EP2787485A1 (en) Method and device for automatic detection of defects in flexible bodies
DE102019211672A1 (en) Training method for an artificial neural network
EP4147101A1 (en) Container treatment machine and method for aligning a container in a container receptacle of a container treatment machine
EP4221930A1 (en) Method, computer program product, and device comprising such a product for displaying the influence of cutting parameters on a cutting edge
DE102021203559A1 (en) Procedure for recognizing an identification number
EP1139285B1 (en) Method and apparatus for testing or inspection of articles
DE10024559B4 (en) Object recognition method
DE102018126212A1 (en) Procedure for aligning a camera according to its extrinsic parameters
DE102012211734A1 (en) Method for determining position of object i.e. workpiece, on desk machine tool, involves matching part of edges at standing area of object with part of transformed edges, and defining position as position of object on support surface
EP4085794B1 (en) Process for preparing and conducting quality testing of brushes, in particular toothbrushes, testing device and brush manufacturing machine
AT511399B1 (en) METHOD FOR THE AUTOMATED CLASSIFICATION OF INCLUSIONS
EP3678868B1 (en) Direct printing device for applying a circumferential printed image
EP4141805A1 (en) Method for the generation, automated annotation and provision of training image data
EP4350539A1 (en) Method and system for automatic image-based recognition of identification information on an object

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20110620

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

DAX Request for extension of the european patent (deleted)
GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

Free format text: NOT ENGLISH

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 573996

Country of ref document: AT

Kind code of ref document: T

Effective date: 20120915

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

Free format text: LANGUAGE OF EP DOCUMENT: GERMAN

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 502009004642

Country of ref document: DE

Effective date: 20121031

REG Reference to a national code

Ref country code: NL

Ref legal event code: VDEP

Effective date: 20120905

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20121205

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

Effective date: 20120905

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20121206

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

REG Reference to a national code

Ref country code: PL

Ref legal event code: T3

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130105

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

BERE Be: lapsed

Owner name: KHS G.M.B.H.

Effective date: 20121130

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20130107

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20121205

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

26N No opposition filed

Effective date: 20130606

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20121130

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 502009004642

Country of ref document: DE

Effective date: 20130606

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20121216

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20121111

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20121130

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20121111

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20131130

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20131130

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20091111

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120905

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 7

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 8

REG Reference to a national code

Ref country code: FR

Ref legal event code: PLFP

Year of fee payment: 9

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: PL

Payment date: 20171023

Year of fee payment: 9

Ref country code: AT

Payment date: 20171121

Year of fee payment: 9

Ref country code: GB

Payment date: 20171123

Year of fee payment: 9

REG Reference to a national code

Ref country code: AT

Ref legal event code: MM01

Ref document number: 573996

Country of ref document: AT

Kind code of ref document: T

Effective date: 20181111

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20181111

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: AT

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181111

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181111

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PL

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20181111

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: IT

Payment date: 20231124

Year of fee payment: 15

Ref country code: FR

Payment date: 20231120

Year of fee payment: 15

Ref country code: DE

Payment date: 20231121

Year of fee payment: 15