DE102022204123A1 - Method for obtaining an identification data set for an industrial plant - Google Patents

Abstract

The invention relates to a method (100) for obtaining an identification data set for a plurality of objects to be inspected that recur along an industrial system based on a sequence of consecutive images recorded along the system. In order to avoid redundant evaluation and an improved allocation of depicted to real objects at the location of the system, it is proposed to assign an identifier to the recognized objects and record this in an identifier data set.

Description

The present invention relates to a method for obtaining an identification data set for a plurality of objects to be inspected that recur along an industrial system based on a sequence of consecutive images recorded along the system. The invention further relates to a method for visually inspecting such objects.

An example of such systems are conveyor belt systems for conveying bulk materials, for example. Here, the bulk material, such as overburden, ore, fuel or building materials, is transported using a closed conveyor belt supported and guided by a plurality of support rollers. Conveyor belt systems can also be referred to as conveyor belt systems or belt conveyor systems. The support rollers are usually rotatably mounted in support structures, so-called support roller chairs, which are evenly spaced apart along the conveyor route. A support roller chair can have one or more support rollers. The support rollers can be arranged in the support roller chair in a rigid or movable manner, for example in the form of a support roller garland. In this sense, support rollers and support roller chairs are understood as recurring objects along the conveyor belt system.

Conveyor belt systems can have conveyor routes several kilometers long, in which several hundred support rollers are regularly provided per kilometer of conveyor route. For each of these support rollers, operational and/or weather-related wear and tear, such as increased running resistance caused by defective bearings, can lead to increased energy consumption of the system or even damage to the conveyor belt. Due to the size of such systems and the associated high number of location-specific support rollers, an inspection of all support rollers in the system always involves a high level of logistical effort.

One way to carry out an inspection of conveyor belt systems is to use a drone or other unmanned vehicle to take images along the system and then evaluate them to determine the functional status of the idlers. Such a procedure is, for example, in DE 10 2020 206 497 A1 described. The thermal images recorded are used to detect increased heat development in the area of a support roller, which indicates increased running resistance and thus a defect.

In order to enable maintenance of the support roller identified as defective on the thermal image, it must be determined where exactly this roller is located in the system so that, for example, maintenance personnel can be dispatched specifically to the location of the roller. The GPS data of a recorded image, which is usually recorded as metadata, cannot be used to determine the location of a support roller depicted therein with sufficient precision, also because the stored GPS data indicates the location of the drone and not the location of the roller.

The DE 10 2020 206 497 A1 proposes as a solution to this problem a preliminary survey of the entire system and the creation of a 3D model in which each role is assigned an exact GPS position. The necessary preparatory work on the system to be inspected and the process of creating the 3D model are extremely time-consuming and must be carried out by specialized specialists, which increases the overall cost of an inspection. Alternatively, in the DE 10 2020 206 497 A1 proposed to attach an RFID transponder to each carrying roller chair and to enable the images to be clearly assigned to the rollers using appropriate readout electronics in the drone. This also involves time-consuming and cost-intensive preparatory work on the system. In addition, RFID transponders can be damaged by weather or during the operation of the system and then have to be replaced at great expense.

The JP 2019 018999 A also discloses an inspection of conveyor belt systems, which is based on the evaluation of images recorded along the system. The images are recorded with a rail-bound vehicle and put together to form a panoramic image, which is intended to enable better positioning of individual rollers. Due to the distance between the vehicle and the system, which is defined by the rail guide and in particular remains constant, the position of the vehicle can be better correlated (e.g. via GPS) with the position of individual rollers. However, rail-bound vehicles always require a rail guide that extends along the entire system, the installation of which requires a lot of work and money. Many systems are unsuitable for inspection using rail-bound vehicles due to their size or area of application. Rail-bound vehicles are also usually much slower and use more energy than free-moving or flying drones. According to the JP 2019 018999 A Processing into a panoramic image also requires that successive images overlap by more than 50%, which is the maximum travel speed of the rail-bound vehicle along the system further limited. Furthermore, such high overlap results in a lot of redundant data having to be recorded and stored.

The object of the present invention is therefore to provide a method that facilitates the evaluation of images of objects to be inspected recorded along an industrial system, enables a precise assignment of the objects depicted and recognized in the images to real objects of the system and requires the lowest possible requirements the type of image capture process and the infrastructure of the system. In particular, image sequences recorded using freely moving or flying drones should be able to be evaluated efficiently and reliably.

This object is achieved by a method for obtaining an identification data set with the features of claim 1 and by a method for visual inspection with the features of claim 4. Preferred features are the subject of the dependent claims. Further advantages and features can be found in the general description and the exemplary embodiments.

• - Empfangen eines Ausgangsbildes der Sequenz von Bildern, auf dem ein Objekt der Mehrzahl von Objekten abgebildet ist;
• - Erkennen des Objekts in dem Ausgangsbild und Zuordnen einer Kennung zu dem Objekt;
• - Empfangen eines Folgebildes der Sequenz von Bildern, das aus einer anderen Perspektive als das Ausgangsbild aufgezeichnet worden ist und auf dem das Objekt sowie ein nicht auf dem Ausgangsbild abgebildetes weiteres Objekt der Mehrzahl von Objekten abgebildet sind;
• - Erkennen des Objekts in dem Folgebild und Zuordnen der Kennung zu dem Objekt;
• - Erkennen des weiteren Objekts und Zuordnen einer neuen Kennung zu dem weiteren Objekt;
• - Optional: Wiederholung der vorangehenden Schritte mit dem Folgebild als neues Ausgangsbild;
• - Erfassen der Kennungen der erkannten Objekte als bildspezifischen Kennungsdatensatz.

The computer-implemented method according to the invention for obtaining an identification data set for a plurality of objects to be inspected, in particular idler rollers and/or idler roller chairs, recurring along an industrial system, in particular a conveyor belt system, on the basis of a sequence of successive images recorded along the system, comprises the following Steps:

• - Receiving an output image of the sequence of images on which one of the plurality of objects is depicted;
• - Detecting the object in the source image and assigning an identifier to the object;
• - Receiving a subsequent image of the sequence of images, which was recorded from a different perspective than the original image and on which the object and a further object of the plurality of objects not depicted on the original image are depicted;
• - Detecting the object in the subsequent image and assigning the identifier to the object;
• - Detecting the further object and assigning a new identifier to the further object;
• - Optional: Repeat the previous steps with the following image as the new initial image;
• - Recording the identifiers of the recognized objects as an image-specific identifier data set.

In this way, each object recognized in one or more images can be assigned a unique identifier, for example in the form of an identification number or character string, and recorded in an identifier data set. The identifier may also be referred to as an identifier or “ID”. The advantage of an identifier is that it can be assigned automatically or mechanically and initially without taking external information (such as position data, etc.) into account. The identifier can be ordered, for example assigned as an ascending sequence of numbers. The identifier data set is image-specific, i.e. the identifier data record shows which identifiers were assigned on the basis of a specific image and which images are the basis for assigning a specific identifier. The identifier record may include one or more databases.

The object recognized in the original image and provided with an identifier is tracked in the subsequent image. In some embodiments of the invention, position and/or movement data from the recording device used to record the sequence of images, for example a freely moving or flying drone, can also be used for tracking purposes. The tracked object is preferably assigned the same identifier as in the original image. Alternatively, the object in the subsequent image can be assigned an identifier that is linked to the identifier of this object in the source image in a way that enables the identifiers to be uniquely assigned. A “new” object detected in the subsequent image that is not shown in the original image is assigned a new identifier.

The method can be applied iteratively to all images of the sequence by selecting the subsequent image as a new initial image and another subsequent image after all identifiers have been assigned and repeating the object recognition and assignment of identifiers. The image of the sequence recorded immediately after the initial image is preferably used as the following image.

The identification of the recognized objects can be recorded as an image-specific identification data set, for example, immediately during the respective assignment or after each iteration step.

Instead of specifying a certain minimum overlap of consecutively recorded images, it is sufficient for the method according to the invention if one and the same object in the system (or recognizable parts of it) is shown in two images taken in particular immediately one after the other. Therefore, the method according to the invention is particularly suitable for image sequences recorded with freely moving or flying drones, in which the recording perspectives between two images can vary more in both translational and rotational terms. In addition, the method delivers easily evaluable results even with a relatively small number of images per longitudinal section of the system to be inspected. This means that the duration of an inspection trip, particularly carried out using a drone, can be significantly shortened and the amount of data produced and subsequently evaluated can be reduced.

The identifier assigned to the objects depicted and recognized in the images is an advantageous auxiliary variable that can be used for more efficient image evaluation and for a more precise assignment to the corresponding real objects in the system. The identification data set makes it possible to avoid redundant determinations about the functional status of an object through clever data management by evaluating only a selection or just one of these images instead of all the images on which an object is depicted and ignoring the remaining images or image areas. In this way, the evaluation of the images can be made more efficient. Furthermore, the assignment of an ordered identifier based on a specific reference or reference point of the system, e.g. in the form of a count of the objects starting from one of the ends of the system, can provide a very simple way of assigning the recognized objects to the real objects of the system. In the present method, such a system-related ordered identifier is preferably assigned. For example, in the case of a conveyor belt system, it can be determined from an assigned identifier and the route along which the sequence of images was recorded that a roller determined to be defective through image evaluation is the 42nd roller on the left side in the conveying direction of the system acts.

The method is based on a sequence of consecutive images recorded along the system. Images are the image data recorded with electronic image sensors, such as CCD or CMOS sensors, which can be further processed using electronic data processing systems. For example, these images may have been taken with one or more cameras on a freely moving or flying drone. The images can be light images (RGB images) and/or thermal images (infrared images). Light images are somewhat better suited for object recognition due to the wider color spectrum. Thermal images are particularly suitable for identifying certain defects that are associated with increased heat generation, such as a bearing defect in a support roller.

The recognition and tracking of objects depicted in the images can be carried out using known, suitable algorithms, in particular trained machine learning algorithms, from the field of digital image processing. For example, “bounding boxes” can be used in object detection. The processing of the images can include segmentation, in particular “semantic segmentation”. There are a number of suitable methods for recognizing objects based on image data, which is why a more detailed explanation is not provided here.

In a preferred embodiment of the invention, an optically recognizable marker attached to the system is recognized on the initial image and/or the subsequent image and the identifier is assigned depending on the recognized marker. For example, in some conveyor belt systems the idler roller chairs are provided with numbers or other markings using plates.

In a preferred embodiment of the invention, a distance between two neighboring objects depicted in the initial image and/or in the subsequent image is determined, with a warning message about a potentially unrecognized object being recorded in the identification data set if the distance is a predetermined one or based on the sequence of The threshold value determined in the images is exceeded. In this way, the identification data set can be used to check whether the sequence of images also depicts all of the objects to be inspected and thus a complete inspection of the system can be guaranteed. For example, based on a warning message received, an image recording can be repeated in an area of the system where the potentially overlooked object is expected. For this purpose, position data of the respective image recording position (e.g. in the form of GPS (meta) data) can be advantageously used. Position data of the respective image recording position of the images is preferably recorded in the identification data set.

The threshold value for the distance can be set based on previously known information about the system. For example, the specification of a conveyor belt system usually contains the nominal distance between the idler rollers and/or idler roller chairs, which can be taken into account when determining the threshold value. The distance between depicted objects can also be determined without any prior information nen statistically determined based on the sequence of images and which are taken into account when determining the threshold value.

According to what has already been described above and what is described below, the task set at the beginning is also solved by a method with the features of claim 4.

• - Gewinnung eines Kennungsdatensatzes nach einem der Ansprüche 1 bis 3;
• - Ermittlung eines Funktionszustandes der Objekte;
• - Erfassen des jeweiligen Funktionszustandes der Objekte in dem Kennungsdatensatz.

The computer-implemented method according to the invention for the visual inspection of a plurality of objects to be inspected, in particular idler rollers and/or idler roller chairs, recurring along an industrial system, in particular a conveyor belt system, comprises the following steps:

• - Obtaining an identification data record according to one of claims 1 to 3;
• - Determination of the functional status of the objects;
• - Recording the respective functional state of the objects in the identification data set.

All of the advantages described above and below are also realized in the inspection process.

In a preferred embodiment of the invention, the functional state of the objects is determined based on the sequence of images. The sequence of images can include light images and/or thermal images. In some embodiments, light images can be used for object recognition and creation of the identification data set and the thermal image can be used to determine the functional state of the objects. Even if light images and thermal images are recorded with different cameras, the relative position of the cameras to one another is usually known, e.g. if both cameras are attached to a drone, so that the light images can be reliably assigned to the thermal images.

The identification data set can of course also be used advantageously in combination with inspection data other than visual, for example acoustic data, which was recorded during the recording of the sequence of images.

In a further preferred embodiment of the invention, the functional state of the objects is determined on the basis of a real subset of the sequence of images, with the subset being selected based on the identification data set. In this way, evaluation of the images can be made more efficient. For example, redundant determinations about the functional status of an object can be avoided by evaluating only a selection or just one of these images instead of all the images on which an object is depicted and ignoring the remaining images or image areas.

The invention is also directed to a computer program product for carrying out the method according to one of claims 1 to 3 and/or the method according to one of claims 4 to 6 on a computer.

In all embodiments of the invention, the step of recording a sequence of successive images along the system, in particular by means of a freely moving or flying drone, can be part of the method according to the invention.

It is expressly pointed out that the embodiments of the invention explained above can each be combined with the subject matter of the independent claims individually or in any technically sensible combination.

• 1 ein Ablaufschema einer Ausführungsform des erfindungsgemäßen Verfahrens zur Gewinnung eines Kennungsdatensatzes;
• 2 ein Ablaufschema eines Kontrollverfahrens zur Überprüfung der Vollständigkeit der erkannten Objekte, das Teil des erfindungsgemäßen Verfahrens gemäß 1 sein kann;
• 3 ein Ausgangsbild einer entlang einer Transportbandanlage aufgenommenen Sequenz von Bildern wie es in einem erfindungsgemäßen Verfahren gemäß 1 verwendet werden kann;
• 4 ein Folgebildes der Sequenz von Bildern gemäß 3;
• 5 ein Bildausschnitt eines Bildes einer entlang einer Transportbandanlage, dessen Tragrollenstühle mit optischen Markern versehen sind, aufgenommenen Sequenz von Bildern wie es in einem erfindungsgemäßen Verfahren gemäß 1 verwendet werden kann;
• 6 ein Ablaufschema einer Ausführungsform des erfindungsgemäßen Verfahrens zur visuellen Inspektion einer Mehrzahl von entlang einer industriellen Anlage wiederkehrenden, zu inspizierenden Objekten.

Modifications and refinements of the invention as well as further advantages and details of the invention can be found in the following description and the drawings. Shown in the schematic figures:

• 1 a flowchart of an embodiment of the method according to the invention for obtaining an identification data record;
• 2 a flowchart of a control method for checking the completeness of the recognized objects, which is part of the method according to the invention 1 can be;
• 3 an initial image of a sequence of images recorded along a conveyor belt system, as in a method according to the invention 1 can be used;
• 4 a subsequent image according to the sequence of images 3 ;
• 5 an image detail of an image of a sequence of images recorded along a conveyor belt system, the support roller chairs of which are provided with optical markers, as in a method according to the invention 1 can be used;
• 6 a flowchart of an embodiment of the method according to the invention for the visual inspection of a plurality of objects to be inspected that recur along an industrial system.

Parts that have the same or similar effect are - where appropriate - provided with identical reference numbers.

Individual technical features of the exemplary embodiments described below can also be combined in combination with previously described exemplary embodiments as well as the features of the independent claims and any further claims to form objects according to the invention.

• - Empfangen eines Ausgangsbildes der Sequenz von Bildern, auf dem ein Objekt der Mehrzahl von Objekten abgebildet ist (Schritt 110);
• - Erkennen des Objekts in dem Ausgangsbild und Zuordnen einer Kennung zu dem Objekt (Schritt 120);
• - Empfangen eines Folgebildes der Sequenz von Bildern, das aus einer anderen Perspektive als das Ausgangsbild aufgezeichnet worden ist und auf dem das Objekt sowie ein nicht auf dem Ausgangsbild abgebildetes weiteres Objekt der Mehrzahl von Objekten abgebildet sind (Schritt 130);
• - Erkennen des Objekts in dem Folgebild und Zuordnen der Kennung zu dem Objekt (Schritt 140);
• - Erkennen des weiteren Objekts und Zuordnen einer neuen Kennung zu dem weiteren Objekt (Schritt 150);
• - Optional: Wiederholung der vorangehenden Schritte mit dem Folgebild als neues Ausgangsbild (Schritt 160);
• - Erfassen der Kennungen der erkannten Objekte als bildspezifischen Kennungsdatensatz (Schritt 170).

1 shows an embodiment of the method 100 according to the invention for obtaining an identification data set for a plurality of objects to be inspected that recur along an industrial system, based on a sequence of successive images recorded along the system, comprising the following steps:

• - receiving an output image of the sequence of images on which one of the plurality of objects is depicted (step 110);
• - Detecting the object in the source image and assigning an identifier to the object (step 120);
• - Receiving a subsequent image of the sequence of images that was recorded from a different perspective than the initial image and on which the object and another object of the plurality of objects not imaged in the initial image are imaged (step 130);
• - Detecting the object in the subsequent image and assigning the identifier to the object (step 140);
• - Detecting the further object and assigning a new identifier to the further object (step 150);
• - Optional: Repeat the previous steps with the following image as the new starting image (step 160);
• - Recording the identifiers of the recognized objects as an image-specific identifier data record (step 170).

The functionality of the method according to the invention is explained below with a view to 3 until 5 explained using an exemplary embodiment of a conveyor belt system 2, in which the objects to be inspected are support rollers 10, 11, 12, 13, 14.

An exemplary output image 300 as used in a method according to 1 or 2 can be used is in 3 shown. An associated follow-up image 400 is in 4 shown. A section of the conveyor belt system 2 is shown on the initial image 300. The conveyor belt system 2 comprises a closed conveyor belt 4, supported and guided by a plurality of support rollers 10, 11, 12, for transporting bulk goods, for example. The support rollers 10, 11, 12 are rotatably mounted in support roller chairs 10', 11', 12' arranged along the conveying route (cf. conveying direction F). The support rollers 10, 11, 12 are evenly spaced from one another, with the upper support rollers 10, 12 having a distance D.

On the received output image 300 ( 3 ) an object in the form of a support roller 10, 11, 12 can now be recognized, which is shown by dashed bounding boxes. The respective support roller 10, 11, 12 can be assigned a respective identifier 30, 31, 32, which can also be referred to as an ID. In the present case, for example, the identification 30 (“15”) is assigned to the support roller 10. The other support rollers 11, 12 are numbered in ascending order from left to right by assigning the identifiers “16” and “17”.

The received follow-up image 400 ( 3 ) is taken from a different perspective than the original image 300, here from a position offset in the conveying direction F. The following image 400 shows the support rollers 12, 13, 14. The support roller 12 is tracked on the subsequent image 400, that is, it is determined that this support roller is the same support roller as is also shown in the initial image 300. The support roller 12 is therefore assigned the same identifier 32 (“17”) as in the initial image 300. Alternatively, another identifier can also be assigned, which can be uniquely assigned to the identifier 32 in the initial image 300. The other support rollers 13, 14 recognized in the following image 400, which were not already recognized in the initial image 300, are assigned new identifiers 33, 34, here increasing numbers “18” and “19”.

Assigning an identifier in this way has many advantages, especially in the area of visual inspection of idlers. The identification data set, which shows, for example, which images in the sequence a specific idler roller is depicted, makes it possible to avoid redundant determinations about the functional status of a idler roller through clever data management by only selecting a selection or. instead of all the images on which the idler roller is depicted only one of these images is evaluated and the remaining images or image areas are ignored. Furthermore, the assignment of an ordered identifier based on a specific reference or reference point of the conveyor belt system, for example as in the 3 until 5 in the form of a count of the idler rollers starting from one of the Ends of the system, provide a very simple way to assign the identified idlers in the images to real idlers at the location of the conveyor belt system. A key advantage of an identifier is that it can be assigned automatically and initially without taking external information (such as position data, etc.) into account.

5 shows an image section 500 in a further embodiment of the invention, in which each support roller chair 10 'of the conveyor belt system 2 is provided with an optically recognizable marker 50 in the form of a license plate. In this case, the information of the marker 50 can be recognized optically and taken into account when assigning the identifier 30. In the present case, the number of the support roller chair is “42”, so that the support roller 10 is also assigned the number “42”.

2 shows a control method 200 for checking the completeness of the recognized objects, which can be part of the method 100 according to the invention. Here, a distance between two neighboring objects depicted in the initial image and/or in the subsequent image is determined (step 210). The distance is then compared with a predetermined threshold (step 220). If the threshold is exceeded, a warning message about a potentially unrecognized object is recorded in the identification data set. For example, with a view to 3 a distance D between the upper support rollers 10, 11 remains constant over the entire conveyor route of the conveyor belt system 2. If there are deviations or the threshold value is exceeded, it can be determined that not all support rollers have been recognized. The threshold value for the distance can be determined based on previously known information. Alternatively or additionally, the distance between imaged objects can be statistically determined based on the sequence of images and taken into account when determining the threshold value.

• - Gewinnung eines Kennungsdatensatzes nach einem der Ansprüche 1 bis 3 (Schritt 610);
• - Ermittlung eines Funktionszustandes der Objekte (Schritt 620);
• - Erfassen des jeweiligen Funktionszustandes der Objekte in dem Kennungsdatensatz (Schritt 630).

6 shows an embodiment of the method 600 according to the invention for the visual inspection of a plurality of objects to be inspected recurring along an industrial system, comprising the following steps:

• - Obtaining an identification data record according to one of claims 1 to 3 (step 610);
• - Determination of a functional state of the objects (step 620);
• - Detecting the respective functional state of the objects in the identification data set (step 630).

QUOTES INCLUDED IN THE DESCRIPTION

This list of documents listed by the applicant was generated automatically and is included solely for the better information of the reader. The list is not part of the German patent or utility model application. The DPMA assumes no liability for any errors or omissions.

Cited patent literature

• DE 102020206497 A1 [0004, 0006]
• JP 2019018999 A [0007]

Claims (7)

Computer-implemented method (100) for obtaining an identification data set for a plurality of objects to be inspected, recurring along an industrial system, in particular a conveyor belt system (2), in particular idler rollers (10, 11, 12, 13, 14) and/or idler roller chairs ( 10', 11', 12', 13', 14'), based on a sequence of consecutive images recorded along the system, comprising the following steps: - receiving an output image of the sequence of images on which one of the plurality of objects is depicted (step 110); - Detecting the object in the source image and assigning an identifier to the object (step 120); - Receiving a subsequent image of the sequence of images that was recorded from a different perspective than the initial image and on which the object and another object of the plurality of objects not imaged in the initial image are imaged (step 130); - Detecting the object in the subsequent image and assigning the identifier to the object (step 140); - Detecting the further object and assigning a new identifier to the further object (step 150); - Optional: Repeat the previous steps with the following image as the new starting image (step 160); - Recording the identifiers of the recognized objects as an image-specific identifier data record (step 170). Procedure according to Claim 1 , where the identifiers are system-related ordered identifiers. Procedure according to Claim 1 or 2 , wherein a distance between two adjacent objects depicted in the initial image and/or in the subsequent image is determined, wherein a warning message about a potentially unrecognized object is recorded in the identification data set if the distance exceeds a predetermined threshold value determined on the basis of the sequence of images . Computer-implemented method (600) for the visual inspection of a plurality of objects to be inspected, in particular idler rollers and/or idler roller chairs, recurring along an industrial system, in particular a conveyor belt system, comprising the following steps: - Obtaining an identification data set according to one of Claims 1 until 3 (step 610); - Determination of a functional state of the objects (step 620); - Detecting the respective functional state of the objects in the identification data set (step 630). Procedure according to Claim 4 , whereby the functional status of the objects is determined based on the sequence of images. Procedure according to Claim 5 , whereby the functional state of the objects is determined on the basis of a real subset of the sequence of images, the subset being selected based on the identification data set. Computer program product with a program code stored on a computer-readable medium for carrying out a method according to one of the Claims 1 until 6 .

Images