WO2024068377A1 - Procédé et dispositif d'annotation interactive de données d'image pendant une opération de service - Google Patents

Procédé et dispositif d'annotation interactive de données d'image pendant une opération de service Download PDF

Info

Publication number
WO2024068377A1
WO2024068377A1 PCT/EP2023/075869 EP2023075869W WO2024068377A1 WO 2024068377 A1 WO2024068377 A1 WO 2024068377A1 EP 2023075869 W EP2023075869 W EP 2023075869W WO 2024068377 A1 WO2024068377 A1 WO 2024068377A1
Authority
WO
WIPO (PCT)
Prior art keywords
machine
image data
component
digital model
coordinate system
Prior art date
Application number
PCT/EP2023/075869
Other languages
German (de)
English (en)
Inventor
Christof Hoppe
Original Assignee
Siemens Mobility GmbH
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Siemens Mobility GmbH filed Critical Siemens Mobility GmbH
Publication of WO2024068377A1 publication Critical patent/WO2024068377A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/98—Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/06—Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q10/00—Administration; Management
    • G06Q10/20—Administration of product repair or maintenance
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06Q—INFORMATION AND COMMUNICATION TECHNOLOGY [ICT] SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES; SYSTEMS OR METHODS SPECIALLY ADAPTED FOR ADMINISTRATIVE, COMMERCIAL, FINANCIAL, MANAGERIAL OR SUPERVISORY PURPOSES, NOT OTHERWISE PROVIDED FOR
    • G06Q50/00—Information and communication technology [ICT] specially adapted for implementation of business processes of specific business sectors, e.g. utilities or tourism
    • G06Q50/40—Business processes related to the transportation industry
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G06T7/0004—Industrial image inspection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30108—Industrial image inspection
    • G06T2207/30164—Workpiece; Machine component

Definitions

  • the invention relates to a method for learning an image recognition algorithm for fault detection in the maintenance of machines with multiple components, a computer program product for carrying out the method and a device for carrying out the method.
  • Artificial intelligence algorithms are trained in a learning phase for specific tasks in order to carry them out in a work phase.
  • input data is often annotated by people. If, for example, people or objects in photos are to be recognized by the algorithms during the work phase, photos with the objects to be recognized are annotated accordingly during the learning phase by manually marking the objects to be recognized on the photos.
  • a reliable algorithm requires a large number of such annotated photos.
  • annotation is difficult for a number of reasons. On the one hand, experts are required for the annotation. On the other hand, the amount of data required for training is often too small, for example because the defects to be detected rarely occur.
  • the publication DE 10 2018 214 210 Al relates to a visualization device and a method for visualizing the interior or exterior of a vehicle, in particular for planning retrofitting of a vehicle.
  • the invention is based on the object of proposing a method for the simplified training of an image recognition algorithm for the maintenance of complex machines.
  • the problem is solved by the subject matter of the independent patent claims. Further developments and refinements of the invention can be found in the features of the dependent patent claims.
  • a method for training an image recognition algorithm for error detection in the maintenance of machines with several components comprises the following method steps: a. Acquiring image data of the machine using at least one camera and storing the image data in a database; b. Creating a digital model of the machine based on the captured image data; c. Creating a machine coordinate system for the digital model of the machine; d. Carrying out maintenance work on at least one faulty component of the machine; e. Detecting a position of the serviced component and generating position data about the serviced component in the machine coordinate system; f. Marking the captured image data for the serviced component from the digital model of the machine based on the assigned position data in the machine coordinate system.
  • the machine is in particular a vehicle, in particular a rail vehicle.
  • Capturing image data from the machine in step a. is advantageously carried out with a large number of high-resolution cameras. A detailed image of the machine can then be created.
  • a three-dimensional digital model (3D model) of the machine can be created in method step b based on the captured image data.
  • the captured image data is first read from the database. Consequently, in method step c. a machine coordinate system is created for the digital 3D model of the machine.
  • the machine coordinate system is created from the image data using a suitable algorithm.
  • the origin could be a reference point on the machine that is easy to identify for an image recognition algorithm, such as a buffer or a wheel on a rail vehicle.
  • the machine is a rail vehicle and the capture of image data of the machine using at least one camera is carried out when the rail vehicle enters a maintenance depot, for example using a so-called camera tunnel.
  • the rail vehicle drives through this tunnel when entering the depot and is photographed from several sides, in particular from all sides.
  • Fewer cameras can then be arranged in the depot itself and aligned with the rail vehicle standing in its maintenance position, for example in accordance with method step e. to record the position of the serviced component.
  • the rail vehicle can be classified into the machine coordinate system, for example, by means of a reference point of the machine that is easy to identify for an image recognition algorithm, for example a buffer or a wheel of the rail vehicle.
  • process step a takes place. in the depot with the rail vehicle in its maintenance position. This can also be done for the process step a. the following process steps b. and c. as well as d. apply .
  • the depot is then equipped with a large number of cameras, particularly high-resolution ones, in different positions and viewing angles of the rail vehicle. Monitoring the maintenance work in accordance with step d. and recording the position of the serviced component according to method step e. can then be done through the same cameras.
  • the maintenance work in particular the servicing, repair or replacement of components of the machine, is carried out by trained service personnel.
  • process step e The position of the serviced component is recorded.
  • This can be further developed using at least one camera, in particular using at least two cameras, for example using the cameras in the depot when the rail vehicle is in its maintenance position.
  • the position of the serviced component is recorded in method step e using a pattern recognition algorithm based on image data that is recorded stereoscopically using at least two cameras.
  • the position of the serviced component can then be recognized using a triangulation of the at least two camera positions. For this purpose, for example, movements of the service personnel servicing the component can be monitored and tracked. Such methods are known from the field of motion capture without markers.
  • process step e. Ahead
  • Marking the faulty component using a marker especially by service personnel.
  • This can also be process step c. follow and process step d. precede or during step d. take place .
  • a technical device for example a pen as a passive marker, or a device that emits a signal, for example a laser pointer, can be used as an active marker, with which the service personnel can physically point out the faulty component.
  • Such methods have become known as position tracking of objects with markers.
  • a tracking unit can be provided for this.
  • process step d. includes :
  • process step c This can also generally follow process step c . and precede process step d .
  • the service personnel can, for example, mark the fault themselves, for example by pointing out the fault or a defective part of the component, for example a crack in the housing of the component or leaking oil .
  • process step e the position of the fault is recorded in addition to the position of the serviced component and, in addition to generating position data for the serviced component in the machine coordinate system, position data for the fault is also generated in the machine coordinate system. Accordingly, in process step f . the image data recorded for the fault from the digital model of the machine can also be marked using the assigned position data in relation to the fault in the machine coordinate system.
  • the recorded position data is used in method step f. transferred into the image data using the machine coordinate system and the captured image data from the digital model of the machine to the serviced component and/or the fault. ler marked based on the assigned position data in the machine coordinate system.
  • At least one image section which includes an image of the faulty component and/or the error, is marked.
  • the machine coordinate system it is known which point or area in the image or the image data shows which part of the machine, for example the rail vehicle.
  • the serviced component and/or the error is therefore assigned to the corresponding image data.
  • Further training includes process step d. :
  • Identifying the faulty or maintained component or fault in particular by maintenance personnel, for example by selecting from a list of machine components and/or faults, and generating information on the identified component and/or fault.
  • the information about the identified component can in turn be in one or stored in the same database.
  • Process step f can then be followed by:
  • Faulty components or errors can then be easily identified using the annotated image data of the machine, in particular by means of an image recognition algorithm.
  • the faulty component or error is marked passively, by tracking the maintenance work on the faulty component, or actively, with little effort, by pointing out the faulty component by the service staff.
  • the maintenance report which also corresponds to the usual activity, the component and/or the error determined using the image recognition algorithm in the marked image data for the serviced component and/or the error can then be annotated using detailed information about the component and/or the error .
  • fault detection in the maintenance of machines with multiple components can be carried out with the following method steps:
  • a computer program product This comprises commands which, when the program is executed by at least one suitable terminal, cause the terminal to carry out the method according to the invention.
  • the computer program product can be stored on a data carrier.
  • a device according to the invention is designed to carry out the method according to the invention.
  • it comprises the means already mentioned which are suitable for carrying out the respective method step:
  • At least one camera for capturing image data from the machine
  • At least one memory for storing the image data in a database
  • At least one processor for creating a digital model of the machine based on the captured image data
  • At least one processor for creating a machine coordinate system for the digital model of the machine
  • At least one tracking unit comprising at least one camera for detecting a position of the serviced component and at least one processor for generating position data for the serviced component in the machine coordinate system;
  • At least one processor for marking the captured image data of the serviced component from the digital model of the machine based on the associated position data in the machine coordinate system.
  • the processors can be one and the same processor in a computing unit.
  • the cameras can also be the same cameras, or different cameras can be used to carry out process steps a. and e.
  • the figure shows a schematic sequence of an embodiment of the method.
  • image data of a machine are captured using at least one camera and the image data are stored in a database.
  • a digital model of the machine is then created from these stored image data in method step b.
  • a machine coordinate system is created for the digital model of the machine (method step c).
  • step d maintenance work is carried out on at least one faulty component of the machine .
  • step e the position of the serviced component is recorded and position data in the machine coordinate system for the serviced component is generated .
  • an external tracking unit comprising the at least one camera and a suitably designed processor, recognizes the position of a marker , for example a pen, with which a maintenance employee points to the faulty component, relative to the rail vehicle .
  • process step f the captured image data for the component being serviced from the digital model of the machine are marked using the associated position data in the machine coordinate system .

Landscapes

  • Engineering & Computer Science (AREA)
  • Business, Economics & Management (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Human Resources & Organizations (AREA)
  • Quality & Reliability (AREA)
  • Strategic Management (AREA)
  • Economics (AREA)
  • Marketing (AREA)
  • General Business, Economics & Management (AREA)
  • Entrepreneurship & Innovation (AREA)
  • Tourism & Hospitality (AREA)
  • Operations Research (AREA)
  • Educational Administration (AREA)
  • Development Economics (AREA)
  • Multimedia (AREA)
  • Game Theory and Decision Science (AREA)
  • Health & Medical Sciences (AREA)
  • General Health & Medical Sciences (AREA)
  • Primary Health Care (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Management, Administration, Business Operations System, And Electronic Commerce (AREA)

Abstract

L'invention concerne un procédé d'apprentissage d'un algorithme de reconnaissance d'image pour une reconnaissance de défaillance dans la maintenance de machines ayant une pluralité de composants, un produit-programme informatique pour mettre en œuvre le procédé et un dispositif pour mettre en œuvre le procédé, le procédé comprenant les étapes de procédé suivantes consistant à : - acquérir des données d'image de la machine au moyen d'au moins une caméra et stocker les données d'image dans une base de données ; - créer un modèle numérique de la machine sur la base des données d'image acquises ; - créer un système de coordonnées de machine pour le modèle numérique de la machine ; - effectuer un travail de maintenance sur au moins un composant défectueux de la machine ; - acquérir une position du composant entretenu et - générer des données de position pour le composant entretenu dans le système de coordonnées de machine ; - marquer les données d'image acquises pour le composant entretenu à partir du modèle numérique de la machine sur la base des données de position attribuées dans le système de coordonnées de machine.
PCT/EP2023/075869 2022-09-30 2023-09-20 Procédé et dispositif d'annotation interactive de données d'image pendant une opération de service WO2024068377A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
DE102022210427.7 2022-09-30
DE102022210427.7A DE102022210427B3 (de) 2022-09-30 2022-09-30 Verfahren und Vorrichtung zur interaktiven Annotation von Bilddaten im Servicebetrieb

Publications (1)

Publication Number Publication Date
WO2024068377A1 true WO2024068377A1 (fr) 2024-04-04

Family

ID=86693324

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/EP2023/075869 WO2024068377A1 (fr) 2022-09-30 2023-09-20 Procédé et dispositif d'annotation interactive de données d'image pendant une opération de service

Country Status (2)

Country Link
DE (1) DE102022210427B3 (fr)
WO (1) WO2024068377A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090290757A1 (en) * 2008-05-22 2009-11-26 Mian Zahid F Inspection using three-dimensional profile information
DE102018214210A1 (de) 2018-08-22 2020-02-27 Siemens Mobility GmbH Visualisierungseinrichtung und Verfahren zum Visualisieren des Inneren oder Äußeren eines Fahrzeugs
CN111855667A (zh) * 2020-07-17 2020-10-30 成都盛锴科技有限公司 一种适用于地铁车辆的新型智慧列检系统和检测方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20090290757A1 (en) * 2008-05-22 2009-11-26 Mian Zahid F Inspection using three-dimensional profile information
DE102018214210A1 (de) 2018-08-22 2020-02-27 Siemens Mobility GmbH Visualisierungseinrichtung und Verfahren zum Visualisieren des Inneren oder Äußeren eines Fahrzeugs
CN111855667A (zh) * 2020-07-17 2020-10-30 成都盛锴科技有限公司 一种适用于地铁车辆的新型智慧列检系统和检测方法

Also Published As

Publication number Publication date
DE102022210427B3 (de) 2023-06-29

Similar Documents

Publication Publication Date Title
EP3768902B1 (fr) Procédé et dispositif de maintenance et/ou de réparation d'un engin de construction
DE102019122865B4 (de) Erfassungssystem, Arbeitssystem, Anzeigeverfahren für ein Erweiterte-Realität-Bild und Programm
WO2018087343A1 (fr) Procédé de commande d'un système de moyens de transport, système de traitement de données
DE102019131284B4 (de) Robotersystem und Koordinatenumwandlungsverfahren
DE112020007421T5 (de) Industriesystem, störungserfassungssystem und störungserfassungsverfahren
EP3767403A1 (fr) Mesure de forme et de surface assistée par apprentissage automatique destinée à la surveillance de production
DE10215885A1 (de) Automatische Prozesskontrolle
DE10197107T5 (de) Rangierbahnhofverfolgungssystem
EP3483682A1 (fr) Élaboration automatisée de documentation concernant une installation industrielle
EP4222039A1 (fr) Détection optique de voie ferrée
WO2024068377A1 (fr) Procédé et dispositif d'annotation interactive de données d'image pendant une opération de service
DE102018008366B4 (de) Verfahren und Anlage zum gestenbasierten Steuern eines Prüfprozesses an einem industriellen teilautomatisierten Sichtprüfplatzes eines industriellen technischen Prozesses oder einer industriellen technischen Produktionsanlage
DE102019208448A1 (de) Verfahren zur Überprüfung eines Geräts oder einer Maschine
EP1487616B1 (fr) Commande de processus automatique
DE102021106584B4 (de) System mit mindestens einem Anlagesystem mit mindestens mehreren Anlageteilen
DE102019209861A1 (de) System und Verfahren zur Visualisierung von Oberflächendefekten auf einem Werkstück
DE102020134680B4 (de) Verfahren und Anordnung zur Qualitätsprüfung eines Objekts
DE19822392C2 (de) Verfahren zur Registrierung von Koordinaten von mindestens einer Fehlerstelle auf einem Prüfobjekt
DE102018112910B4 (de) Herstellungsverfahren für eine Antriebseinrichtung und Prüfeinrichtung
DE102022109528A1 (de) Vorrichtung zum bestimmen von unregelmässigkeiten, verfahren zum bestimmen von unregelmässigkeiten, programm und system zum bestimmen von unregelmässigkeiten
WO2020120192A1 (fr) Procédé d'échange d'un composant dans une installation de transport de personnes ainsi que dispositif à mettre en œuvre avec le procédé
DE102005014979B4 (de) Verfahren und Anordnung zur Planung von Fertigungseinrichtungen
DE102019209117A1 (de) Verfahren zur Lokalisierung eines Fahrzeugs
DE102019127322A1 (de) Verfahren zur Erfassung von Objekten in einer Fahrzeugumgebung, Vorrichtung zur Datenverarbeitung, Computerprogrammprodukt und computerlesbarer Datenträger
DE102018002621A1 (de) Fahrzeugerfassungssystem

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23786180

Country of ref document: EP

Kind code of ref document: A1