EP2998184B1 - System and method for automatically locating and recording a beacon - Google Patents

System and method for automatically locating and recording a beacon Download PDF

Info

Publication number
EP2998184B1
EP2998184B1 EP14380028.2A EP14380028A EP2998184B1 EP 2998184 B1 EP2998184 B1 EP 2998184B1 EP 14380028 A EP14380028 A EP 14380028A EP 2998184 B1 EP2998184 B1 EP 2998184B1
Authority
EP
European Patent Office
Prior art keywords
track
autonomous system
beacon
camera
processing unit
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active
Application number
EP14380028.2A
Other languages
German (de)
French (fr)
Other versions
EP2998184A1 (en
Inventor
David Celestino Salmeron
Jose Luis Munoz Munoz
Maria Saiz Munoz
Rocio Sanchiz Redondo
Faustino Frechiall Daza
Jaime Rodriguez Urbieta
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Siemens Rail Automation SA
Original Assignee
Siemens Rail Automation SA
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Siemens Rail Automation SA filed Critical Siemens Rail Automation SA
Priority to EP14380028.2A priority Critical patent/EP2998184B1/en
Priority to PT143800282T priority patent/PT2998184T/en
Priority to ES14380028T priority patent/ES2780623T3/en
Priority to PCT/EP2015/060496 priority patent/WO2016041645A1/en
Publication of EP2998184A1 publication Critical patent/EP2998184A1/en
Application granted granted Critical
Publication of EP2998184B1 publication Critical patent/EP2998184B1/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Images

Classifications

    • BPERFORMING OPERATIONS; TRANSPORTING
    • B61RAILWAYS
    • B61LGUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L3/00Devices along the route for controlling devices on the vehicle or vehicle train, e.g. to release brake, to operate a warning signal
    • B61L3/02Devices along the route for controlling devices on the vehicle or vehicle train, e.g. to release brake, to operate a warning signal at selected places along the route, e.g. intermittent control simultaneous mechanical and electrical control
    • B61L3/08Devices along the route for controlling devices on the vehicle or vehicle train, e.g. to release brake, to operate a warning signal at selected places along the route, e.g. intermittent control simultaneous mechanical and electrical control controlling electrically
    • B61L3/12Devices along the route for controlling devices on the vehicle or vehicle train, e.g. to release brake, to operate a warning signal at selected places along the route, e.g. intermittent control simultaneous mechanical and electrical control controlling electrically using magnetic or electrostatic induction; using radio waves
    • B61L3/121Devices along the route for controlling devices on the vehicle or vehicle train, e.g. to release brake, to operate a warning signal at selected places along the route, e.g. intermittent control simultaneous mechanical and electrical control controlling electrically using magnetic or electrostatic induction; using radio waves using magnetic induction
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B61RAILWAYS
    • B61LGUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L23/00Control, warning, or like safety means along the route or between vehicles or vehicle trains
    • B61L23/04Control, warning, or like safety means along the route or between vehicles or vehicle trains for monitoring the mechanical state of the route
    • B61L23/041Obstacle detection
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B61RAILWAYS
    • B61LGUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L25/00Recording or indicating positions or identities of vehicles or vehicle trains or setting of track apparatus
    • B61L25/02Indicating or recording positions or identities of vehicles or vehicle trains
    • B61L25/025Absolute localisation, e.g. providing geodetic coordinates
    • BPERFORMING OPERATIONS; TRANSPORTING
    • B61RAILWAYS
    • B61LGUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L2205/00Communication or navigation systems for railway traffic
    • B61L2205/04Satellite based navigation systems, e.g. GPS

Definitions

  • the present invention concerns a system and a method for automatically locating and recording beacons installed at points along a route followed by a guided vehicle.
  • the present invention is directed to the detection, localization and recording of beacons or balises installed on the route or way followed by the guided vehicle and which are configured for exchanging data with the guided vehicle by means of an electromagnetic signal each time the guided vehicle passes near, for example above/over, said balise or beacon.
  • said balise is an Eurobalise, i.e. a balise which complies with the European Train Control System, and is installed between rails of a railway followed by the guided vehicle.
  • Guided vehicle refers to public transport means such as buses, trol-leybuses, streetcars, subways, trains or train units, etc., as well as load transporting means such as, for example, overhead traveling cranes, freight trains, for which safety is a very important factor and which are guided along a route or railway by guiding means, for example at least one rail, in particular by two rails between which beacons/balises are placed.
  • public transport means such as buses, trol-leybuses, streetcars, subways, trains or train units, etc.
  • load transporting means such as, for example, overhead traveling cranes, freight trains, for which safety is a very important factor and which are guided along a route or railway by guiding means, for example at least one rail, in particular by two rails between which beacons/balises are placed.
  • balises are recorded using a manual procedure, wherein at least one operator has to be physically present on the track or route followed by the guided vehicle in order to record each balise by means of a portable device that has to be carried to the railway track, placed on top of the balise and then, via induction, used to upload the balise data into the portable device.
  • This task is time consuming and might be risky for the operator.
  • Benedikt Wenzel, et al. in "SAT.VALID - A New Data Validation Tool For Communication Based Train Control Systems” (Proceedings, IRSE ASPECT Conference - 2012 ) disclose a system mounted on-board a train for capturing an updated topology model of a track.
  • An objective of the present invention is to propose a method and system for automatically locating and recording balises or beacons installed at points along a route followed by a guided vehicle.
  • the present invention proposes to make use of an autonomous system, e.g. a robot, for automatically carrying out the detection, the localization and the recording of each beacon/balise installed at points along a route followed by a guided vehicle.
  • an autonomous system e.g. a robot
  • the invention concerns an autonomous system for automatically detecting, locating and recording at least one beacon or balise installed at a point along a route designed for a guided vehicle, according to claim 1.
  • the present invention also concerns a method for automatically detecting, locating and recording at least one beacon or balise installed at a point along a route designed for a guided vehicle like a train or metro, according to claim 10.
  • the present invention refers thus to an autonomous system, i.e. a system that does not require to be driven by an operator.
  • the processing unit according to the invention is configured for driving the autonomous system along the track by controlling the moving means, for detecting, locating and recording balises identified by means of the camera, free of any operator input.
  • the autonomous system comprises at least two cameras, a first camera pointing forward for taking images of the track in front of the autonomous system and identifying approaching balises, and a second camera pointing perpendicularly to the track, i.e. to the ground, used for a precise location of the balise.
  • FIG. 1 shows a preferred embodiment of an autonomous system 1 according to the invention, wherein the autonomous system 1 is in particular a robot.
  • Said autonomous system 1 comprises a frame or body 11 in, or on, which is installed at least one camera 12 configured for taking images of a route or track 2, said track 2 comprising guiding means for guiding a guided vehicle along a path defines by said track 2, said guiding means being for example one or several rails configured for guiding the guided vehicle, and in particular the autonomous system 1 according to the invention.
  • the autonomous system 1 according to the invention is thus configured for using the same guiding means provided by the track 2 as those used by the guided vehicle, e.g. said rails 21, in order to move on the same path as the guided vehicle along the track 2.
  • One or several balises 3 are installed at points along the track 2 and are configured for exchanging data with the guided vehicle when the latter passes at proximity, e.g. above, said balise 3.
  • the autonomous system 1 comprises moving means for autonomously moving its body 11 along the track 2.
  • Said moving means may comprise a connection to a power source and/or a power source, a motor 10 for converting the power of the power source into mechanical energy capable of moving the body 11, and therefore the autonomous system 1.
  • the motor converts the power of the power source, e.g. electric power, into a mechanical energy that makes a shaft rotating, and wherein the rotation of said shaft drives at least one wheel 13 into rotation.
  • the moving means of the autonomous system 1 according to the invention are configured for making use of an identical type of interaction with the track 2 as the guided vehicle has with said track 2 for moving the body 11, and therefore the autonomous system 1.
  • the autonomous system 1 comprises wheel 13, that are configured for being supported by the same rails 21.
  • the moving means according to the invention are in particular configured for cooperating with the guiding means of the track 2 that are configured for guiding the guided vehicle along the path defined by said track 2.
  • the wheels 13 of the moving means are configured for being supported and guided by the rails 21 of the track 2.
  • the present invention is not restricted to autonomous systems 1 using the guiding means of the track 2 for being guided along said track 2, but is also open to autonomous systems 1 like a drone capable of identifying the track 2 by means of the camera 12, and to follow the path described by said track 2.
  • the moving means according to the invention are in particular controlled by a processing unit 14 of the autonomous system 1 according to the invention.
  • said processing unit 14 is connected to the camera 12 for processing each image taken by the camera 12 during the displacement of the autonomous system 1 along the track 2.
  • the processing unit 14 preferentially comprises communication means for communicating with a remote communication device installed for instance in a remote command center.
  • the camera 12 is notably configured for taking images of track sections in front of the body 11 (hereafter also called downstream section) compared to its direction of displacement (indicated by the arrow A in the particular case of Fig. 1 ).
  • the processing unit 14 is connected to storing means 15 which may comprise a database for storing data and/or uploading data and/or modifying data already recorded in said storing means 15 and/or database.
  • said camera 12 is configured for capturing in real time at least one image of a downstream section of said track 2, said downstream section being a track portion extending from a first point of said track 2 located downstream of said body 11 to a second point of said track 2 located downstream of said body 11, the distance separating the camera 12 from the second point being greater than the distance separating the camera 12 from said first point.
  • the processing unit 14 is able to automatically analyze each image taken by each camera 12, to identify and locate said downstream section therein and to determine in real time if a balise 3 is located in said downstream section, notably between said first point and second point.
  • the processing unit 14 is able to digitize each image captured by the camera 12 for transforming said captured image into a format that can be used for localizing the balise 3.
  • the processing unit 14 is configured for detecting if a balise 3 is comprised in the image acquired by the camera and optionally for detecting and identifying the track 2.
  • the processing unit 14 uses in particular an object recognition algorithm.
  • the identification of the track 2 in each image may help detecting balises 3, since the position of a balise 3 compared to the track position might be used as a parameter for searching balises in the images acquired by the camera, said parameter being for example used by the object recognition algorithm.
  • said camera 12 might be configured and installed for taking images of portions of track that are located under the body 11 of the autonomous system 1.
  • said camera 12 may point perpendicularly to the track or ground (optical axis of the camera perpendicular to the track).
  • This configuration of the camera 12 may help in determining the position of the balise 3 compared to the position of the body 11. Indeed, it is then easier to determine the center of the balise 3 by making at least one part of the balise coinciding with the optical axis of the camera 12.
  • the autonomous system 1 comprises two cameras, a first camera pointing forward as previously described, and a second camera having its optical axis pointing perpendicularly to the track 2 as described above, so that the first camera provides a first approximation of the location and position of the balise and is used for controlling the approach of the autonomous system 1 toward the balise 3, and the second camera is used for the precise determination of the location/position of the balise under the body 11 of the autonomous system 1.
  • the images of both cameras might be processed by means of object recognition algorithm in order to determine balise characteristics and position.
  • first camera will refer to a camera pointing forwards
  • second camera will refer to a camera pointing perpendicularly to the track as previously described.
  • the processing unit 14 uses said object recognition algorithm for identifying a presence or absence of balise 3 in each image captured by the camera 12, for example by the first and/or second camera.
  • the recognition algorithm uses geometric recognition techniques applied to each image or to a part of each image (for example only to the part comprising the track, or only to the part comprised between two rails 21 of a track 2) for identifying for example the balise 3, and in particular the downstream section of the track 2.
  • the object recognition algorithm is in particular capable of first searching for a track 2 (e.g.
  • the object recognition algorithm preferentially uses automatic learning techniques such as boosting for identifying objects, like the track 2 or the balise 3, in the images taken by the camera 12. Techniques such as boosting for identifying objects in images are well known by the skilled man and do not need further explanations.
  • the recognition algorithm is able to make a correlation or matching between a set of pixels of the image captured by said camera and a set of features of the balise 3, and preferentially also of the track 2, wherein said features might be previously saved in the database of the autonomous system 1 according to the invention.
  • said features define for example specific geometric shapes of the balise 3, and preferentially also of the track 2, said features facilitating the track/balise identification by means of the object recognition algorithm by comparing features of the images with the features recorded for the balise/track in said database.
  • said method according to the invention may in particular include a learning phase intended to create said database.
  • the learning phase may comprise an acquisition of images by the camera of the autonomous system 1 when the latter is moving on a track 2 comprising beacons/balises installed at known points along the track 2. Said images acquired during the learning phase might be used as "learning" images (i.e. typical images representing a track comprising balises) for the detection by the object recognition algorithm of the balise and/or the track during the learning phase.
  • track/balise features like specific geometric shapes of the balise 3 and/or of the track 2, might be automatically stored in the database by the processing unit 14.
  • said database is used by the object recognition algorithm for the identification in real time of balise/track features in images captured by the camera.
  • said database might be upgradable and adaptive in that, once said learning phase is completed, it can be updated in real time using new images acquired by said camera during subsequent runs of the autonomous system 1 along the track 2 in order to complete the database and to improve the identification process of balises 3 by means of the object recognition algorithm.
  • the processing unit 14 may carry out at least one of the following steps:
  • the present invention proposes a system and a method for automatically detecting beacons at points along a route of a guided vehicle, wherein a processing unit is used for detecting beacons in images taken by at least two cameras on-board said system.

Description

  • The present invention concerns a system and a method for automatically locating and recording beacons installed at points along a route followed by a guided vehicle.
  • The present invention is directed to the detection, localization and recording of beacons or balises installed on the route or way followed by the guided vehicle and which are configured for exchanging data with the guided vehicle by means of an electromagnetic signal each time the guided vehicle passes near, for example above/over, said balise or beacon. In particular, said balise is an Eurobalise, i.e. a balise which complies with the European Train Control System, and is installed between rails of a railway followed by the guided vehicle. "Guided vehicle" according to the present invention refers to public transport means such as buses, trol-leybuses, streetcars, subways, trains or train units, etc., as well as load transporting means such as, for example, overhead traveling cranes, freight trains, for which safety is a very important factor and which are guided along a route or railway by guiding means, for example at least one rail, in particular by two rails between which beacons/balises are placed.
  • Nowadays, balises are recorded using a manual procedure, wherein at least one operator has to be physically present on the track or route followed by the guided vehicle in order to record each balise by means of a portable device that has to be carried to the railway track, placed on top of the balise and then, via induction, used to upload the balise data into the portable device. This task is time consuming and might be risky for the operator. Benedikt Wenzel, et al. in "SAT.VALID - A New Data Validation Tool For Communication Based Train Control Systems" (Proceedings, IRSE ASPECT Conference - 2012) disclose a system mounted on-board a train for capturing an updated topology model of a track.
  • An objective of the present invention is to propose a method and system for automatically locating and recording balises or beacons installed at points along a route followed by a guided vehicle.
  • The present invention proposes to make use of an autonomous system, e.g. a robot, for automatically carrying out the detection, the localization and the recording of each beacon/balise installed at points along a route followed by a guided vehicle.
  • More precisely, the invention concerns an autonomous system for automatically detecting, locating and recording at least one beacon or balise installed at a point along a route designed for a guided vehicle, according to claim 1.
  • The present invention also concerns a method for automatically detecting, locating and recording at least one beacon or balise installed at a point along a route designed for a guided vehicle like a train or metro, according to claim 10.
  • The present invention refers thus to an autonomous system, i.e. a system that does not require to be driven by an operator. The processing unit according to the invention is configured for driving the autonomous system along the track by controlling the moving means, for detecting, locating and recording balises identified by means of the camera, free of any operator input. Of course, it might be possible for an operator to remotely take the lead and drive the system according to the invention if needed. According to the invention, the autonomous system comprises at least two cameras, a first camera pointing forward for taking images of the track in front of the autonomous system and identifying approaching balises, and a second camera pointing perpendicularly to the track, i.e. to the ground, used for a precise location of the balise.
  • Further aspects of the present invention will be better understood through the following drawing:
  • Figure 1
    schematic representation of an autonomous system according to the invention.
  • Figure 1 shows a preferred embodiment of an autonomous system 1 according to the invention, wherein the autonomous system 1 is in particular a robot. Said autonomous system 1 comprises a frame or body 11 in, or on, which is installed at least one camera 12 configured for taking images of a route or track 2, said track 2 comprising guiding means for guiding a guided vehicle along a path defines by said track 2, said guiding means being for example one or several rails configured for guiding the guided vehicle, and in particular the autonomous system 1 according to the invention. Preferentially, the autonomous system 1 according to the invention is thus configured for using the same guiding means provided by the track 2 as those used by the guided vehicle, e.g. said rails 21, in order to move on the same path as the guided vehicle along the track 2. One or several balises 3 are installed at points along the track 2 and are configured for exchanging data with the guided vehicle when the latter passes at proximity, e.g. above, said balise 3.
  • The autonomous system 1 comprises moving means for autonomously moving its body 11 along the track 2. Said moving means may comprise a connection to a power source and/or a power source, a motor 10 for converting the power of the power source into mechanical energy capable of moving the body 11, and therefore the autonomous system 1. Preferentially, the motor converts the power of the power source, e.g. electric power, into a mechanical energy that makes a shaft rotating, and wherein the rotation of said shaft drives at least one wheel 13 into rotation. Preferentially, the moving means of the autonomous system 1 according to the invention are configured for making use of an identical type of interaction with the track 2 as the guided vehicle has with said track 2 for moving the body 11, and therefore the autonomous system 1. For example, if the guided vehicle is a train comprising wheels that are supported by rails 21, then preferentially, the autonomous system 1 comprises wheel 13, that are configured for being supported by the same rails 21. Additionally, the moving means according to the invention are in particular configured for cooperating with the guiding means of the track 2 that are configured for guiding the guided vehicle along the path defined by said track 2. For example, the wheels 13 of the moving means are configured for being supported and guided by the rails 21 of the track 2. Of course, the present invention is not restricted to autonomous systems 1 using the guiding means of the track 2 for being guided along said track 2, but is also open to autonomous systems 1 like a drone capable of identifying the track 2 by means of the camera 12, and to follow the path described by said track 2.
  • The moving means according to the invention are in particular controlled by a processing unit 14 of the autonomous system 1 according to the invention. In particular said processing unit 14 is connected to the camera 12 for processing each image taken by the camera 12 during the displacement of the autonomous system 1 along the track 2. The processing unit 14 preferentially comprises communication means for communicating with a remote communication device installed for instance in a remote command center. The camera 12 is notably configured for taking images of track sections in front of the body 11 (hereafter also called downstream section) compared to its direction of displacement (indicated by the arrow A in the particular case of Fig. 1). Preferentially, the processing unit 14 is connected to storing means 15 which may comprise a database for storing data and/or uploading data and/or modifying data already recorded in said storing means 15 and/or database.
  • Preferentially, said camera 12 is configured for capturing in real time at least one image of a downstream section of said track 2, said downstream section being a track portion extending from a first point of said track 2 located downstream of said body 11 to a second point of said track 2 located downstream of said body 11, the distance separating the camera 12 from the second point being greater than the distance separating the camera 12 from said first point. The processing unit 14 is able to automatically analyze each image taken by each camera 12, to identify and locate said downstream section therein and to determine in real time if a balise 3 is located in said downstream section, notably between said first point and second point.
  • Preferentially, the processing unit 14 is able to digitize each image captured by the camera 12 for transforming said captured image into a format that can be used for localizing the balise 3. The processing unit 14 is configured for detecting if a balise 3 is comprised in the image acquired by the camera and optionally for detecting and identifying the track 2. For this purpose, the processing unit 14 uses in particular an object recognition algorithm. Advantageously, the identification of the track 2 in each image may help detecting balises 3, since the position of a balise 3 compared to the track position might be used as a parameter for searching balises in the images acquired by the camera, said parameter being for example used by the object recognition algorithm.
  • Alternatively, said camera 12 might be configured and installed for taking images of portions of track that are located under the body 11 of the autonomous system 1. In other words, said camera 12 may point perpendicularly to the track or ground (optical axis of the camera perpendicular to the track). This configuration of the camera 12 may help in determining the position of the balise 3 compared to the position of the body 11. Indeed, it is then easier to determine the center of the balise 3 by making at least one part of the balise coinciding with the optical axis of the camera 12.
  • According to the invention, the autonomous system 1 comprises two cameras, a first camera pointing forward as previously described, and a second camera having its optical axis pointing perpendicularly to the track 2 as described above, so that the first camera provides a first approximation of the location and position of the balise and is used for controlling the approach of the autonomous system 1 toward the balise 3, and the second camera is used for the precise determination of the location/position of the balise under the body 11 of the autonomous system 1. The images of both cameras might be processed by means of object recognition algorithm in order to determine balise characteristics and position. In the following text, "first camera" will refer to a camera pointing forwards and "second camera" will refer to a camera pointing perpendicularly to the track as previously described.
  • Preferably, the processing unit 14 uses said object recognition algorithm for identifying a presence or absence of balise 3 in each image captured by the camera 12, for example by the first and/or second camera. In particular, the recognition algorithm uses geometric recognition techniques applied to each image or to a part of each image (for example only to the part comprising the track, or only to the part comprised between two rails 21 of a track 2) for identifying for example the balise 3, and in particular the downstream section of the track 2. For example, the object recognition algorithm is in particular capable of first searching for a track 2 (e.g. a track downstream section) in the image acquired by the camera, preferentially by the first camera, and second, once a track 2 or track downstream section has been identified in said image, searching for a balise 3 only on an area of the image wherein the track 2 or track downstream section has been identified. The object recognition algorithm preferentially uses automatic learning techniques such as boosting for identifying objects, like the track 2 or the balise 3, in the images taken by the camera 12. Techniques such as boosting for identifying objects in images are well known by the skilled man and do not need further explanations.
  • Preferably, and in order to identify a balise 3 in an image, an in particular in order to recognize said track 2 in the image acquired by the camera, e.g. by the first camera and/or the second camera, the recognition algorithm is able to make a correlation or matching between a set of pixels of the image captured by said camera and a set of features of the balise 3, and preferentially also of the track 2, wherein said features might be previously saved in the database of the autonomous system 1 according to the invention. Preferentially, said features define for example specific geometric shapes of the balise 3, and preferentially also of the track 2, said features facilitating the track/balise identification by means of the object recognition algorithm by comparing features of the images with the features recorded for the balise/track in said database.
  • Preferably, said method according to the invention may in particular include a learning phase intended to create said database. The learning phase may comprise an acquisition of images by the camera of the autonomous system 1 when the latter is moving on a track 2 comprising beacons/balises installed at known points along the track 2. Said images acquired during the learning phase might be used as "learning" images (i.e. typical images representing a track comprising balises) for the detection by the object recognition algorithm of the balise and/or the track during the learning phase. During said learning phase, track/balise features, like specific geometric shapes of the balise 3 and/or of the track 2, might be automatically stored in the database by the processing unit 14. In particular, once said learning phase is completed, then said database is used by the object recognition algorithm for the identification in real time of balise/track features in images captured by the camera. Advantageously, said database might be upgradable and adaptive in that, once said learning phase is completed, it can be updated in real time using new images acquired by said camera during subsequent runs of the autonomous system 1 along the track 2 in order to complete the database and to improve the identification process of balises 3 by means of the object recognition algorithm.
  • Preferentially, each time a balise 3 is identified by the object recognition algorithm in an image taken by the camera 12, in particular by the first camera and/or the second camera, then the processing unit 14 may carry out at least one of the following steps:
    • decreasing the moving speed of the body 1, i.e. of the autonomous system 1, by controlling the moving means, e.g. the motor, so that the speed of the autonomous system 1 according to the invention when passing above/close to a balise 3 is smaller than its speed when no balise is detected in order to improve the determination of the location of the balise 3. For example, the images taken by the first camera are used by the processing unit for controlling the approach of the autonomous system 1 towards the balise 3, and once the autonomous system 1 passes above the balise 3, the second camera and/or a device 16 (see below) might be used for determining the exact location of the balise, wherein the processing unit is able to determine for example the time T corresponding to an alignment of the optical axis of the second camera with a part of the balise (e.g. an edge) and/or the alignment of a receiving loop of the device 16 with a transmitting loop of the balise 3, and to deduct from the position of the autonomous system 1 at said time T the position of the balise, wherein said balise position might then be recorded in the database and/or transmitted to a remote control center;
    • controlling a device 16 for communicating with the balise 3, said device 16 comprising for example an emitter and a receiver and being installed on the body 11 so that in particular at least its emitter and receiver face the balise 3 when the body passes over/above the balise 3. Preferentially, the emitter is configured for remotely powering the balise 3, in particular by means of radiant energy. Said emitter comprises for example an antenna comprising an emitting loop for radiating energy, in particular radio frequency energy, the balise being then powered by said radiated energy and able to transmit, in return, an electromagnetic signal. The receiver comprises an antenna incorporating at least one receiving loop for picking up the electromagnetic signal produced by the balise 3, more precisely the electromagnetic signal produced by a transmitting loop of the balise 3, the receiver comprising thus a receiving loop configured for picking up the electromagnetic signal sent by the balise 3 in response to its powering by the emitter and for delivering a signal SR to the processing unit 14, said signal SR being current induced by the electromagnetic signal in said receiving loop, wherein said current provides a measure of the amplitude of the electromagnetic signal in function of the position of the receiving loop (e.g. in function of the position of the center of the receiving loop) compared to the position of the balise 3, more precisely compared to the position of the center of the transmitting loop of the balise 3. In the context of the present invention, the center of the balise 3 refers to the center of its transmitting loop, and centers of receiving or transmitting loops are notably geometric centers. The processing unit 14 is capable of processing the signal SR delivered by the receiving loop in order to collect information sent by the balise 3 in response to its powering and is able to send information to the balise 3, wherein said information may comprise the position of the balise 3 and/or a reference number. The processing unit is capable to determine the time T at which the center of the receiving loop coincides with the center of the transmitting loop from the signal SR. Such technique of balise center determination is known by the persons skilled in the art and does not need further explanations;
    • controlling a geolocation system for determining the position of each balise 3. Said geolocation system being preferentially installed in or on the body 11. Said geolocation system may make use of the camera installed on or in the body 11 and preferentially configured for having its optical axis pointing perpendicularly to the track 2, i.e. of said second camera, or of the device 16, for determining the position of the balise 3. For example, the position of the optical axis of the second camera might be used for determining the position of the balise on the track 2. Indeed, the processing unit 14 is able to identify said balise 3 in images taken by the second camera by means of said object recognition algorithm and to determine when the second camera optical axis points to or coincides with a part of the balise 3 from analyzing said images. For example, the database may comprise pre-recorded images of the balise, wherein data regarding the distance between the balise center and said part is recorded and used by the processing system for determining the position of the center of the balise 3 at the time T from the position of said part at said time T compared to the position of the autonomous system at said time T, wherein the position of the autonomous system at said time T is determined by means of the geolocation system. By position of the autonomous system, it has to be understood the position of a reference point (for example its geographic coordinates and/or the distance separating said reference point from a fixed reference as described below) of the autonomous system, and therefore, the determination of the position of the balise center compared to the position of the autonomous system means the determination of the position of the balise center compared to the position of the reference point. For example, said reference point might be the projection of the optical axis of the second camera on a horizontal plane. Said geolocation system is in particular configured for determining the position of the reference point of the autonomous system, e.g. the position of the optical axis of said second camera, compared to the fixed reference, the fixed reference being in particular a fixed position on a guided vehicle network, for instance a starting point on the track 2, by using techniques such as odometry and/or Global Positioning System techniques. The same applies mutatis mutandis for the determination of the position of the center of the receiving loop compared to said fixed reference, wherein the center of the receiving loop might be chosen as reference point. Once the position of the balise 3 is determined, then the processing unit 14 may write information related to the balise 3 position on a memory of the balise by means of the device 16 for communicating and/or may write said information in said database, and/or may compare an information related to the balise position provided by the balise 3 to the determined position, and/or may automatically communicate said position to a remote control center.
  • To summarize, the present invention proposes a system and a method for automatically detecting beacons at points along a route of a guided vehicle, wherein a processing unit is used for detecting beacons in images taken by at least two cameras on-board said system.

Claims (15)

  1. Autonomous system (1) for automatically detecting, locating and recording at least one beacon (3) installed at a point along a track (2) designed for a guided vehicle, the system comprising:
    - a body (11);
    - moving means for autonomously moving said body (11) along said track (2);
    - a first and a second camera installed on said body (11) and configured for taking images of the track (2), wherein the first camera is configured for pointing forward for taking images of the track (2) in front of the autonomous system, and the second camera is configured for having its optical axis pointing perpendicularly to the track (2);
    - a processing unit (14) for driving the moving means (14), analyzing the images, detecting in said images beacon (3) installed at points along the track (2), and determining the position of each beacon (3) ;
    - recording means (15) for recording information related to each detected beacon (3);
    wherein the first camera is used by the autonomous system for identifying approaching beacons and the second camera is used for a precise location of the beacon, wherein the processing unit (14) is configured for using the images acquired by the first camera for controlling the approach of the autonomous system (1) towards the beacon (3), and once the autonomous system (1) passes above the beacon (3), the processing unit (14) is configured for using the second camera for determining the exact location of the beacon (3).
  2. Autonomous system (1) according to claim 1, configured for cooperating with guiding means of the track (2) designed for guiding the guided vehicle in order to be guided along said track (2).
  3. Autonomous system according to claim 1 or 2, wherein the processing unit (14) comprises communication means.
  4. Autonomous system according to one of the claims 1-3, wherein the recording means (15) includes a database.
  5. Autonomous system according to one of the claims 1-4, wherein the processing unit (14) is configured for identifying at least one portion of track (2) in each image.
  6. Autonomous system according to one of the claims 1-5, wherein the processing unit (14) comprises an object recognition algorithm.
  7. Autonomous system according to one of the claims 4 to 6, wherein the database comprises information related to features of the beacon (3) and/or the track (2).
  8. Autonomous system according to one of the claims 1 to 7, comprising a device (16) for communicating and exchanging information with the beacon (3).
  9. Autonomous system according to one of the claims 1 to 8, comprising a geolocation system for determining the position of the beacon (3).
  10. Method for automatically detecting, locating and recording at least one beacon (3) installed at a point along a track (2) designed for guiding a guided vehicle, the method comprising:
    - moving an autonomous system (1) along said track (2), wherein the autonomous system (1) is capable of autonomously moving along said track (2);
    - taking images of said track (2) by means of a first and a second camera on-board the autonomous system (1), wherein the first camera is configured for pointing forward for taking images of the track (2) in front of the autonomous system, and the second camera is configured for having its optical axis pointing perpendicularly to the track (2);
    - detecting and locating beacons (3) installed at a point along said track (2) by analyzing, by means of a processing unit (14) of the autonomous system (1), the images of said track (2) taken by the cameras when the autonomous system (1) is moving along said track (2), wherein the first camera is used by the autonomous system for identifying approaching beacons and the second camera is used for a precise location of the beacon, wherein the processing unit (14) is configured for using the images acquired by the first camera for controlling the approach of the autonomous system (1) towards the beacon (3), and once the autonomous system (1) passes above the beacon (3), the processing unit (14) is configured for using the second camera for determining the exact location of the beacon (3);
    - recording information related to each detected beacon (3) in recording means (15) of the autonomous system (1).
  11. Method according to claim 10, comprising controlling a device (16) for communicating with the beacon (3) by means of the processing unit (14).
  12. Method according to claim 10 or 11, comprising controlling a geolocation system for determining the position of each beacon (3).
  13. Method according to one of the claims 10-12, comprising using an object recognition algorithm for detecting at least one part of the track (2) and/or the beacon (3) in the images taken by the cameras (12).
  14. Method according to one of the claims 10-13, comprising a learning phase for automatically collecting information related to the track (2) and/or beacon and storing said information in a database.
  15. Method according to claim 14, wherein the object recognition algorithm uses said information stored in the database for identifying the track (2) and/or beacon (3) in the images acquired by the cameras (12).
EP14380028.2A 2014-09-18 2014-09-18 System and method for automatically locating and recording a beacon Active EP2998184B1 (en)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP14380028.2A EP2998184B1 (en) 2014-09-18 2014-09-18 System and method for automatically locating and recording a beacon
PT143800282T PT2998184T (en) 2014-09-18 2014-09-18 System and method for automatically locating and recording a beacon
ES14380028T ES2780623T3 (en) 2014-09-18 2014-09-18 System and method to automatically locate and register a beacon
PCT/EP2015/060496 WO2016041645A1 (en) 2014-09-18 2015-05-12 System and method for automatically locating and recording a beacon

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
EP14380028.2A EP2998184B1 (en) 2014-09-18 2014-09-18 System and method for automatically locating and recording a beacon

Publications (2)

Publication Number Publication Date
EP2998184A1 EP2998184A1 (en) 2016-03-23
EP2998184B1 true EP2998184B1 (en) 2020-01-08

Family

ID=51795591

Family Applications (1)

Application Number Title Priority Date Filing Date
EP14380028.2A Active EP2998184B1 (en) 2014-09-18 2014-09-18 System and method for automatically locating and recording a beacon

Country Status (4)

Country Link
EP (1) EP2998184B1 (en)
ES (1) ES2780623T3 (en)
PT (1) PT2998184T (en)
WO (1) WO2016041645A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024049638A1 (en) * 2022-08-31 2024-03-07 Siemens Mobility, Inc. System and method for managing operations of a train relative to a fouling mark

Families Citing this family (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111923965B (en) * 2019-05-13 2022-02-11 株洲中车时代电气股份有限公司 Automatic driving method, system and equipment for locomotive
CN110641522A (en) * 2019-09-30 2020-01-03 中铁第四勘察设计院集团有限公司 Train positioning system of high-speed magnetic levitation track traffic
CN112829799B (en) * 2021-02-02 2023-04-28 中铁第四勘察设计院集团有限公司 Signal transponder deflection system of automatic vehicle base and control method thereof
CN113022643B (en) * 2021-03-23 2022-09-27 上海电气泰雷兹交通自动化系统有限公司 Fault-oriented safety transponder detection system and method

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004028881A1 (en) * 2002-09-23 2004-04-08 Siemens Aktiengesellschaft Sensor system and method for monitoring the travel path of a mobile unit
DE102012203483A1 (en) * 2011-04-27 2012-10-31 Trimble Navigation Ltd. Track railway track monitoring

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
DE102006062549B4 (en) * 2006-12-29 2016-05-19 Deutsches Zentrum für Luft- und Raumfahrt e.V. Method and device for the automatic positioning of trackside infrastructures
DE102011106345B4 (en) * 2011-06-08 2017-01-12 Deutsches Zentrum für Luft- und Raumfahrt e.V. path validation

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2004028881A1 (en) * 2002-09-23 2004-04-08 Siemens Aktiengesellschaft Sensor system and method for monitoring the travel path of a mobile unit
DE102012203483A1 (en) * 2011-04-27 2012-10-31 Trimble Navigation Ltd. Track railway track monitoring

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2024049638A1 (en) * 2022-08-31 2024-03-07 Siemens Mobility, Inc. System and method for managing operations of a train relative to a fouling mark

Also Published As

Publication number Publication date
ES2780623T3 (en) 2020-08-26
WO2016041645A1 (en) 2016-03-24
EP2998184A1 (en) 2016-03-23
PT2998184T (en) 2020-03-02

Similar Documents

Publication Publication Date Title
EP2998184B1 (en) System and method for automatically locating and recording a beacon
JP3421768B2 (en) Autonomous vehicle route guidance method, autonomous vehicle route guidance device, and autonomous vehicle equipped with route guidance device
CN108762276B (en) Automatic deviation rail correction device and automatic deviation rail correction method for AGV trolley
US8583313B2 (en) Robotic vehicle for performing rail-related actions
CN109552367B (en) Train running protection system and method based on combination of communication and active identification
US10300804B2 (en) Apparatus and method for automated positioning of a vehicle
US8466777B2 (en) Target object detection server
CN108196552A (en) A kind of GPS vision navigation systems of intelligent carriage
US10136106B2 (en) Train asset tracking based on captured images
US20170166207A1 (en) Method and system for automatically controlling a following vehicle with a front vehicle
CN106155066B (en) Carrier capable of detecting road surface obstacle and carrying method
JP7289184B2 (en) Automatic train operation system
EP3333043B1 (en) Rail inspection system and method
CN111273673A (en) Automatic driving following method and system of unmanned vehicle and unmanned vehicle
CN109415071A (en) For the method, apparatus and rail vehicle of the signal identification in rail traffic, particularly railway traffic, especially rolling stock
US10782419B2 (en) Method to determine clearance of an obstacle
JP3448088B2 (en) Obstacle detection system
CN105593103A (en) Device and method for rail-side monitoring of position of stabled rail-based vehicle
US20190373218A1 (en) Information processing system
US11017553B2 (en) Information processing system
US20220101723A1 (en) Annotation and Mapping for Vehicle Operation in Low-Confidence Object Detection Conditions
CN113985867A (en) Carrier control method, system, electronic device, medium and carrier
JP2021107994A (en) Autonomously traveling device and autonomous-travel managing system
JP7227936B2 (en) Drone guidance method and device
Hong et al. An Indoor Location-Tracking Using Wireless Sensor Networks Cooperated with Relative Distance Finger Printing

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

AX Request for extension of the european patent

Extension state: BA ME

17P Request for examination filed

Effective date: 20160819

RBV Designated contracting states (corrected)

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: EXAMINATION IS IN PROGRESS

17Q First examination report despatched

Effective date: 20170330

GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: GRANT OF PATENT IS INTENDED

INTG Intention to grant announced

Effective date: 20191002

RIN1 Information on inventor provided before grant (corrected)

Inventor name: CELESTINO SALMERON, DAVID

Inventor name: SANCHIZ REDONDO, ROCIO

Inventor name: RODRIGUEZ URBIETA, JAIME

Inventor name: SAIZ MUNOZ, MARIA

Inventor name: MUNOZ MUNOZ, JOSE LUIS

Inventor name: FRECHIALL DAZA, FAUSTINO

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE PATENT HAS BEEN GRANTED

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AL AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO RS SE SI SK SM TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602014059623

Country of ref document: DE

REG Reference to a national code

Ref country code: IE

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 1222339

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200215

REG Reference to a national code

Ref country code: PT

Ref legal event code: SC4A

Ref document number: 2998184

Country of ref document: PT

Date of ref document: 20200302

Kind code of ref document: T

Free format text: AVAILABILITY OF NATIONAL TRANSLATION

Effective date: 20200219

REG Reference to a national code

Ref country code: NO

Ref legal event code: T2

Effective date: 20200108

REG Reference to a national code

Ref country code: NL

Ref legal event code: MP

Effective date: 20200108

REG Reference to a national code

Ref country code: LT

Ref legal event code: MG4D

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: NL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

REG Reference to a national code

Ref country code: ES

Ref legal event code: FG2A

Ref document number: 2780623

Country of ref document: ES

Kind code of ref document: T3

Effective date: 20200826

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200508

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200408

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602014059623

Country of ref document: DE

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: SM

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

26N No opposition filed

Effective date: 20201009

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

REG Reference to a national code

Ref country code: BE

Ref legal event code: MM

Effective date: 20200930

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200918

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: BE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200930

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200930

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200930

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20200918

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

Ref country code: AL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20200108

REG Reference to a national code

Ref country code: AT

Ref legal event code: UEP

Ref document number: 1222339

Country of ref document: AT

Kind code of ref document: T

Effective date: 20200108

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: TR

Payment date: 20230915

Year of fee payment: 10

Ref country code: NO

Payment date: 20230906

Year of fee payment: 10

Ref country code: IT

Payment date: 20230920

Year of fee payment: 10

Ref country code: AT

Payment date: 20230814

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: PT

Payment date: 20230823

Year of fee payment: 10

Ref country code: FR

Payment date: 20230918

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: GB

Payment date: 20231009

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: ES

Payment date: 20231218

Year of fee payment: 10

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20231120

Year of fee payment: 10