EP4059805A2

EP4059805A2 - Real-time computer vision-based track monitoring

Info

Publication number: EP4059805A2
Application number: EP22162767.2A
Authority: EP
Inventors: Archie Marshall
Original assignee: Siemens Mobility Ltd
Current assignee: Siemens Mobility Ltd
Priority date: 2021-03-17
Filing date: 2022-03-17
Publication date: 2022-09-21
Also published as: GB2604882A; GB202103661D0; EP4059805A3

Abstract

A rail-vehicle mounted system arranged to receive data concerning a state of a track ahead of the vehicle, and to analyse the data in real time to indicate conditions of the track ahead of the vehicle, the system comprising a sensor unit mounted on the rail vehicle and directed at a track ahead of the rail vehicle in the direction of motion.

Description

The present invention provides methods and apparatus for performing real-time track monitoring, for example identifying the route set ahead of a rail vehicle from visual cues, such as the position of a switch plate in a set of railway points, bypassing the need to interface with other systems, using a type of computer vision.
Conventionally, the driver of a rail vehicle is responsible for monitoring of the track for obstructions or persons in danger. The condition of railway trackside assets has been monitored by site surveys, periodical maintenance inspections and requests for maintenance raised in response to detection of a defect in a track. Solutions such as those in CH693308 and WO2018/104454 have been proposed to automate the process of determining the presence of objects on or near the track. This may be done by detecting an object using the reflected component of a laser beam, or by recognising objects in a set region using optical imaging.
However, the identification of a route ahead of a train from visual cues is not currently performed. Setting of a route, in the sense of controlling operation of sets of points, may be performed automatically after being requested by the signaller or requested indirectly by the driver. Such route setting also relies on the infrastructure functioning correctly, for example, points moving and signals being set appropriately. Should there be an error in either of these the consequences could be catastrophic. Although such instances are minimal the risk to safety from infrastructure issues still exists, and it would be desirable to find a way to reduce this yet further without installing further trackside apparatus.
The present invention may be used in conjunction with the alarm apparatus and method described in co-pending UK patent application, GB2103662.9 .
The present invention accordingly provides methods and apparatus as set out in the appended claims.
The above, and further, characteristics and advantages of the present application will become more apparent from the following description of certain embodiments thereof, in conjunction with the accompanying drawing, wherein:

Fig. 1 illustrates an example embodiment of a system as may be mounted to a rail vehicle;
Figs. 2A-2F illustrate image capture steps of a system according to an embodiment of the present invention;
Fig. 3 is a schematic layout of a rail vehicle mounted rail track detection system in accordance with embodiments of the present invention; and
Fig. 4 is a flowchart illustrating the steps of a computer-implemented method of detecting the state of a rail track ahead of a rail vehicle in accordance with embodiments of the present invention.

The basic outline of embodiments of the present invention are set out below with reference to Figs. 1 and 2A-2F. Fig. 1 shows an example embodiment of equipment according to the present invention. The system comprises a main unit 10, itself comprising a data storage log 14 and computer vision processing equipment 12. The system also comprises a sensor unit 20. In some embodiments, sensor unit 20 may be built into a same enclosure with the main unit 10. The sensor unit 20 may be mounted externally to a rail vehicle. The main unit 10 may be mounted externally or internally of the rail vehicle, according to the designer. The sensor unit 20 may include a flexible phased array, mounted externally to the vehicle 30. It may be attached by magnets and calibrated in a known manner. With such an embodiment, the sensor unit 20 may be removable, and may be taken from one vehicle 30 to another according to requirements. The main unit 10 may be a compact computational unit, and may be mounted internally or externally. Similarly, it may be designed to be removable, and so able to be taken from one vehicle 30 to another according to requirements.
The main unit 10 receives data from sensor unit 20, which may be image data derived from LIDAR or SONAR imaging performed by sensor unit 20. Video cameras may be used instead, or indeed any representation of visual data. Other sensors 16, or data sources, may provide data to data log 14, and may be stored with corresponding entries of data provided by the sensor unit 20. An on board power supply 18 may be provided to the main unit 10 and sensor unit 20. An output 22 may be provided to supply outputs to other system units. For example, detection of an object on the track may give rise to generation of an alarm signal to an emergency system, which may cause an alarm to be sounded, or for brakes to be applied; or both.
By collecting and logging various data, the system of Fig. 1 can build up a detailed description of the track ahead of the rail vehicle. This allows the rail vehicle to anticipate the upcoming track layout. The behaviour of the rail vehicle may be adapted according to the anticipated track layout. For example, the vehicle speed may be modulated accordingly, or an alarm sounded, and so on.
As discussed, sensor unit 20 may include an image sensor, such as a video camera, LIDAR or SONAR sensors. Image data obtained by the image sensor may be used as data for track monitoring: to detect the route set ahead of the vehicle from visual cues; to detect defects in the track ahead of the vehicle; to detect obstructions on the track ahead of the vehicle; to detect and monitor the state of rail assets at trackside; or as input to a directional alarm system, such as described in co-pending UK patent application, GB2103662.9 . These aspects are described in further detail below.
Where sensor unit 20 includes an image sensor, processing of captured image data may be limited to an image region corresponding to an elongate rectangle covering sections of both rails of the track. Such an image region may be considered a region of interest, and reduction of image processing to such a limited image region may be found to reduce system and power requirements.
Use of such a region of interest may be useful in determining a direction of the track ahead of the vehicle: for example as illustrated in Figs. 2A-2F
Figs. 2A-2F represent image data derived from a region of interest. In each case, an upper sketch represents a location of a rail vehicle 30 on a track 32 carrying an imaging sensor 20, such as a LIDAR image sensor detecting a region of interest.
In the illustrated example, the image data will be used to determine which of two alternative paths the vehicle will take at points 34 in the track ahead of the train. In other embodiments, the image data will be used to determine a curvature of the track. Once the future path of the vehicle is determined, in can be used to direct a directional alarm carried by the vehicle, to protect a region ahead of the vehicle in the direction that it will be travelling, while ignoring regions in directions which the vehicle will not take.
In Fig. 2A, rail vehicle 30 is shown proceeding along a track 32. At the front of the vehicle, it carries a sensor unit 20, in this case an imaging unit, which emits an imaging beam 24. A region of interest 40 is defined in the image field. The lower part of Fig. 2A represents the content of the region of interest. The region of interest extends over both rails 42, so that sections of both rail heads are visible in the region of interest, and sleepers 44 or parts thereof and ballast 46 appear in the region of interest, providing an opportunity to monitor the condition of sleepers, and to store image data which may be evaluated later in regard to the condition of the sleepers.
Fig. 2B shows, in similar format, corresponding regions of interest captured corresponding to locations 40-3, 40-2, 40-1 shown in the drawing. The length of track covered by these images is a straight-ahead section, which accordingly represents the future path of the vehicle over the length of track represented by regions of interest 40-1, 40-2, 40-3.
Fig. 2C shows, again in similar format, corresponding regions of interest captured corresponding to locations 40-4, 40-5, 40-6 shown in the drawing. The length of track covered by these images is a straight-ahead section, leading into points 34. The point tips (switch plates) 38 are just visible in the region of interest 40-6. From this region of interest, it is just possible to deduce which direction the vehicle will be travelling in after the points. As the point tip 38 on the left is closed against the adjacent rail 42, it is apparent that the vehicle will be taking the track on the right out of the points 34. The systems of the invention may begin to direct detector beam 24, and any associated directional equipment such as a directional alarm system, to the track on the right.
Fig. 2D shows, again in similar format, corresponding regions of interest captured corresponding to locations 40-7, 40-8, 40-9 shown in the drawing. Locations 40-7, 40-8 and the corresponding images essentially correspond to locations 40-5 and 40-6 of Fig. 2C. Location 40-9 and the corresponding image are further along the track and show a widening separation in the two possible paths after the points 34. In a preferred embodiment, the system has deduced that the vehicle 30 will be travelling on the right-hand of the two possible paths. The detector beam 24 may therefore be directed along the right-hand path to provide optimal viewing of the path ahead of the vehicle 30. As it is by now known that the vehicle 30 will be taking the right-hand path, it is not necessary to image the left-hand path out of the points 34.
Figs. 2E and 2F continue the sequence, with imaging locations 40-10, 40-11, 40-12, 40-13, 40-14, 40-15 following the right-hand path out of the points 34. As can be clearly seen in Fig. 2F, the imaging proceeds along the right-hand path even before the vehicle 30 has reached the points 34.
The example described with reference to Figs. 2A-2F illustrates how the present invention can determine the shape of the track ahead of the train. While the described example relies on detection of the position of points, other arrangements will simply direct the detector beam 24 to follow the location of the rails 42. Other associated equipment, such as a directional alarm, may similarly be directed along the path to be taken by the vehicle 30.
Although illustrated as separate regions of interest at respective points on the track, the detector beam 24 and the received images will preferably represent the continuous surfaces of the rails 42 and nearby equipment. By using a detector of some sort, as will be apparent to those of skill in the art, anomalies in the expected images may be detected. Image recognition software, possibly including elements of artificial intelligence, may be found useful in such applications, as will be apparent to those skilled in the art. For example, a person, animal or debris on the track may be detected. Such detection may be communicated 22 to other systems aboard the vehicle, and may prompt a response such as applying brakes or sounding an alarm.
In another example, a trackside asset such as an axle counter, may be identified using computer vision such as described above. Means to identify location such as GPS may then be used by the system to identify the asset by reference to stored map data with asset information. In this example, a defect may be observed in captured image data of the asset, such as a cable may be detected to be disconnected in image collected. A corresponding maintenance request may then be raised against this asset, using the determined location or other asset identifier and with image(s) of the detected defect. The maintenance request can then be automatically communicated to maintenance staff systems so that the issue can be addressed.
In a further example, the rail head may be identified using imagery such as LiDAR or video to have buckled or warped in a specific location. This information can be collated and communicated as per the previous example.
In a particular embodiment, the stored map data of assets and the railway routes may be automatically updated to include information on detected defects and may be used to impose a temporary speed restriction or issue other notifications to passing vehicles until the issue has been addressed. Such notification(s) may be transmitted to signallers and/or other vehicles in the area. If suitably equipped, route map data stored on the other vehicles may be updated with information from the notification. Otherwise, printouts or text notifications of advice may be shared.
Accordingly, in certain embodiments of the invention, the system may be provided with means for updating a stored map with the information determined by the system. Means may also, or alternatively be provided for communicating the information determined to other systems in the vicinity or central information stores. Means may be provided for receiving updates from other systems in the vicinity or central information stores and acting on that information. Means may be provided for confirming and/or updating the information collected by other systems in the vicinity or central information stores.
The rail condition data derived from the described imaging may be provided to other systems as input data. In some embodiments, the images and data captured may be transmitted to a control centre, to enable other vehicles to use the derived information on track state. In addition, the captured images may be received and examined to determine a suitable response to an object on the track, for example.
In preferred embodiments, the vehicle 30 may change its behaviour in response to objects or damage identified in the captured images. For example, if an object is detected on the track, the vehicle brakes may be applied and an alarm sounded. Where a degradation in the condition of the track is identified, the vehicle may slow down until the degraded section of track has been passed.
The images obtained of the rails 42 may be used to detect and diagnose rail defects; to assess overall health of the rails; monitor the condition of various rail assets shown in the images, in addition to the described functions of identifying points positions and directing a directional alarm which may be carried by the vehicle.
The imaging arrangement as described above allows the system to anticipate the upcoming track layout for directing imaging and other directional equipment such as a directional alarm to persons in danger on the track ahead.
A directional alarm system which may be provided to the vehicle 30 may be arranged to emit a highly directional focused alarm sound beam. At low speeds, this beam's focus can be fixed straight ahead and will cover the track ahead of the vehicle sufficiently to detect and warn persons in danger.
At higher speeds, the protection offered by such alarm systems will be poor if the track diverges from the "straight-ahead" orientation. The sound beam's focus therefore needs to be adjusted to follow the future path of the vehicle 30.
The route ahead of the vehicle may diverge from the "straight ahead" orientation, due to a curvature in the track, or due to points, as examples. Once the path of the track ahead of the vehicle is calculated, the alarm sound beam may be directed along that path, to provide adequate warning to persons in danger on the track ahead. The alarm system may also rely on a focussed detection beam, to locate objects in the path of the vehicle. The detection beam is also preferably detected along the direction of the derived path ahead of the vehicle 30.
In an additional embodiment, an infra-red camera may be provided to support the identification of living beings in positions of danger. This can be in conjunction with other imaging sources such as the mentioned LiDAR or SONAR. However, alone, infra-red camera would also allow body heat signatures to be identified, by artificial intelligence or otherwise, in a configured imaging area. Alarm systems may be focussed to the detected body, or other emergency systems may be triggered as appropriate.
In optional features, trackside nodes may be provided, and passing vehicles may pass data to the node for storage, and communication to other nodes or the control centre, and a later passing vehicle may receive that data from the node, to indicate a condition of the track ahead. The nodes are preferably networked together - so that information can be relayed to the correct node by a train that has just passed a section and needs information to be passed back to a signaller/control centre, or to allow information to be communicated from such signaller/control centre to a train. Derivation of data from such a node ensures that it is provided in the relevant geographic location. Similarly, one vehicle 30 may communicate data concerning track conditions directly to another vehicle, if the two vehicles are in proximity; or such communication may be provided over a network for vehicles distanced from one another, such as to a central controller or signalling authority.
However, whilst the embodiments of the present invention may be advantageous in relation to the non-claimed examples outlined above, embodiments of the present invention are particularly advantageous when used in the anticipation of track layout. For example, a complete route network map may be provided, and the shape of the path ahead of the vehicle may be compared against the route network map, so that a current position of the vehicle may be calculated or deviations from the stored route network map may be identified and actioned. The system may capture images, as described above, and capture data from other sensors, to indicate the state of the track at the present position of the train. The captured data may be stored, and referred to when the vehicle passes over the same track again some time later, so that a condition of the rails ahead may be anticipated. Calculation of a current location of the vehicle may also employ the detection of trackside equipment, and the detected trackside equipment may be compared to a set of pre-loaded data in an attempt to find a match for trackside equipment imaged by the vehicle, in order to provide a precise location for the vehicle. Pre-loaded data aboard the vehicle 30 can also be used to identify additional information for the area the rail vehicle is in such as line speed or upcoming junction layouts.
Fig. 3 is a schematic layout of a rail vehicle mounted rail track detection system in accordance with embodiments of the present invention. A rail vehicle 50 is shown on a rail track 51 comprising a first rail 52a and a second rail 52b fixed to a plurality of sleepers 53a...n laid on ballast 54. The rail vehicle mounted rail track detection system 55 comprises a sensor unit 56, mounted on the rail vehicle, and image processing equipment 57, shown within the rail vehicle in this example. Housing the image processing equipment within the rail vehicle provides increased security and reduced likelihood of damage from everyday rail vehicle operation. However, both the sensor unit 56 and the image processing equipment 57 may fitted to the rail vehicle together in a single housing, if desired. The sensor unit 56 comprises a housing 58, within which is mounted a LiDAR system 59. The LiDAR system 59 comprises a laser 60 having a wavelength in the infra-red region of the electromagnetic spectrum, a phased array of optical antennas 61 adapted to illuminate at least one region of interest on the rail track 51, a moveable mount 62 adapted to direct the laser in the direction of the rail track ahead of the rail vehicle, and a photodetector 63 connected to the image processing equipment. A power source (not shown) may be provided, or the sensor unit 56 may be linked to a power source on the rail vehicle (also not shown).
The laser 60 is preferably a semiconductor laser diode operating in a wavelength range of 700nm to 1mm, and most preferably around a LiDAR suitable wavelength such as 905nm or 1550nm, and able to operate in a pulsed mode. The quality of the image received by the photodetector 63 is speed-dependent. This is compensated for by the number of individual beams and sensor channels in the sensor unit 56 set up. The phased array of optical antennas 61 comprises at least eight individual optical antennas 61a...n, a, matched with a corresponding number of channels in the photodetector 63. The field of view of the LiDAR system 59 does not need to be 360° as there is a limited range of vision required (to fit the width of the rail track 51), which aids in reducing the number of optical antennas 61a...n and channels required. Preferably, the field of view of the LiDAR system 59 is in the range of 60° to 120° degrees. The optical antennas 61a...n may be gratings fabricated on waveguides for the laser 60, and preferably the photodetector 63 is a CCD (Charge-Coupled Device) or CMOS (Complementary Metal-Oxide-Semiconductor) device. The moveable mount 62 comprises a motor and base, such that the base is tiltable by the motor to direct the laser 60, phased optical antenna array 61 and photodetector 63 in use.
A central controller 64 connects both the sensor unit 56 and the image processing equipment 57. The image processing equipment 57 comprises a processor 65 with I/O connections to a memory 66 and the sensor unit 56 via a communications interface 67, such as a transceiver, to receive and transmit information to a central control (not show). The central controller 64 also comprises a processor able to send commands to both the image processing equipment 57 and the sensor unit 56. This may be done automatically, or under the command of a rail vehicle driver. The central controller 64 links with other systems on the rail vehicle, such as GNSS (Global Navigation Satellite System), GSM-R (Global System for Mobile Communications-Railway) and cellular communications networks.
Fig. 4 is a flowchart illustrating the steps of a computer-implemented method of detecting the state of a rail track ahead of a rail vehicle in accordance with embodiments of the present invention. The set direction of a set of points is determined by detection of one of: a set of points, a switch plate, a junction route indicator or a signal. The method 400 may be used to activate the LiDAR system 59 of the sensor unit 56 continuously whilst the rail vehicle is in motion. Alternatively, the method 400 may be used to activate the LiDAR system 59 of the sensor unit 56 intermittently, for example, only when a set of points is expected on the track ahead from GNSS data. Initially, at step 402, LiDAR system 59 is activated to detect image data received from the rail track 61 when the rail vehicle 50 is in motion. At step 404, image data is received from at least one region of interest on the rail track. Figs. 2A to 2F illustrate three regions of interest within the scope of the LiDAR system 59. During motion of the rail vehicle 50, the direction of the LiDAR system 59 may be altered to match that of the upcoming rail track 51 by tilting the moveable base 62. Alternatively, the LiDAR system 59 may have a wide enough field of view to omit the moveable base 62 as all reasonable rail track 51 configurations will sit within that field of view. Whilst data is received continuously from the LiDAR system 59, regions of interest are analysed specifically to reduce the on-board computing cost required to implement the overall system. At step 406, the image data is processed to identify the set direction of a set of points in the rail track 51 ahead of the rail vehicle 50. Finally, at step 408, a state of the rail track 51 is output. This may be to the driver of the rail vehicle 50 or to a central control (not shown).
The steps of image detection and processing will now be described in more detail. Step 402, of activating a LiDAR system to detect image data from the rail track, comprises firstly, at step 4020 illuminating at least one region of interest on the rail track 51 using the phased array of optical antennas 61, and, secondly, if required at step 4022, directing the laser 60 in the direction of the rail track 51 ahead of the rail 51 vehicle a using the moveable mount 62. In situations where the LiDAR system is not activated continuously whilst the rail vehicle 50 is in motion, the laser 60 is turned on by a signal from the central controller 64 in order to create the multi-beam LiDAR feed required for image detection when needed. Otherwise, the laser 60 is on the entire time that the rail vehicle 50 is in motion.
The step 406 of image processing preferably takes place at the processor 65, although in certain circumstances data may be transmitted to a central control (not shown) via a cellular connection for analysis off-rail vehicle. The image processing steps are, however, the same, regardless of location. At step 4060, the presence of a set of points is detected using an edge detection algorithm. Suitable algorithms include first-order edge detection, such as Canny edge detection, Sobel edged detection, and second order edge detection, such as differential or phase transform techniques. At step 4062, if a set of points is detected, this is analysed using a supervised machine-learning algorithm to determine the state of the rail track 51. Initial training data may be obtained from a series of LiDAR scans of a rail network, with continual updating of data from LiDAR scans obtained whilst the rail vehicle 50 is in service. Data from other rail vehicles 50 in service on the same section of rail track 51 may also be included in the overall dataset. Training data may also be used in image matching, where the training data forms an initial template to allow either a feature-based approach of a template-based approach to be used in matching received image data and reference data. Analysing the set of points at step 4062 also comprises obtaining location data from the rail vehicle, accessing a database of sets of points matched to location data; and comparing the detected set of points and the location data to a corresponding entry in the database. If this step of comparing indicates a mismatch between the detected set of points and the location data and the corresponding entry in the database, or that there is no corresponding entry in the database, the step of outputting of the state of the rail track 51 comprises outputting an alarm. The alarm may be output to the driver of the rail vehicle 50 and/or a central control.
Alternatively, if may be that at step 406 a set of points is not detected. If this is the case, at step 4062, location data is obtained from the rail vehicle 50 and a database of sets of points matched to location data is accessed, and the step of outputting of the state of the rail track 51 comprises outputting an alarm. Again, this may be to the driver of the rail vehicle 50 and/or to a central control.
Whilst outputting an alarm leads to a warning on the state of the rail track 51, the continuous image detection may also be used to update not only datasets of real-time information on rail track 51 status but the database of sets of points and location data. The embodiments of the system and method described above therefore enable the identification of a route ahead of a train from visual cues as well as alerting to possible safety issues and danger on the rail track 51.

Claims

A rail vehicle mounted rail track detection system arranged to receive data concerning the state of a rail track ahead of the rail vehicle and to analyse the data in real time to indicate conditions of the track ahead of the rail vehicle, comprising:
a sensor unit mounted on the rail vehicle and directed at the rail track ahead of the rail vehicle in the direction of motion, the sensor unit being arranged to detect image data received from the rail track when the rail vehicle is in motion; and

image processing equipment arranged to perform object detection on the received image data to detect the direction in which the rail track is proceeding and to detect the set direction of a set of points in the rail track ahead of the rail vehicle.
A system according to claim 1, further comprising:
a light source arranged to provide a light beam to illuminate the rail track ahead of the rail vehicle.
A system according to claim 2, wherein the sensor unit comprises a LiDAR system, wherein the light source is a laser having a wavelength in the infra-red region of the electromagnetic spectrum and wherein the LiDAR system comprises a phased array of optical antennas adapted to illuminate at least one region of interest on the rail track, a MEMS mirror adapted to direct the laser in the direction of the rail track ahead of the rail vehicle, and a photodetector connected to the image processing equipment.
A computer-implemented method of detecting the state of a rail track ahead of a rail vehicle, comprising;
activating a LiDAR system to detect image data received from the rail track when the rail vehicle is in motion;

receiving image data from at least one region of interest on the rail track;

processing the image data to identify the set direction of a set of points in the rail track ahead of the rail vehicle; and

outputting a state of the rail track.
A method according to claim 4, wherein the step of processing the image data comprises:
determining the presence of a set of points using an edge detection algorithm; and, if a set of points is detected

analysing the set of points using a supervised machine-learning algorithm to determine the state of the rail track.
A method according to claim 5, wherein, if a set of points is detected, the step of analysing the set of points comprises:
obtaining location data from the rail vehicle;

accessing a database of sets of points matched to location data; and

comparing the detected set of points and the location data to a corresponding entry in the database.
A method according to claim 6, wherein if the step of comparing indicates a mismatch between the detected set of points and the location data and the corresponding entry in the database, or that there is no corresponding entry in the database, the step of outputting of the state of the rail track comprises outputting an alarm.
A method according to claim 4, wherein, if a set of points is not detected, the step of analysing the set of points comprises:
obtaining location data from the rail vehicle;

accessing a database of sets of points matched to location data;

; and, if a set of points is not detected

the step of outputting of the state of the rail track comprises outputting an alarm.
A method according to any of claims 4 to 8, wherein the step of activating a LiDAR system to detect image data from the rail track comprises the steps of:
illuminating at least one region of interest on the rail track using a phased array of optical antennas; and

directing the laser in the direction of the rail track ahead of the rail vehicle a using a moveable mount.
A method according to any of claims 4 to 9 wherein the set direction of a set of points is determined by detection of one of: a set of points, a switch plate, a junction route indicator or a signal.
A method according to any of claims 4 to 10, wherein the LiDAR system is activated continuously whilst the rail vehicle is in motion.
A method according to any of claims 4 to 10, wherein the LiDAR system is activated intermittently whilst the rail vehicle is in motion.