WO2016118672A2 - Vision industrielle et analyse de nuage de points en temps réel pour la télédétection et la commande de véhicule - Google Patents

Vision industrielle et analyse de nuage de points en temps réel pour la télédétection et la commande de véhicule Download PDF

Info

Publication number
WO2016118672A2
WO2016118672A2 PCT/US2016/014196 US2016014196W WO2016118672A2 WO 2016118672 A2 WO2016118672 A2 WO 2016118672A2 US 2016014196 W US2016014196 W US 2016014196W WO 2016118672 A2 WO2016118672 A2 WO 2016118672A2
Authority
WO
WIPO (PCT)
Prior art keywords
vehicle
data
map
information
assets
Prior art date
Application number
PCT/US2016/014196
Other languages
English (en)
Other versions
WO2016118672A3 (fr
Inventor
Fabien Chraim
Shanmukha PUTTAGUNTA
Anuj Gupta
Scott Harvey
Jason CREADORE
Graham Mills
Original Assignee
Solfice Research, Inc.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Solfice Research, Inc. filed Critical Solfice Research, Inc.
Priority to JP2017556798A priority Critical patent/JP2018508418A/ja
Priority to EP16740714.7A priority patent/EP3248140A4/fr
Priority to CN201680006431.6A priority patent/CN107533630A/zh
Publication of WO2016118672A2 publication Critical patent/WO2016118672A2/fr
Publication of WO2016118672A3 publication Critical patent/WO2016118672A3/fr

Links

Classifications

    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B61—RAILWAYS
    • B61L—GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L23/00—Control, warning or like safety means along the route or between vehicles or trains
    • B61L23/34—Control, warning or like safety means along the route or between vehicles or trains for indicating the distance between vehicles or trains by the transmission of signals therebetween
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B61—RAILWAYS
    • B61L—GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L23/00—Control, warning or like safety means along the route or between vehicles or trains
    • B61L23/04—Control, warning or like safety means along the route or between vehicles or trains for monitoring the mechanical state of the route
    • B61L23/041—Obstacle detection
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B61—RAILWAYS
    • B61L—GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L25/00—Recording or indicating positions or identities of vehicles or trains or setting of track apparatus
    • B61L25/02—Indicating or recording positions or identities of vehicles or trains
    • B61L25/025—Absolute localisation, e.g. providing geodetic coordinates
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B61—RAILWAYS
    • B61L—GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L25/00—Recording or indicating positions or identities of vehicles or trains or setting of track apparatus
    • B61L25/02—Indicating or recording positions or identities of vehicles or trains
    • B61L25/026—Relative localisation, e.g. using odometer
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B61—RAILWAYS
    • B61L—GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L27/00—Central railway traffic control systems; Trackside control; Communication systems specially adapted therefor
    • B61L27/04—Automatic systems, e.g. controlled by train; Change-over to manual control
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B61—RAILWAYS
    • B61L—GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L3/00—Devices along the route for controlling devices on the vehicle or train, e.g. to release brake or to operate a warning signal
    • B61L3/02—Devices along the route for controlling devices on the vehicle or train, e.g. to release brake or to operate a warning signal at selected places along the route, e.g. intermittent control simultaneous mechanical and electrical control
    • B61L3/08—Devices along the route for controlling devices on the vehicle or train, e.g. to release brake or to operate a warning signal at selected places along the route, e.g. intermittent control simultaneous mechanical and electrical control controlling electrically
    • B61L3/12—Devices along the route for controlling devices on the vehicle or train, e.g. to release brake or to operate a warning signal at selected places along the route, e.g. intermittent control simultaneous mechanical and electrical control controlling electrically using magnetic or electrostatic induction; using radio waves
    • B61L3/127—Devices along the route for controlling devices on the vehicle or train, e.g. to release brake or to operate a warning signal at selected places along the route, e.g. intermittent control simultaneous mechanical and electrical control controlling electrically using magnetic or electrostatic induction; using radio waves for remote control of locomotives
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/93—Lidar systems specially adapted for specific applications for anti-collision purposes
    • G01S17/931—Lidar systems specially adapted for specific applications for anti-collision purposes of land vehicles
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/10—Terrestrial scenes
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/59—Context or environment of the image inside of a vehicle, e.g. relating to seat occupancy, driver state or inner lighting conditions
    • B—PERFORMING OPERATIONS; TRANSPORTING
    • B61—RAILWAYS
    • B61L—GUIDING RAILWAY TRAFFIC; ENSURING THE SAFETY OF RAILWAY TRAFFIC
    • B61L2205/00—Communication or navigation systems for railway traffic
    • B61L2205/04—Satellite based navigation systems, e.g. global positioning system [GPS]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V2201/00—Indexing scheme relating to image or video recognition or understanding
    • G06V2201/02—Recognising information on displays, dials, clocks

Definitions

  • GPS Global Positioning System
  • supplemental sensing systems may be desirable, as well as highly detailed infrastructure and landmark maps, potentially including three-dimensional semantic maps.
  • Radio towers still require signaling equipment to be deployed in order for the radio communication to take place.
  • additional transponders have to be deployed along tracks for the train to reliably determine the position of the train and the track it is currently occupying.
  • ECS Error Control System
  • trackside equipment which relies on trackside equipment and a train-mounted control that reacts to the information related to the signaling. That system relies heavily on infrastructure that has not been deployed in the United States or in developing countries.
  • a solution that requires minimal deployment of wayside signaling equipment would be beneficial for establishing Positive Train Control throughout the United States and in the developing world.
  • Deploying millions of balises—the transponders used to detect and communicate the presence of trains and their location— every 1-15 km along tracks is less effective because balises are negatively affected by environmental conditions, theft, and require regular maintenance, and the data collected may not be used in real time.
  • Obtaining positional data through only trackside equipment is not a scalable solution considering the costs of utilizing balises throughout the entire railway network PTC.
  • train control and safety systems cannot rely solely on a global positioning system (GPS) as it not sufficiently accurate to distinguish between tracks, thereby requiring wayside signaling for position calibration.
  • GPS global positioning system
  • Local environment sensors which may include a machine vision system such as LiDAR, can be mounted on a vehicle.
  • a GPS receiver may also be included to provide a first geographical position of the vehicle.
  • a remote database and processor stores and processes data collected from multiple sources, and an on-board vehicle processor downloads data relevant for operation, safety, and/or control of the moving vehicle.
  • the local environmental sensors generate data describing a surrounding environment, such as point-cloud data generated by a LiDAR sensor. Collected data can be processed locally, on board the vehicle, or uploaded to a remote data system for storage, processing and analysis. Analysis mechanisms (on-board and/or
  • An exemplary embodiment of a system described herein includes a hardware component mounted on railroad or other vehicles, a remote database, and analysis components to process data collected regarding information about a transportation system, including moving and stationary vehicles, infrastructure, and transit pathway (e.g. rail or road) condition.
  • the system can accurately estimate the precise position of the vehicle traveling down the transit pathway, such as by comparing the location of objects detected in the vehicle's on-board sensors relative to the known location of objects. Additional attributes about the exemplary components are detailed herein and include the following:
  • the Hardware informs the movement of vehicles for safety, including: in railroad applications, identifying the track upon which they are traveling, obstructions, health of track and rail system, among other features; and in automotive applications, the lane upon which the vehicle is traveling, the texture and health of the road, the identification of assets in the vicinity, amongst other features.
  • the Remote Database contains information about assets, and which can be queried remotely to obtain additional asset information.
  • methods include machine vision data collected by the traveling vehicle itself, or by another vehicle (such as road-rail vehicles, track inspection vehicles, aerial vehicles, mobile mapping platforms, etc.). This data is then processed to generate the asset information (location, features, road/track health, among other information).
  • FIG. 1 is a representative flow diagram of a Train Control System
  • FIG. 2 is a representative flow diagram of the on board ecosystem
  • FIG. 3 is a representative flow diagram for obtaining positional information
  • FIG. 4 is an exemplary depiction of a train extrapolating the signal state
  • FIG. 5 is an exemplary depiction of the various interfaces available to the conductor as feedback
  • FIG. 6 is a representative flow diagram for obtaining the track ID occupied by the train
  • FIG. 7 is a representative flow diagram which describes the track ID algorithm
  • FIG. 8 is a representative flow diagram which describes the signal state algorithm
  • FIG. 9 is a representative flow diagram which depicts sensing and feedback.
  • FIG. 10 is a representative flow diagram of image stitching techniques for relative track positioning.
  • FIGS. 11 A and 1 IB are flow diagrams of point-cloud analysis processes.
  • FIG. 12 is a schematic block diagram of an apparatus for point-cloud analysis.
  • FIG. 13 is a flow diagram of a process for analyzing point-cloud data.
  • FIG. 14 is a further flow diagram of a process for analyzing point-cloud data.
  • FIG. 15 is a chart illustrating point cloud tile size and density distribution in an exemplary point-cloud survey.
  • FIG. 16 is a schematic block diagram of a point-cloud processing cluster.
  • FIG. 17 is a plot of characteristics for compression mechanisms usable with point- cloud data.
  • FIG. 18 is a plot of characteristics for compression mechanisms usable with point- cloud data.
  • FIG. 19 is a plot of characteristics for compression mechanisms usable with point- cloud data.
  • FIG. 20 is a flow diagram of a process for track detection.
  • FIG. 21 is a visualization of a point-cloud section with extracted rail information.
  • FIG. 22 is a histogram of point-cloud intensity levels in an exemplary point-cloud segment.
  • FIG. 23 is a visualization of track detection mechanism output.
  • FIG. 24 is a schematic block diagram of a map generation system utilizing supervised machine learning.
  • FIG. 25 is a schematic block diagram of a run-time system for automobile localization, automobile control and map auditing.
  • methods and apparatuses are provided for determining the position of one or more moving vehicles, e.g., trains or autonomous driving vehicles, without depending on balises/transponders distributed throughout the operating environment for accurate positional data.
  • embodiments are sometimes referred to herein as BVRVB-PTC, a PTC vision system, or a machine vision system.
  • railway embodiments can use a series of sensor fusion and data fusion techniques to obtain the track position with improved precision and reliability.
  • Such embodiments can also be used for auto-braking of trains for committing red light violations on the track, for optimizing fuel based on terrain, synchronizing train speeds to avoid red lights, anti-collision systems, and for preventative maintenance of not only the trains, but also the tracks, rails, and gravel substrate underlying the tracks.
  • Some embodiments may use a backend processing and storage component for keeping track of asset location and health information (accessible by the moving vehicle or by railroad operators through reports).
  • embodiments to take advantage of highly detailed infrastructure and landmark maps can be utilized to direct the flow of traffic in the real world and plan routes for vehicles to travel from source to destination.
  • the three-dimensional nature of the maps in addition to their accuracy in representing the physical world, assist the vehicles in anticipating events beyond their sensing range, foveating their sensors to the assets of interest, and localizing the vehicles in relation to the landmarks.
  • highly detailed three-dimensional (semantic) maps for the pseudo-static assets, the vehicle's resources are liberated to observe the dynamic objects around it.
  • the PTC vision system may include modules that handle communication, image capture, image processing, computational devices, data aggregation platforms that interface with the train signal bus and inertial sensors (including on-board and positional sensors).
  • FIG. 1 illustrates an exemplary flow operation of a Train Control System.
  • step SI 00 a train undergoes normal operation.
  • step SI 05 the train state is retrieved from the Data Aggregation Platform (described below).
  • step SI 10 the train position is refined.
  • step SI 15 semaphore signal states are identified from local environment sensor information.
  • step S120 feedback is applied.
  • the train speed can be adjusted (step S125), alarms and/or notifications can be raised (step S130). Further detail concerning of each of these steps is described hereinbelow.
  • a PTC vision system may include one or more of the following: Data Aggregation Platform (DAP) 215, Vision Apparatus (VA) 230, Positive Train Control Computer (PTCC) 210, Human Machine Interface (HMI) 205, GPS Receiver 225, and the Vehicular Communication Device (VCD) 220, typically communicating via LAN or WAN communications network 240.
  • DAP Data Aggregation Platform
  • VA Vision Apparatus
  • PTCC Positive Train Control Computer
  • HMI Human Machine Interface
  • GPS Receiver 225 GPS Receiver 225
  • VCD Vehicular Communication Device
  • the components e.g., VCD, HMI, PTCC, VA, DAP, GPS
  • Each component in the PTC vision system may have its own power supply or share one with the PTCC.
  • the power supplies used for the components in the PTC vision system may include non-interruptible components for power outages.
  • the PTCC module maintains the state of information passing in between the modules of the PTC vision system.
  • the PTCC communicates with the HMI, VA, VCD, GPS, and DAP. Communication may include providing information (e.g., data) and/or receiving information.
  • An interface e.g., bus, connection
  • Modules of the ecosystem may communicate with each other, a human operator, and/or a third party (e.g., another train, conductor, train operator) using any conventional communication protocol. Communication may be accomplished via wired and/or wireless communication link (e.g., channel).
  • the PTCC may be implemented using any conventional processing circuit including a microprocessor, a computer, a signal processor, memory, and/or buses.
  • a PTCC may perform any computation suitable for performing the functions of the PTC vision system.
  • the HMI module may receive information from the PTCC module.
  • Information received by the HMI module may include: Geolocation (e.g., GPS Latitude & Longitude coordinates); Time; Recommended speeds; Directional Heading (e.g., azimuth); Track ID;
  • the HMI module may provide information to the PTCC module.
  • Information provided to the PTCC may include information and/or requests from an operator.
  • the HMI may process (e.g., format, reduce, adjust, correlate) information prior to providing the information to an operator or the PTCC module.
  • the information provided by the HMI to the PTCC module may include: Conductor commands to slow down the train; Conductor requests to bypass certain parameters (e.g., speed restrictions); Conductor acknowledgement of messages (e.g., faults, state information); Conductor requests for additional information (e.g., diagnostic procedures, accidents along the railway track, or other points of interest along the railway track); and Any other information of interest relevant to a conductor's train operation.
  • the HMI provides a user interface (e.g., GUI) to a human user (e.g., conductor, operator).
  • a human user may operate controls (e.g., buttons, levers, knobs, touch screen, keyboard) of the HMI module to provide information to the HMI module or to request information from the vision system.
  • controls e.g., buttons, levers, knobs, touch screen, keyboard
  • An operator may wear the user interface to the HMI module.
  • the user interface may communicate with the HMI module via tactile operation, wired communication, and/or wireless communication.
  • Information provided to a user by the HMI module may include: Recommended speed, Present speed, Efficiency score or index, Driver profile, Wayside signaling state, Stations of interest, Map view of inertial metrics, Fault messages, Alarms, Conductor interface for actuation of locomotive controls, and Conductor interface for acknowledgement of messages or notifications.
  • the VCD module performs communication (e.g., wired, wireless).
  • the VCD module enables the PTC vision system to communicate with other devices on and off the train.
  • the VCD module may provide Wide Area Network ("WAN") and/or Local Area Network ("LAN") communications.
  • WAN communications may be performed using any conventional communication technology and/or protocol (e.g., cellular, satellite, dedicated channels).
  • LAN communications may be performed using any conventional communication technology and/or protocol (e.g., Ethernet, WiFi, Bluetooth, WirelessHART, low power WiFi, Bluetooth low energy, fibre optics, IEEE 802.15.4e).
  • Wireless communications may be performed using one or more antennas suitable to the frequency and/or protocols used.
  • the VCD module may receive information from the PTCC module.
  • the VCD may transmit information received from the PTCC module.
  • Information may be transmitted to headquarters (e.g., central location), wayside equipment, individuals, and/or other trains.
  • Information from the PTCC module may include: Packets addressed to other trains; Packets addressed to common backend server to inform operators of train location; Packets addressed to wayside equipment; Packets addressed to wayside personnel to communicate train location; Any node to node arbitrary payload; and Packets addressed to third party listeners of PTC vision system.
  • the VCD module may also provide information to the PTCC module. The VCD may receive information from any source to which the VCD may transmit information.
  • Information provided by the VCD to the PTCC may include: Packets addressed from other trains; Packets addressed from common backend server to give feedback to a conductor or a train; Packets addressed from wayside equipment; Packets addressed from wayside personnel to communicate personnel location; Any node to node arbitrary payload; and Packets addressed from third party listeners of PTC vision system.
  • the GPS modules may include a conventional global positioning system ("GPS") receiver.
  • the GPS module receives signals from GPS satellites and determines a geographical position of the receiver and time (e.g., UTC time) using the information provided by the signals.
  • the GPS module may include one or more antennas for receiving the signals from the satellites. The antennas may be arranged to reduce and/or detect multipath signals and/or error.
  • the GPS module may maintain a historical record of geographical position and/or time.
  • the GPS module may determine a speed and direction of travel of the train.
  • a GPS module may receive correction information (e.g., WAAS, differential) to improve the accuracy of the geographic coordinates determined by the GPS receiver.
  • the GPS module may provide information to PTCC module.
  • the information provided by the GPS module may include: Time (e.g., UTC, local); Geographic coordinates (e.g., latitude & longitude, northing & easting); Correction information (e.g., WAAS, differential); Speed; and Direction of travel.
  • Time e.g., UTC, local
  • Geographic coordinates e.g., latitude & longitude, northing & easting
  • Correction information e.g., WAAS, differential
  • Speed e.g., Speed
  • Direction of travel e.g., direction of travel.
  • the DAP may receive (e.g., determine, detect, request) information regarding a train, the systems (e.g., hardware, software) of a train, and/or a state of operation of a train (e.g., train state).
  • the DAP may receive information from the systems of a train regarding the speed of the train, train acceleration, train deceleration, braking effort (e.g., force applied), brake pressure, brake circuit status, train wheel traction, inertial metrics, fluid (e.g., oil, hydraulic) pressures, and energy consumption.
  • Information from a train may be provided via a signal bus used by the train to transport information regarding the state and operation of the systems of the train.
  • a signal bus includes one or more conventional signal busses such as Fieldbus (e.g., IEC 61158), Multifunction Vehicle Bus (“MVB”), wire train bus (“WTB”), controller area network bus (“CanBUS”), Train Communication Network (“TCN”) (e.g., IEC 61375), and Process Field Bus (“Profibus”).
  • a signal bus may include devices that perform wired and/or wireless (e.g., TTEthernet) communication using any conventional and/or proprietary protocol.
  • the DAP may further include any conventional sensor to detect information not provided by the systems of the train. Sensors may be deployed (e.g., attached, mounted) at any location on the train. Sensors may provide information to the DAP directly and/or via another device or bus (e.g., signal bus, vehicle control unit, wide train bus, multifunction vehicle bus). Sensors may detect any physical property (e.g., density, elasticity, electrical properties, flow, magnetic properties, momentum, pressure, temperature, tension, velocity, viscosity). The DAP may provide information regarding the train to the other modules of the PTC ecosystem via the PTCC module.
  • Sensors may be deployed (e.g., attached, mounted) at any location on the train. Sensors may provide information to the DAP directly and/or via another device or bus (e.g., signal bus, vehicle control unit, wide train bus, multifunction vehicle bus). Sensors may detect any physical property (e.g., density, elasticity, electrical properties, flow, magnetic properties, momentum, pressure, temperature, tension, velocity, viscosity).
  • the DAP may receive information from any module of the PTC ecosystem via the PTCC module.
  • the DAP may provide information received from any source to other modules of the PTC ecosystem via the PTCC module.
  • Other modules may use information provided by or through the DAP to perform their respective functions.
  • the DAP may store received data.
  • the DAP may access stored data.
  • the DAP may create a historical record of received data.
  • the DAP may relate data from one source to another source.
  • the DAP may relate data of one type to data of another type.
  • the DAP may process (e.g., format, manipulate, extrapolate) data.
  • the DAP may store data that may be used, at least in part, to derive a signal state of the track on which the train travels, geographic position of the train, and other information used for positive train control.
  • the DAP may receive information from the PTCC module.
  • Information received by the DAP from the PTCC module may include: Requests for train state data; Requests for braking interface state; Commands to actuate train behavior (speed, braking, traction effort); Requests for fault messages; Acknowledgement of fault messages; Requests to raise alarms in the train; Requests for notifications of alarms raised in the train; and Requests for wayside equipment state.
  • the DAP may provide information to the PTCC module.
  • Information provided by the DAP to the PTCC module may include: Data from the signal bus of the train regarding train state; Acknowledge of requests; Fault messages on train bus; and Wayside equipment state.
  • the VA module detects the environment around the train.
  • the VA module detects the environment through which a train travels.
  • the VA module may detect the tracks upon which the train travels, tracks adjacent to the tracks traveled by the train, the aspect (e.g., appearance) of wayside (e.g., along tracks) signals (semaphore, mechanical, light, position), infrastructure (e.g., bridges, overpasses, tunnels), and/or objects (e.g., people, animals, vehicles).
  • Additional examples include: PTC assets, ETCS assets, Tracks, Signals, Signal lights, Permanent speed restrictions, Catenary structures, Catenary wires, Speed limit Signs, Roadside safety structures, Crossings, Pavements at crossings, Clearance point locations for switches installed on the main and siding tracks, Clearance/ structure gauge/ kinematic envelope, Beginning and ending limits of track detection circuits in non-signaled territory, Sheds, Stations, Tunnels, Bridges, Turnouts, Cants, Curves, Switches, Ties, Ballast, Culverts, Drainage structures, Vegetation ingress, Frog (crossing point of two rails), Highway grade crossings, Integer mileposts, Interchanges,
  • the VA module may detect the environment using any type of conventional sensor that detects a physical property and/or a physical characteristic.
  • Sensors of the VA module may include cameras (e.g., still, video), remote sensors (e.g., Light Detection and Ranging), radar, infrared, motion, and range sensors.
  • Operation of the VA module may be in accordance with a geographic location of the train, track conditions, environmental conditions (e.g., weather), speed of the train. Operation of the VA may include the selection of sensors that collect information and the sampling rate of the sensors.
  • the VA module may receive information from the PTCC module.
  • Information provided by the PTCC module may provide parameters and/or settings to control the operation of the VA module.
  • the PTCC may provide information for controlling the sampling frequency of one or more sensors of the VA.
  • the information received by the VA from the PTCC module may include: The frequency of the sampling, The thresholds for the sensor data, and Sensor configurations for timing and processing.
  • the VA module may provide information to the PTCC module.
  • the information provided by the VA module to the PTCC module may include: Present sensor configuration parameters, Sensor operational status, Sensor capability (e.g., range, resolution, maximum operating parameters), Raw or processed sensor data, Processing capability, and Data formats.
  • Raw or processed sensor data may include a point cloud (e.g., two-dimensional, three-dimensional), an image (e.g., jpg), a sequence of images, a video sequence (e.g., live, recorded playback), scanned map (e.g., two-dimensional, three-dimensional), an image detected by Light Detection and Ranging (e.g., LIDAR), infrared image, and/or low light image (e.g., night vision).
  • the VA module may perform some processing of sensor data. Processing may include data reduction, data augmentation, data extrapolation, and object identification.
  • Sensor data may be processed, whether by the VA module and/or the PTCC module, to detect and/or identify: Track used by the train, Distance to tracks, objects and/or infrastructure, Wayside signal indication (e.g., meaning, message, instruction, state, status), Track condition (e.g., passable, substandard), Track curvature, Direction (e.g., turn, straight) of upcoming segment, Track deviation from horizontal (e.g., declivity, acclivity), Junctions, Crossings, Interlocking exchanges, Position of train derived from environmental information, and Track identity (e.g., track ID).
  • Wayside signal indication e.g., meaning, message, instruction, state, status
  • Track condition e.g., passable, substandard
  • Track curvature e.g., Direction (e.g., turn, straight) of upcoming segment
  • Track deviation from horizontal e.g., declivity, acclivity
  • Junctions e.g., Crossings
  • the VA module may be coupled (e.g., mounted) to the train.
  • the VA module may be coupled at any position on the train (e.g., top, inside, underneath).
  • the coupling may be fixed and/or adjustable.
  • An adjustable coupling permits the viewpoint of the sensors of the VA module to be moved with respect to the train and/or the environment. Adjustment of the position of the VA may be made manually or automatically. Adjustment may be made responsive to a geographic position of the train, track condition, environmental conditions around the train, and sensor operational status.
  • the PTCC utilizes its access to all subsystems (e.g., modules) of the PTC system to derive (e.g., determine, calculate, extrapolate) track ID and signal state from the sensor data obtained from the VA module.
  • the PTCC module may utilize the train operating state information, discussed above, and data from the GPS receiver to refine geographic position data.
  • the PTCC module may also use information from any module of the PTC environment, including the PTC vision system, to qualify and/or interpret sensor information provided by the VA module. For example, the PTCC may use geographic position information from the GPS module to determine whether the infrastructure or signaling data detected by the VA corresponds to a particular location.
  • Speed and heading (e.g., azimuth) information derived from video information provided by the VA module may be compared to the speed and heading information provided by the GPS module to verify accuracy or to determine likelihood of correctness.
  • the PTCC may use images provided by the VA module with position information from the GPS module to prepare map information provided to the operator via the user interface of the HMI module.
  • the PTCC may use present and historical data from the DAP to detect the position of the train using dead reckoning, position determination may be correlated to the location information provided by the VA module and/or GPS module.
  • the PTCC may receive communications from other trains or wayside radio transponders (e.g., balises) via the VCD module for position determination that may be correlated and/or corrected (e.g., refined) using position information from the VA module and/or the GPS module or even dead reckoning position information from the DAP. Further, track ID, signal state, or train position may be requested to be entered by the operator via the HMI user interface for further correlation and/or verification.
  • trains or wayside radio transponders e.g., balises
  • the VCD module for position determination that may be correlated and/or corrected (e.g., refined) using position information from the VA module and/or the GPS module or even dead reckoning position information from the DAP.
  • track ID, signal state, or train position may be requested to be entered by the operator via the HMI user interface for further correlation and/or verification.
  • the PTCC module may also provide information and calls to action (e.g., messages, warnings, suggested actions, commands) to a conductor via the HMI user interface.
  • action e.g., messages, warnings, suggested actions, commands
  • the PTCC may bypass the conductor and actuate a change in train behavior (e.g., function, operation) utilizing the integration with the braking interface or the traction interface to adjust the speed of the train.
  • PTCC handles the routing of information by describing the recipient(s) of interest, the payload, frequency, route and duration of the data stream to share the train state with third party listeners and devices.
  • the PTCC may also dispatch/receive packets of information automatically or through calls to action from the common backend server in the control room or from the railway operators or from the control room terminal or from the conductor or from wayside signaling or modules in the PTC vision system or other third party listeners subscribed to the data on the train.
  • the PTCC may also receive information concerning assets near the location of the moving vehicle.
  • the PTCC may use the VA to collect data concerning PTC and other assets.
  • the PTCC may also process the newly collected data (or forward it) to audit and augment the information in the backend database.
  • the Track Identification Algorithm depicted in FIGs. 6 - 7 determines which track the rolling stock is currently utilizing.
  • the TIA creates a superimposed feature dataset by overlaying the features from the 3D LIDAR scanners and FLIR Cameras onto the onboard camera frame buffer.
  • the superset of features allows for three orthogonal measurements and perspectives of the tracks.
  • Thermal features from the FLIR Camera may be used to identify (e.g., separate, locate, isolate) the thermal signature of the railway tracks to generate a region of interest (spatial & temporal filters) in the global feature vector.
  • Range information from the 3D LIDAR scanner's 3D point cloud dataset may be utilized to identify the elevation of the railway track to also generate a region of interest (spatial & temporal filters) in the global feature vector.
  • Line detection algorithms may be utilized on the onboard camera, FLIR cameras and 3D LIDAR scanner's 3D point cloud dataset to further increase confidence in identifying tracks.
  • Color information from the onboard camera and the FLIR cameras may be used to also create a region of interest (spatial & temporal filter) in the global feature vector.
  • the TIA may look for overlaps in the regions of interest from multiple orthogonal measurements on the global feature vector to increase redundancy and confidence in track identification data.
  • the TIA may utilize the region of interest data to filter out false positives when the regions of interest do not overlap in the global feature vector.
  • the TIA may process the feature vectors in a region of interest to identify the width, distance, and curvature of a track.
  • the TIA may examine the rate at which a railway track is converging towards a point to further validate the track identification process; furthermore the slope of a railway track may also be used to filter out noise in the global feature vector dataset.
  • the TIA may take into consideration the spatial and temporal consistency of feature vectors prior to identifying the relative offset position of a train amongst multiple railway tracks.
  • Directional heading may be obtained by sampling the GPS receiver multiple times to create a temporal profile of movement in geographic coordinates.
  • the list of potential absolute track IDs may be obtained through a query to a locally cached GIS dataset or a remotely hosted backend server.
  • the odometer and directional heading may be used to calculate the dead reckoning offset.
  • the TIA compares the relative offset position of the train among multiple railway tracks and references to the list of potential absolute track IDs to identify the absolute track ID that the train is utilizing.
  • the global feature vector samples may be annotated with the geolocation (e.g., geographic coordinate) information and track ID. This allows the TIA to utilize the global feature vector datasets to directly determine a track position in the future. This machine learning approach reduces the computational cost of searching for an absolute track ID.
  • the TIA may further match global feature vector samples from a local or backend database with spatial transforms.
  • the parameters of the spatial transform may be utilized to calculate an offset position from a reference position generated from the query match.
  • the TIA may utilize the global feature vectors to stitch together features from multiple points in space or from a single point in space using various image processing techniques (e.g., image stitching, geometric registration, image calibration, image blending). This results in a superset of feature data that has collated global feature vectors from multiple points or a single point in space.
  • image processing techniques e.g., image stitching, geometric registration, image calibration, image blending.
  • the TIA can normalize the offset position for a relative track ID prior to determining an absolute track ID. This is useful when there are tracks outside the range of the vision apparatus (VA). This functionality is depicted in FIG. 10.
  • the TIA is a core component in the PTC vision system that eliminates the need for wireless transponders, beacons or balises to obtain positional data. TIA may also enable railway operators to annotate newly constructed railway tracks for their network wide GIS datasets that are authoritative in mapping the wayside equipment and infrastructure assets.
  • the Signal State Algorithm determines the signal state of the track a train is currently utilizing.
  • the purpose of this component is to ensure a train's operation is in compliance with the expected operational parameters of the railway operators or modal control rooms or central control rooms.
  • the compliance of a train's inertial metrics along a railway track can be audited in a distributed environment many backend servers or a centralized environment with a common backend server.
  • a train's ability to obtain the absolute track ID is important for correlating the semaphore signal state to the track ID utilized by a train. Auditing signal compliance is possible once the correlation between the semaphore signal state and the absolute track ID is established. Placement of sensors is important for efficiently determining a semaphore signal state.
  • FIG. 4 depicts one example wherein the 3D LIDAR scanner is forward facing and mounted on top of a train's roof.
  • the SSA takes into account an absolute track ID utilized by a train in order to audit the signal compliance of the train. Once the correlation of a track to a semaphore signal is complete, the signal state from that semaphore signal may actuate calls to action as feedback to a train or conductor.
  • Correlation of a railway track to a semaphore signal state may be possible by analyzing the regulatory specifications for wayside signaling from a railway operator. Utilizing the regulatory documentation, the spatial-temporal consistency of a semaphore signal may be compared to the spatial-temporal consistency of a railway track. A scoring mechanism may be used to choose the best candidate semaphore signal for the current railway track utilized by the train.
  • a local or remote GIS dataset may be queried to confirm the geolocation of a semaphore signal.
  • a local or remote signaling server may be queried to confirm the signal state in the semaphore signal matches what the PTC vision system is extrapolating.
  • Areas wherein the signal state is available to the train via radio communication may be utilized to confirm the accuracy of the PTC vision system and additionally augment the feedback provided to a machine learning apparatus that helps tune the PTC vision system.
  • a 3D point cloud dataset obtained from a PTC vision system may be utilized to analyze the structure of the semaphore signal. If the structure of an object of interest matches the expected specifications as defined by the regulatory body for a semaphore signal in that rail corridor, the object of interest may be annotated and added as a candidate for the scoring mechanism referenced above.
  • An infrared image captured through an FLIR camera may be utilized to identify the light being emitted from a wayside semaphore signal.
  • a call to action will be dispatched to the HMI onboard the train for signal compliance.
  • a call to action will be dispatched directly to the braking interface onboard the train for signal compliance.
  • the color spectrum in an image captured through the PTC vision system may be segmented to compute centroids that are utilized to identify blobs that resemble signal green, red, yellow or double yellow lights.
  • a centroid's spatial coordinates and size of its blob may be utilized to validate the spatial-temporal consistency of the semaphore signal with specifications from a regulatory body.
  • a spatial-temporal consistency profile of a track may be created by analyzing the curvature of a track, spacing between the rails on a track, and rate of convergence of the track spacing towards a point on the horizon.
  • a spatial-temporal consistency profile of a semaphore signal may be created by analyzing the following components: the height of a semaphore signal, the relative spatial distance between points in space, and the orientation and distance with respect to a track a train is currently utilizing.
  • the backend server may be queried to inform a train of an expected semaphore signal state along a railway track segment that the train is currently utilizing.
  • the backend server may be queried to inform a train of an expected semaphore signal state along a railway track segment identified by an absolute track ID and geolocation coordinates.
  • the Position Refinement Algorithm provides a high confidence geolocation service onboard the train.
  • the purpose of this algorithm is to ensure that loss of geolocation services does not occur when a single sensor fails.
  • the PRA relies on redundant geolocation services to obtain the track position.
  • GPS or Differential GPS may be utilized to obtain fairly accurate geolocation coordinates.
  • Tachometer data along with directional heading information can be utilized to calculate an offset position.
  • a WiFi antenna may scan SSIDs along with signal strength of each SSID while
  • GPS is working and later use the Medium Access Control (MAC) addresses (or any unique identifier associated with an SSID) to quickly determine the geolocation coordinates.
  • MAC Medium Access Control
  • the signal strength of the SSID during the scan by a WiFi antenna may be utilized to calculate the position relative to the original point of measurement.
  • the PTC vision system may choose to insert the SSID profile (SSID name, MAC address, geolocation coordinates, signal strength) as a reference point into a database based on the confidence in the current train's geolocation.
  • Global feature vectors created by the PTC vision system may be utilized to lookup geolocation coordinates to further ensure accuracy of the geolocation coordinates.
  • a scoring mechanism that takes samples from all the components described above would filter out for inconsistent samples that might inhibit a train's ability to obtain geolocation information. Furthermore, the samples may carry different weightage based on the performance and accuracy of each subcomponent in the PRA.
  • the PTC vision system samples the train state from the various subsystems described above.
  • the train state is defined as a comprehensive overview of track, signal and on-board information.
  • the state consists of track ID, signal state of relevant signals, relevant on-board information, location information (pre- and post-refinement, reference PRA, TIA and SSA algorithms described above), and information obtained from backend servers.
  • These backend servers hold information pertaining to the railroad infrastructure.
  • a backend database of assets is accessed remotely by the moving vehicle as well as railroad operators and officers. The moving train and its conductor for example use this information to anticipate signals along the route. Operator and maintenance officers have access to track information for example.
  • These reports and notifications are relevant to signals and signs, structures, track features and assets, safety information.
  • the PTC vision system issues notifications (local or remote), possibly raises alarms on-board the train, and can automatically control the train's inertial metrics by interfacing with various subsystems on-board (e.g., traction interface, braking interface, traction slippage system).
  • notifications local or remote
  • subsystems on-board e.g., traction interface, braking interface, traction slippage system.
  • On-board data represents a unit where all the data extracted from the various train systems is collected and made available. This data usually includes but is not limited to: Time information, Diagnostics information from various onboard devices, Energy monitoring information, Brake interface information, Location information, Signaling state obtained from train interfaces to wayside equipment, Environmental state obtained through the VA devices on board or on other trains, and Any other data from components that would help in Positive Train Control.
  • This data is made available within the PTC vision system for other components and can be transmitted to remote servers, other trains, or wayside equipment.
  • Location data is strategic to ensure that trains are operating within a safety envelope that meets the Federal Railroad Administration's PTC criteria.
  • wayside equipment is currently being utilized by the industry to accurately determine vehicle position.
  • the output of location services described above e.g., TIA & SSA
  • TIA & SSA provides the relative track position based on computer vision algorithms.
  • the relative position can be obtained through using a single sensor or multiple sensors.
  • the position we obtain is returned as an offset position, usually denoted as a relative track number.
  • Directional heading can also be a factor in building a query to obtain the absolute position from the feedback to the train.
  • the absolute position can be obtained either from a cached local database, or cached local dataset, remote database, remote dataset, relative offset position using on board inertial metric data, GPS samples, Wi-Fi SSIDs and their respective signal strength or through synchronization with existing wayside signaling equipment.
  • the various types of datasets we use include but are not limited to: 3D point cloud datasets, FLIR imaging, Video buffer data from on-board cameras.
  • this information can be utilized to correlate signal state from wayside signaling to the corresponding track.
  • the location services can also be exposed to third party listeners.
  • the on board components defined in the PTC vision system can act as listeners to the location services.
  • the train can scan the MAC IDs of the networked devices in the surrounding areas and utilize MAC ID filtering for any application these networked devices are utilizing. This is useful for creating context aware applications that depend on the pairing the MAC ID of a third party device (e.g., mobile phones, laptops, tablets, station servers, and other computational devices) with a train's geolocation information.
  • a third party device e.g., mobile phones, laptops, tablets, station servers, and other computational devices
  • the track signal state is important for ensuring the train complies with the PTC safety envelope at all times.
  • the PTC vision system's functional scope includes extrapolating the signal value from wayside signaling (semaphore signal state).
  • the communication module or the vision apparatus may identify the signal values of the wayside equipment.
  • a central back end server can relay the information to the train as feedback.
  • this information can also augment the vision-based signal extrapolation algorithms (e.g., TIA & SSA).
  • Datasets are used at the discretion of the PTC vision system.
  • data necessary for updating asset information is generated by the vision apparatus. This data then gets processed to verify the integrity of certain asset information, as well as update other asset information. Missing assets, damaged assets or ones that have been tampered with can then be detected and reported. The status of the infrastructure can also be verified, and the operational safety can be assessed, every time a vehicle with the vision apparatus travels down the track. For example, clearance measurements are performed making sure that no obstacles block the path of trains. The volume of ballast supporting the track is estimated and monitored over time.
  • the backend component has many purposes. For one, it receives, annotates, stores and forwards the data from the trains and algorithms to the various local or remote subscribers.
  • the backend also hosts many processes for analyzing the data (in real-time or offline), then generating the correct output. This output is then sent directly to the train as feedback, or relayed to command and dispatch centers or train stations.
  • Some of the aforementioned processes can include: Algorithms to reduce headways between trains to optimize the flow on certain corridors; Algorithms that optimize the overall flow of the network by considering individual trains or corridors; and Collision avoidance algorithms that constantly monitor the location and behavior of the trains.
  • the backend also hosts the asset database queried by the moving train to obtain asset and infrastructure information, as required by rolling stock movement regulations.
  • This database holds the following assets with relevant information and features: PTC assets, ETCS assets, Tracks, Signals, Signal lights, Permanent speed restrictions, Catenary structures, Catenary wires, Speed limit Signs, Roadside safety structures, Crossings, Pavements at crossings, Clearance point locations for switches installed on the main and siding tracks, Clearance/ structure gauge/ kinematic envelope, Beginning and ending limits of track detection circuits in non-signaled territory, Sheds, Stations, Tunnels, Bridges, Turnouts, Cants, Curves, Switches, Ties, Ballast, Culverts, Drainage structures, Vegetation ingress, Frog (crossing point of two rails), Highway grade crossings, Integer mileposts, Interchanges, Interlocking/control point locations, Maintenance facilities, Milepost signs, and Other signs and signals.
  • the rolling stock vehicle utilizes the information queried from the database to refine the track identification algorithm, the position refinement algorithm and the signal state detection algorithm.
  • the train (or any other vehicle utilizing the machine vision apparatus) moving along/in close proximity to the track collects data necessary to populate, verify and update the information in the database.
  • the backend infrastructure also generates alerts and reports concerning the state of the assets for various railroad officers.
  • the train can be controlled using the PTC vision system (e.g., Applications in Fig 5).
  • the output of the sensory stage might trigger certain actions independently of the any other system. For example, upon the detection of a red-light violation, the braking interface might be triggered automatically to attempt to bring the train to a stop.
  • Certain control commands can also arrive to the train through its VCD.
  • the backend system can for example instruct the train to increase its speed thereby reducing the headway between trains.
  • Other train subsystems might also be actuated through the PTC vision system, as long as they are accessible on the locomotive itself.
  • Feedback can also reach the locomotive and conductor through alarms.
  • an alarm can be displayed on the HMI.
  • the alarms can accompany any automatic control or exist on its own.
  • the alarms can stop by being acknowledged or halt independently.
  • Feedback can be in the form of notifications to the conductor through the user interface of the HMI module. These notifications may describe the data sensed and collected locally through the PTC vision system, or data obtained from the backend systems through the VCD. These notifications may require listeners or may be permanently enabled. An example of a notification can be about speed recommendations for the conductor to follow.
  • the backend may have two modules: data aggregation and data processing.
  • Data aggregation is one module whose role is to aggregate and route information between trains and a central backend.
  • the data processing component is utilized to make recommendations to the trains.
  • the communication is bidirectional and this backend server can serve all of the various possible applications from the PTC vision system.
  • Possible applications for PTC vision system include the following: Signal detection; Track detection; Speed synchronization; Extrapolating interlocking state of track and relaying it back to other trains in the network; Fuel optimization; Anti-Collision system; Rail detection algorithms; Track fault detection or preventative derailment detection; Track performance metric; Image stitching algorithms to create comprehensive reference datasets using samples from multiple runs; Cross Train imaging for, e.g., Preventative maintenance, Fault detection, and/or Vibration signature of passerby trains; Imaging based geolocation or geofiltering services; SSID based geolocation or geofiltering; and Sensory fusion of GPS + Inertial Metrics + Computer Vision-based algorithms.
  • FIG. 25 is a schematic block diagram of an exemplary in-vehicle system for vehicle localization and/or control.
  • In-vehicle runtime engine (“IVRE") 2500 and vehicle decision engine 2510 are computation and control modules, typically microprocessor-based, implemented locally on board a vehicle.
  • Local 3D map cache 2530 stores map data associated with the area surrounding the vehicle's rough position, as determined by GPS and EVIU sensors 2520, and can be periodically or continuously updated from a remote map store via communications module 2540 (which may include, e.g., a cellular data transceiver).
  • Machine vision sensors 2550 may include one or more mechanisms for sensing a local environment proximate the vehicle, such as LiDAR, video cameras and/or radar.
  • IVRE 2500 implements vehicle localization by obtaining a rough vehicle position from onboard GPS and IMU sensors 2520.
  • Machine vision sensors 2550 generate environmental signatures indicative of the local environment surrounding the vehicle, which are passed to IVRE 2500.
  • IVRE 2500 queries local 3D map cache 2530 using
  • Detailed vehicle position and other observed or calculated information can be utilized to implement other functionality, such as vehicle control and/or map auditing.
  • data from machine vision sensors 2550 can be analyzed using graphs and other data analysis mechanisms, as described elsewhere herein, for IVRE 2500 to determine a centerline for a lane in which the vehicle is traveling.
  • IVRE 2500 can also operate to obtain semantics (such as events and triggers) along the vehicle's route.
  • Available compute resources can be used to audit centralized map data sources by comparing previously-observed asset information obtained from centralized maps (and, e.g., stored in local 3D map cache 2530) to asset information derived from real time data captured by machine vision sensors 2550.
  • IVRE 2500 can thereby identify errors of omission (i.e. observed assets omitted from centralized map data) as well as errors of commission (i.e. assets in centralized map data that are not observed by machine vision sensors 2550). Such errors can be stored in cache 2530, and subsequently communicated to a central map repository via communications module 2540.
  • auditing of map data by a local vehicle may be initiated by a centralized control server, communicating with the vehicle via communications module 2540. For example, if the time elapsed since last auditing of a map section exceeds a threshold, a centralized control server can request auditing from a local vehicle traveling through the target region. In another example, if one vehicle reports discrepancies between centralized map data and locally-observed conditions, the centralized control server may request confirmation auditing by one or more other vehicles moving within the area of the discrepancy. Auditing requests may pertain to various combinations of geographic regions and/or mapping layers.
  • vehicle decision engine 2510 can operate to control various other systems and functions of the vehicle. For example, in an autonomous driving implementation, vehicle decision engine 2510 may utilize lane center line information and precise vehicle position information in order to steer the vehicle and maintain a centered lane position. These and other vehicle control operations may be beneficially implemented using systems and processes described herein.
  • Maps are collections of objects, their location and their properties. Maps can be divided into layers, where each layer is a grouping of objects of the same type. The location of each object is defined, along with a geometric attribute (example: the location of a pole could be a point in three-dimensional space, whereas a signal can be located by drawing a polygon around it).
  • a map becomes "semantic" when the semantic associations between different objects and layers are also recorded. For example, a map composed of the centerlines of various lanes on a roadway as well as the signs located around the infrastructure is labeled semantic, when the associations between the various signs and centerlines are recorded.
  • the semanticization of a map creates more context for the vehicle or user consuming the map.
  • the semantic map can also be packaged with regulatory information from various transportation authorities.
  • Geometric features used to describe shapes include points, lines, polygons, and arcs. The features are typically in three dimensions, but they can be projected into two-dimensional spaces where depth/elevation is lost.
  • semantic maps can be recorded and delivered in different coordinate and reference frames. There are also transformations allowing to project maps from one coordinate reference frame to the next. These maps can be packaged and delivered in different formats. Common formats include GeoJSON, KML, shapefiles, and the like.
  • the geospatial data used for semantic map creation comes from LiDAR, visible spectrum cameras, infrared cameras, and other optical equipment.
  • the output is a set of data points in three dimensions, along with images and video feeds in the visible spectrum and other frequencies.
  • the collection vehicle is also variable (aerial, mobile, terrestrial).
  • the geospatial data is collected initially with the collection vehicle being the origin of the reference frame.
  • IMU Inertial Measurement Unit
  • GPS Global Positioning Systems
  • Semantic maps derived from point cloud survey data may provide a vehicle with high levels of detail and information regarding the vehicle's current or anticipated local environment, which may be used, for example, to assist in relative vehicle localization, or serve as input data to autonomous control decision-making systems (e.g. automated braking, steering, speed control, etc.). Additionally, or alternatively, point-cloud data measured by a vehicle may be compared to previously-measured point cloud data to detect conditions or changes in a local environment, such as a fallen tree, overgrown vegetation, changed signage, lane closures, track or roadway obstructions, or the like. The detected changes in the environment can be used to further update the semantic maps.
  • FIG. 11 A illustrates a typical prior art process for extracting asset information from point cloud data.
  • surveying procedures generate point cloud data sets, such as using a LiDAR surveying apparatus.
  • step SI 105 the raw point cloud data is visualized.
  • GIS Geographical Information Systems
  • the first step in the GIS analysts' process is to separate the terabytes of point cloud data into smaller manageable sections. This is due to the fact that contemporary personal computers are limited (memory/computational power) and are unable to manage the terabytes of LiDAR data at once.
  • the GIS analysts use 3D visualization software to traverse each of the smaller sections of point cloud. As they progress through their respective sections, the GIS analysts delineate and annotate the important assets. Finally, the annotated assets of each GIS analyst are combined into one map (step SI 110). Varying file formats and software systems can create additional difficulties in merging the separate datasets.
  • FIG. 1 IB illustrates an alternative approach to extracting asset information from raw point cloud data.
  • step SI 150 surveying is conducted to generate the raw point cloud data.
  • step SI 155 asset maps are generated directly from the raw point cloud data, without requiring visualization of the large, complex data set, or manual annotation of that data.
  • Figure 12 illustrates a computing apparatus for rapidly and efficiently extracting asset information from large point-cloud data sets.
  • Figure 13 illustrates a process for using the apparatus of Figure 12.
  • the components within the apparatus of Figure 12 are implemented using Internet-connected cloud computing resources, which may include one or more servers.
  • Front-End component 1200 includes data upload tool 1205, configuration tool 1210, and map retrieval tool 1215.
  • Front-End component 1200 provides a mechanism for end users to interact with and control the computing apparatus.
  • a user can upload LiDAR and other surveying data from a local data storage device to data storage component 1220 (step S1300).
  • Data storage component 1220 may implement a distributed file system (such as the Hadoop Distributed File System) or other mechanism for storing data.
  • Configuration tool 1210 can be accessed via a user's network-connected computing device (not shown), and enables a user to define the format of uploaded data as well as other survey details, and specify assets to search for and annotate (step SI 305). After a user interacts with configuration tool 1210 to select desired assets, the user is provided with various options to configure the output map format.
  • configuration tool 1210 then solicits a desired turnaround time from a configuring user, and presents the user with an estimated cost for the analysis (step S1310).
  • the cost estimate is determined based on, e.g., the size of the uploaded data set to be analyzed, the number (and complexity) of selected assets, the output format, and the selected turnaround time.
  • the user interacts with configuration tool 1210 to initiate an analysis job (step S1315).
  • the geospatial data uploaded through front end 1200 is tracked in database collections. This data is organized by category, geographic area, and other properties. As the data evolves through various stages of execution, the relevant database entries get updated.
  • Point-cloud data uploaded through the front-end tool is stored in a secure and replicated manner.
  • the data is tiled into different size tiles in a Cartesian coordinate system.
  • the tiles themselves are limited in two dimensions and namespaced accordingly.
  • tiles are limited in X and Y dimensions, and unlimited in a Z dimension that is vertical or parallel to the direction of the Earth's gravitational pull, such that a tile defines a columnar area, unlimited in height (i.e. limited only to the extent of available geospatial data) and having a rectangular cross-section.
  • tiles which are 1000m on the side (in the horizontal plane) can be utilized.
  • the files representing the tiles would then hold all the points which belong to the particular geographic area delimited by the tile, and no other.
  • tree structures (such as quadtrees and octrees) are implemented depending on the traversal style for the data.
  • Processing of the data to automatically extract semantic maps from geospatial data occurs on computation clusters, implemented within processing unit 1240 (embodiments of which are described further with reference to Figure 16, below). These have access to the point cloud and other data through the network accessible storage unit 1220. Intermediary results as well as finalized ones are stored similarly.
  • Figure 14 illustrates a process that may be performed by the apparatus of Figure
  • the point-cloud data is subdivided into chunks (step S1400) by data storage / preprocessing component 1220. These chunks can be subsets of tiles or combinations thereof, potentially selected to optimize for, e.g., the desired processing method, available memory and other runtime considerations. Individual nodes in the computation cluster (i.e. within processing unit 1240) are then capable of processing geospatial and other data associated with a given data chunk, i.e., selected subsets or combinations of tiles.
  • the density of the point-cloud may be an important factor in determining the number of tiles (or the size of tile subsets) to process within the same computation node.
  • Figure 15 illustrates the size of tiles with respect to the number of points within (represented by the diagonal line), as well as the distribution of tiles sizes for an exemplary dataset comprising LiDAR point-cloud data measured along a 2km section of railway (each tile represented by hatches across the diagonal line).
  • Data storage and preprocessing component 1220 performs tile aggregation, and/or subdivision, prior to feeding data to processing unit 1240, in order to optimize the analysis performance.
  • job scheduler 1225 creates a queue containing tasks pertaining to the job, as configured in steps S1305 and S1310.
  • Job scheduler 1225 associates one or more of analysis mechanisms 1250 (typically implementing various different data analysis algorithms) with the task (step S1405), and creates a cluster of machines within processing unit 1240 to process the data (step S1410).
  • the size of the cluster i.e. the number of computation nodes
  • job scheduler 1225 can initiate a cluster of 20 machines with four cores each, and process the same dataset in approximately 24 hours instead.
  • Processing unit 1240 is composed of a collection of compute clusters.
  • the size of the cluster depends on the number of jobs.
  • Figure 16 illustrates an exemplary compute cluster.
  • Each cluster contains: a master instance 1605, responsible for managing the cluster; a set number of principal computation nodes 1610, which also store data in data storage system 1220; and a variable number of "spot" instances 1620.
  • compute clusters consisting entirely of spot instances, or entirely of principal nodes, may be utilized.
  • data storage and preprocessor component 1220 directs a stream of data chunks (e.g. aggregations of tiles satisfying a desired data subset size) to processing unit 1240 (step S1415).
  • data chunks e.g. aggregations of tiles satisfying a desired data subset size
  • processing unit 1240 execute appropriate data analysis mechanisms 1250 to, e.g., extract asset or feature information from the 3D point-cloud tiles.
  • map generator 1230 combines the output of nodes within processing unit 1240 into semantic maps (step S1420). Reporting analytics can be derived from the semantic maps by running queries to analyze particular assets and their combinations.
  • Map generator 1230 may also include an annotation integrity verifier operating to verify the integrity of annotated datasets over time.
  • locations may be surveyed repeatedly at different times.
  • trains equipped with LiDAR or other railway surveying vehicles may periodically survey the same length of railway, such as to monitor the health or status of assets along a track.
  • LiDAR-equipped survey vehicles may travel along a given portion of road at different times.
  • data captured by LiDAR equipped automobiles, such as autonomous driving cars may be regularly analyzed, providing potentially frequent analyses of the local environment in a given location.
  • Each time a new map is generated by map generator 1230 concerning a given area asset or local feature information can be compared to such information contained in older maps. Alarms, notifications or events can be triggered when discrepancies are detected.
  • map generator 1230 The output of map generator 1230 is ultimately made available to the user, via front end 1200 and map retrieval tool 1215 (step S1425). Once a job is completed and a map is generated, scheduler 1225 (monitoring the status of tasks and jobs) generates notifications for the end user.
  • Feature maps (containing only the location, geometry and features of various assets), as well as semantic ones can also be stored in remotely accessible geodatabases.
  • the map data can be retrieved either directly or through a server to facilitate the querying and collection of results.
  • the maps can be retrieved in their entirety or by selecting a specific area of interest.
  • data upload step SI 300 employs end-to-end encryption (such as AES encryption) from the user data source to the cloud computing platform.
  • AES encryption may also be utilized for communications between a user's system and front-end 1200.
  • data storage component 1220 may include a compression mechanism to compress point-cloud data before storage.
  • LAZ LempelZivOberhumer
  • GZIP also based on LempelZiv methods
  • LASzip released by rapidlasso GmbH
  • LZO and GZIP are optimized for decompression, and therefore present a superior alternative to LAZ in terms of CPU time required for decompression.
  • Data analysis mechanisms 1250 are typically selected based on the nature of the information desired to be extracted from the point-cloud data. It may be desirable to design mechanisms 1250 with very low false positive rates, while maintaining acceptable detection rates. For added confidence in generated maps, in some applications, a subset of results may be verified manually by inspecting the original point-cloud and raw imaging data.
  • track detection may be an important first step. Track detection can be important because knowledge of the track position facilitates identification of assets, since regulations often assign specific locations for each asset in relation to the track.
  • Figure 20 illustrates a process for track detection and traversal that can be implemented by processing unit 1240, e.g. in step S1415 of Figure 14.
  • step S2000 a 100m x 100m section of point-cloud data is identified for analysis.
  • step S2010 the geometry of the 10,000m 2 point cloud section is analyzed to extract a subset of points which are associated with the track. Many techniques can be employed to achieve the desired result. In some
  • previously-classified tracks from similar data sets can be studied to identify properties of data in the vicinity of the tracks, with those properties serving as an indicia of track location in newly-analyzed data.
  • Other techniques include projecting points in two-dimensional space (based on, e.g., height or pulse intensity) and utilizing edge detection mechanisms and transforms to isolate regions belonging to the track.
  • the 10,000 m 2 point cloud section in step S2000 may consist of about 1 GB of data
  • the extracted track subset output in step S2010 may consist of about 1 MB of data.
  • Figure 21 is a visualization of the 10,000 m2 point cloud section input to step
  • Lines 2100 represent track that is visible in the point-cloud.
  • Line 2110 represent track that was obscured during the LiDAR data collection process, having a position that is estimated. This is typically the result of shadowing, a process which occurs when the object of interest is hidden from direct line of sight of the measuring instrument.
  • Dots 2120 correspond to problematic positioning of a LiDAR tripod system which resulted in some track sections being obstructed.
  • the location of the invisible track can be inferred by utilizing known spatial continuity properties of the infrastructure (such as spacing relative to other observed elements) (step S2020).
  • Geospatial data presents many dimensionalities that can be taken advantage of during asset extraction.
  • Imagery, infrared, video feeds and/or multispectral sensors can be combined to increase detection confidence and accuracy.
  • Most LiDAR systems include an intensity measurement for each point.
  • classification mechanisms and filters can be added to the system, for an increased track detection rate.
  • Figure 22 is a histogram of point-cloud intensity levels in an exemplary track detection implementation.
  • Figure 22a illustrates quantity of each measured intensity level in an analyzed body of point cloud data, as a whole.
  • Figure 22b illustrates the same histogram, for points within the point cloud identified as corresponding to track.
  • a simple band pass filter can be effective in some cases to further narrow a search space for points belonging to the rail.
  • Other classification methods can also be utilized.
  • Figure 23 is a visualization of a portion of the output of an implementation including a track detection mechanism and other asset detection mechanisms.
  • track segments 2300 are identified first, then for each track, centerline markers 2310 are established. Once the tracks and track centerlines are identified, subsequent analysis components can traverse the track within the point-cloud data, while enjoying a 360 degree view of high resolution point cloud data around each point in the centerline.
  • Other analysis mechanisms identify and locate other assets or features for inclusion in a sematic map. For example, an overhead wire detection mechanism identifies and locates overhead wires, and demarcates them with overhead wire centerline indicia 2320.
  • a pole detection mechanism identifies and locates trackside poles, and locates them with indicia 2330.
  • analysis mechanisms may be applied sequentially, with an output of one mechanism serving as an input to another mechanism.
  • assets and elements of the local environment regularly are replaced, added, removed or shifted. It may be desirable to regularly check clearance above and around a track to ensure safe operation, and that train cars do not come into contact with any obstructions.
  • a track detection mechanism such as that described above, may be implemented as part of a sequence of analysis mechanisms.
  • the output of a track detection mechanism that includes the track centerline may be subsequently used as an input to a track clearance check mechanism.
  • a bounding box is defined with respect to the track center line, and any objects that encroach within that bound are reported. The dimensions of the bounding box can be modified to fit various standards.
  • Overhead wires can be identified within point-cloud data. The height of the wire in comparison with the track is assessed. Areas with saggy lines are reported. By using pole location information, the catenary shape of the wire can also be assessed.
  • the automated extraction of maps can be achieved by combining computation blocks into directed acyclic graphs (hereafter referred to as "graphs").
  • the blocks contained in these graphs have a varying degree of complexity, ranging from simple averaging and thresholding to transforms, filters, decompositions, etc.
  • the output of one stage of the graph can feed into any other subsequent stage.
  • the stages need not run in sequence but can be parallelized given sufficient information per stage.
  • a graph is generally used to classify points within a point cloud belonging to the same category, or to vectorize.
  • Vectorization refers to the creation of an (often imaginary) line or polygon going through a set of points delimiting their center, boundary, location, etc.
  • computation graphs can be used to implement classifiers, clustering methods, fitting routings, neural networks and the like.
  • Rotations and projections are also used, often in conjunction with machine vision processing techniques.
  • the creation of semantic maps from geospatial data may be parallelized. There are many levels of parallelization that can be implemented. At the highest level, the survey data can be divided into regularly-shaped regions of interest which get streamed to different machines and CPU processes. The results coming from each area need to then be merged in a "reduce” step once all the processes finish, similarly to the process of Figure 14. Since boundary conditions arise, padding the regions of interest with extra data which is truncated at the end of the process usually removes those deformities near the edges. The size of the region of interest, as well as the padding thickness is determined by the graph extracting the assets or features.
  • parallelism can occur when processing is taking place along a pre-extracted vector. For example, when searching for signs in the vicinity of a railroad track, the data can be traversed by extracting regions around waypoints along the previously extracted track centerline. Multiple processes can then be used in parallel along different waypoints of the track.
  • each point when analyzing a particular region, each point can be considered individually.
  • a voxel surrounding that point is usually extracted and analyzed.
  • This process can also be made parallel, in those cases when the outcome of one point's operation does not affect that of any other point.
  • GPU graphics processing units
  • Geospatial data is not limited to point cloud, but extends to imagery, video feeds, multispectral data, RADAR, etc.
  • some embodiments may utilize any additional data sources that are available.
  • datasets can be combined in a pre-processing stage (e.g. step S1400), before feeding into the computation graphs. This approach provides computation graphs with data from multiple sources for processing.
  • one set of data may be used to generate a hypothesis concerning an asset and its properties; data from other sources can then be used to validate and/or augment the hypothesis via other analysis mechanisms.
  • Figure 24 illustrates an embodiment of a system implementing supervised machine learning, including training component 2400 and map generation component 2410.
  • Training component 2400 receives as inputs, raw point cloud data 2420 and sample output 2422.
  • sample output 2422 may be verified output data associated with approximately 1% of the total data set.
  • Sample output 2422 may include classified point cloud data (where points belonging to a particular asset category are grouped together), and/or a vectorized map (with points, lines and polygons drawn over assets of interest).
  • Training component output 2424 defines an optimized categorization mechanism, such as algorithm coefficients for an analysis mechanism comparable to mechanisms 1250 in the map generation system of Figure 12.
  • Training component output 2424 may also define a region of interest for the algorithms to be most effective, define functional blocks within a computation graph which should be utilized, and/or define features of interest for a particular asset under consideration. Training component output 2424 is fed into map generation component 2410, along with the full corpus of raw point cloud data 2420. Map generation component 2410 then operates to generate map output 2426.
  • Unsupervised methods can also be implemented for generating maps. Such processes can rely on scale-dependent features to describe contextual information for individual map points. They can also rely on deep learning to design feature transformations for use with map point features. Ensembles of feature transformations generated by deep learning are used to encode map point context information. Asset membership for points can then be based on features transformed by deep learning algorithms. Another method revolves around curriculum- based learning where assets are described in a curriculum, then learned in computation graphs. This method can be effective when the assets of interest are regular in shape and properties, and do not exhibit a lot of spatial complexity.
  • a neural network is often trained in a primary step, then applied to the remainder of the geospatial data for extraction of the map.
  • Machine learning techniques can therefore assist in optimizing and refining computation graphs. These graphs can be engineered manually or learned using the above methods.
  • a parameter search component is useful for accuracy improvements and reductions in false positives and negatives.
  • various parameters of the computation graph (from the region of interest, to the parameters of each function, to the number and nature of features used in a classifier) can all be modulated and the output monitored.
  • search methodologies the best performance combination of parameters can be found and applied to the remainder of the data. This step assumes the availability of previously annotated semantic maps.
  • computation graphs When computation graphs are refined to an acceptable performance level, they can be used directly in the vehicles. This would correspond to streaming of the intelligence from the cloud to the vehicles, as opposed to the more conventional streaming of data from local environments to cloud systems. With geospatial data, the sheer size of the sensor data can be prohibitive. Therefore, in some embodiments, locally-obtained sensor data (e.g. data obtain by vehicle-mounted sensors) is summarized via local computation resources, with only a subset of collected information and/or extracted content being sent back to remote data systems. For example, resources comparable to data storage / preprocessor component 1220, processing unit 1240 and data analysis mechanisms 1250, can be implemented in-vehicle to extract semantic map data from onboard sensor systems. Computation graphs analogous to those described above for implementation in a cloud-based processing structure, can be optimized and tested in a machine learning framework, while presenting an opportunity for local in-vehicle
  • Such embodiments can utilize the vehicles as a distributed computing platform, constantly updating the contents of a centrally-maintained map, while consuming most of the remotely-sensed data in place, rather than streaming all of it to a central, cloud-based system.
  • a simulation environment can be utilized.
  • maps are programmatically generated in large numbers of permutations of parameters, to replicate the variability of terrains and landmarks on the face of the planet.
  • Three dimensional models are then generated from the maps and raytraced to create a point cloud in as similar a way to real data collection as possible. Since the location of every asset is known a priori, a perfect map extracted from the point cloud is then available.
  • the variability of the data, and the fact that a perfect ground truth exists for each point cloud greatly increases the scope of the computation graphs and their accuracy. It also provides a mechanism to understand the limitations of the current computing paradigms.
  • Quality control can be performed in multiple ways. Similar to creating a semantic map, a GIS analyst can use conventional visualization tools and overlay the raw survey data with the automatically extracted map. Any discrepancies can then be identified and corrected. Another method for QC would be to crowd source the effort amongst multiple agents online. Since each one of those agents might not be entirely skilled in semantic map creation, the QC work would need to be replicated. Hypotheses can then be confirmed or denied by each QC result, and a final conclusion reached with enough trials.

Landscapes

  • Engineering & Computer Science (AREA)
  • Mechanical Engineering (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • Electromagnetism (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Train Traffic Observation, Control, And Security (AREA)
  • Image Analysis (AREA)
  • Traffic Control Systems (AREA)
  • Electric Propulsion And Braking For Vehicles (AREA)

Abstract

La présente invention concerne des procédés et un appareil pour la vision industrielle et l'analyse de données de nuage de points en temps réel pour la télédétection et la commande de véhicule. Les données de nuage de points peuvent être analysées par des systèmes infonuagiques extensibles et centralisés pour l'extraction d'information d'actifs et la génération de cartes sémantiques. Une mémoire/un préprocesseur de données (1220) subdivise un ensemble de données pour la diffusion en continu vers une unité distribuée de traitement (1240) et une opération par des mécanismes d'analyse de données (1250). Les données de sortie de l'unité de traitement (1240) sont agrégées par un générateur de cartes (1230). Des composants d'apprentissage automatique peuvent optimiser des mécanismes d'analyse de données afin d'améliorer l'extraction d'actifs et de caractéristiques à partir de données de capteur. Des mécanismes d'analyse de données optimisés peuvent être téléchargés vers des véhicules pour utilisation dans des systèmes embarqués d'analyse des données de capteur de véhicule. Des données de carte sémantique peuvent être utilisées localement dans des véhicules, conjointement avec des détecteurs embarqués, pour dériver une localisation précise et fournir une entrée au véhicule pour des systèmes de commande.
PCT/US2016/014196 2015-01-20 2016-01-20 Vision industrielle et analyse de nuage de points en temps réel pour la télédétection et la commande de véhicule WO2016118672A2 (fr)

Priority Applications (3)

Application Number Priority Date Filing Date Title
JP2017556798A JP2018508418A (ja) 2015-01-20 2016-01-20 リモートセンシング及び車両制御のための実時間マシンビジョン並びに点群分析
EP16740714.7A EP3248140A4 (fr) 2015-01-20 2016-01-20 Vision industrielle et analyse de nuage de points en temps réel pour la télédétection et la commande de véhicule
CN201680006431.6A CN107533630A (zh) 2015-01-20 2016-01-20 用于远程感测和车辆控制的实时机器视觉和点云分析

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201562105696P 2015-01-20 2015-01-20
US62/105,696 2015-01-20

Publications (2)

Publication Number Publication Date
WO2016118672A2 true WO2016118672A2 (fr) 2016-07-28
WO2016118672A3 WO2016118672A3 (fr) 2016-10-20

Family

ID=56417920

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2016/014196 WO2016118672A2 (fr) 2015-01-20 2016-01-20 Vision industrielle et analyse de nuage de points en temps réel pour la télédétection et la commande de véhicule

Country Status (4)

Country Link
EP (1) EP3248140A4 (fr)
JP (1) JP2018508418A (fr)
CN (1) CN107533630A (fr)
WO (1) WO2016118672A2 (fr)

Cited By (43)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018134535A1 (fr) 2017-01-20 2018-07-26 Messulam Pierre Systeme d'aide a la conduite de vehicules ferroviaires
CN108571974A (zh) * 2017-03-14 2018-09-25 福特全球技术公司 使用摄像机的车辆定位
WO2018191613A1 (fr) * 2017-04-14 2018-10-18 Bayer Cropscience Lp Procédé et système de détection de végétation et d'alerte pour un véhicule ferroviaire
CN109286946A (zh) * 2018-11-13 2019-01-29 山东建筑大学 基于无依托定位的移动通信室内无线网络优化方法和系统
EP3459812A1 (fr) * 2017-09-22 2019-03-27 Aurora Flight Sciences Corporation Système de détection pour véhicule
US10426393B2 (en) 2017-09-22 2019-10-01 Aurora Flight Sciences Corporation Systems and methods for monitoring pilot health
WO2019211302A1 (fr) * 2018-05-03 2019-11-07 Thales Systeme de localisation integre et autonome d'un train dans un referentiel de reseau ferroviaire
WO2019226782A1 (fr) * 2018-05-25 2019-11-28 Bayer Cropscience Lp Système et procédé d'évaluation et de résolution du risque associé à la végétation
EP3589527A4 (fr) * 2017-02-28 2020-02-19 Thales Canada Inc. Système de localisation de véhicule monté sur voie de guidage
US10599138B2 (en) 2017-09-08 2020-03-24 Aurora Flight Sciences Corporation Autonomous package delivery system
CN111046765A (zh) * 2019-12-02 2020-04-21 北京深测科技有限公司 一种用于高铁的危险预警方法和系统
US10699135B2 (en) 2017-11-20 2020-06-30 Here Global B.V. Automatic localization geometry generator for stripe-shaped objects
CN111832548A (zh) * 2020-06-29 2020-10-27 西南交通大学 一种列车定位方法
CN112147635A (zh) * 2020-09-25 2020-12-29 北京亮道智能汽车技术有限公司 一种检测系统、方法及装置
CN112184869A (zh) * 2020-10-09 2021-01-05 北京理工大学 基于绝对高斯曲率估计的保持几何特征的点云简化方法
CN112363837A (zh) * 2020-11-19 2021-02-12 北京航天泰坦科技股份有限公司 基于集群环境的点云数据并行处理方法和装置、设备及存储介质
WO2021028082A1 (fr) * 2019-08-09 2021-02-18 Siemens Mobility GmbH Reconnaissance des rails par lidar
WO2021067935A1 (fr) * 2019-10-04 2021-04-08 Waymo Llc Intégrations spatio-temporelles
CN112669461A (zh) * 2021-01-07 2021-04-16 中煤航测遥感集团有限公司 机场净空安全检测方法、装置、电子设备及存储介质
WO2021076251A1 (fr) * 2019-10-16 2021-04-22 Bnsf Railway Company Systèmes et procédés de vérification d'actifs
CN113039579A (zh) * 2018-11-19 2021-06-25 华为技术有限公司 一种生成高分辨率高精度点云的系统、设备和方法
EP3851872A1 (fr) * 2020-01-16 2021-07-21 Outsight Détection d'objets sur un trajet de déplacement et détection d'obstacles sur des voies ferrées à l'aide d'informations sur l'espace libre
US11136120B2 (en) 2018-10-05 2021-10-05 Aurora Flight Sciences Corporation Ground operations for autonomous object pickup
CN113536883A (zh) * 2021-03-23 2021-10-22 长沙智能驾驶研究院有限公司 障碍物检测方法、车辆、设备及计算机存储介质
CN113677583A (zh) * 2020-01-21 2021-11-19 深圳元戎启行科技有限公司 基于图计算的车辆驾驶数据处理方法、装置和计算机设备
WO2021232160A1 (fr) * 2020-05-22 2021-11-25 Profound Positioning Inc. Système et procédé de localisation de véhicule
US11340080B2 (en) 2018-10-29 2022-05-24 Motional Ad Llc Automatic annotation of environmental features in a map during navigation of a vehicle
EP4002090A1 (fr) * 2020-11-24 2022-05-25 GEOTAB Inc. Mise en tampon et simplification de données à conservation de valeurs extrêmes
EP4002091A1 (fr) * 2020-11-24 2022-05-25 GEOTAB Inc. Mise en tampon et simplification de données à conservation de valeurs extrêmes
US20220165065A1 (en) * 2019-10-31 2022-05-26 Aptiv Technologies Limited Multi-Domain Neighborhood Embedding and Weighting of Sampled Data
IT202000030677A1 (it) * 2020-12-15 2022-06-15 Ecosud S R L Dispositivo dinamico e relativo metodo per la ispezione cartografica e per il trasferimento di know-how
EP4016231A1 (fr) * 2020-12-16 2022-06-22 Siemens Mobility GmbH Procédé de commande d'un dispositif
US11556509B1 (en) 2020-07-31 2023-01-17 Geotab Inc. Methods and devices for fixed interpolation error data simplification processes for telematic
US11585664B2 (en) 2020-06-16 2023-02-21 Geotab Inc. Dataset simplification of n-dimensional signals captured for asset tracking
WO2023019709A1 (fr) * 2021-08-19 2023-02-23 中国铁路设计集团有限公司 Procédé de détection automatique de hauteur de conducteur et de valeur de retrait d'un système de ligne aérienne sur la base d'un nuage de points laser mobile monté sur véhicule
US11593329B2 (en) 2020-07-31 2023-02-28 Geotab Inc. Methods and devices for fixed extrapolation error data simplification processes for telematics
US11609888B2 (en) 2020-07-31 2023-03-21 Geotab Inc. Methods and systems for fixed interpolation error data simplification processes for telematics
US11623673B2 (en) 2019-04-12 2023-04-11 Thales Management & Services Deutschland Gmbh Method for safely and autonomously determining the position information of a train on a track
EP4129742A4 (fr) * 2020-03-30 2023-05-24 Mitsubishi Electric Corporation Dispositif d'extraction de données, procédé d'extraction de données et programme d'extraction de données
WO2024068204A1 (fr) * 2022-09-29 2024-04-04 GTS France Procédé de gestion des reconfigurations réseau d'un système de communication bord/sol hybride d'un véhicule roulant ferroviaire durant un trajet
US11999396B2 (en) 2018-09-14 2024-06-04 Siemens Mobility GmbH Automated on-vehicle control system for a rail vehicle
EP4183660A4 (fr) * 2020-07-20 2024-07-31 Toshiba Kk Dispositif d'acquisition d'informations de ligne de chemin de fer et procédé d'acquisition d'informations de ligne de chemin de fer
EP4149817A4 (fr) * 2020-05-11 2024-08-07 Mtr Corporation Ltd Systèmes embarqués pour trains et procédés de détermination de vitesses de sécurité et d'emplacements de trains

Families Citing this family (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN108416808B (zh) * 2018-02-24 2022-03-08 斑马网络技术有限公司 车辆重定位的方法及装置
US10468062B1 (en) * 2018-04-03 2019-11-05 Zoox, Inc. Detecting errors in sensor data
US11050933B2 (en) * 2018-04-27 2021-06-29 Continenal Automotive Systems, Inc. Device and method for determining a center of a trailer tow coupler
CN108648150A (zh) * 2018-05-10 2018-10-12 句容康泰膨润土有限公司 一种图像拼接方法
US20200103233A1 (en) * 2018-10-01 2020-04-02 GM Global Technology Operations LLC Decentralized distributed map using blockchain
US20190061771A1 (en) * 2018-10-29 2019-02-28 GM Global Technology Operations LLC Systems and methods for predicting sensor information
CN111198904A (zh) * 2018-11-16 2020-05-26 千寻位置网络有限公司 数据处理方法及装置、处理系统
US11536844B2 (en) * 2018-12-14 2022-12-27 Beijing Voyager Technology Co., Ltd. Dynamic sensor range detection for vehicle navigation
CN109948661B (zh) * 2019-02-27 2023-04-07 江苏大学 一种基于多传感器融合的3d车辆检测方法
CN111935642B (zh) * 2019-05-13 2022-10-21 北京地平线机器人技术研发有限公司 一种可移动设备的定位方法及装置
US11581022B2 (en) * 2019-05-29 2023-02-14 Nokia Technologies Oy Method and apparatus for storage and signaling of compressed point clouds
KR102043405B1 (ko) * 2019-06-20 2019-11-12 주식회사 첨단공간정보 지아이에스를 기반으로 하는 지형지물 변형 편집용 영상처리 편집시스템
CN112241129A (zh) * 2019-07-17 2021-01-19 中车长春轨道客车股份有限公司 控制系统、方法、装置、计算机设备和存储介质
JP7343153B2 (ja) * 2019-08-08 2023-09-12 独立行政法人自動車技術総合機構 列車位置検出装置
CN112441075A (zh) * 2019-08-30 2021-03-05 比亚迪股份有限公司 轨道交通外部环境感知系统、方法和轨道交通设备
CN110481601B (zh) * 2019-09-04 2022-03-08 深圳市镭神智能系统有限公司 一种轨道检测系统
CN111145166B (zh) * 2019-12-31 2023-09-01 北京深测科技有限公司 安全监控的方法及系统
CN111223107A (zh) * 2019-12-31 2020-06-02 武汉中海庭数据技术有限公司 基于点云深度学习的点云数据集制作系统及方法
CN111091700A (zh) * 2020-03-23 2020-05-01 北京全路通信信号研究设计院集团有限公司 用于轨道电路室外监测设备的综合配置智能终端
CN111427985B (zh) * 2020-03-25 2023-04-11 北京小马慧行科技有限公司 地图更新的方法及装置、存储介质、处理器
EP3919344B1 (fr) * 2020-06-05 2024-08-28 Mitsubishi Electric R&D Centre Europe B.V. Procédé de géolocalisation d'une source d'interférence dans un système de transport basé en communication
CN112132896B (zh) * 2020-09-16 2024-05-10 北京埃福瑞科技有限公司 一种轨旁设备状态检测方法及系统
US20220091266A1 (en) * 2020-09-18 2022-03-24 Denso International America, Inc. Systems and methods for enhancing outputs of a lidar
CN113514030B (zh) * 2021-07-12 2022-08-23 石家庄铁道大学 一种道床断面检测方法及系统
CN115348253A (zh) * 2022-06-28 2022-11-15 高德软件有限公司 数据传输系统的数据传输方法及数据传输系统
CN117311283B (zh) * 2023-10-24 2024-03-19 风凯换热器制造(常州)有限公司 换热器内部预装体的车间行车控制智能化监测方法及系统

Family Cites Families (15)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP4086994B2 (ja) * 1999-02-08 2008-05-14 株式会社デンソー 画像データ供給装置及び画像圧縮装置
JP2006321277A (ja) * 2005-05-17 2006-11-30 Hitachi Ltd 鉄道設備情報提供システム、鉄道設備情報提供方法、および鉄道設備情報提供プログラム
US8605947B2 (en) * 2008-04-24 2013-12-10 GM Global Technology Operations LLC Method for detecting a clear path of travel for a vehicle enhanced by object detection
JP2010202017A (ja) * 2009-03-03 2010-09-16 Mitsubishi Electric Corp データ解析装置及びデータ解析方法及びプログラム
US20140379254A1 (en) * 2009-08-25 2014-12-25 Tomtom Global Content B.V. Positioning system and method for use in a vehicle navigation system
CN101697229B (zh) * 2009-10-30 2012-06-13 宁波大学 一种医学图像的感兴趣区域提取方法
US8525835B1 (en) * 2010-02-24 2013-09-03 The Boeing Company Spatial data compression using implicit geometry
US20110216063A1 (en) * 2010-03-08 2011-09-08 Celartem, Inc. Lidar triangular network compression
US20130096886A1 (en) * 2010-03-31 2013-04-18 Borys Vorobyov System and Method for Extracting Features from Data Having Spatial Coordinates
US9102341B2 (en) * 2012-06-15 2015-08-11 Transportation Technology Center, Inc. Method for detecting the extent of clear, intact track near a railway vehicle
CN102799898A (zh) * 2012-06-28 2012-11-28 浙江大学 大背景下高分辨率遥感影像目标识别高效处理方法
US9221461B2 (en) * 2012-09-05 2015-12-29 Google Inc. Construction zone detection using a plurality of information sources
US9846025B2 (en) * 2012-12-21 2017-12-19 Wabtec Holding Corp. Track data determination system and method
JP2014185947A (ja) * 2013-03-25 2014-10-02 Geo Technical Laboratory Co Ltd ３次元復元のための画像撮影方法
DE102013104088A1 (de) * 2013-04-23 2014-10-23 Deutsches Zentrum für Luft- und Raumfahrt e.V. Verfahren zur automatischen Detektion von charakteristischen Elementen, insbesondere eines Bahnübergangs, und Einrichtung dafür

Cited By (73)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018134535A1 (fr) 2017-01-20 2018-07-26 Messulam Pierre Systeme d'aide a la conduite de vehicules ferroviaires
FR3062103A1 (fr) * 2017-01-20 2018-07-27 Pierre Messulam Systeme d'aide a la conduite de vehicules ferroviaires
EP3589527A4 (fr) * 2017-02-28 2020-02-19 Thales Canada Inc. Système de localisation de véhicule monté sur voie de guidage
US11608097B2 (en) 2017-02-28 2023-03-21 Thales Canada Inc Guideway mounted vehicle localization system
CN108571974B (zh) * 2017-03-14 2023-12-22 福特全球技术公司 使用摄像机的车辆定位
CN108571974A (zh) * 2017-03-14 2018-09-25 福特全球技术公司 使用摄像机的车辆定位
EP4095012A3 (fr) * 2017-04-14 2023-05-03 Bayer CropScience LP Procédé et système de détection de végétation et d'alerte pour un véhicule ferroviaire
US10370014B2 (en) 2017-04-14 2019-08-06 Bayer Cropscience Lp Vegetation detection and alert system for a railway vehicle
WO2018191613A1 (fr) * 2017-04-14 2018-10-18 Bayer Cropscience Lp Procédé et système de détection de végétation et d'alerte pour un véhicule ferroviaire
AU2022263574B2 (en) * 2017-04-14 2023-06-15 Discovery Purchaser Corporation Vegetation detection and alert method and system for a railway vehicle
US11046340B2 (en) 2017-04-14 2021-06-29 Bayer Cropscience Lp Vegetation detection and alert system for a railway vehicle
EP4049914A3 (fr) * 2017-04-14 2023-02-22 Bayer CropScience LP Procédé et système de détection d'objet et d'alerte pour un véhicule ferroviaire
CN110662687B (zh) * 2017-04-14 2021-12-28 拜耳作物科学有限合伙公司 用于铁路载具的植被检测和警报方法和系统
CN110662687A (zh) * 2017-04-14 2020-01-07 拜耳作物科学有限合伙公司 用于铁路载具的植被检测和警报方法和系统
AU2018253281B2 (en) * 2017-04-14 2022-08-04 Discovery Purchaser Corporation Vegetation detection and alert method and system for a railway vehicle
US10599138B2 (en) 2017-09-08 2020-03-24 Aurora Flight Sciences Corporation Autonomous package delivery system
US10426393B2 (en) 2017-09-22 2019-10-01 Aurora Flight Sciences Corporation Systems and methods for monitoring pilot health
EP3459812A1 (fr) * 2017-09-22 2019-03-27 Aurora Flight Sciences Corporation Système de détection pour véhicule
US10699135B2 (en) 2017-11-20 2020-06-30 Here Global B.V. Automatic localization geometry generator for stripe-shaped objects
WO2019211302A1 (fr) * 2018-05-03 2019-11-07 Thales Systeme de localisation integre et autonome d'un train dans un referentiel de reseau ferroviaire
FR3080823A1 (fr) * 2018-05-03 2019-11-08 Thales Systeme de localisation integre et autonome d'un train dans un referentiel de reseau ferroviaire
AU2019275041B2 (en) * 2018-05-25 2022-03-24 Discovery Purchaser Corporation System and method for vegetation management risk assessment and resolution
US11010608B2 (en) 2018-05-25 2021-05-18 Bayer Cropscience Lp System and method for vegetation management risk assessment and resolution
WO2019226782A1 (fr) * 2018-05-25 2019-11-28 Bayer Cropscience Lp Système et procédé d'évaluation et de résolution du risque associé à la végétation
US11999396B2 (en) 2018-09-14 2024-06-04 Siemens Mobility GmbH Automated on-vehicle control system for a rail vehicle
US11136120B2 (en) 2018-10-05 2021-10-05 Aurora Flight Sciences Corporation Ground operations for autonomous object pickup
US11774261B2 (en) 2018-10-29 2023-10-03 Motional Ad Llc Automatic annotation of environmental features in a map during navigation of a vehicle
US11340080B2 (en) 2018-10-29 2022-05-24 Motional Ad Llc Automatic annotation of environmental features in a map during navigation of a vehicle
CN109286946A (zh) * 2018-11-13 2019-01-29 山东建筑大学 基于无依托定位的移动通信室内无线网络优化方法和系统
CN113039579A (zh) * 2018-11-19 2021-06-25 华为技术有限公司 一种生成高分辨率高精度点云的系统、设备和方法
US11623673B2 (en) 2019-04-12 2023-04-11 Thales Management & Services Deutschland Gmbh Method for safely and autonomously determining the position information of a train on a track
WO2021028082A1 (fr) * 2019-08-09 2021-02-18 Siemens Mobility GmbH Reconnaissance des rails par lidar
US11657291B2 (en) 2019-10-04 2023-05-23 Waymo Llc Spatio-temporal embeddings
WO2021067935A1 (fr) * 2019-10-04 2021-04-08 Waymo Llc Intégrations spatio-temporelles
US11427232B2 (en) 2019-10-16 2022-08-30 Bnsf Railway Company Systems and methods for auditing assets
US11932290B2 (en) 2019-10-16 2024-03-19 Bnsf Railway Company Systems and methods for auditing assets
WO2021076251A1 (fr) * 2019-10-16 2021-04-22 Bnsf Railway Company Systèmes et procédés de vérification d'actifs
US11693090B2 (en) * 2019-10-31 2023-07-04 Aptiv Technologies Limited Multi-domain neighborhood embedding and weighting of sampled data
US20220165065A1 (en) * 2019-10-31 2022-05-26 Aptiv Technologies Limited Multi-Domain Neighborhood Embedding and Weighting of Sampled Data
CN111046765A (zh) * 2019-12-02 2020-04-21 北京深测科技有限公司 一种用于高铁的危险预警方法和系统
CN111046765B (zh) * 2019-12-02 2023-07-14 北京深测科技有限公司 一种用于高铁的危险预警方法和系统
EP3851872A1 (fr) * 2020-01-16 2021-07-21 Outsight Détection d'objets sur un trajet de déplacement et détection d'obstacles sur des voies ferrées à l'aide d'informations sur l'espace libre
CN113677583B (zh) * 2020-01-21 2023-08-15 深圳元戎启行科技有限公司 基于图计算的车辆驾驶数据处理方法、装置和计算机设备
CN113677583A (zh) * 2020-01-21 2021-11-19 深圳元戎启行科技有限公司 基于图计算的车辆驾驶数据处理方法、装置和计算机设备
EP4129742A4 (fr) * 2020-03-30 2023-05-24 Mitsubishi Electric Corporation Dispositif d'extraction de données, procédé d'extraction de données et programme d'extraction de données
EP4149817A4 (fr) * 2020-05-11 2024-08-07 Mtr Corporation Ltd Systèmes embarqués pour trains et procédés de détermination de vitesses de sécurité et d'emplacements de trains
WO2021232160A1 (fr) * 2020-05-22 2021-11-25 Profound Positioning Inc. Système et procédé de localisation de véhicule
US11867512B2 (en) 2020-06-16 2024-01-09 Geotab Inc. Dataset simplification of n-dimensional signals captured for asset tracking
US11585664B2 (en) 2020-06-16 2023-02-21 Geotab Inc. Dataset simplification of n-dimensional signals captured for asset tracking
CN111832548A (zh) * 2020-06-29 2020-10-27 西南交通大学 一种列车定位方法
CN111832548B (zh) * 2020-06-29 2022-11-15 西南交通大学 一种列车定位方法
EP4183660A4 (fr) * 2020-07-20 2024-07-31 Toshiba Kk Dispositif d'acquisition d'informations de ligne de chemin de fer et procédé d'acquisition d'informations de ligne de chemin de fer
US11593329B2 (en) 2020-07-31 2023-02-28 Geotab Inc. Methods and devices for fixed extrapolation error data simplification processes for telematics
US11609888B2 (en) 2020-07-31 2023-03-21 Geotab Inc. Methods and systems for fixed interpolation error data simplification processes for telematics
US11556509B1 (en) 2020-07-31 2023-01-17 Geotab Inc. Methods and devices for fixed interpolation error data simplification processes for telematic
CN112147635B (zh) * 2020-09-25 2024-05-31 北京亮道智能汽车技术有限公司 一种检测系统、方法及装置
CN112147635A (zh) * 2020-09-25 2020-12-29 北京亮道智能汽车技术有限公司 一种检测系统、方法及装置
CN112184869A (zh) * 2020-10-09 2021-01-05 北京理工大学 基于绝对高斯曲率估计的保持几何特征的点云简化方法
CN112363837A (zh) * 2020-11-19 2021-02-12 北京航天泰坦科技股份有限公司 基于集群环境的点云数据并行处理方法和装置、设备及存储介质
EP4002091A1 (fr) * 2020-11-24 2022-05-25 GEOTAB Inc. Mise en tampon et simplification de données à conservation de valeurs extrêmes
US11546395B2 (en) 2020-11-24 2023-01-03 Geotab Inc. Extrema-retentive data buffering and simplification
EP4002090A1 (fr) * 2020-11-24 2022-05-25 GEOTAB Inc. Mise en tampon et simplification de données à conservation de valeurs extrêmes
US11838364B2 (en) * 2020-11-24 2023-12-05 Geotab Inc. Extrema-retentive data buffering and simplification
US20220166834A1 (en) * 2020-11-24 2022-05-26 Geotab Inc. Extrema-retentive data buffering and simplification
IT202000030677A1 (it) * 2020-12-15 2022-06-15 Ecosud S R L Dispositivo dinamico e relativo metodo per la ispezione cartografica e per il trasferimento di know-how
EP4016231A1 (fr) * 2020-12-16 2022-06-22 Siemens Mobility GmbH Procédé de commande d'un dispositif
CN112669461B (zh) * 2021-01-07 2024-01-26 中煤航测遥感集团有限公司 机场净空安全检测方法、装置、电子设备及存储介质
CN112669461A (zh) * 2021-01-07 2021-04-16 中煤航测遥感集团有限公司 机场净空安全检测方法、装置、电子设备及存储介质
CN113536883B (zh) * 2021-03-23 2023-05-02 长沙智能驾驶研究院有限公司 障碍物检测方法、车辆、设备及计算机存储介质
CN113536883A (zh) * 2021-03-23 2021-10-22 长沙智能驾驶研究院有限公司 障碍物检测方法、车辆、设备及计算机存储介质
WO2023019709A1 (fr) * 2021-08-19 2023-02-23 中国铁路设计集团有限公司 Procédé de détection automatique de hauteur de conducteur et de valeur de retrait d'un système de ligne aérienne sur la base d'un nuage de points laser mobile monté sur véhicule
WO2024068204A1 (fr) * 2022-09-29 2024-04-04 GTS France Procédé de gestion des reconfigurations réseau d'un système de communication bord/sol hybride d'un véhicule roulant ferroviaire durant un trajet
FR3140506A1 (fr) * 2022-09-29 2024-04-05 Thales Procédé de gestion des reconfigurations réseau d'un système de communication bord/sol hybride d'un véhicule roulant ferroviaire durant un trajet

Also Published As

Publication number Publication date
JP2018508418A (ja) 2018-03-29
EP3248140A2 (fr) 2017-11-29
EP3248140A4 (fr) 2018-12-05
WO2016118672A3 (fr) 2016-10-20
CN107533630A (zh) 2018-01-02

Similar Documents

Publication Publication Date Title
US10549768B2 (en) Real time machine vision and point-cloud analysis for remote sensing and vehicle control
WO2016118672A2 (fr) Vision industrielle et analyse de nuage de points en temps réel pour la télédétection et la commande de véhicule
US11959740B2 (en) Three-dimensional data creation method and three-dimensional data creation device
US20180370552A1 (en) Real time machine vision system for vehicle control and protection
CN110832474B (zh) 更新高清地图的方法
US11988518B2 (en) Updating high definition maps based on lane closure and lane opening
US20210004363A1 (en) Updating high definition maps based on age of maps
US10832502B2 (en) Calibration for autonomous vehicle operation
EP3371797B1 (fr) Cartographie adaptative pour le pilotage de véhicules autonomes réagissant aux changements d'environnement physique
US11346682B2 (en) Augmented 3D map
CN108369775B (zh) 响应于物理环境的改变自适应制图以对自主车辆进行导航
JP2019502214A (ja) 物理的環境の変化に応答して自律車両をナビゲートするための適応マッピング
JP2019501468A (ja) 遠隔操作および／またはプランナ決定を最適化するための機械学習システムおよび技法
JP2024025803A (ja) センサを利用して獲得した空間情報を活用する車両、センサを利用して獲得した空間情報を活用するセンシング装置、及びサーバ
US20220351526A1 (en) Multi-frame image segmentation
JP2020510941A (ja) コネクテッド自動運転車のハイウエイシステムとそれを用いた方法
US20220012899A1 (en) Systems and methods for deriving an agent trajectory based on tracking points within images
US20240104757A1 (en) Systems and methods for using image data to identify lane width
WO2024086778A1 (fr) Réseaux neuronaux de graphe pour analyser des routes
US12044784B2 (en) Device and method for autonomously locating a mobile vehicle on a railway track
US12097880B2 (en) Augmented 3D map
US20220234621A1 (en) Augmented 3d map
US20240183684A1 (en) Using robot observations
US20240104939A1 (en) Systems and methods for using image data to identify lane width
CN116912436A (zh) 生成图像地图的方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16740714

Country of ref document: EP

Kind code of ref document: A2

ENP Entry into the national phase

Ref document number: 2017556798

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

REEP Request for entry into the european phase

Ref document number: 2016740714

Country of ref document: EP

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16740714

Country of ref document: EP

Kind code of ref document: A2