RU2812263C2 - Automated monitoring system of signal conformity and emergency notification - Google Patents

Abstract

FIELD: mobile assets.

SUBSTANCE: invention relates to a system for automated monitoring of signals and emergency notifications in mobile assets. The technical result is achieved by the fact that the claimed solution includes image measurement devices, range measurement devices, and a microphone; means of recording data on board the mobile asset; database; data sources remote from the mobile asset; as well as the first artificial intelligence-based model to determine the triggering condition; and the second AI-based model with the ability to identify an episode based on video content.

EFFECT: ensuring the safe operation of mobile assets in real time.

21 cl, 28 dwg

Description

Cross reference to related application

[0001] This application claims priority to U.S. Provisional Patent Application No. 62/825,943, filed March 29, 2019, claims priority to U.S. Provisional Patent Application No. 62/829,730, filed April 5, 2019, and claims to priority to U.S. Non-Provisional Patent Application No. 16/833590, filed March 28, 2020, to the extent permitted by law, the entire contents of which are incorporated herein by reference. This application also claims priority to U.S. Provisional Patent Application No. 62/337,227, filed May 16, 2016; non-provisional patent application (US) number 15/595650, filed May 15, 2017, now patent (US) number 9934623, issued April 3, 2018; non-provisional patent application (US) number 15/907486, filed February 28, 2018, now patent (US) number 10445951, issued October 15, 2019; US Provisional Patent Application No. 62/337225, filed May 16, 2016; non-provisional patent application (US) (US) number 15/595689, filed May 15, 2017, now patent (US) number 10410441, issued September 10, 2019; US patent application number 16/385745, filed April 16, 2019; US Provisional Patent Application No. 62/337228, filed May 16, 2016; non-provisional patent application (US) number 15/595712, filed May 15, 2017, now patent (US) number 10392038, issued August 27, 2019; US Provisional Patent Application No. 62/680907, filed June 5, 2018; and non-provisional patent application (US) number 16/431466, filed June 4, 2019. Disclosures of each of the above documents are contained in their entirety herein by reference. All patent applications, patents and printed publications cited herein are incorporated herein by reference in their entirety, excluding any definitions, disclaimers or disclaimers of subject matter and except to the extent that included material is inconsistent with the express disclosure herein, in which case terminology prevails in this disclosure.

Field of technology to which the invention relates

[0002] This disclosure relates to equipment used in high-value assets, and in particular to an automated signal compliance monitoring and alarm system used in high-value mobile assets.

State of the art

[0003] High-value mobile assets such as locomotives, aircraft, various forms of public transport, mining equipment, transportable medical equipment, cargo, marine vessels and military vessels typically use on-board "black box" systems to receive and record data and/or or "event recording" systems. These data acquisition and recording systems, such as event data recorders or flight data recorders, record a variety of system parameters used for incident investigation, crew performance evaluation, fuel efficiency analysis, maintenance planning, and predictive diagnostics. A typical data acquisition and recording system contains digital and analog inputs, as well as pressure switches and pressure transducers that record data from various onboard sensor devices. Recorded data may include parameters such as speed, distance traveled, location, fuel level, engine revolutions per minute (RPM), fluid levels, operator controls, pressures, current and predicted weather and environmental conditions. In addition to basic event data and operational data, video and audio event/data recording capabilities are also being deployed on many of these identical mobile assets. Typically, data is retrieved from the data recorder after an asset incident occurs and an investigation is required after the data recorder is recovered. There may be some situations in which the data writer cannot be recovered or the data is otherwise unavailable. In these situations, data such as event data and operating data, video data and audio data obtained through the data acquisition system are required quickly regardless of whether physical access to the acquisition system or the data is available.

The essence of the invention

[0004] This disclosure generally relates to real-time data acquisition and recording systems and automated alarm monitoring and alerting systems used in high-value mobile assets. The teachings herein may provide real-time or near real-time access to data, such as event and operational data, video data, and audio data, recorded by a real-time data acquisition and recording system in a high-value mobile asset. One implementation is a method for processing data from at least one mobile asset, which includes the steps of: receiving, using one of a data processing and storage center remote from at least one mobile asset, and a processing center and storing data on board the at least one mobile asset, data based on at least one data signal from at least one of the following: at least one data source on board the at least one mobile asset; and at least one data source remote from the at least one mobile asset; determining, using a first data center artificial intelligence-based model, that at least one mobile asset is at least one of a head mobile asset and a control mobile asset, provided that at least at least one trigger condition is detected using a data center based on the data; receiving, using the data center, video content from at least one of the head mobile asset and the control mobile asset, wherein the video content comprises a configurable predetermined amount of data collected over a configurable predetermined amount of time to at least one triggering condition; saving, using the database of the data processing and storage center, video content; determining, using a second data center artificial intelligence-based model, an episode based on the video content; saving, using the database of the data processing and storage center, the episode; and sending, using the data processing center, an electronic message to a predetermined number of users.

[0005] One implementation is a system for processing data from at least one mobile asset, which includes: at least one, at least one image measurement device, at least one video measurement device at least one ranging device and at least one microphone; means for recording data on board at least one mobile asset, adapted to receive at least one data signal from at least one at least one image measuring device at least one video measuring device, at least one ranging device, at least one microphone, at least one on-board data source, at least one mobile asset, and at least one a data source remote from at least one mobile asset; a data processing and storage center adapted to receive data based on at least one data signal; a first artificial intelligence-based model of the data center, wherein the first artificial intelligence-based model is adapted to determine that the at least one mobile asset is at least one of a head mobile asset and a control mobile asset provided that at least one triggering condition is detected by the data processing center; a data processing and storage center database, wherein the database is adapted to store video content obtained from at least one, at least one image measurement device, at least one video measurement device, at least one devices for measuring the range of at least one microphone, at least one data source on board, at least one mobile asset and at least one data source remote from at least one mobile asset; and a second AI-based model of the data center, the second AI-based model being adapted to detect an episode based on the video content.

[0006] Variations of these and other aspects of the disclosure are described in more detail below.

Brief description of drawings

[0007] The description herein refers to the accompanying drawings, in which like reference numbers indicate like parts in several views and in which:

[0008] FIG. 1 illustrates a field implementation of a first embodiment of an exemplary real-time data acquisition and recording system in accordance with implementations of this disclosure;

[0009] FIG. 2 illustrates a field implementation of a second embodiment of an exemplary real-time data acquisition and recording system in accordance with implementations of this disclosure;

[0010] FIG. 3 is a flowchart of a method for recording data and/or information from a mobile asset in accordance with implementations of this disclosure;

[0011] FIG. 4 is a flowchart for a process for appending data and/or information from a mobile asset after a power failure in accordance with implementations of this disclosure;

[0012] FIG. 5 is a diagram that illustrates exemplary intermediate record blocks and complete record blocks stored in a durable storage module in accordance with implementations of this disclosure;

[0013] FIG. 6 is a diagram that illustrates exemplary intermediate blocks of records in an impact resistant memory module before power failure and after power restoration in accordance with implementations of this disclosure;

[0014] FIG. 7 is a diagram that illustrates an example segment of records in an impact resistant memory module after power is restored in accordance with implementations of this disclosure;

[0015] FIG. 8 illustrates a field implementation of a first embodiment of a real-time data acquisition and recording system viewer in accordance with implementations of this disclosure;

[0016] FIG. 9 is a flowchart of a method for recording video data, audio data and/or information from a mobile asset in accordance with implementations of this disclosure;

[0017] FIG. 10 is a flowchart of a method for recording video data, audio data and/or information from a mobile asset in accordance with implementations of this disclosure;

[0018] FIG. 11 is a flowchart that illustrates an exemplary fisheye view of a 360-degree camera viewer of a real-time data acquisition and recording system in accordance with implementations of this disclosure;

[0019] FIG. 12 is a diagram that illustrates an exemplary 360-degree panoramic camera view of a real-time data acquisition and recording system viewer in accordance with implementations of this disclosure;

[0020] FIG. 13 is a diagram that illustrates an exemplary 360-degree quadratic camera view of a real-time acquisition and recording system viewer in accordance with implementations of this disclosure;

[0021] FIG. 14 is a diagram that illustrates an exemplary distortion-free view of a 360-degree camera viewer of a real-time data acquisition and recording system in accordance with implementations of this disclosure;

[0022] FIG. 15 illustrates a field implementation of a first embodiment of a video content analysis system of a data acquisition and recording system in accordance with implementations of this disclosure;

[0023] FIG. 16A is a diagram that illustrates exemplary rut detection in accordance with implementations of this disclosure;

[0024] FIG. 16B is a diagram that illustrates exemplary rut detection and arrow detection in accordance with implementations of this disclosure;

[0025] FIG. 16C is a diagram that illustrates exemplary rut detection, rut counting, and signal detection in accordance with implementations of this disclosure;

[0026] FIG. 16D is a diagram that illustrates exemplary level crossing and rut detection in accordance with implementations of this disclosure;

[0027] FIG. 16E is a diagram that illustrates exemplary dual air signal detection in accordance with implementations of this disclosure;

[0028] FIG. 16F is a diagram that illustrates exemplary multi-rut detection in accordance with implementations of this disclosure;

[0029] FIG. 16G is a diagram that illustrates exemplary switch and track detection in accordance with implementations of this disclosure;

[0030] FIG. 16H is a diagram that illustrates exemplary arrow detection in accordance with implementations of this disclosure;

[0031] FIG. 17 is a flowchart of a method for determining the internal state of a mobile asset in accordance with implementations of this disclosure;

[0032] FIG. 18 is a flowchart for a process for determining object detection and obstacle detection occurring externally to a mobile asset, in accordance with implementations of this disclosure;

[0033] FIG. 19 illustrates a field implementation of a seventh embodiment of an exemplary real-time data acquisition and recording system in accordance with implementations of this disclosure;

[0034] FIG. 20 is a diagram that illustrates exemplary signal detection of an automated signal compliance monitoring and alarm system in accordance with implementations of this disclosure; And

[0035] FIG. 21 is a flowchart of a method of a first embodiment of a process for determining signal compliance in accordance with implementations of this disclosure.

Detailed Description of the Invention

[0036] The first embodiment of the real-time data acquisition and recording system described herein provides real-time or near real-time access to a wide range of data, such as event and operational data, video data and audio data associated with expensive asset to remotely located users such as asset owners, machinists, and investigators. The data acquisition and recording system records data, through a data recorder, associated with an asset and streams the data to a remote data repository and remotely located users before, during and after an incident occurs. Data is streamed to a remote data repository in real time or near real time, making information available at least up to the time of the incident or emergency, thereby virtually eliminating the need to locate and download a "black box" to investigate an incident involving an asset, and eliminating the need to interact with the data recorder on the asset to request the download of specific data, locate and transfer files, and use a user application to view the data. The present entity discovery system retains the typical recording capability and adds the ability to stream data to a remote data repository and to a remote end user before, during and after an incident. In the vast majority of situations, the information written to the data recorder is redundant and unnecessary because the data is already being retrieved and stored in a remote data repository.

[0037] Prior to the present disclosure system, data is retrieved from a "black box" or "event recorder" after an incident occurs and an investigation is required. Data files containing time segments recorded by the black box must be loaded into and retrieved from the black box and then viewed by the user using the user's own software. The user must gain physical or remote access to the asset, select the required data to be downloaded from the black box, upload a file containing the required information to the computing device, and locate the corresponding file containing the required data using a user application that runs on the computing device. device. The system of the present disclosure eliminates the need for the user to perform these steps, requiring the user only to use a general web browser in order to navigate to the required data. A remotely located user can access a common web browser to navigate to the required data associated with a selected asset to view and analyze asset performance and safety in real time or near real time.

[0038] A remotely located user, such as an asset owner, machinist, and/or investigator, can access a common web browser to navigate to live and/or statistical desired data associated with a selected asset to view and analyze operational performance and real-time or near-real-time asset security. The ability to view transactions in real time or near real time allows for rapid assessment and regulation of behavior. During an incident, for example, real-time information and/or data can make it easier to triage the situation and provide valuable information to emergency responders. During normal operation, for example, real-time information and/or data may be used to audit crew performance and contribute to network-wide situational awareness.

[0039] The data may include, but is not limited to, analog and frequency parameters such as speed, pressure, temperature, current, voltage and acceleration that originate from the asset and/or nearby assets, Boolean data such as arrow positions, actuator position, warning lights and actuator control commands, global positioning system (GPS) data and/or geographic information system (GIS) data such as position, speed and altitude, internally generated information such as legal speed limit for asset based on its current position, video and image information from cameras located at various locations in, on or near the asset, audio information from microphones located at various locations in, on or near the asset, information regarding the operational plan for the asset that is being sent to asset from a data center, such as routing, schedule and freight manifest information, information regarding environmental conditions, including current and forecast weather conditions for the area in which the asset is currently operating or planned to operate, condition data and operational asset management data generated through systems such as Precision Train Control (PTC) systems in locomotives, and data derived from a combination of any of the above, including, but not limited to, additional data, video and audio analysis and analytics.

[0040] FIG. 1 and 2 illustrate a field implementation of a first embodiment and a second embodiment, respectively, of an exemplary real-time data acquisition and recording system (DARS) 100, 200 in which aspects of entity disclosure may be implemented. DARS 100, 200 is a system that delivers real-time information to remotely located end users from a data recorder. DARS 100, 200 includes a data recorder 154, 254 that is installed on a vehicle or mobile asset 148, 248 and communicates with any number of different information sources via any combination of on-board wired and/or wireless transmission links 170, 270 data, such as a wireless gateway/router, or off-board information sources via the DARS data processing and storage center 150, 250 100, 200 via data links, for example, via wireless data links 146. The data recorder 154, 254 includes an on-board data manager 120, 220, a data encoder 122, 222, a vehicle event detector 156, 256, a queuing repository 158, 258, and a wireless gateway/router 172, 272. Additionally, in this implementation , the data recording means 154, 254 may include a high-impact memory module 118, 218 and/or an Ethernet switch 162, 262 with/without power over Ethernet (POE). The exemplary secure memory module 118, 218, for example, may be a crash-resistant event recorder memory module that complies with the Code of Federal Regulations and/or Federal Railroad Administration regulations, a crash survivable memory unit that complies with the Code of Federal Regulations and/or FAA regulations, an impact resistant memory module in accordance with any applicable Code of Federal Regulations, or any other suitable secure storage device known in the art. In the second embodiment shown in FIG. 2, the data recording means 254 may further include an optional non-shockproof removable data storage device 219.

[0041] The wired and/or wireless data communication lines 170, 270 may include any one or combination of discrete signal inputs, standard or proprietary Ethernet, serial connections, and wireless connections. Ethernet-connected devices may use an Ethernet switch 162, 262 to record data 154, 254 and may use POE. The Ethernet switch 162, 262 may be internal or external and may support POE. Additionally, data from remote data sources such as the mapping component 164, 264, the route/crew manifest processing component 124, 224, and the weather component 126, 226 in the implementation of FIG. 1 and 2 are accessible to the on-board data manager 120, 220 and vehicle event detector 156, 256 from the data processing and storage center 150, 250 via a wireless data link 146, 246 and a wireless gateway/router 172, 272.

[0042] The data recorder 154, 254 collects data or information from a wide range of sources, which may vary significantly based on the configuration of the asset, through on-board data communications links 170, 270. The data encoder 122, 222 encodes at least a minimum set of data, which is typically specified by the regulator. In this implementation, the data encoder 122, 222 receives data from a wide range of asset sources 148, 248 and data center sources 150, 250. The information sources may include any number of components in the asset 148, 248, for example, any of analog inputs 102, 202, digital inputs 104, 204, I/O module 106, 206, vehicle controller 108, 208, controller 110, 210 engine, inertial sensors 112, 212, global positioning system (GPS) 114, 214, camera 116, 216, signal data 166, 266 precision train traffic control system, fuel system data 168, 268, cellular detectors (not shown) , internally managed data and any additional data signals, and any of a number of components in the data processing and storage center 150, 250, for example, any of the route/team manifest processing component 124, 224, the weather component 126, 226, the mapping component 164, 264 and any additional data signals. The data encoder 122, 222 compresses or encodes the data and time-synchronizes the data to facilitate efficient transfer and real-time replication to the remote data repository 130, 230. The data encoder 122, 222 transmits the encoded data to the on-board data manager 120, 220, which then stores the encoded data in a high-impact storage module 118, 218 and a repository 158, 258 and is queued for replication to a remote data repository 130, 230 via the remote manager 132 , 232 data, located in the center 150, 250 data processing and storage. Optionally, the on-board data manager 120, 220 may store a tertiary copy of the encoded data in the non-impact removable data storage device 219 of the second embodiment shown in FIG. 2. The onboard data manager 120, 220 and the remote data manager 132, 232 operate in concert to manage the data replication process. A single remote data manager 132, 232 at a data center 150, 250 can manage data replication from multiple assets 148, 248.

[0043] Data from various input components and data from the audio/graphical user interface (GUI) 160, 260 in the driver's station are sent to the vehicle event detector 156, 256. The vehicle event detector 156, 256 processes data to determine whether or not an event, incident, or other predetermined situation involving the asset 148, 248 occurs. When the vehicle event detector 156, 256 detects signals that indicate that that the predetermined event occurs, the vehicle event detector 156, 256 sends processed data regarding that the predetermined event occurs, along with confirmation data accompanying the predetermined event, to the on-board data manager 120, 220. The vehicle event detector 156, 256 detects events based on data from a wide variety of sources, such as analog inputs 102, 202, digital inputs 104, 204, I/O module 106, 206, vehicle controller 108, 208, controller 110, 210 engine, inertial sensors 112, 212, GPS 114, 214, cameras 116, 216, route/team declaration processing component 124, 224, weather component 126, 226, mapping component 164, 264, PTC signaling data 166, 266 and data 168, 268 fuel system, which may vary based on asset configuration. When the vehicle event detector 156, 256 detects an event, the detected asset event information is stored in a queued repository 158, 258 and can optionally be presented to the asset crew 148, 248 via an audio/graphical user interface (GUI) 160, 260 in the driver's station. .

[0044] The onboard data manager 120, 220 also sends data to the repository 158 for queuing. In near real time, the onboard data manager 120, 220 stores the encoded data received from the data encoder 122, 222 and any event information in a high-impact memory module 118, 218 and a queued repository 158, 258. In the second embodiment of FIG. 2, the on-board data manager 220 may optionally store the encoded data in a non-shatterproof removable data storage device 219. After five minutes of encoded data have accumulated in the queuing repository 158, 258, the onboard data manager 120, 220 stores the five minutes of encoded data in the remote data repository 130, 230 via the remote data manager 132, 232 at the processing center 150, 250 and storing data over a wireless data link 146, 246 accessible through the wireless gateway/router 172, 272. In real time, the on-board data manager 120, 220 stores encoded data received from the data encoder 122, 222 and any information events to the impact-resistant memory module 118, 218 and optionally to the non-impact-resistant removable storage device 219 of FIG. 2 and to a remote data repository 130, 230 via a remote data manager 132, 232 at a data processing and storage center 150, 250 over a wireless data link 146, 246 accessible through a wireless gateway/router 172, 272. Airborne dispatcher 120, The data 220 and the remote data manager 132, 232 may communicate over multiple wireless communication links, such as Wi-Fi, cellular, satellite and private wireless systems using a wireless gateway/router 172, 272. Wireless communication link 146, 246 for data transmission, for example, may be a wireless local area network (WLAN), a wireless metropolitan area network (WMAN), a wireless wide area network (WWAN), a private wireless system, a cellular telephone network, or any other means of transmitting data from means 154, 254 records of DARS data 100, 200, in this example, to a remote manager 130, 230 of data from DARS 100, 200. When a wireless data connection is not available, the data is stored in a storage device and queued in a repository 158, 258 with staging queued until the wireless connection is restored and the data replication process can resume.

[0045] In parallel with writing the data, the data writer 154, 254 continuously and offline replicates the data to the remote data repository 130, 230. The replication process has two modes, real-time and near-real-time. In real time, data is replicated to a remote repository 130, 230 data every second. In near real time, data is replicated to a remote data repository 130, 230 every five minutes. The frequency used for near real time mode is configurable, and the frequency used for real time mode can be adjusted to support high resolution data by replicating the data to the remote data repository 130, 230 every 0.10 seconds. When DARS 100, 200 is in near real time, the onboard data manager 120, 220 queues the data in the repository 158, 258, queuing before replicating the data to the remote data manager 132, 232. The onboard data manager 120, 220 also replicates the vehicle event detector information queued in the queuing repository 158, 258 to the remote data manager 132, 232. Near real-time mode is used during normal operation, under most conditions, to improve the efficiency of the data replication process.

[0046] The real-time mode may be initiated based on events generated and detected by the vehicle event detector 156, 256 on board the asset 148, 248, or by a request initiated from the data processing and storage center 150, 250. A typical real-time request initiated by the data processing and storage center 150, 250 is initiated when a remotely located user 152, 252 requests real-time information from the web client 142, 242. A typical reason that real-time originates on board the asset 148, 248, is to detect an event or incident through the vehicle event detector 156, 256, such as the driver's initiation of an emergency stop request, emergency braking activity, rapid acceleration or deceleration in any axis, or loss of power input to the means 154. 254 data records. When transitioning from near real-time mode to real-time mode, all data not already replicated to the remote data repository 130, 230 is replicated and stored in the remote data repository 130, 230, and then live replication is initiated. The transition between near real-time mode and real-time mode typically occurs in less than five seconds. After a predetermined amount of time has passed since an event or incident, after a predetermined amount of time of inactivity, or when the user 152, 252 no longer requires real-time information from the asset 148, 248, the data recorder 154, 254 returns to mode. almost real time. The predetermined amount of time required to initiate a transition is configurable and is typically set to ten minutes.

[0047] When the data recorder 154, 254 is in real time, the on-board data manager 120, 220 attempts to continuously empty its queue to the remote data manager 132, 232, storing the data in the impact-resistant storage module 118, 218 and optionally in the non-impact-resistant removable device. 219 data storage of FIG. 2 and sending data to the remote data manager 132, 232 simultaneously. The onboard data manager 120, 220 also sends detected vehicle information queued in the queuing repository 158, 258 to the remote data manager 132, 232.

[0048] Upon receiving data to be replicated from data recorder 154, 254, along with data from map component 164, 264, route/crew manifest processing component 124, 224, and weather component 126, 226, remote dispatcher 132. Data 232 stores the compressed data in a remote data repository 130, 230 at the DARS data processing and storage center 150, 250 100, 200. The remote data repository 130, 230, for example, may be a cloud data store or any other suitable remote data store. When data is received, a process is initiated that instructs the data decoder 136, 236 to decode the latest replicated data to/from the remote data repository 130, 230 and send the decoded data to the remote event detector 134, 234. The remote data manager 132, 232 stores vehicle event information in a remote data repository 130, 230. When the remote event detector 134, 234 receives the decoded data, it processes the decoded data to determine whether or not an event of interest is detected in the decoded data. The decoded information is then used by the remote event detector 134, 234 to detect events, incidents, or other predefined situations in the data occurring with the asset 148, 248. When an event of interest is detected from the decoded data, the remote event detector 134, 234 stores event information and supporting data in the remote data repository 130, 230. When the remote data manager 132, 232 receives information from the remote event detector 134, 234, the remote data manager 132, 232 stores the information in the remote data repository 130, 230.

[0049] A remotely located user 152, 252 may access information, including vehicle event detector information, associated with a specific asset 148, 248 or multiple assets using a standard web client 142, 242, such as a web a browser or virtual reality style device (not shown) which, in this implementation, can display thumbnail images from selected cameras. The web client 142, 242 transmits user requests 152, 252 for information to the web server 140, 240 via the network 144, 244 using common web standards, protocols and technologies. The network 144, 244, for example, may be the Internet. The network 144, 244 may also be a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network, or any other means of transmitting data from the web server 140 , 240 , in this example, to the web client 142 , 242 . The web server 140 , 240 requests the required data from the data decoder 136 , 236 . The data decoder 136, 236 receives the requested data associated with a specific asset 148, 248 or multiple assets from the remote data repository 130, 230 when requested from the web server 140, 240. The data decoder 136, 236 decodes the requested data and sends the decoded data to the localizer 138, 238. Localization is the process of converting data into formats required by the end user, for example, converting the data into the user's preferred language and units of measure. The localizer 138, 238 identifies profile settings specified by the user 152, 252 by accessing the web client 142, 242, and uses the profile settings to prepare information sent to the web client 142, 242 for presentation to the user 152, 252 , as raw encoded data, and the detected event information is stored in a remote data repository 130, 230 using Coordinated Universal Time (UTC) and the International System of Units (SI units). The localizer 138, 238 converts the decoded data into the format required by the user 152, 252, such as the user's preferred language and units of measurement 152, 252. The localizer 138, 238 sends the localized data in the user's preferred format 152, 252 to the web server 140, 240 by request. The web server 140, 240 then sends the localized data of the asset or multiple assets to the web client 142, 242 for viewing and analysis, providing real-time playback and display of standard and 360-degree video. The web client 142, 242 may display, and the user 152, 252 may view, data, video, and audio for a single asset or simultaneously view data, video, and audio for multiple assets. The web client 142, 242 may also provide synchronous playback and real-time display of data along with a variety of video and audio data from both standard video sources and 360-degree video sources on, at, or near an asset, nearby assets, and/or remotely located sites.

[0050] FIG. 3 is a flowchart showing a process 300 for recording data and/or information from an asset 148, 248 in accordance with an implementation of this disclosure. The data recorder 154, 254 receives data signals from various input components, which include physical or calculated data elements from the asset 148, 248 and data processing and storage center 150, 250, such as speed, latitude coordinates, longitude coordinates, horn detection, throttle position, weather data, map data or crew data (302). Data encoder 122, 222 creates a record that includes a structured sequence of bits used to configure and record data signal information (304). The encoded recording is then sent to the on-board data manager 120, 220, which sequentially combines the sequence of recordings in chronological order into blocks of recordings that include up to five minutes of data (306). An intermediate block of records includes less than five minutes of data, while a full block of records includes a full five minutes of data. Each block of records includes all the data required to fully decode the enabled signals, including performing data integrity checks. At a minimum, a block of records must begin with a start record and end with an end record.

[0051] To ensure that all encoded signal data is stored in the impact-resistant storage module 118 and optionally in the non-impact-resistant removable data storage device 219 of FIG. 2, if the data recorder 154, 254 loses power or is exposed to extreme temperatures or mechanical stress due to a collision or other catastrophic event, the flight data controller 120, 220 stores intermediate blocks of records in the shock-resistant memory module 118 at a predetermined frequency (308) and optionally in the non-impact-resistant removable data storage device 219 of FIG. 2, wherein the predetermined frequency is configurable and/or variable, as shown in FIG. 5 in an approximate representation. Intermediate blocks of records are stored at least once per second, but may also be stored as frequently as once every tenth of a second. The frequency at which intermediate blocks of records are saved depends on the sampling rates of each signal. Each intermediate block of records includes the complete set of records since the last complete block of records. The data recording means 154, 254 may alternate between two temporary storage locations in the impact-resistant storage module 118, 218 and optionally in the non-impact-resistant removable data storage device 219 of FIG. 2, when writing each intermediate block of records to prevent corruption or loss of more than one second of data when the data writer 154, 254 loses power while storing the data in the impact-resistant memory module 118, 218 or the optional non-shockproof removable data storage device 219 of the data writer 254 according to fig. 2. Each time a new intermediate record block is stored in a temporary location of the impact storage device, it shall overwrite the existing previously stored intermediate record block in that location.

[0052] Every five minutes, in this implementation, when the data recorder 154, 254 is in near real time, the on-board data manager 120, 220 stores a complete block of records, including the last five minutes of encoded signal data, in a records segment in shockproof memory module 118, 218 shown in FIG. 7, and sends a copy of the complete block of records to the remote data manager 132, 232 for storage in the remote data repository 130, 230 for a predetermined retention period, for example, two years (310). The impact-resistant memory module 118, 218 and/or the optional non-impact-resistant removable data storage device 219 of the data recording means 254 of FIG. 2 stores the recording segment of the last blocks of recordings for a prescribed retention duration, which in this implementation is the federally mandated duration during which the data recorder 154, 254 must store operational data or video data in the impact-resistant memory module 118, 218 with an additional 24- hour buffer and then overwrite.

[0053] FIG. 4 is a flowchart showing a process 400 for appending data and/or information from an asset 148, 248 after a power loss in accordance with an implementation of this disclosure. After power is restored, data writer 154, 254 identifies the last intermediate record block that is stored in one of two temporary locations of the impact memory (402) and checks the validity of the last intermediate record block using 32-bit CRC. code that is included in the final record of each block of records (404). The validated intermediate block of records is then added to the end of the record segment of the impact storage device, and the record segment, which can contain up to five minutes of data before loss of power, is sent to the remote data manager 132, 232 for storage during the retention period (406). The encoded signal data is stored in the high-impact memory module 118, 218 and/or in the optional non-high-impact removable data storage device 219 of the data recorder 254 of FIG. 2, in a ring buffer for a prescribed storage duration. Because a segment of records of an impact-resistant memory device is divided into several blocks of records, the data writer 154, 254 deletes obsolete blocks of records as necessary to free up storage space each time a complete block of records is stored in the impact-resistant memory module 118, 218 and/or in an optional non-impact-resistant removable memory module. the data storage device 219 of the data recording means 254 of FIG. 2.

[0054] FIG. 6 is a diagram that illustrates exemplary intermediate write blocks before power loss and after power is restored to data writer 154, 254. When the intermediate record block stored in temporary location 2 at (02/01/2016 10:10:08 AM) (602) is valid, this intermediate record block is added to the end of the records segment (702) (FIG. 7) in the impact storage module 118 , 218 and/or the optional non-shockproof removable data storage device 219 of the data recording means 254 of FIG. 2, as shown in FIG. 7. When the intermediate block of records stored in temporary location 2 at (02/01/2016 10:10:08 AM) is not valid, the intermediate block of records in temporary location 1 at (02/01/2016 10:10:07 AM) passes the validity check, and if valid, is added to the end of the record segment in the impact-resistant memory module 118, 218 and/or the optional non-impact-resistant removable data storage device 219 of the data recorder 254 of FIG. 2.

[0055] Whenever any block of records is to be stored in the impact-resistant memory module 118, 218 and/or the optional non-impact-resistant removable data storage device 219 of the data recording means 254 of FIG. 2, the recording segment is immediately flushed to disk. Since the data recording means 154, 254 alternates between two different temporary storage locations while storing intermediate record blocks, there is always one temporary storage location that is not modified or dumped into the impact storage device or non-impact removable storage device, thereby ensuring that that at least one of the two intermediate blocks of records stored in temporary storage locations is valid, and that the data writer 154, 254 should not lose more than one second for most of the data each time the data writer 154, 254 data is losing power. Likewise, when the data writer 154, 254 writes data to the impact resistant memory module 118, 218 and/or the optional non-impact removable data storage device 219 of the data writer 254 of FIG. 2, every tenth of a second, the data writer 154, 254 should not lose more than one tenth of a second for the majority of the data each time the data writer 154, 254 loses power.

[0056] For ease of explanation, process 300 and process 400 are illustrated and described as a series of steps. However, the steps in accordance with this disclosure may be performed in other orders and/or in parallel. Additionally, the steps in accordance with this disclosure may be performed with other steps not presented and described herein. Moreover, not all of the illustrated steps may be required in order to implement technologies in accordance with the claimed subject matter.

[0057] A third embodiment of the real-time data acquisition and recording system and viewer described herein provides real-time or near-real-time access to a wide range of data, such as event and operational data, video data, and audio data of expensive asset, remotely located users such as asset owners, operators and investigators. The data acquisition and recording system records data, through a data recorder, associated with an asset and streams the data to a remote data repository and remotely located users before, during and after an incident occurs. Data is streamed to a remote data repository in real time or near real time, making information available at least up to the time of the incident or emergency, thereby virtually eliminating the need to locate and download a "black box" to investigate an incident involving an asset, and eliminating the need to interact with the data recorder on the asset to request the download of specific data, locate and transfer files, and use a user application to view the data. The present entity discovery system retains typical recording capabilities and adds the ability to stream data to a remote data repository and to a remote end user before, during and after an incident. In the vast majority of situations, the information written to the data recorder is redundant and unnecessary because the data is already being retrieved and stored in a remote data repository.

[0058] Prior to the present entity discovery system, data is retrieved from a "black box" or "event recorder" after an incident occurs and an investigation is required. Data files containing time segments recorded by the black box must be loaded into and retrieved from the black box and then viewed by the user using the user's own software. The user must gain physical or remote access to the asset, select the required data to be downloaded from the black box, upload a file containing the required information to the computing device, and locate the corresponding file containing the required data using a user application that runs on the computing device. device. The system of the present disclosure eliminates the need for the user to perform these steps, requiring the user only to use a general web browser in order to navigate to the required data. A remotely located user can access a common web browser to navigate to the required data associated with a selected asset to view and analyze asset performance and safety in real time or near real time.

[0059] A remotely located user, such as an asset owner, machinist, and/or investigator, can access a common web browser to navigate to live and/or statistical desired data associated with a selected asset to view and analyze operational performance and real-time or near-real-time asset security. The ability to view transactions in real time or near real time allows for rapid assessment and regulation of behavior. During an incident, for example, real-time information and/or data can make it easier to triage the situation and provide valuable information to emergency responders. During normal operation, for example, real-time information and/or data may be used to audit crew performance and contribute to network-wide situational awareness.

[0060] The real-time data acquisition and recording system of the third embodiment uses at least one of or any combination of an image measurement device, a video measurement device, and a range measurement device in, on, or near the mobile asset as part of the acquisition and recording system data. Image measurement devices and/or video measurement devices include, but are not limited to, 360-degree cameras, stationary cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye cameras and/or other cameras. Ranging devices include, but are not limited to, radar and optical detection and ranging (LDAR). Lidar is a surveillance technique that measures the distance to a target by illuminating the target with pulsed laser light and measuring the reflected pulses with a sensor. Prior to the present disclosure, the "black box" and/or "event recorder" did not include 360-degree cameras or other cameras in, on, or near the mobile asset. True Disclosure System adds the ability to use and record video using 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye cameras, radar, lidar and/or other cameras as parts of the data acquisition and recording system by providing 360 degree views, narrow angle views, wide angle views, fisheye views and/or other views in, on or near the mobile asset to a remote data repository and to the remote user and investigator before, during and after an incident involving a mobile asset occurs. The ability to view operations, 360-degree videos and/or other videos in real time or near real time allows for rapid assessment and management of team behavior. Owners, operators and investigators can view and analyze operational efficiency, safety of people, vehicles and infrastructure and can investigate or inspect an incident. The ability to view 360-degree video and/or other videos from a mobile asset allows for rapid assessment and management of crew behavior. During an incident, for example, 360-degree and/or other videos can make it easier to triage the situation and provide valuable information to emergency responders and investigators. During normal operation, for example, 360-degree and/or other videos may be used to audit crew performance and promote network-wide situational awareness. 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye cameras, radar, lidar and/or other cameras provide a complete picture of situations to provide video surveillance systems for law enforcement agencies and/or railway police, inspection of critical infrastructure, monitoring of level crossings, monitoring the progress of track works, auditing the crew inside the driver's booth and in the depot and remote monitoring in real time.

[0061] Prior art systems require users to download video files containing time segments in order to view the video files using their own software application or other external video playback applications. The data acquisition and recording system of the present disclosure provides 360 degree video, other video, image information and audio information and ranging information, which can be displayed to a remote user using a virtual reality style device and/or through a standard web client, for This eliminates the need to download and use external applications in order to view videos. Additionally, remote users can view 360-degree and/or other videos in various modes using a virtual reality-style device or through a standard web client, such as a web browser, thereby eliminating the need to download and use external applications to watch videos. Previous video systems require the user to download video files containing temporary segments of data that are only viewable using native application software or other external video playback applications that the user must purchase separately.

[0062] The data may include, but is not limited to, video and image information from cameras located at various locations in, on, or near the asset, and audio information from microphones located at various locations in, on, or near the asset. A 360-degree camera is a camera that provides a 360-degree spherical field of view, a 360-degree hemispherical field of view, and/or a 360-degree fisheye field of view. Using 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye cameras, and/or other cameras in, on, or near the asset provides the ability to use and record video using 360-degree cameras 360-degree cameras, stationary cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye cameras, and/or other cameras as part of DARS, thereby making the 360-degree and/or other views available in, on, or near the asset for a remote data repository, remotely located users, and investigators before, during, and after an incident.

[0063] FIG. 8 illustrates a field implementation of a third embodiment of an exemplary real-time data acquisition and recording system (DARS) 800 in which aspects of entity discovery may be implemented. DARS 800 is a system that delivers real-time information, video information, and audio information from the data recorder 808 on the mobile asset 830 to remotely located end users through a data processing and storage center 832. The data recorder 808 is installed on the vehicle or mobile asset 830 and communicates with any number of different information sources through any combination of wired and/or wireless data links, such as a wireless gateway/router (not shown). The data recorder 808 includes a high-impact memory module 810, an on-board data manager 812, and a data encoder 814. In a fourth embodiment, the data recording means 808 may also include a non-shockproof removable data storage device (not shown). The exemplary secure memory module 810, for example, may be a crash-resistant event recorder memory module that complies with the Code of Federal Regulations and/or Federal Railroad Administration regulations, a crash survivable memory unit that complies with the Code of Federal Regulations and/or or FAA regulations, an impact resistant memory module in accordance with any applicable Code of Federal Regulations, or any other suitable secure storage device known in the art. Wired and/or wireless data links may include any one or combination of discrete signal inputs, standard or proprietary Ethernet, serial connections, and wireless connections.

[0064] Data recorder 808 collects video data, audio data, and other data and/or information from a wide range of sources, which may vary based on asset configuration, through onboard data links. In this implementation, the data recording engine 808 receives data from the video management system 804, which continuously records video data and audio data from 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye cameras , radar, lidar, and/or other cameras 802 and fixed cameras 806 that are placed in, on, or near the asset 830, and the video management system 804 stores video and audio data in a high-impact storage module 810 and may also store video and audio data in a non-shock-resistant the removable storage device of the second embodiment. Different versions of video data are created using different bit rates or spatial resolutions, and these versions are divided into variable length segments such as thumbnails, five-minute low-resolution segments, and five-minute high-resolution segments.

[0065] Data encoder 814 encodes at least a minimum set of data, which is typically specified by a regulator. The data encoder 814 receives video and audio data from the video management system 804 and compresses or encodes the data and time-synchronizes the data to facilitate efficient transfer and real-time replication to the remote data repository 820. Data encoder 814 transmits encoded data to on-board data manager 812, which then sends encoded video and audio data to remote data repository 820 via remote data manager 818 located at data center 830 in response to an on-demand request by remotely located user 834, or in response to certain operating conditions observed on board the asset 830. The onboard data manager 812 and the remote data manager 818 operate in concert to manage the data replication process. A remote data manager 818 at a data center 832 can manage the replication of data from multiple assets. Video and audio data stored in remote data repository 820 are available to web server 822 for access by remotely located user 834.

[0066] The on-board data manager 812 also sends data to a queued repository (not shown). The on-board data manager 812 monitors the video and audio data stored in the high-impact memory module 810 and the optional non-high-impact removable storage device of the second embodiment through the video management system 804, and determines whether it is in near real time or real time or No. In near real time, the on-board data manager 812 stores encoded data, including video data, audio data, and any other data or information received from the data encoder 814, and any event information, in a high-impact memory module 810 and an optional non-shock-resistant removable data storage device. of the second embodiment and in a queued repository. After five minutes of encoded data have accumulated in the queuing repository, the on-board data manager 812 stores the five minutes of encoded data to the remote data repository 820 via the remote data manager 818 at the data processing and storage center 832 via wireless data link 816 . In real time, the onboard data manager 812 stores encoded data, including video data, audio data, and any other data or information received from the data encoder 814, and any event information, to a remote data repository 820 via the remote data manager 818 at the center 832 processing and storing data via wireless data link 816. The onboard data manager 812 and the remote data manager 818 may communicate over multiple wireless communication links. The wireless data link 816, for example, may be a wireless local area network (WLAN), a wireless metropolitan area network (WMAN), a wireless wide area network (WWAN), a private wireless system, a cellular telephone network, or any other data transmission medium. from the data writer 808, in this example, to the remote data manager 818. The process of sending and retrieving video and audio data remotely from asset 830 requires a wireless data connection between asset 830 and data processing center 832 . When a wireless data connection is not available, the data is stored and queued in the high-impact memory module 810 and the optional non-high-impact removable storage device of the fourth embodiment until the wireless connection is restored. The process of retrieving video data, audio data, and any other additional data resumes as soon as the wireless connection is restored.

[0067] In parallel with writing data, data writer 808 continuously and offline replicates data to remote data repository 820. The replication process has two modes, real-time and near-real-time. In real time, data is replicated to a remote data repository 820 every second. In near real time, data is replicated to a remote 820 data repository every five minutes. The frequency used for near real time is configurable, and the frequency used for real time can be adjusted to support high resolution data by replicating the data to the remote data repository 820 every 0.10 seconds. Near real-time mode is used during normal operation, under most conditions, to improve the efficiency of the data replication process.

[0068] The real-time mode can be initiated based on events occurring on board the asset 830 or through a request initiated from the data processing and storage center 832. A typical data center 832-initiated real-time request is initiated when a remotely located user 834 requests real-time information from the web client 826. A typical reason for real-time originating on board an asset 830 is when an event is detected or an incident, such as a driver-initiated emergency stop request, emergency braking activity, rapid acceleration or deceleration in any axis, or loss of power input to the data recording means 808. When transitioning from near real-time mode to real-time mode, all data not already replicated to the remote data repository 820 is replicated and stored in the remote data repository 820, and then live replication is initiated. The transition between near real-time mode and real-time mode typically occurs in less than five seconds. After a predetermined amount of time has passed since an event or incident, after a predetermined amount of time of inactivity, or when the user 834 no longer requires the real-time information from the asset 830, the data recorder 808 returns to near real-time mode. The predetermined amount of time required to initiate a transition is configurable and is typically set to ten minutes.

[0069] When the data recorder 808 is in real time, the onboard data manager 812 attempts to continuously empty its queue to the remote data manager 818 by storing the data in the impact-resistant storage module 810 and the optional non-impact-resistant removable storage device of the second embodiment and sending the data to remote manager of 818 data simultaneously.

[0070] Upon receiving video data, audio data, and any other data or information that needs to be replicated from the data recorder 808, the remote data manager 818 stores the data in a remote data repository 820 at the data processing center 830. The remote data repository 820, for example, may be a cloud data store or any other suitable remote data store. When data is received, a process is initiated that instructs the data decoder (not shown) to decode the latest replicated data from the remote data repository 820 and send the decoded data to the remote event detector (not shown). The remote data manager 818 stores vehicle event information in a remote data repository 820 . When the remote event detector receives decoded data, it processes the decoded data to determine whether or not an event of interest is detected in the decoded data. The decoded information is then used by the remote event detector to detect events, incidents, or other predefined situations in the data occurring with the asset 830. When an event of interest is detected from the decoded data previously stored in the remote data repository 820, the remote event detector stores the information events and supporting data in a remote 820 data repository.

[0071] Video data, audio data, and any other data or information are made available to user 834 in response to a request on demand by user 834 and/or sent by on-board data manager 812 to remote data repository 820 in response to certain operating conditions observed on board asset 830 The video data, audio data, and any other data or information stored in the remote data repository 820 are available on the web server 822 for access by the user 834. The remotely located user 834 can access the video data, audio data, and any other data or information associated with a specific asset 830 or multiple assets stored in a remote data repository 820 using a standard web client 826 such as a web browser or a virtual reality style device 828 which, in this implementation, can display thumbnail images of selected cameras. Web client 826 transmits user 834's request for video, audio, and/or other information to web server 822 via network 824 using common web standards, protocols, and technologies. Network 824, for example, may be the Internet. The network 824 may also be a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network, or any other means of transporting data from the web server 822 to In this example, to the web client 826. The web server 822 requests the required data from the remote data repository 820. Web server 822 then sends the requested data to web client 826, which provides playback and real-time display of standard video, 360-degree video, and/or other video. Web client 826 renders video data, audio data, and any other data or information to user 834, who can interact with the 360-degree video data and/or other video data and/or still image data for viewing and analysis. The user 834 can also download video data, audio data, and any other data or information using the web client 826 and can then use a virtual reality style device 828 to interact with the 360 degree video data for viewing and analysis.

[0072] Web client 826 may be enhanced by a software application that provides playback of 360-degree video and/or other video in a variety of different modes. The user 834 can select the mode in which the software application presents the video playback, such as, for example, a fisheye view, as shown in FIG. 11 is a panoramic view as shown in FIG. 12, double panoramic view (not shown), square view, as shown in FIG. 13 and a view without distortion as shown in FIG. 14.

[0073] FIG. 9 is a flowchart showing a process 840 for recording video data, audio data and/or information from asset 830 in accordance with an implementation of this disclosure. The video management system 804 receives data signals from various input components (842), such as 360-degree cameras, fixed cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye cameras, radar, lidar, and/or or other cameras 802 and fixed cameras 806 on, in, or near the asset 830. The video management system 804 then stores video data, audio data, and/or information in a high-impact storage module 810 and an optional non-impact-resistant removable storage device of the fourth embodiment (844), using any combination of standardized formats, such as, for example, still images, thumbnails, still image sequences or compressed video formats. Data encoder 814 creates a record that includes a structured sequence of bits used to configure and record data signal information (846). In near real time, the video management system 804 stores video data in the high-impact memory module 810 and the optional non-impact-resistant removable storage device of the fourth embodiment, while sending only limited video data, such as thumbnails or very short low-resolution video segments, off-board to a remote repository. 820 data (848).

[0074] In another implementation, the encoded record is then sent to the on-board data manager 812, which sequentially combines the sequence of records in chronological order into blocks of records that include up to five minutes of data. An intermediate block of records includes less than five minutes of data, while a full block of records includes a full five minutes of data. Each block of records includes all the data required to fully decode the enabled signals, including performing data integrity checks. At a minimum, a block of records must begin with a start record and end with an end record.

[0075] To ensure that all encoded signal data is stored in the high-impact memory module 810 and the optional non-impact-resistant removable data storage device of the fourth embodiment, if the data writer 808 loses power, the on-board data manager 812 stores intermediate blocks of records in the high-impact memory module 810 and an optional non-shockproof removable data storage device of the fourth embodiment, with a predetermined frequency, wherein the predetermined frequency is configurable and/or variable. Intermediate blocks of records are stored at least once per second, but may also be stored as frequently as once every tenth of a second. The frequency at which intermediate blocks of records are saved depends on the sampling rates of each signal. Each intermediate block of records includes the complete set of records since the last complete block of records. The data writer 808 may alternate between two temporary storage locations in the impact storage module 810 when writing each intermediate block of records to prevent corruption or loss of more than one second of data when the data writer 808 loses power while storing data in the impact storage module 810. Each whenever a new intermediate record block is stored in a temporary location of the impact storage device, it must overwrite the existing previously stored intermediate record block in that location.

[0076] Every five minutes, in this implementation, when the data recorder 808 is in near real time, the on-board data manager 812 stores a complete block of records including the last five minutes of encoded signal data into a records segment in the impact memory module 810 and the optional non-shockproof removable data storage device of the fourth embodiment, and sends a copy of the complete block of recordings containing five minutes of video data, audio data and/or information to the remote data manager 818 for storage in the remote data repository 820 for a predetermined retention period, for example , two years. The impact-resistant memory module 810 and the optional non-impact-resistant removable data storage device of the fourth embodiment retain the record segment of the last blocks of records for a prescribed retention duration, which in this implementation is the federally mandated duration for which the data recorder 808 must retain operational data or video data in a high-impact 810 memory module with an additional 24-hour buffer, and then overwritten.

[0077] FIG. 10 is a flowchart showing a process 850 for viewing data and/or information from asset 830 through a web browser or virtual reality style device. When an event occurs, or when a remotely located authorized user 834 requests a segment of video data stored in the impact-resistant memory module 810 via the web client 826, the on-board data manager 812, depending on the event, must begin sending video data off-board in real time with the best available resolution taking into account the bandwidth of the wireless data link 816. A remotely located user 834 initiates a request for specific video and/or audio data in a specific viewing mode (852) via web client 826, which transmits the request to web server 822 via network 824. Web server 822 requests specific video and/or audio data. or audio data from remote data repository 820 and sends the requested video and/or audio data to the web client 826 (854) via the network 824. The web client 826 displays the video and/or audio data in a view mode specified by the user 834 (856). The user 834 can then download specific video and/or audio data to view on the device 828 in a virtual reality style. In another implementation, in real time, thumbnails are sent first at one-second intervals, then short segments of lower resolution video, and then short segments of higher resolution video.

[0078] For ease of explanation, process 840 and process 850 are illustrated and described as a series of steps. However, the steps in accordance with this disclosure may be performed in other orders and/or in parallel. Additionally, the steps in accordance with this disclosure may be performed with other steps not presented and described herein. Moreover, not all of the illustrated steps may be required in order to implement technologies in accordance with the claimed subject matter.

[0079] A fifth embodiment of the real-time data acquisition and recording system and video analytics system described herein provides real-time or near-real-time access to a wide range of data, such as event and operational data, video data and audio data of high value asset to remotely located users. The data acquisition and recording system records data associated with an asset and streams the data to a remote data repository and remotely located users before, during and after an incident occurs. Data is streamed to a remote data repository in real time or near real time, making information available at least up to the time of the incident or emergency, thereby virtually eliminating the need to locate and download a "black box" to investigate an incident involving an asset by streaming information to a remote data repository in real time, or near real time, and making the information available at least up to the time of the catastrophic event. DARS performs video analysis of the mobile asset's recorded video data to determine, for example, driver's booth occupancy, rut detection, and detection of objects near ruts. A remotely located user can use a general web browser to navigate and view the required data associated with the selected asset, and does not necessarily need to interact with the asset's data acquisition system to request specific data downloads, locate or transfer files, and use a custom application to view the data.

[0080] DARS provides remotely located users with access to video data and video analysis performed through a video analytics system by streaming data to a remote data repository and remotely located user before, during and after an incident, thereby eliminating the need for the user to manually download ,extract and playback video to analyze video data to determine ,occupancy of the driver's booth, whether or not a crew member or ,unauthorized personnel is present during an incident, rut detection, ,object detection near ruts, investigation, or at some other time of ,interest. Additionally, the video analytics system provides driver station occupancy status detection, rut detection, near-rut object detection, head and tail detection by processing image data and video data in real time, thereby ensuring that the correct data is always available to the user. For example, real-time image processing ensures that a locomotive designated as a tail locomotive is not operated as a lead locomotive to improve railway safety. Prior art systems provide the position of a locomotive in a train by using train formation functionality in dispatch systems. At times, dispatch system information may be out of date because the information is not updated in real time and crew personnel may change locomotives as needed.

[0081] Prior to the present disclosure, inspection teams and/or asset personnel must manually inspect track conditions, manually verify whether a vehicle is in the head or tail position, manually observe the locations of each individual object of interest, manually create a database of geographic locations of all features of interest, periodically perform manual field observations of each feature of interest to verify its location, and identify any changes in geographic location that differ from the original observation, manually update the database when features of interest change location due to renovations or additional infrastructure development over time, when the initial database is created, select and download the required data from the digital video recorder and/or data recorder and inspect the downloaded data and/or video offline and check the tracks for any obstructions, and the vehicle operator must physically inspect any obstructions and/or changing the arrow position. The present disclosure system eliminates the need for users to perform these steps, requiring the user only to use a general web browser to navigate to the required data. Asset owners and operators can automate and improve the efficiency and safety of mobile assets in real time and can actively monitor track conditions and can receive real-time warning information. The true disclosure system eliminates the need for asset owners and operators to download data from a data recorder to monitor track conditions and investigate incidents. As an active protection system, DARS can help the driver check any obstacles, send real-time emergency alerts and/or store information offline and send emergency information for remote monitoring and storage. Both current and past rut detection information and information related to object detection near ruts can be stored in a remote real-time data repository to assist the user in reviewing the information when needed. A remotely located user can access a common web browser to navigate to the required data associated with a selected asset to view and analyze asset performance and safety in real time or near real time.

[0082] The real-time data acquisition and recording system of the fifth embodiment may be used to continuously monitor objects of interest and identify in real time when they have been moved or damaged, become obstructed by foliage and/or are in disrepair, and require maintenance. DARS uses video information, image information and/or audio information to detect and identify various infrastructure objects, such as railway tracks, in video, has the ability to track the tracks as the mobile asset moves, and has the ability to create, audit and periodically update a database of objects of interest with geographical location. The real-time data acquisition and recording system of the fifth embodiment uses at least one of or any combination of an image measurement device, a video measurement device, and a range measurement device in, on, or near the mobile asset as part of the data acquisition and recording system. Image measurement devices and/or video measurement devices include, but are not limited to, 360-degree cameras, stationary cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye cameras and/or other cameras. Ranging devices include, but are not limited to, radar and optical detection and ranging (LDAR). Lidar is a surveillance technique that measures the distance to a target by illuminating the target with pulsed laser light and measuring the reflected pulses with a sensor.

[0083] DARS can automatically inspect the condition of the track, for example, counting the number of tracks present, identifying the current track on which the mobile asset is moving, and detecting any obstacles or defects present, such as washed out ballast, broken tracks, out of line tracks , misaligned arrows, overrunning arrows, flooded ruts, snow accumulation, etc., and plan preventative maintenance to avoid catastrophic events. DARS can also detect railway track switches and track track changes. DARS can further detect a change in data location, including whether an object is missing, occluded, and/or not present at the expected location or not. Track detection, infrastructure diagnostic information and/or infrastructure monitoring information can be displayed to the user using any standard web client such as a web browser, thereby eliminating the need to download files from the data recorder and use proprietary application software or other external applications to view information as required by prior art systems. This process can be extended to automatically create, audit and/or update a database of geographic locations of interest and ensure compliance with Federal regulations. Through a true disclosure system, cameras pre-installed to comply with federal regulations are used to perform a variety of tasks that previously required human interaction, specialized vehicles and/or alternative equipment. DARS provides the ability to perform these tasks automatically as the mobile asset moves through an area, as part of normal commercial operation and daily operations. DARS can be used to save countless man-hours of manual labor by using normal vehicle operation and pre-installed cameras to perform tasks that previously required manual effort. DARS can also perform tasks that were previously performed using specialized vehicles, preventing the closure of track segments in order to inspect and locate tracks and objects of interest, which often results in lost commercial operations and the need to purchase and maintain expensive equipment. DARS further reduces the amount of time people must remain in close proximity to railroad tracks, resulting in fewer overall accidents and possible fatalities.

[0084] The data may include, but is not limited to, measured analog and frequency parameters such as speed, pressure, temperature, current, voltage and acceleration that originate from mobile assets and/or nearby mobile assets; measured Boolean data such as pointer positions, actuator positions, warning lights, and actuator control commands; position, speed and altitude information from global positioning system (GPS) data and additional data from geographic information system (GIS) such as latitude and longitude of various points of interest; internally generated information, such as the regulatory speed limit for a mobile asset taking into account its current position; train traffic control system status and performance data generated by systems such as Precision Train Control (PTC); vehicle parameters and inertial parameters such as speed, acceleration and location, for example, received from GPS; GIS data such as latitude and longitude of various points of interest; video and image information from at least one camera located at various locations in, on or near the mobile asset; audio information from at least one microphone located at various locations in, on or near the mobile asset; information regarding the operational plan for the mobile asset that is sent to the mobile asset from the data center, such as routing, schedule and freight manifest information; information regarding environmental conditions, such as the current and forecasted weather of the area in which the mobile asset is currently operating or planned to operate; and data derived from a combination of any of the above sources, including additional data, video and audio analysis and analytics.

[0085] "Track" may include, but is not limited to, the rails and sleepers of railroads used to transport a locomotive and/or train. “Points of interest” may include, but are not limited to, various infrastructure assets installed and maintained in the immediate vicinity of railroad tracks that may be identified using artificial intelligence, such as supervised learning or reinforcement learning, for camera-captured images and videos assets. Supervised learning and/or reinforcement learning uses pre-labeled datasets specified as “training” data to enable the remote and autonomous identification of objects within a camera view in, on, or near a mobile asset. Supervised learning and/or reinforcement learning trains neural network models to identify patterns that occur in visual images obtained from cameras. These patterns, such as people, crossing barriers, cars, trees, signals, arrows, etc., are contained only in single images. Consecutive frames in a video can also be analyzed for patterns, such as flashing lights, moving cars, people falling asleep, etc. DARS may or may not require human interaction at any stage of implementation, including, but not limited to, labeling training datasets required for supervised learning and/or reinforcement learning. Objects of interest include, but are not limited to, tracks, track centerline points, waymarks, signals, crossing barriers, arrows, level crossings, and text signs. “Video analytics” means any understandable information collected through the analysis of video and/or images recorded from image, video and/or ranging devices, such as at least one camera, for example, 360-degree cameras, stationary cameras, narrow-angle cameras, wide-angle cameras, 360-degree fisheye cameras, radar, lidar and/or other cameras in, on or near the mobile asset, such as, but not limited to, objects of interest, geographic locations of objects , obstacles on the track, distances between objects of interest and the mobile asset, incorrect alignment of the track, etc. A video analytics system can also be used in any mobile asset, residential area, space or facility containing a surveillance camera to improve video surveillance. In mobile assets, a video analytics system cost-effectively and efficiently provides autonomous event detection in a busy driver's booth for remotely located users.

[0086] FIG. 15 illustrates a field implementation of a fifth embodiment of an exemplary real-time data acquisition and recording system (DARS) 900 in which aspects of entity discovery may be implemented. DARS 900 is a system that delivers real-time information, video information, and audio information from a data recorder 902 on a mobile asset 964 to remotely located end users 968 through a data processing and storage center 966. The data recorder 902 is installed on the vehicle or mobile asset 964 and communicates with any number of different information sources via any combination of wired and/or wireless data links 942, such as a wireless gateway/router (not shown). Data recorder 902 collects video data, audio data, and other data or information from a wide range of sources, which may vary based on asset configuration, through on-board data communications links 942. The data recorder 902 includes a local storage component such as a high-impact memory module 904, an on-board data manager 906, and a data encoder 908 in the asset 964. In a sixth embodiment, the data recorder 902 may also include a non-impact removable data storage device (not shown). The exemplary secure memory module 904, for example, may be a crash-resistant event recorder memory module that complies with the Code of Federal Regulations and/or Federal Railroad Administration regulations, a crash survivable memory unit that complies with the Code of Federal Regulations and/or or FAA regulations, an impact resistant memory module in accordance with any applicable Code of Federal Regulations, or any other suitable secure storage device known in the art. Wired and/or wireless data links may include any one or combination of discrete signal inputs, standard or proprietary Ethernet, serial connections, and wireless connections.

[0087] DARS 900 further includes a video analytics system 910, which includes a track and/or object detection and infrastructure monitoring component 914. The rut detection and infrastructure monitoring component 914 includes a supervised learning and/or reinforcement learning or other neural network or artificial intelligence component 924, an object detection and location component 926, and an obstacle detection component 928 that detects obstacles present on or near ruts, and/or obstructions to the cameras, such as personnel blocking the cameras' view. In this implementation, live video data is captured by at least one camera 940 mounted in the driver's shack of asset 964, on or near asset 964. Cameras 940 are positioned at the appropriate height and angle to capture video data in and around asset 964 and obtain a sufficient quantity of the species for subsequent processing. Live video and image data is captured in front of and/or around the asset 964 via cameras 940 and fed to the rut and/or object detection and infrastructure monitoring component 914 for analysis. The track detection and infrastructure monitoring component 914 of the video analytics system 910 processes live video and image data on a frame-by-frame basis to detect the presence of railroad tracks and any objects of interest. Either the camera position parameters, such as height, angle, offset, focal length, and field of view, may be provided to the track and/or object detection and infrastructure monitoring component 914, or the camera 940 may be configured to allow the video analytics system 910 to detect and determine the position and camera settings.

[0088] To perform condition determination, for example, detecting the occupancy of a driver's booth, the video analytics system 910 uses a supervised learning and/or reinforcement learning component 924 and/or other artificial intelligence and learning algorithms to evaluate, for example, video data from cameras 940, asset data 934 such as speed, GPS and inertial sensor data, weather component data 936 and route/crew processing component 938 data, manifests and GIS data. Driver shack occupancy detection is inherently susceptible to environmental noise sources such as light reflecting off clouds and sunlight passing through buildings and trees while the asset is moving. To process environmental noise, supervised learning and/or reinforcement learning component 924 data, object detection and locating component 926, obstacle detection component, asset component 934, which may include speed, GPS, and inertial sensor data, data weather component 936 and other learning algorithms are put together to form an internal and/or external state determination comprising the mobile asset 964. The rut and/or object detection and infrastructure monitoring component 914 may also include a facial recognition system adapted from the ability to allow authorization of access to the locomotive as part of the locomotive's security system, a fatigue detection component adapted to monitor the crew's attention span, and an activity detection component to detect unauthorized activities such as smoking.

[0089] Additionally, video analytics system 910 may receive location information, including latitude and longitude coordinates of a signal, such as a stop signal, traffic light, speed limit signal, and/or rut signal, from the asset owner. The video analytics system 910 then determines whether or not the location information received from the asset owner is correct. If the location information is correct, the video analytics system 910 stores the information and does not have to re-check the location information again for a predetermined amount of time, for example, check the location information on a monthly basis. If the location information is not correct, the video analytics system 910 determines the correct location information and reports the correct location information to asset owners, stores the location information, and does not have to re-check the location information again for a predetermined amount of time, for example, check the location information on a monthly basis. . Storing location information allows for easier detection of signals such as stop lights, traffic lights, speed limits, and/or rut signals.

[0090] Supervised learning and/or reinforcement learning, using the supervised learning and/or reinforcement learning component 924, for tracks is performed by using various information obtained from successive frames of video and/or images, as well as using additional information, received from the data processing and storage center 966 and the vehicle data processing component 934, which includes inertial sensor data and GPS data to determine the training data. Object detection and location component 926 uses training data received from supervised learning and/or reinforcement learning component 924 and specific information regarding the mobile asset 964 and the railroad, such as track widths and curvatures, sleeper positioning, and vehicle speed, to distinguish railway tracks, signs, signals, etc. from other objects in order to determine object detection data. Obstacle detection component 928 uses object detection data received from object detection and locating component 926, such as information regarding obstacles present on or near tracks, and/or obstacles to cameras, such as personnel blocking the view of cameras, and additional information from a weather component 936, a route/team manifest and GIS data processing component 938, and a vehicle data processing component 934 that includes inertial sensor data and GPS data to improve accuracy and determine obstacle detection data. The mobile asset data from the vehicle processing component 934 includes, but is not limited to, speed, location, acceleration, yaw rate/pitch rate, and railroad crossings. Any additional information received and used from the data processing and storage center 966 includes, but is not limited to, 24/7 information and the geographic location of the mobile asset 964.

[0091] The infrastructure objects of interest, information processed by the track and/or object detection and infrastructure monitoring component 914, and diagnostic and monitoring information are sent to the data encoder 908 of the data recording means 902 via onboard data communication lines 942 to encode the data. Data recording means 902 stores encoded data in a high-impact memory module 904 and optionally in an optional non-impact-resistant removable storage device and sends the encoded information to a remote data manager 946 at a data processing and storage center 966 via a wireless data link 944. The remote data manager 946 stores the encoded data in a remote data repository 948 at the data center 966 .

[0092] To determine obstacle detection 928 or object detection 926, for example, the presence of a rut in front of an asset, objects on and/or near the ruts, obstacles on or near the ruts, and/or obstructions blocking the view of the cameras 964, vehicle analytics system 910 uses supervised learning and/or reinforcement learning or other artificial intelligence component 924, object detection and locating component 926, and hazard detection component 928 and other image processing algorithms to process and evaluate image data and camera video data from cameras 940 in real-time time. The rut and/or asset detection and infrastructure monitoring component 914 uses processed video data along with asset component 934 data, which may include speed, GPS and inertial sensor data, weather component 936 data, and route/crew processing component 938 data. and GIS data to determine external state definitions, such as head and tail mobile assets, in real time. When processing image data and video data to detect ruts and/or objects, for example, the video analytics system 910 automatically configures the parameters of the cameras 940 necessary to detect the ruts, detects passage through switches, counts the number of ruts, detects any additional ruts along the side of the asset 964, determines the rut , on which the asset 964 is currently traveling, detects defects in the track geometry, detects track erosion scenarios, for example, detection of water near the track within the specified track limits, and detects missing ramp or track scenarios. The accuracy of object detection depends on the existing lighting conditions in and around the asset 964. DARS 900 must handle varying lighting conditions by using additional data collected on board the asset 964 and at the data processing center 966. DARS 900 is enhanced with the ability to operate in different lighting conditions, operate in different weather conditions, detect more objects of interest, integrate with existing database systems to create, audit and update data automatically, detect multiple tracks, work consistently with curved tracks, detect any obstructions, detect any track defect that may cause safety issues, and work in low-cost embedded systems.

[0093] Determining internal and/or external status from the video analytics system 910, such as driver's station occupancy, detecting and locating objects, such as rut detection and detecting objects near ruts, and detecting obstacles, such as obstacles on or near ruts and obstacles blocking the cameras are provided to the data recorder 902, along with any data from the fleet management system (VMS) or digital video recorder component 932, via on-board data communication lines 942. The data recording means 902 stores the determination of the internal and/or external state, the data of the object detection and locating component 926, and the data of the obstacle detection component 928 in an impact-resistant memory module 904 and optionally in a non-impact-resistant removable data storage device of the second embodiment and a remote data repository 948 via a remote a data manager 946 located at a data processing and storage center 966. Web server 958 provides internal and/or external state determination, object detection and locating component 926 information, and obstacle detection component 928 information to remotely located user 968 via web client 962 upon request.

[0094] Data encoder 908 encodes at least a minimum set of data, which is typically specified by the regulator. Data encoder 908 receives video data, image data, and audio data from any of cameras 940, video analytics system 910, and video management system 932 and compresses or encodes and time synchronizes the data to facilitate efficient transfer and real-time replication to remote data repository 948 . The data encoder 908 transmits the encoded data to the on-board data manager 906, which then sends the encoded video data, image data, and audio data to the remote data repository 948 via the remote data manager 946 located at the data processing and storage center 966, in response to an on-demand request by the user 968 or in response to certain operating conditions observed on board the asset 964. The onboard data manager 906 and the remote data manager 946 operate in concert to manage the data replication process. A remote data manager 946 at a data center 966 may manage the replication of data from multiple assets 964 .

[0095] The on-board data manager 908 determines whether a detected event, internal and/or external state determination, object detection and locating, and/or obstacle detection should be queued or sent immediately, based on the prioritization of the detected event. For example, in a normal operating situation, detecting an obstruction on a track is much more urgent than detecting whether or not someone is in the driver's box of asset 964. Onboard data manager 908 also sends data to a queuing repository (not shown). In near real time, the on-board data manager stores encoded data received from data encoder 908 and any event information in a high-impact storage module 904 and in a queuing repository. After five minutes of encoded data have accumulated in the queuing repository, the onboard data manager 906 stores the five minutes of encoded data to the remote data repository 948 via the remote data manager 946 at the data processing and storage center 966 over the wireless data link 944 . In real time, the onboard data manager 908 stores the encoded data received from the data encoder 908 and any event information into a high-impact storage module 904 and into a remote data repository 948 via a remote data manager 946 at the data processing and storage center 966 over a wireless link 944 communications for data transmission.

[0096] In this implementation, the on-board data manager 906 sends video data, audio data, internal and/or external state determinations, object detection and location information, obstacle detection information, and any other data or event information to the remote data repository 948 via the remote controller 946 data in the data processing and storage center 966 via a wireless data link 944. The wireless data link 944, for example, may be a wireless local area network (WLAN), a wireless metropolitan area network (WMAN), a wireless wide area network (WWAN), a wireless virtual private network (WVPN), a cellular telephone network, or any another means for transmitting data from data writer 902, in this example, to remote data manager 946. The process of retrieving data remotely from asset 964 requires a wireless connection between asset 964 and data center 966 . When a wireless data connection is not available, the data is stored and queued until the wireless connection is restored.

[0097] In parallel with writing data, data writer 902 continuously and offline replicates data to remote data repository 948. The replication process has two modes, real-time and near-real-time. In real time, data is replicated to a remote data repository 10 times every second. In near real time, data is replicated to a remote data repository 15 times every five minutes. The frequency used for near real-time mode is configurable, and the frequency used for real-time mode can be adjusted to support high resolution data by replicating data to a remote data repository 15 every 0.10 seconds. Near real-time mode is used during normal operation, under most conditions, to improve the efficiency of the data replication process.

[0098] The real-time mode may be initiated based on events occurring on board the asset 964, or through a request initiated from the data processing center 966. A typical data center 966-initiated real-time request is initiated when a remotely located user 968 requests real-time information from a web client 962. A typical reason for real-time originating on board an asset 964 is when an event is detected or an incident involving an asset 964, such as a driver initiation of an emergency stop request, emergency braking activity, rapid acceleration or deceleration in any axis, or loss of input power to the data recording means 902. When transitioning from near real-time mode to real-time mode, all data not already replicated to the remote data repository 948 is replicated and stored in the remote data repository 948, and then live replication is initiated. The transition between near real-time mode and real-time mode typically occurs in less than five seconds. After a predetermined amount of time has passed since an event or incident, after a predetermined amount of time of inactivity, or when the user 968 no longer requires the real-time information from the asset 964, the data recorder 902 returns to near real-time mode. The predetermined amount of time required to initiate a transition is configurable and is typically set to ten minutes.

[0099] When the data recorder 902 is in real time, the on-board data manager 906 attempts to continuously empty its queue to the remote data manager 946, storing the data in the impact-resistant storage module 940 and optionally in the optional non-impact-resistant removable storage device of the sixth embodiment and sending data to the remote data manager 946 simultaneously.

[00100] Upon receiving video data, audio data, internal and/or external state determination, object detection and location information, obstacle detection information, and any other data or information that needs to be replicated from the data recorder 902, the remote data manager 946 stores the data that it receives from the on-board data manager 906, such as encoded data and detected event data, into a remote data repository 948 at the data processing and storage center 966. The remote data repository 948, for example, may be a cloud data store or any other suitable remote data store. When data is received, a process is initiated that instructs data decoder 954 to decode the latest replicated data from remote data repository 948 and send the decoded data to track/object detection/location information component 950, which checks the stored data for additional "post-processed" events. Track/object detection/locating information component 950 includes an object/obstacle detection component for determining internal and/or external state determinations, object detection and locating information, and obstacle detection information, in this implementation. When internal and/or external information, object detection and locating information, and/or obstacle detection information are detected, track/object detection/locating information component 950 stores the information in remote data repository 948 .

[00101] The remotely located user 968 can access video data, audio data, internal and/or external state determinations, object detection and location information, obstacle detection information, and any other information stored in the remote data repository 948, including track information. , asset information, and driver's booth occupancy information associated with a specific asset 964 or multiple assets using a standard web client 962 such as a web browser or virtual reality style device (not shown), which, in this implementation, can Display thumbnail images of selected cameras. Web client 962 transmits user 968's request for information to web server 958 via network 960 using common web standards, protocols, and technologies. Network 960, for example, may be the Internet. The network 960 may also be a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network, or any other means of transmitting data from the web server 958, to in this example, to the web client 962. The web server 958 requests the required data from the remote data repository 948, and the data decoder 954 receives the requested data associated with a particular asset 964 from the remote data repository 948 when requested from the web server 958. The decoder Data 954 decodes the requested data and sends the decoded data to localizer 956. Localizer 956 identifies profile settings specified by user 968 by accessing web client 962, and uses the profile settings to prepare information sent to web client 962 for presented to the user 968 as raw encoded data, and the detected track/object detection/location information is stored in a remote data repository 948 using Coordinated Universal Time (UTC) and the International System of Units (SI units). The localizer 956 converts the decoded data into the format required by the user 968, such as the preferred unit and language of the user 968. The localizer 956 sends the localized data in the user 968's preferred format to the web server 958 upon request. Web server 958 then sends localized data to web client 962 for viewing and analysis, providing real-time playback and display of standard and 360-degree video, along with internal and/or external state determination, detection and location information objects and obstacle detection information such as rut and/or object detection (FIG. 16A), rut and switch detection (FIG. 16B), rut and/or object detection, rut counting and signal detection (FIG. 16C), detection crossings and tracks and/or objects (Fig. 16D), detection of double air signals (Fig. 16E), detection of multiple tracks and/or multiple objects (Fig. 16F), detection of switches and tracks and/or objects (Fig. 16G) and arrow detection (FIG. 16H).

[00102] Web client 962 is enhanced by a software application that provides playback of 360-degree video and/or other video in a variety of different modes. The user 968 can select the mode in which the software application presents the video playback, such as, for example, fisheye view, undistorted view, panoramic view, dual panorama view, and square view.

[00103] FIG. 17 is a flowchart showing a process 970 for determining the internal state of an asset 964 in accordance with an implementation of this disclosure. The video analytics system 910 receives data signals from various input components (972), such as cameras 940, including, but not limited to, 360-degree cameras, stationary cameras, narrow-angle cameras, wide-angle cameras, fish-eye cameras with 360-degree view, radar, lidar, and/or other cameras, on, in, or near asset 964, vehicle data processing component 934, weather component 936, and route/manifestation and GIS data processing component 938. The video analytics system 910 processes the data signals using a supervised learning and/or reinforcement learning component (974) and determines an internal state (976), such as whether the driver's booth is occupied.

[00104] FIG. 18 is a flowchart showing a process 980 for detecting/locating objects and detecting obstacles occurring externally and internally to asset 964 in accordance with an implementation of this disclosure. The video analytics system 910 receives data signals from various input components (982), such as cameras 940, including, but not limited to, 360-degree cameras, stationary cameras, narrow-angle cameras, wide-angle cameras, fish-eye cameras with 360-degree view, radar, lidar, and/or other cameras, on, in, or near asset 964, vehicle data processing component 934, weather component 936, and route/manifestation and GIS data processing component 938. The video analytics system 910 processes the data signals using a supervised and/or reinforcement learning component 924, an object detection/location component 926, and an obstacle detection component 928 984 and determines obstacle detection (986) and object detection and location (988), such like the presence of a rut.

[00105] For ease of explanation, process 970 and process 980 are illustrated and described as a series of steps. However, the steps in accordance with this disclosure may be performed in other orders and/or in parallel. Additionally, the steps in accordance with this disclosure may be performed with other steps not presented and described herein. Moreover, not all of the illustrated steps may be required in order to implement technologies in accordance with the claimed subject matter.

[00106] A seventh embodiment of the real-time data acquisition and recording system and automated signal compliance and alarm monitoring system described herein provides real-time or near real-time access to a wide range of data, such as event and operational data. data, video data and audio data associated with a high-value asset to remotely located users such as asset owners, machinists and investigators. The Automated Signal Compliance Monitoring and Alert System records data through a data recorder associated with an asset and streams the data to a remote data repository and remotely located users before, during and after an incident occurs. Data is streamed to a remote data repository in real time or near real time, making information available at least up to the time of the incident or emergency, thereby virtually eliminating the need to locate and download a "black box" to investigate an incident that contains an asset, and eliminates the need to interact with the data recorder in the asset to request the download of specific data, locate and transfer files, and use a user application to view the data. The present entity discovery system retains the typical recording capability and adds the ability to stream data to a remote data repository and to a remote end user before, during and after an incident. In the vast majority of situations, the information written to the data recorder is redundant and unnecessary because the data is already being retrieved and stored in a remote data repository.

[00107] The Automated Signal Monitoring and Alert System also automatically monitors and provides statistical and real-time alarm alerts to mobile assets such as locomotives, trains, aircraft and automobiles when a signal such as a brake light is violated. traffic signal and/or speed limit signal, or when the mobile asset is operating unsafely in an attempt to maintain signal compliance, such as a stop light, traffic signal, and/or speed limit signal. The Automated Signal Monitoring and Alert System combines the use of image analytics, GPS location, braking forces and rail car speeds, and automated electronic notifications to alert personnel on board and/or offboard a mobile asset in real time when the mobile asset is in violation. safe operating rules, for example, when a stop signal is transmitted by a mobile asset prior to stopping and being authorized (red light violation), when a limit signal indicating reduced speed limits is violated by a mobile asset traveling at a higher speed, and when a mobile asset Applies late and/or excessive braking force to stop before passing a stop/red light.

[00108] Prior to the automated alarm monitoring and alert system of the present disclosure, control center personnel relied on mobile asset crews to report when safe operating rules were violated. Sometimes the result is a catastrophic head-on collision of mobile assets and subsequent investigations are conducted to understand whether safe operating procedures have been violated. Additionally, excessive braking forces may cause mechanical failure of a portion of the mobile asset, and in situations where the mobile asset is a locomotive and/or train, excessive braking forces may result in a derailment, with subsequent investigations finding failure to operate safely as the root cause. . The present disclosure system allows users to monitor and be alerted when a violation of safe operating procedures occurs before a mechanical failure, collision, crash and/or other emergency occurs.

[00109] The end user can subscribe to an emergency alert when a safety violation occurs and must receive email, text message, and/or browser email notifications within minutes of the actual event occurring. The end user can use statistical records to analyze data to identify patterns, such as problematic locations, broken line of sight, faulty equipment and unqualified crews, which can be helpful in implementing new and safer operating procedures. or training and educational programs of teams for continuous improvement. The present disclosure's system enables the end user to leverage continuous electronic monitoring and extensive image analytics to understand any and all instances where a mobile asset is operating unsafely due to safety violations and/or signal failures.

[00110] The automated alarm monitoring and alert system is used by railcar and/or mobile asset owners, operators and investigators to view and analyze the operational efficiency and safety of mobile assets in real time. The ability to view transactions in real time allows for rapid assessment and regulation of behavior. During an incident, real-time information can make it easier to triage the situation and provide valuable information to emergency responders. During normal operation, real-time information can be used to audit crew performance and improve safety and competency throughout the network.

[00111] An automated signal monitoring and alert system uses outward-facing cameras and/or other cameras, GPS location, speed and acceleration, and brake pressure sensor data from cars, trains, and/or mobile assets in a fully integrated automated system with time synchronization to identify unsafe and potentially catastrophic operating practices to provide real-time feedback to mobile asset crews and management. The Automated Alarm Monitoring and Alert System also provides automated data and video downloads to users through various data sources to provide complete visibility into the operating environment during an emergency alert.

[00112] The data may include, but is not limited to, analog and digital parameters such as speed, pressure, temperature, current, voltage and acceleration that originate from the asset and/or nearby assets, Boolean data such as arrow positions, actuator position, warning lights and actuator control commands, global positioning system (GPS) data and/or geographic information system (GIS) data such as position, speed and altitude, internally generated information such as legal speed limit for asset based on its current position, video and image information from cameras located at various locations in, on or near the asset, audio information from microphones located at various locations in, on or near the asset, information regarding the operational plan for the asset that is being sent to asset from a data center such as route, schedule and freight manifest information, information regarding environmental conditions including current and forecast weather conditions for the area in which the asset is currently operating or planned to operate, status data and operating asset management data generated through systems such as Precision Train Control (PTC) systems in locomotives, and data derived from a combination of any of the above, including, but not limited to, additional data, video and audio analysis and analytics.

[00113] FIG. 19 illustrates a field implementation of a seventh embodiment of an exemplary real-time data acquisition and recording system (DARS) 1000 and automated alarm monitoring and alert system 1080, in which aspects of entity discovery may be implemented. DARS 1000 is a system that delivers real-time information to remotely located end users from a data recorder. DARS 1000 includes a data recorder 1054 that is installed on the car or mobile asset 1048 and communicates with any number of different information sources through any combination of onboard wired and/or wireless data links 1070, such as a wireless gateway/router or off-board information sources via the DARS data processing and storage center 1050 1000 via data links, such as wireless data links 1046. The data writer 1054 includes an on-board data manager 1020, a data encoder 1022, a car event detector 1056, a queuing repository 1058, and a wireless gateway/router 1072. Additionally, in this implementation, the data writer 1054 may include a crash-resistant storage device. 1018 module and/or 1062 Ethernet switch with or without Power over Ethernet (POE). The exemplary secure memory module 1018 may, for example, be a crash-resistant event recorder memory module that complies with CFR and/or Federal Railroad Administration regulations, a crash survivable memory unit that complies with CFR requirements, and /or FAA regulations, a crash-resistant storage module in accordance with any applicable Code of Federal Regulations, or any other suitable rugged storage device known in the art. In an eighth embodiment, the data recording means may further include an optional non-crash-proof removable data storage device.

[00114] Wired and/or wireless data links 1070 may include any one or combination of discrete signal inputs, standard or proprietary Ethernet, serial connections, and wireless connections. Ethernet-connected devices may use an Ethernet switch 1062 data recorder 1054 and may use POE. The 1062 Ethernet switch can be internal or external and can support POE. Additionally, data from remote data sources such as the map component 1064, the route/team manifest processing component 1024, and the weather component 1026 in the implementation of FIG. 19 are accessible to onboard data manager 1020 and car event detector 1056 from data processing and storage center 1050 via wireless data link 1046 and wireless gateway/router 1072.

[00115] Data recorder 1054 collects data or information from a wide range of sources, which may vary significantly based on asset configuration, through on-board data link 1070. Data encoder 1022 encodes at least a minimum set of data, which is typically specified by the regulator. In this implementation, data encoder 1022 receives data from a wide range of sources in asset 1048 and sources in data processing and storage center 1050. Sources of information may include any number of components in asset 1048, for example, any of analog inputs 1002, digital inputs 1004, I/O module 1006, car controller 1008, engine controller 1010, inertial sensors 1012, global positioning system (GPS) 1014 ), cameras 1016, precision train control (PTC) signal data 1066, fuel system data 1068, cellular detectors (not shown), internal data and any additional data signals, and any number of components in the processing and storage center 1050 data, for example, any of the route/team manifest processing component 1024, the weather component 1026, the mapping component 1064, and any additional data signals. In addition, information sources in asset 1048 may be coupled to data recorder 1054 via any combination of wired or wireless data links 1070. The data encoder 1022 compresses or encodes the data and time-synchronizes the data to facilitate efficient transfer and real-time replication to the remote data repository 1030. The data encoder 1022 transmits the encoded data to the on-board data manager 1020, which then stores the encoded data in a crash-proof storage module 1018 and a repository 1058 and is queued for replication to a remote data repository 1030 via a remote data manager 1032 located at the processing center 1050 and data storage. Optionally, the on-board data manager 1020 may store a tertiary copy of the encoded data in the non-crash-proof removable storage device of the eighth embodiment. The onboard data manager 1020 and the remote data manager 1032 work in concert to manage the data replication process. A single remote data manager 1032 at a data center 1050 can manage data replication from multiple assets 1048 .

[00116] Data from various input components and data from the audio/graphical user interface (GUI) 1060 in the driver's station are sent to the car event detector 1056. Car event detector 1056 processes data to determine whether or not an event, incident, or other predetermined situation involving asset 1048 occurs. When car event detector 1056 detects signals that indicate that a predetermined event has occurred, car event detector 1056 sends processed data that a predetermined event occurs, along with confirmation data accompanying the predetermined event, to on-board data manager 1020. Car event detector 1056 detects events based on data from a wide variety of sources, such as analog inputs 1002, digital inputs 1004, I/O module 1006, car controller 1008, engine controller 1010, inertial sensors 1012, GPS 1014, cameras 1016, component 1024 route/crew declaration processing, weather component 1026, mapping component 1064, PTC signaling data 1066, and fuel system data 1068, which may vary based on asset configuration. When the car event detector 1056 detects an event, the detected asset event information is stored in a queued repository 1058 and may optionally be presented to the asset crew 1048 via an audio/graphical user interface (GUI) 1060 in the driver's booth.

[00117] When the location of asset 1048 indicates that signal 1082 is crossed, excessive braking occurs, and asset 1048 is stopped very close to signal 1082, or speed limits are applied via signal reading, on-board data manager 1020 must initiate image analysis of outward-facing cameras. to determine the meaning or indication of signal 1082, as shown in FIG. 20. Using state-of-the-art image processing technologies, video footage from outward-facing cameras can be analyzed by a pre-trained neural network or artificial intelligence component to decipher the signal reading and the consequences of operating rules. Analysis and/or processing by a neural network or artificial intelligence component, in this example implementation, is performed in the back office. In another embodiment, analysis and/or processing by a neural network or artificial intelligence component is performed on the asset 1048. The signal decoding output is combined with other sensor data to determine whether the asset 1048 is seriously violating the traffic signal indicator by occupancy of the railroad. tracks, in this exemplary implementation, which may result in head-on collisions between trains, or operate in an unsafe manner in order to achieve signal compliance. When an asset 1048 is found to be non-compliant, an electronic alert must be stored in the back office and also delivered to users who sign up to receive such alerts after associating railroad business rules with signal and asset operations. These alarm alerts can then be retrieved either directly through the database or through the use of a web site graphical user interface or web client 1042 provided to users.

[00118] Additionally, an audible emergency alert may be added to the operator's booth 1048, which should provide an emergency alert to the crew regarding an impending signal violation, an impending bad situation that the crew can respond to more quickly in the event that the crew is distracted or otherwise unable to pays attention to an obstruction in the rut, a brake light, and/or if the asset 1048 accelerates in an area in which the signal requires a lower speed limit.

[00119] Automated signal monitoring and alert system 1080 is also enhanced with the ability to automatically perform video analytics to determine the meaning of a signal each time a monitored asset crosses a signal, with the ability to automatically perform video analytics to determine the meaning of a signal each time an asset is crossed excessive braking forces and stops within a predetermined distance, and with the ability to monitor the speed of the asset to determine whether or not the asset is traveling above the authorized speed, as determined by a signal reading. Image analytics are performed on board the asset 1048 to reduce the delay between the actual event and electronic notification to users and/or subscribers. The functionality of the automated alarm monitoring and alert system 1080 is enhanced with the ability to provide automated downloads of inward and outward facing video during an emergency alert to improve the user experience and reduce the amount of work required to investigate an event. The functionality of the automated alarm monitoring and alert system 1080 is also enhanced with the ability to provide real-time audible alarm cues on the non-compliant asset 1048 to alert the crew in the event of a distraction or other cause of deviation from following safe operating practices regarding the application and meaning of the alarms.

[00120] Additionally, the automated signal monitoring and emergency alert system 1080 and/or the video analytics system 910 may receive location information including the latitude and longitude coordinates of a signal such as a stop light, a traffic light, a speed limit signal, and/or signal of an object near the tracks, from the asset owner. The video analytics system 910 then determines whether or not the location information received from the asset owner is correct. If the location information is correct, the video analytics system 910 stores the information and does not have to re-check the location information again for a predetermined amount of time, for example, checking the location information on a monthly basis. If the location information is not correct, the video analytics system 910 determines the correct location information and reports the correct location information to asset owners, stores the location information, and does not have to re-check the location information again for a predetermined amount of time, for example, check the location information on a monthly basis. . Storing location information allows for easier detection of signals such as stop lights, traffic lights, speed limits, and/or rut signals.

[00121] The onboard data manager 1020 also sends data to the repository 1058 for queuing. In near real time, the onboard data manager 1020 stores the encoded data received from the data encoder 1022 and any event information in a crash-proof storage module 1018 and a queued repository 1058. In an eighth embodiment, the on-board data manager 1020 may optionally store the encoded data in a non-crash-proof removable storage device. After five minutes of encoded data have accumulated in the queuing repository 1058, the onboard data manager 1020 stores the five minutes of encoded data in the remote data repository 1030 via the remote data manager 1032 at the data processing and storage center 1050 over the wireless communication link 1046 for transmission. data available through the wireless gateway/router 1072. In real time, the on-board data manager 1020 stores the encoded data received from the data encoder 1022 and any event information in the crash-proof storage module 1018 and optionally in a non-crash-proof removable device data storage and in a remote data repository 1030 through a remote data manager 1032 at a data processing and storage center 1050 over a wireless data link 1046 accessible through a wireless gateway/router 1072. The process of replicating data to a remote data repository 1030 requires a wireless connection for transmission data between the asset 1048 and the data processing and storage center 1050. The onboard data manager 1020 and the remote data manager 1032 may communicate over multiple wireless communications links, such as Wi-Fi, cellular, satellite, and private wireless systems, using a wireless gateway/router 1072. Wireless data link 1046 , for example, may be a wireless local area network (WLAN), a wireless metropolitan area network (WMAN), a wireless wide area network (WWAN), a private wireless system, a cellular telephone network, or any other means of transmitting data from the DARS 1000 data recorder 1054 in, in this example, the remote DARS 1000 data manager 1030. When a wireless data connection is not available, the data is stored in a storage device and queued in a repository 1058, queuing until the wireless connection is restored, and the process data replication can resume.

[00122] In parallel with writing data, data writer 1054 continuously and offline replicates data to remote data repository 1030. The replication process has two modes, real-time and near-real-time. In real time, data is replicated to a remote data repository 1030 every second. In near real time, data is replicated to a remote data repository 1030 every five minutes. The frequency used for near real-time mode is configurable, and the frequency used for real-time mode can be adjusted to support high resolution data by replicating the data to the remote data repository 1030 every 0.10 seconds. When the DARS 1000 is in near real time, the onboard data manager 1020 queues the data in the repository 1058 to be queued before replicating the data to the remote data manager 1032. The onboard data manager 1020 also replicates the car event detector information queued in the queuing repository 1058 to the remote data manager 1032 . Near real-time mode is used during normal operation, under most conditions, to improve the efficiency of the data replication process.

[00123] The real-time mode may be initiated based on events occurring and detected by the car event detector 1056 on board the asset 1048, or by a request initiated from the data processing and storage center 1050. A typical real-time request initiated by data center 1050 is initiated when a remotely located user 1052 requests real-time information from web client 1042. A typical reason that real-time originates on board asset 1048 is when an event is detected. or an occurrence through the car event detector 1056, such as the driver's initiation of an emergency stop request, emergency braking activity, rapid acceleration or deceleration in any axis, or loss of input power to the data recording means 1054. When transitioning from near real-time mode to real-time mode, all data not already replicated to the remote data repository 1030 is replicated and stored in the remote data repository 1030, and then live replication is initiated. The transition between near real-time mode and real-time mode typically occurs in less than five seconds. After a predetermined amount of time has passed since an event or incident, after a predetermined amount of time of inactivity, or when the user 1052 no longer requires the real-time information from the asset 1048, the data recorder 1054 returns to near real-time mode. The predetermined amount of time required to initiate a transition is configurable and is typically set to ten minutes.

[00124] When the data recorder 1054 is in real time, the onboard data manager 1020 attempts to continuously empty its queue to the remote data manager 1032, storing the data in the failsafe storage module 1018 and optionally in the non-breakdown removable storage device. and sending data to the remote data manager 1032 simultaneously. The onboard data manager 1020 also sends the detected car information queued in the repository 1058 to the remote data manager 1032 .

[00125] Upon receiving data to be replicated from the data writer 1054, along with data from the map component 1064, the route/team manifest processing component 1024, and the weather component 1026, the remote data manager 1032 stores the compressed data in the remote data repository 1030 at the data processing and storage center 1050 of the DARS 1000. The remote data repository 1030, for example, may be a cloud data store or any other suitable remote data store. When data is received, a process is initiated that instructs data decoder 1036 to decode the latest replicated data to/from remote data repository 1030 and send the decoded data to remote event detector 1034. The remote data manager 1032 stores car event information in a remote data repository 1030 . When the remote event detector 1034 receives the decoded data, it processes the decoded data to determine whether or not an event of interest is detected in the decoded data. The decoded information is then used by the remote event detector 1034 to detect events, incidents, or other predetermined situations in the data occurring with the asset 1048. When an event of interest is detected from the decoded data, the remote event detector 1034 stores the event information and supporting data in remote repository 1030 data. When the remote data manager 1032 receives information from the remote event detector 1034, the remote data manager 1032 stores the information in the remote data repository 1030.

[00126] A remotely located user 1052 may access information, including car event detector information, associated with a specific asset 1048 or multiple assets using a standard web client 1042, such as a web browser or a virtual reality style device. (not shown) which, in this implementation, can display thumbnail images from selected cameras. Web client 1042 transmits user 1052's request for information to web server 1040 via network 1044 using common web standards, protocols, and technologies. Network 1044, for example, may be the Internet. The network 1044 may also be a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a virtual private network (VPN), a cellular telephone network, or any other means of transmitting data from the web server 1040, in which example, to the web client 1042. The web server 1040 requests the required data from the data decoder 1036. Data decoder 1036 receives requested data associated with a specific asset 1048 or multiple assets from a remote data repository 1030 upon request from web server 1040. Data decoder 1036 decodes the requested data and sends the decoded data to localizer 1038. Localization is the process of transforming the data. into the formats required by the end user, for example converting data into the user's preferred language and units of measurement. Localizer 1038 identifies profile settings specified by user 1052 by accessing web client 1042, and uses the profile settings to prepare information sent to web client 1042 for presentation to user 1052 as raw encoded data, and detected event information is stored in a remote data repository 1030 using Coordinated Universal Time (UTC) and the International System of Units (SI units). The localizer 1038 converts the decoded data into the format required by the user 1052, such as the preferred language and unit of measurement of the user 1052. The localizer 1038 sends the localized data in the preferred format of the user 1052 to the web server 1040 upon request. Web server 1040 then sends localized data of the asset or multiple assets to web client 1042 for viewing and analysis, providing playback and real-time display of standard video, 360-degree video, and/or other video. Web client 1042 may display, and user 1052 may view, data, video, and audio for a single asset, or simultaneously view data, video, and audio for multiple assets. The web client 1042 may also provide synchronous playback and real-time display of data along with a variety of video and audio data from image measurement sources, standard video sources, 360-degree video sources, and/or other video and/or ranging sources, at, at or near an asset, nearby assets and/or remote sites.

[00127] FIG. 21 is a flowchart showing a first illustrated embodiment of a process 1100 for determining signal compliance in accordance with an implementation of this disclosure. After the DARS 1000 and cameras 1016 are installed and connected to various sensors in the asset 1048, such as analog inputs 1002, digital inputs 1004, I/O module 1006, car controller 1008, engine controller 1010, inertial sensors 1012, global system 1014 positioning (GPS), cameras 1016, precision train control (PTC) signal data 1066, fuel system data 1068, cellular detectors (not shown), internal data and any additional data signals (1102), on-board data from various sensors and/or event-triggered video or still images are sent to the back office data processing and storage center 1074 every five minutes, and camera images are stored on board the asset 1048 for more than the last 72 hours (1104). The back office data center service 1074 continuously scans the data for trigger conditions (1106). If the trigger conditions for episode-based business logic are not satisfied, the processing thread is suppressed and the episode event is not logged (1108). If the asset 1048 moves past the path signal 1082, as indicated by the latitude and longitude coordinates of all signals stored in the back office processing and storage center 1074 (1110), and/or the asset 1048 stops within a certain distance before the signal 1082 and uses excessive braking force to allow a stop before passing signal 1082 (1112), the back office data center 1074 scans the data to determine whether or not a train car is present, in this illustrated embodiment, in head and/or managing position in the asset-railway train 1048 (1114). If the train car is not in the head and/or control position in the train asset 1048, the trigger conditions for the episode-based business logic are not satisfied, the processing flow is suppressed, and the episode event is not logged (1108). If a train car is in the head and/or control position in the asset train 1048, the back office data processing and storage center 1074 uses the first artificial intelligence model to determine whether or not the train car is in the head and/or control position. or a management position in the railway asset 1048 (1116). If a train car is in the head and/or control position in the active train 1048, the back office data processing and storage center 1074 requests video content from the locomotive in the head and/or control position, taken in the short period of time before the intersection of the signal 1082 and /or during the stop of the asset 1048 (1118). The extracted video content is transferred and/or stored in a back office processing and storage center 1074 and forwarded to a second AI-based model that scans the video content to determine the signal reading 1082, for example, the color combination of each signal light, to determine whether , whether or not signal 1082 indicates the semantic meaning "stop" (1120). The back office data center 1074 determines whether or not the signal 1082 indicates that the asset 1048 must stop and cannot pass through the signal 1082 (1122). If the indication of signal 1082 does not indicate that asset 1048 should stop and cannot pass through signal 1082, the trigger conditions for episode-based business logic are not satisfied, processing flow is suppressed, and the episode event is not logged (1108). If the reading of the signal 1082 indicates that the asset 1048 must stop and cannot pass through the signal 1082, and the stop signal is present, the episode is initiated, stored in the back office data center 1074 database, and mail messages are sent to users who pre-selected to notify when such conditions exist (1124).

[00128] For ease of explanation, process 1100 is illustrated and described as a series of steps. However, the steps in accordance with this disclosure may be performed in other orders and/or in parallel. Additionally, the steps in accordance with this disclosure may be performed with other steps not presented and described herein. Moreover, not all of the illustrated steps may be required in order to implement technologies in accordance with the claimed subject matter.

[00129] When used in this application, the term “or” is intended to mean an inclusive “or” rather than an exclusive “or”. Thus, unless otherwise stated or obvious from the context, "X uses A or B" is intended to mean any of the natural inclusive permutations. In other words, if X includes A; X includes B; or X includes both A and B, then "X includes A or B" is satisfied in either of the above cases. Additionally, "X includes at least one of A and B" is intended to mean any of the natural containing permutations. In other words, if X includes A; X includes B; or X includes both A and B, then "X includes at least one of A and B" is satisfied in any of the above cases. The articles "a" and "an" when used in this application and the accompanying claims are generally to be construed to mean "one or more" unless otherwise stated or obvious from the context, which is intended to refer to the singular form numbers. In addition, the use of the term “implementation” or “one implementation” throughout the application is not intended to mean an identical embodiment, aspect, or implementation unless actually described.

[00130] Although the present disclosure has been described in connection with specific embodiments, it should be understood that the disclosure is not intended to be limited to the disclosed embodiments, but rather is intended to cover various modifications and equivalent arrangements included within the scope of the appended claims, which the scope shall be consistent with the broadest interpretation so that it covers all such modifications and equivalent structures to the extent permitted by law.

Claims (56)

1. A method for processing data from at least one mobile asset, comprising the steps of: receive, using one of a data processing and storage center remote from the at least one mobile asset, and a data processing and storage center on board the at least one mobile asset, data based on at least one signal data from at least one of the following: at least one data source on board the at least one mobile asset; And at least one data source remote from at least one mobile asset; determining, using a first data center artificial intelligence-based model, that the at least one mobile asset is at least one of a head mobile asset and a managing mobile asset, provided that at least at least one trigger condition is detected using a data center on board the at least one mobile asset, based on the data; in response to the determination, video content is obtained, using a data center on board the at least one mobile asset, from at least one of the head mobile asset and the control mobile asset, wherein the video content contains a configurable, predetermined amount of data collected over a configurable predetermined amount of time until at least one triggering condition; storing, using a data center database on board the at least one mobile asset, video content; determining, using a second artificial intelligence-based model of a data processing center on board the at least one mobile asset, an episode based on the video content; storing, using a data center database on board the at least one mobile asset, the episode; And sending, using a data processing and storage center on board the at least one mobile asset, an electronic message to a predetermined number of users. 2. The method of claim 1, wherein the at least one trigger condition comprises at least one of the following: the data indicates that at least one mobile asset is moving past a signal of the plurality of signals, the signal comprising a location indicated by latitude and longitude coordinates of the plurality of signals stored in a database of a data center on board the at least one mobile asset; the data indicates that at least one mobile asset is stopped within a predetermined distance from a signal of the plurality of signals, and at least one mobile asset uses excessive braking force to allow stopping to the signal location; And data indicates speed limits. 3. The method according to claim 2, wherein the episode contains at least one triggering condition. 4. The method of claim 1, wherein the data comprises at least one of the following: data from at least one camera on, in or about at least one mobile asset, image analytics, analog parameters, analog inputs, digital parameters, digital inputs, I/O module, car controller, engine controller, inertial sensors, cameras, Precision Train Control (PTC) signal data, fuel system data, cellular detectors, internal data, map data, speed, pressure, temperature, current, voltage, acceleration, Boolean data, pointer position, actuator position, warning lights, actuator control command, global positioning system (GPS) data, braking forces, automated electronic notifications, geographic information system (GIS) data, position, speed, altitude, internally generated information, regulatory speed limit, video information, image information, audio information, route information, schedule information, freight manifest information, environmental information, current weather conditions, forecast weather conditions, asset management status, operating data and data generated by the Precision Train Control (PTC) system. 5. The method of claim 1, wherein the data comes from at least one of at least one mobile asset and at least one nearby asset. 6. The method of claim 1, wherein the data is received using one of a data processing and storage center remote from the at least one mobile asset, and a data processing and storage center on board the at least one mobile asset at least every five minutes. 7. The method of claim 1, wherein the data comprises at least one of the following: video information from cameras located at at least one of the locations in the mobile asset, on the mobile asset, and near the mobile asset; image information from cameras located at at least one of locations in the mobile asset, on the mobile asset, and near the mobile asset; And audio information from microphones located at at least one of the locations in the mobile asset, on the mobile asset, and near the mobile asset. 8. The method of claim 1, wherein the video content contains at least one of the following: video information from cameras located at at least one of the locations in the mobile asset, on the mobile asset, and near the mobile asset; image information from cameras located at at least one of locations in the mobile asset, on the mobile asset, and near the mobile asset; And audio information from microphones located at at least one of the locations in the mobile asset, on the mobile asset, and near the mobile asset. 9. The method of claim 1, wherein the at least one data source on board the at least one mobile asset comprises at least one of analog inputs, digital inputs, an input/output module, a car controller, an engine controller, inertial sensors, a global positioning system (GPS), at least one camera, precision train control (PTC) signal data, fuel system data, cellular detectors, and internal data. 10. The method of claim 1, wherein the at least one data source remote from the at least one mobile asset comprises at least one of a route/team declaration processing component, a weather component, and a mapping component. component. 11. The method of claim 1, wherein at least one of at least one on-board data source, at least one mobile asset, and at least one remote data source , from one mobile asset, contains at least one of at least one 360-degree camera, at least one fixed camera, at least one narrow-angle camera, at least one wide-angle camera at least one 360-degree fisheye camera and at least one radar and an optical detection and ranging (LIDAR) device. 12. A system for processing data from at least one mobile asset, comprising: at least one of at least one image measuring device, at least one video measuring device, at least one range measuring device, and at least one microphone; means for recording data on board the at least one mobile asset, adapted to receive at least one data signal from at least one, from at least one image measuring device of the at least one device video measurements, at least one ranging device, at least one microphone, at least one on-board data source, at least one mobile asset and at least one remote data source, at least at least from one mobile asset; a data processing and storage center adapted to receive data based on at least one data signal; a first artificial intelligence-based model of a data center, wherein the first artificial intelligence-based model is adapted to determine that at least one mobile asset is at least one of a head mobile asset and a control mobile asset provided that at least one triggering condition is detected by the data processing center; a data processing and storage center database, wherein the database is adapted to store video content obtained from at least one, at least one image measurement device, at least one video measurement device, at least one devices for measuring the range of at least one microphone, at least one data source on board, at least one mobile asset and at least one data source remote from at least one mobile asset; And a second AI-based model of the data center, wherein the second AI-based model is adapted with the ability to identify an episode based on video content. 13. The system of claim 12, wherein the at least one trigger condition comprises at least one of the following: the data indicates that at least one mobile asset is moving past a signal of the plurality of signals, the signal comprising a location indicated by latitude and longitude coordinates of the plurality of signals stored in a data center database; the data indicates that at least one mobile asset is stopped within a predetermined distance from a signal of the plurality of signals, and at least one mobile asset uses excessive braking force to allow stopping to the location of the signal; And data indicates speed limits. 14. The system of claim 12, wherein the image measuring device comprises at least one of at least one 360-degree camera, at least one fixed camera, at least one narrow-angle camera, at least one wide-angle camera and at least one 360-degree fisheye camera. 15. The system of claim 12, wherein the ranging device comprises at least one of a radar and an optical detection and ranging (LIDAR) device. 16. The system of claim 12, wherein the data comprises at least one of the following: data from at least one camera on, in or about at least one mobile asset, image analytics, analog parameters, analog inputs, digital parameters, digital inputs, I/O module, car controller, engine controller, inertial sensors, cameras, Precision Train Control (PTC) signal data, fuel system data, cellular detectors, internal data, map data, speed, pressure, temperature, current, voltage, acceleration, Boolean data, pointer position, actuator position, warning lights, actuator control command, global positioning system (GPS) data, braking forces, automated electronic notifications, geographic information system (GIS) data, position, speed, altitude, internally generated information, regulatory speed limit, video information, image information, audio information, route information, schedule information, freight manifest information, environmental information, current weather conditions, forecast weather conditions, asset management status, operating data and data generated by the Precision Train Control (PTC) system. 17. The system of claim 12, wherein the data comes from at least one mobile asset and at least one nearby asset. 18. The system of claim 12, wherein the data comprises at least one of the following: video information from cameras located at at least one of the locations in the mobile asset, on the mobile asset, and near the mobile asset; image information from cameras located at at least one of locations in the mobile asset, on the mobile asset, and near the mobile asset; And audio information from microphones located at at least one of the locations in the mobile asset, on the mobile asset, and near the mobile asset. 19. The system of claim 12, wherein the video content contains at least one of the following: video information from cameras located at at least one of the locations in the mobile asset, on the mobile asset, and near the mobile asset; image information from cameras located at at least one of locations in the mobile asset, on the mobile asset, and near the mobile asset; And audio information from microphones located at at least one of the locations in the mobile asset, on the mobile asset, and near the mobile asset. 20. The system of claim 12, wherein the at least one data source on board the at least one mobile asset comprises at least one of analog inputs, digital inputs, an input/output module, a car controller, an engine controller, inertial sensors, a global positioning system (GPS), at least one camera, precision train control (PTC) signal data, fuel system data, cellular detectors, and internal data. 21. The system of claim 12, wherein at least one data source remote from at least one mobile asset contains at least one of a route/crew declaration processing component, a weather component, and a mapping component. component.