WO2022140486A1

WO2022140486A1 - Distributed intellgent snap informatics

Info

Publication number: WO2022140486A1
Application number: PCT/US2021/064754
Authority: WO
Inventors: Shaobo HAN; Yuheng Chen; Ming-Fang Huang; Tingfeng LI
Original assignee: NEC Laboratories America Inc
Current assignee: NEC Laboratories America Inc
Priority date: 2020-12-22
Filing date: 2021-12-21
Publication date: 2022-06-30
Anticipated expiration: 2023-06-22
Also published as: WO2022140486A9; US12038320B2; DE112021006611T5; JP2023553479A; JP7539582B2; US20220196463A1

Abstract

A fiber optic sensing technology for vehicle run-off-road incident automatic detection by an indicator of sonic alert pattern (SNAP) vibration patterns. A machine learning method is employed and trained and evaluated against a variety of heterogeneous factors using controlled experiments, demonstrating applicability for future field deployment. Extracted events resulting from operation of our system may be advantageously incorporated into existing management systems for intelligent transportation and smart city applications, facilitating real-time alleviation of traffic congestion and/or providing a quick response rescue and clearance operation

Description

DISTRIBUTED INTELLGENT SNAP INFORMATICS

TECHNICAL FIELD

[0001] This disclosure relates generally to distributed fiber optic sensing (DFOS) of highways / roadways. More particularly, it pertains to a method for determining real time highway conditions and activity from vibrational events resulting from vehicular / sonic alert pattern (SNAP) interactions.

BACKGROUND

[0002] As will be understood by those skilled in the art, one highway development that has enhanced highway / road safety is the shoulder rumble strip - also known in the art as Sonic Alert Patterns (SNAP) - which produce mechanical vibrations and accompanying warning sounds to notify drowsy or distracted drivers when their vehicle travel over the patterns (SNAP events). Despite the utility of notifying a driver (or passenger) of such activity, highway planners and emergency personnel may also benefit from notification of SNAP events - particularly in real time when such events occur.

SUMMARY

[0003] An advance in the art is made according to aspects of the present disclosure directed to systems, methods, and structures that detect and report sonic alert pattern (SNAP) events in real time such that real time response may result.

[0004] In sharp contrast to the prior art, systems, methods, and structures according to aspects of the present disclosure employ a fiber optic sensing technology for vehicle run-off-road incident automatic detection by an indicator of sonic alert pattern (SNAP) vibration patterns. A machine learning method is employed and trained and evaluated against a variety of heterogeneous factors using controlled experiments. Extracted events resulting from operation of our system may be advantageously incorporated into existing management systems for intelligent transportation and smart city applications, facilitating real-time alleviation of traffic congestion and/or providing a quick response rescue and clearance operation(s).

BRIEF DESCRIPTION OF THE DRAWING

[0005] A more complete understanding of the present disclosure may be realized by reference to the accompanying drawing in which:

[0006] FIG. 1 is a schematic diagram of an illustrative distributed fiber optic sensing system (DFOS) generally known in the art.

[0007] FIG. 2(A) and FIG. 2(B) show illustrative system layout of a sensing layer overlaid on a deployed optical sensing fiber and collected SNAP signals displayed as schematic waterfall images for both a sunny and rainy day according to aspects of the present disclosure;

[0008] FIG. 3(A) and FIG. 3(B) are flow diagrams illustrating model training and inference procedures according to aspects of the present disclosure;

[0009] FIG. 4 is a schematic diagram showing an illustrative architecture of our model according to aspects of the present disclosure;

[0010] FIG. 5 is a schematic waterfall plot with SNAP events denoted therein according to aspects of the present disclosure; and

[0011] FIG. 6 is a flow diagram of an overall procedure according to aspects of the present disclosure.

[0012] The illustrative embodiments are described more fully by the Figures and detailed description. Embodiments according to this disclosure may, however, be embodied in various forms and are not limited to specific or illustrative embodiments described in the drawing and detailed description. DESCRIPTION

[0013] The following merely illustrates the principles of the disclosure. It will thus be appreciated that those skilled in the art will be able to devise various arrangements which, although not explicitly described or shown herein, embody the principles of the disclosure and are included within its spirit and scope.

[0014] Furthermore, all examples and conditional language recited herein are intended to be only for pedagogical purposes to aid the reader in understanding the principles of the disclosure and the concepts contributed by the inventor(s) to furthering the art and are to be construed as being without limitation to such specifically recited examples and conditions.

[0015] Moreover, all statements herein reciting principles, aspects, and embodiments of the disclosure, as well as specific examples thereof, are intended to encompass both structural and functional equivalents thereof. Additionally, it is intended that such equivalents include both currently known equivalents as well as equivalents developed in the future, i.e., any elements developed that perform the same function, regardless of structure.

[0016] Thus, for example, it will be appreciated by those skilled in the art that any block diagrams herein represent conceptual views of illustrative circuitry embodying the principles of the disclosure.

[0017] Unless otherwise explicitly specified herein, the FIGs comprising the drawing are not drawn to scale.

[0018] By way of some additional background - we again note that in recent years, distributed fiber optic sensing (DFOS) systems including distributed vibration sensing (DVS) and distributed acoustic sensing (DAS) have found widespread acceptance in numerous applications including - but not limited to - infrastructure monitoring, intrusion detection, and earthquake detection. For DAS and DVS, backward Rayleigh scattering effects are used to detect changes in the fiber strain, while the fiber itself acts as the transmission medium for conveying the optical sensing signal back to an interrogator for subsequent analysis.

[0019] By way of some additional background - and with reference to FIG. 1 which is a schematic diagram of an illustrative distributed fiber optic sensing system generally known in the art - we begin by noting that distributed fiber optic sensing (DFOS) is an important and widely used technology to detect environmental conditions (such as temperature, vibration, stretch level etc.) anywhere along an optical fiber cable that in turn is connected to an interrogator. As is known, contemporary interrogators are systems that generate an input signal to the fiber and detects / analyzes the reflected/scattered and subsequently received signal(s). The signals are analyzed, and an output is generated which is indicative of the environmental conditions encountered along the length of the fiber. The signal(s) so received may result from reflections in the fiber, such as Raman backscattering, Rayleigh backscattering, and Brillion backscattering. It can also be a signal of forward direction that uses the speed difference of multiple modes. Without losing generality, the following description assumes reflected signal though the same approaches can be applied to forwarded signal as well.

[0020] As will be appreciated, a contemporary DFOS system includes an interrogator that periodically generates optical pulses (or any coded signal) and injects them into an optical fiber. The injected optical pulse signal is conveyed along the optical fiber.

[0021] At locations along the length of the fiber, a small portion of signal is reflected and conveyed back to the interrogator. The reflected signal carries information the interrogator uses to detect, such as a power level change that indicates - for example - a mechanical vibration. [0022] The reflected signal is converted to electrical domain and processed inside the interrogator. Based on the pulse injection time and the time signal is detected, the interrogator determines at which location along the fiber the signal is coming from, thus able to sense the activity of each location along the fiber.

[0023] We note that one guiding principle of systems, methods, and structures according to aspects of the present disclosure is using existing fiber infrastructure as sensing media - along with our associated Distributed Intelligent SNAP Informatics (DISI) system - to detect highway / roadway incident events such that the road incident events can be detected/determined and subsequently prevented by distributed optical sensing (DFOS) and Al technologies. As those skilled in the art will understand and appreciate, since SNAPs are oftentimes installed between traffic lanes of a highway/roadway, systems, methods, and structures according to aspects of the present disclosure enable the discovery and reporting of highway incidents vehicles crossing SNAPs and stopping on the shoulder of the highway.

[0024] Advantageously, our illustrative systems, methods, and structures according to aspects of the present disclosure advantageously utilize an in-place infrastructure of optical fiber cables to detect/determine valuable highway transportation related data. When a DISI system is combined with DFOS it advantageously may include both Distributed Acoustic Sensing (DAS) and Distributed Vibration Sensing (DVS) coupled with machine learning technologies thereby providing accident or other event coverage to tens of kilometers of roadway /highway by one integrated system.

[0025] As we shall describe further, DISI is an Al-based traffic analysis and management unit that reports drift-off-road traffic incidents and emergency stop events in real time and/or near-real-time. Collected data can be incorporated into Geographic Information Systems (GIS) for highway/roadway infrastructure service / management / maintenance operation, and other decision-making purposes. Our distributed fiber optic sensing-based approach has the advantages of being non-destructive while exhibiting relatively low deployment cost, high spatial-resolution and low latency characteristics. [0026] As those skilled in the art will understand and appreciate, a key technical challenge with the systems, structures, and methods disclosed herein is the identification of distinct data patterns associated with events resulting from vehicles crossing rumble stripes, as conveyed by spatio-temporal data streams from a DFOS (DAS) system.

[0027] In a typical deployment/operation, SNAP crossing signals are buried or otherwise obfuscated by background noises caused by normal traffic and environmental noise. Although nimble strips produce audible rumbling in audio frequencies, the range of the frequency changes with the spacing of grooves and the driving speed of the vehicles. Furthermore, the vehicle type, weather condition, sensing distance, and soil type can all make the patterns different in the received sensor data. In addition, the change of speed before and after crossing the rumble stripes also contains useful information for SNAP event detection.

[0028] As we shall show and describe further, to address these challenges, we employ a deep learning-based approach on the DAS data. The deep learning system and method software learns what constitute such distinctive patterns by comparing normal traffic signals against those with SNAP events. To facilitate such learning, we create a local spatial-temporal window, such that more focus is directed to features that makes SNAP events look different among other high intensity events. Comparing to traditional frequency-based approach, the local spatial-temporal window can also exploit the context information before and after SNAP vibrations.

[0029] More specifically, our model includes at least two main components - an embedding network for representation learning and a Support Vector Machine (SVM) for classification. As currently implemented, the embedding network is a Convolutional Neural Network (CNN) with three convolutional layers and three fully connected layers. We also add drop out and rectified linear unit (ReLU) after each fully connected layer for non-linearity and overfitting alleviation. The embedding network is trained with a triplet loss to enlarge distance between SNAP patches and other irrelevant patches, while shortening the distance between different SNAP patches in representation space. Based on the learned representation, we can train a more accurate SVM classifierto distinguish SNAP events from other events.

[0030] FIG. 2(A) and FIG. 2(B) show illustrative system layout of a sensing layer overlaid on a deployed optical sensing fiber and collected SNAP signals displayed as schematic waterfall images for both a sunny and rainy day according to aspects of the present disclosure.

[0031] With simultaneous reference to these figures, FIG. 2(A) shows the illustrative configuration having a sensing layer overlaid on deployed fiber. The optical sensing system (DFOS) and Distributed Intelligent SNAP Informatics (DISI) detector is shown located in a control office / central office for remote monitoring of entire fiber optic sensor cable route. The DFOS system is connected to the field optical fiber to provide sensing functions.

[0032] As shown illustratively in the figure, there are two sections of buried fiber, under grass and another under asphalt pavement. To increase the variabilities of fiber detection length, an additional 10km and 5km single-mode fiber (SMF) are inserted in the route.

[0033] The rumble strips installed on the pavement are in an indentation spacing of one every foot (30.5 cm) along the direction of travel of vehicle tires. Each strip exhibits a height of substantially 0.5 inches (1.27 cm). Such a strip is sufficient to produce vibration signals when a vehicle tire travels over it.

[0034] FIG. 2(B) shows illustrative collected SNAP signals in waterfall traces on a sunny and a rainy day. The signal strength and patterns of SNAP are different with weather-ground conditions (e.g., sunny/rainy day), sensing distance (e.g., hundreds or thousands of meters), vehicle type (e.g., sedan/truck) and soil type (e.g., grass/pavement).

[0035] FIG. 3(A) and FIG. 3(B) are flow diagrams illustrating model training and inference procedures according to aspects of the present disclosure. As shown in the figure, during a training phase, a preprocessing step extracts regions of waterfall images exhibiting high vibration intensity, and a model is trained using data augmented from many waterfall patches labeled as SNAP or Background. During the inference phase, the input waterfall images are converted into individual rectangle patches by applying sliding windows with overlaps. The identified windows that contain SNAP events are then merged into one single box via a boxes-fusion step. The timestamp, cable location, event type, and confidence score are provided as the output

[0036] Shown further in the figure it may be observed that our inventive two-stage model combines deep representation learning (RL) with support vector machine (SVM). The embedding network performs a verification task, which determines whether the vibration patterns in a pair of patches are caused by the same event (either normal traffic or SNAP). The SVM classifier performs an identification task which assigns a class label to each patch. A convolutional neural network architecture is well- suited for detecting patterns from two-dimensional waterfall data viewed as images.

[0037] FIG. 4 is a schematic diagram showing an illustrative architecture of our model according to aspects of the present disclosure.

[0038] Our model can learn to achieve >91.8% average classification accuracy, consistently against various heterogeneous factors, such as sunny-rainy, 0-10-15km, Grass- Pavement, Truck- Sedan-Minivan, single or two vehicles. Experimental results are summarized in Tables 1 - 4, each entry reports the mean +/- standard deviation out of 10 independent runs

[0039] To perform SNAP event localization, we take a sliding window-based approach and apply the trained classifier to each image patch. SNAP vibration may appear in multiple overlapping boxes.

[0040] FIG. 5 is a schematic waterfall plot with SNAP events denoted therein according to aspects of the present disclosure. As illustrated in FIG .5, a single fused bounding box is derived from taking union of multiple individual bounding boxes. The timestamp and location of SNAP events within each fused bounding box is estimated via a peak-finding algorithm

[0041] FIG. 6 is a flow diagram of an overall procedure according to aspects of the present disclosure. With reference to that figure, it may be observed that our inventive method involves first, measuring and/or determining a normal, baseline vibration level from road or other traffic or events and event characteristics using a fiber optic sensor cable of a deployed fiber link configured as a distributed fiber optic sensor (DFOS) system.

[0042] Second, the location(s) of the DFOS system fiber optic sensor cable is related to a geographic map.

[0043] Third, operating the DFOS system and collecting data of vehicles passing through SNAP and generating detectable vibration events, such data collection including different classes of vehicles with different weather-ground conditions) by DFOS and generating an Al model comprising the distributed intelligent SNAP informatics (DISI) system.

[0044] Fourth, when an incident is detected, an alarm is triggered by the DISI system using anomaly scores and outputting a map display of the anomaly location on map(s).

[0045] Fifth, when the alarm is triggered and it exhibits a pre-determined confidence score, the incident event is reported to a control room or supervisory system for further action.

[0046] Sixth, road maintenance crews or first responders may be dispatched to the location at which the incident occurred.

[0047] To evaluate our inventive systems, methods, and structures, we conducted a field trial using a setup substantially as that shown previously in FIG. 2(A). As shown and deployed, the fiber sensing system (with Al engines for real-time inference) is optically connected to the fiber optic sensor cable buried underground at a depth of approximately 15-20 inches. The fiber optic sensing system employed in this trial utilizes a Distributed Acoustic Sensing (DAS) methodology, which measures a change in Rayleigh scattering via interferometric phase beating in fiber. The DAS system employed uses short optical pulses along with on-chip fast processing to enable an equivalent spatial resolution as small as 1 meter at 2000-Hz sampling rate. The rumbling stripes were installed on the pavement with a spacing of substantially 7 inches (17.8cm) every foot (30.5 cm) along the travelling direction of test vehicles and exhibited a height of 0.5 inches (1.27 cm). Such configuration will create a sufficient height/depth for a tire drop when the tire overrides the strip sufficient to produce detectable vibration signals.

[0048] To sufficiently emulate practical challenges of SNAP detection, a number of heterogeneous factors are considered in our field trial experimental design. These include (1) ground type - fiber buried under grass (”G”) or embedded within asphalt pavement (”P”), (2) weather conditions - sunny day (dry ground) and rainy day (wet ground), (3) vehicle types - sedan, minivan, and truck, and (4) sensing distance - 300 m, 10.4 km and 15.8 km.

[0049] In our trial configuration an additional 10km and5 km of single-mode fibers (SMF) are inserted in the fiber route. As a result, every SNAP event is sensed simultaneously by 6 different segments of fiber in one driving test. Similar to that shown in FIG 3(A), exemplary SNAP signals in the spatial-time domain, collected from a sedan on a sunny day and a truck on a rainy day, at 3 different sensing distances under 2 ground types (”G” and ”P”). The recorded sensing data exhibit distinct signal and noise characteristics, highlighting the difficulties in feature engineering using signal processing techniques.

[0050] As previously discussed, we consider the spatiotemporal locality of sensing signals, and employ a sliding-window paradigm for SNAP event detection - independently classify image patches as being labeled as SNAP or Others. The classification is performed in the embedding space.

[0051] In particular, the data representation is optimized by triplet loss, such that patch embeddings with the same label are closer to each other than those with different labels. As noted, the embedding function is trained using convolutional neural network (CNN) on a verification task, which determines whether the vibration patterns in a pair of patches are caused by the same kind of events or not. Based on the learned representation, the support vector machine (SVM) classifier performs an identification task, which assigns the class label to the patch embedding.

[0052] Our overall training and inference architecture is like that shown previously. To improve training performance, data augmentation is performed by randomly sampling patches around regions with high vibrations, promoting contrast between SNAP vibration and other strong vibrations such as normal driving in a learning process. In the inference stage, the input waterfall images are converted into local patches with overlaps by sliding window'. Neighboring windows with SNAP events detected are then merged into one single bounding box. The Al engine takes continuous streams of input data and runs inference in real-time on GPU, providing a timestamp, cable location, event type, and confidence score as output.

[0053] In performing our field trial experiments, we collected data from multiple independent rounds (20 or 25 for sunny/rainy day), in each of the 12 controlled experiments covering 2 weather conditions, 3 vehicle types, and 2 event types (crossing SNAP or passing by). In each round, we obtained data directly at 3 sensing distances under 2 ground types. To train a SNAP event detector that works under different practical scenarios, we aggregated all the training data collected under all 36 conditions. Testing was performed under each unique condition, such that the effects of individual factors can be isolated. To ensure there is no overlap between training and test set, a split ratio of 8 : 2 was applied on the rounds. The performance was measured by three classification metrics: 1) Accuracy; 2) Area Under the ROC Curve (AUC); and 3) Area Under Precision- Recall Curve (AUPRC). Advantageously, our model achieved high average classification accuracy, consistently against various heterogeneous factors. As would be expected however, wedetorrinedtethat longer sen sing distance, higher damping coefficient under grass, and lighter vehicle weight can slightly reduce the model performance, while weather condition had no obvious effects.

Classification Performance (Accuracy, AUC, AUPRC) with different factors on a sunny day.

Classification Performance (Accuracy, AUC, AUPRC) with different factors on a rainy day.

Comparison of test accuracy (%) to different factors between sunny and rainy day.

Comparison of test accuracy (%) to different factors between single vehicle and two vehicles on a rainy day.

[0054] At this point, while we have presented this disclosure using some specific examples, those skilled in the art will recognize that our teachings are not so limited. Accordingly, this disclosure should be only limited by the scope of the claims attached hereto.

Claims

Claims:

1. A method of operating a distributed fiber optic sensing (DFOS) system employing distributed intelligent sonic alert pattern (SNAP) informatics, the method comprising: operating the DFOS system by generating by an interrogator, interrogating light and sending it into a length of optical sensing fiber located proximate to a roadway; detecting backscattered light that results from the interrogating light sent into the optical sensing fiber; generating and outputting signals indicative of detected backscattered light; and analyzing, the outputted signals indicative of detected backscattered light and determining strain signals associated with location(s) along the length of the optical sensing fiber that are indicative of a vibration and acoustic environment of the optical sensing fiber at the location(s); and determining normal, baseline characteristics including baseline vibration levels from road traffic operating on the roadway under normal conditions of the optical sensing; and determining when a vehicle operating on the roadway passes through a SNAP area of the roadway.

2. The method of claim 1 wherein a portion of the optical sensing fiber is located under grass and another portion of the optical sensing fiber is located under a portion of the roadway.

3. The method of claim 2 wherein said DFOS system analyzing is performed by a two- stage model comprising an embedding network providing deep representation learning and a support vector machine classifier.

4. The method of claim 3 wherein the embedding network comprises a convolutional neural network and detects patterns from two-dimensional waterfall images generated from the determined strain signals and locations.

5. The method of claim 4 wherein the convolutional neural network is trained on a set of training images including both normal waterfall images indicative of normal traffic patterns and SNAP waterfall images indicative of traffic encountering SNAP areas of the roadway.

6. The method of claim 5 wherein the training images are preprocessed to extract regions exhibiting a high vibration intensity relative to other regions.

7. The method of claim 6 wherein the training of the convolutional neural network includes training waterfall images of different weather conditions, vehicle types, and overlying ground types.

8. The method of claim 7 wherein the DFOS system generates an alarm when a detected SNAP event exceeds a predetermined vibrational threshold.

9. The method of claim 8 wherein the DFOS system triggers a notification event when the alarm exhibits a predetermined confidence score.

10. The method of claim 1 wherein the optical sensing fiber simultaneously carries telecommunications traffic.