WO2021152534A1 - Procédé et système de surveillance de milieux d'intérêt troubles destinés à prédire des événements - Google Patents

Procédé et système de surveillance de milieux d'intérêt troubles destinés à prédire des événements Download PDF

Info

Publication number
WO2021152534A1
WO2021152534A1 PCT/IB2021/050725 IB2021050725W WO2021152534A1 WO 2021152534 A1 WO2021152534 A1 WO 2021152534A1 IB 2021050725 W IB2021050725 W IB 2021050725W WO 2021152534 A1 WO2021152534 A1 WO 2021152534A1
Authority
WO
WIPO (PCT)
Prior art keywords
turbid
interest
data
turbid media
cameras
Prior art date
Application number
PCT/IB2021/050725
Other languages
English (en)
Inventor
Jayavardhana Rama GUBBI LAKSHMINARASIMHA
Karthik Seemakurthy
Mahesh Rangarajan
Balamuralidhar Purushothaman
Ashutosh Raj
Vishnu Hariharan Anand
Srinivas Kotamraju
Original Assignee
Tata Consultancy Services Limited
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Tata Consultancy Services Limited filed Critical Tata Consultancy Services Limited
Publication of WO2021152534A1 publication Critical patent/WO2021152534A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/23—Clustering techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/25—Determination of region of interest [ROI] or a volume of interest [VOI]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/30—Noise filtering

Definitions

  • the embodiments herein generally relate to the field of harsh environment monitoring and alarming systems and, more particularly, to method and system for monitoring of turbid media of interest in the harsh environments to predict events such as possible underground geyser outbursts.
  • Embodiments of the present disclosure present technological improvements as solutions to one or more of the above-mentioned technical problems recognized by the inventors in conventional systems. For example, in one embodiment, a method for monitoring of turbid media of interest to predict outbursts is provided.
  • the method comprises receiving input data comprising camera data and sensor data from a multimodal sensing unit providing long distance surveillance, wherein the input data provides information of one or more turbid media present in a Region of Interest (ROI), and wherein the multimodal sensing unit comprises: a plurality of multimodal cameras, comprising a combination of portable and fixed visual cameras and thermal cameras, covering a plurality of subregions within the ROI, and positioned to capture and stream the camera data comprising visual image frames and thermal image frames, wherein the camera data provides multiple views of the plurality of subregions comprising local views and global views; and a plurality of multimodal sensors positioned to capture the sensor data comprising a plurality of parameters associated with the plurality of subregions.
  • ROI Region of Interest
  • the method comprises preprocessing the camera data and the sensor data to eliminate noise, wherein the camera data is preprocessed using a video stabilization approach to eliminate noise induced in the camera data due to undesired motion of the plurality of multimodal cameras. Further, the method comprises deriving a flow vector information from the visual image frames or thermal image frames to extract temporal features, of turbid regions of the one or more turbid media within every image frame, from the flow vector information.
  • the method comprises processing the sensor data and the temporal features extracted from the flow vector information to identify a plurality of turbid regions corresponding to a turbid media of interest among the one or more turbid media present in the ROI, wherein the plurality of turbid regions corresponding to the turbid media of interest are identified based on a set of features uniquely defining the turbid media of interest.
  • the method comprises clustering the plurality of turbid regions into a plurality of clusters based on the temporal features and a thermal profile obtained from the thermal image frames, wherein each cluster among the plurality of clusters comprises one or more turbid regions among the plurality of turbid regions.
  • the method comprises fitting a convex hull to each turbid region of each cluster to determine a plurality of minimum points of the convex hull fitted to each turbid region, wherein each minimum point is identified by corresponding location coordinates. Further, the method comprises grouping the plurality of minimum points in accordance to corresponding location coordinates to identify a plurality of source locations of a plurality of turbid activities corresponding to the turbid media of interest, wherein each source location among the plurality of source locations corresponds to one or more minimum points which have mapping location coordinates.
  • the method comprises monitoring temporal variations occurring in the plurality of turbid activities corresponding to the plurality of source locations by performing frame by frame analysis of the temporal features extracted from flow vector information and corresponding thermal profile of the ROI obtained from the thermal cameras. Further, the method comprises predicting one or more source locations among the plurality of source locations as target locations for occurrence of events based on a prediction model, wherein the prediction model utilizes a clustered historical data the plurality of source locations, the detected temporal variations and the thermal profile of each frame for prediction. [005] In another aspect, a system for monitoring of turbid media of interest to predict outbursts is provided.
  • the system comprises a memory storing instructions; one or more Input/Output (I/O) interfaces; and one or more hardware processors coupled to the memory via the one or more I/O interfaces, wherein the one or more hardware processors are configured by the instructions to receive input data comprising camera data and sensor data from a multimodal sensing unit providing long distance surveillance, wherein the input data provides information of one or more turbid media present in a Region of Interest (ROI), and wherein the multimodal sensing unit comprises: a plurality of multimodal cameras, comprising a combination of portable and fixed visual cameras and thermal cameras with pan- tilt- zoom feature, covering a plurality of subregions within the ROI, and positioned to capture and stream the camera data comprising visual images and thermal image frames, wherein the camera data provides multiple views of the plurality of subregions comprising local views and global views; and a plurality of multimodal sensors positioned to capture the sensor data comprising a plurality of parameters associated with the plurality of subregions.
  • I/O Input/Output
  • the one or more hardware processors are configured to preprocess the camera data and the sensor data to eliminate noise, wherein the camera data is preprocessed using a video stabilization approach to eliminate noise induced in the camera data due to undesired motion of the plurality of multimodal cameras. Further, the one or more hardware processors are configured to derive a flow vector information from the visual images to extract temporal features, of turbid regions of the one or more turbid media within every image frame, from the flow vector information.
  • the one or more hardware processors are configured to process the sensor data and the temporal features extracted from the flow vector information to identify a plurality of turbid regions corresponding to a turbid media of interest among the one or more turbid media present in the ROI, wherein the plurality of turbid regions corresponding to the turbid media of interest are identified based on a set of features uniquely defining the turbid media of interest. Furthermore, the one or more hardware processors are configured to cluster the plurality of turbid regions into a plurality of clusters based on the temporal features and a thermal profile obtained from the thermal image frames , wherein each cluster among the plurality of clusters comprises one or more turbid regions among the plurality of turbid regions.
  • the one or more hardware processors are configured to fit a convex hull to each turbid region of each cluster to determine a plurality of minimum points of the convex hull fitted to each turbid region, wherein each minimum point is identified by corresponding location coordinates. Furthermore, the one or more hardware processors are configured to group the plurality of minimum points in accordance to corresponding location coordinates to identify a plurality of source locations of a plurality of turbid activities corresponding to the turbid media of interest, wherein each source location among the plurality of source locations corresponds to one or more minimum points which have mapping location coordinates.
  • the one or more hardware processors are configured to monitor temporal variations occurring in the plurality of turbid activities corresponding to the plurality of source locations by performing frame by frame analysis of the temporal features extracted from flow vector information and corresponding thermal profile of the ROI obtained from the thermal cameras. Further, the one or more hardware processors are configured to predict one or more source locations among the plurality of source locations as target locations for occurrence of events based on a prediction model, wherein the prediction model utilizes a clustered historical data the plurality of source locations, the detected temporal variations and the thermal profile of each frame for prediction.
  • one or more non-transitory machine readable information storage mediums comprising one or more instructions, which when executed by one or more hardware processors causes a method for monitoring of turbid media of interest to predict outbursts.
  • the method comprises receiving input data comprising camera data and sensor data from a multimodal sensing unit providing long distance surveillance, wherein the input data provides information of one or more turbid media present in a Region of Interest (ROI), and wherein the multimodal sensing unit comprises: a plurality of multimodal cameras, comprising a combination of portable and fixed visual cameras and thermal cameras, covering a plurality of subregions within the ROI, and positioned to capture and stream the camera data comprising visual images and thermal image frames, wherein the camera data provides multiple views of the plurality of subregions comprising local views and global views; and a plurality of multimodal sensors positioned to capture the sensor data comprising a plurality of parameters associated with the plurality of subregions.
  • ROI Region of Interest
  • the method comprises preprocessing the camera data and the sensor data to eliminate noise, wherein the camera data is preprocessed using a video stabilization approach to eliminate noise induced in the camera data due to undesired motion of the plurality of multimodal cameras. Further, the method comprises deriving a flow vector information from the visual image frames and the thermal image frames to extract temporal features, of turbid regions of the one or more turbid media within every image frame, from the flow vector information.
  • the method comprises processing the sensor data and the temporal features extracted from the flow vector information to identify a plurality of turbid regions corresponding to a turbid media of interest among the one or more turbid media present in the ROI, wherein the plurality of turbid regions corresponding to the turbid media of interest are identified based on a set of features uniquely defining the turbid media of interest.
  • the method comprises clustering the plurality of turbid regions into a plurality of clusters based on the temporal features and a thermal profile obtained from the thermal image frames, wherein each cluster among the plurality of clusters comprises one or more turbid regions among the plurality of turbid regions.
  • the method comprises fitting a convex hull to each turbid region of each cluster to determine a plurality of minimum points of the convex hull fitted to each turbid region, wherein each minimum point is identified by corresponding location coordinates. Further, the method comprises grouping the plurality of minimum points in accordance to corresponding location coordinates to identify a plurality of source locations of a plurality of turbid activities corresponding to the turbid media of interest, wherein each source location among the plurality of source locations corresponds to one or more minimum points which have mapping location coordinates.
  • the method comprises monitoring temporal variations occurring in the plurality of turbid activities corresponding to the plurality of source locations by performing frame by frame analysis of the temporal features extracted from flow vector information and corresponding thermal profile of the ROI obtained from the thermal cameras. Further, the method comprises predicting one or more source locations among the plurality of source locations as target locations for occurrence of events based on a prediction model, wherein the prediction model utilizes a clustered historical data the plurality of source locations, the detected temporal variations and the thermal profile of each frame for prediction.
  • FIG. 1 is a functional block diagram of a system for monitoring of a turbid media of interest in a Region of Interest (ROI) to predict events , in accordance with some embodiments of the present disclosure.
  • ROI Region of Interest
  • FIG. 2A and FIG. 2B is a flow diagram illustrating a method for monitoring of the turbid media of interest in the ROI to predict the events, using the system of FIG. 1, in accordance with some embodiments of the present disclosure.
  • FIG. 3 illustrates an example ROI of a harsh environment, wherein the system of FIG 1 is deployed for monitoring of the turbid media of interest to predict the events, in accordance with some embodiments of the present disclosure.
  • FIG. 4 is a functional block diagram of the system of FIG. 1 depicting prediction model providing pattern mining for predictive analytics for event prediction in the ROI, in accordance with some embodiments of the present disclosure.
  • Embodiments herein provide a method and system for monitoring of turbid media of interest to predict events in a Region of Interest (ROI) of a harsh environment.
  • the events refer to sudden or abrupt changes in environment such as outbursts.
  • the system disclosed herein utilizes a multimodal sensing unit for monitoring the turbid media of interest in the ROI.
  • the multimodal sensing unit is a combination of a plurality of multimodal cameras such as visual camera and thermal cameras, along with multimodal sensors such as pressure sensors, doppler radars and the like.
  • the visual cameras, the thermal cameras, the doppler radars and so on are positioned to provide long distance surveillance, while the sensors such as pressure sensors and the like are positioned across the ROI to provide proximity surveillance to capture various parameters.
  • the visual cameras provide passive sensing, utilizing reflected ambient light to capture the dynamic features of a one or more turbid media present in the ROI.
  • the thermal cameras provide passive sensing to capture thermal signature of the ROI.
  • the system provides mechanism to effectively compensate for noise induced camera data and sensor data capturing information of the turbid media.
  • the method and system disclosed herein utilizes the spatio-temporal features extracted from the videos captured by visual cameras in fusion with features extracted from other sensors and cameras of different modalities to rightly identify turbid regions.
  • a set of parameters defining parameters relevant to the turbid media of interest enables the system to rightly identify the turbid media of interest from plurality of turbid media captured by the visual cameras from the ROI. This eliminates possibility of false negatives.
  • the system disclosed utilizes historical data along with current data from multiple sensors, to more accurately predict potential event regions well in advance to raise alerts.
  • FIGS. 1 through 4 where similar reference characters denote corresponding features consistently throughout the figures, there are shown preferred embodiments and these embodiments are described in the context of the following exemplary system and/or method.
  • FIG. 1 is a functional block diagram of a system for monitoring of a turbid media of interest to predict events, in accordance with some embodiments of the present disclosure.
  • the system 100 includes a processor(s) 104, communication interface device(s), alternatively referred as input/output (I/O) interface(s) 106, and one or more data storage devices or a memory 102 operatively coupled to the processor(s) 104.
  • the system 100 with one or more hardware processors is configured to execute functions of one or more functional blocks of the system 100.
  • the system further includes a multimodal sensing unit 110 comprising a plurality of multimodal cameras such as visual cameras and thermal cameras for long distance surveillance.
  • the multimodal sensing unit 110 further comprises multimodal sensors such as doppler radars or the like for long distance surveillance and pressure sensors and the like for close surveillance.
  • the system 100 utilizes spatio-temporal features extracted from the images captured by visual cameras of the multimodal cameras in fusion with features extracted from other sensors and cameras of different modalities to rightly identify turbid regions in the ROI.
  • the processor(s) 104 can be one or more hardware processors 104.
  • the one or more hardware processors 104 can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions.
  • the one or more hardware processors 104 is configured to fetch and execute computer- readable instructions stored in the memory 102.
  • the system 100 can be implemented in a variety of computing systems including laptop computers, notebooks, hand-held devices such as mobile phones, workstations, mainframe computers, servers, a network cloud and the like.
  • the I/O interface(s) 106 can include a variety of software and hardware interfaces, for example, a web interface, a graphical user interface, a touch user interface (TUI) and the like and can facilitate multiple communications within a wide variety of networks N/W and protocol types, including wired networks, for example, LAN, cable, etc., and wireless networks, such as WLAN, cellular, or satellite.
  • the I/O interface (s) 106 can include one or more ports for connecting a number of devices (nodes) of the system 100 to one another or to another server. Further the I?0 interface 106 provides interface for interfacing the multimodal cameras, doppler radars, and sensors of the multimodal sensing unit 110 to the system 100.
  • the memory 102 may include any computer-readable medium known in the art including, for example, volatile memory, such as static random access memory (SRAM) and dynamic random access memory (DRAM), and/or non-volatile memory, such as read only memory (ROM), erasable programmable ROM, flash memories, hard disks, optical disks, and magnetic tapes. Further, the memory 102 may include a database 108, which may store the current and historical camera data, sensor data, any meta data and the like. In an embodiment, the database 108 may be external to the system 100 (not shown) and coupled to the system via the I/O interface 106.
  • the memory 102 may comprise information pertaining to input(s)/output(s) of each step performed by the processor(s) 104 of the system 100 and methods of the present disclosure. Functions of the components of system 100 are explained in conjunction with flow diagram of FIGS. 2 A and 2B, and FIG. 3 depicting an example system 100 deployed in an example harsh environment such as a coal mining region, to monitor geyser activities or steam flows (turbid media of interest) and predict possible events such as geyser outburst, with sudden heavy and high speed steam flows, which are risky events for people working in the ROI.
  • a harsh environment such as a coal mining region
  • FIG. 2A and FIG. 2B is a flow diagram illustrating a method for monitoring of the turbid media of interest in the ROI to predict events, using the system of FIG. 1, in accordance with some embodiments of the present disclosure.
  • the system 100 comprises one or more data storage devices or the memory 102 operatively coupled to the processor(s) 104 and is configured to store instructions for execution of steps of the method 200 by the processor(s) or one or more hardware processors 104.
  • the steps of the method 200 of the present disclosure will now be explained with reference to the components or blocks of the system 100 as depicted in FIG. 1, the steps of flow diagram as depicted in FIG. 2 A through FIG. 2B, the example system of FIG. 3 and functional block of the system 100 for predictive analytics as in FIG. 4.
  • process steps, method steps, techniques or the like may be described in a sequential order, such processes, methods and techniques may be configured to work in alternate orders. In other words, any sequence or order of steps that may be described does not necessarily indicate a requirement that the steps to be performed in that order.
  • the steps of processes described herein may be performed in any order practical. Further, some steps may be performed simultaneously.
  • the one or more hardware processors 104 are configured to receive input data comprising camera data and sensor data from the multimodal sensing unit 110.
  • the input data provides information of one or more turbid media present in a Region of Interest (ROI), which represents a harsh environment.
  • ROI Region of Interest
  • the harsh environment may for example a coal mining region, oil refineries, areas of underwater sea exploration for energy sources and the like.
  • An illustrative example ROI in FIG. 3, depicts the coal mining region.
  • the ROI spread across several kilometers, is shown to have steam flows rising above the ground at multiple locations and presence of workers within the harsh environment. The presence of workers is implicitly depicted by movement of load carrying vehicles.
  • a control unit at the periphery of the ROI includes the system 100 with the associated multimodal sensing unit 110.
  • the entire ROI is monitored by the system 100 via the multimodal sensing unit 110 comprising a plurality of multimodal cameras, which include a combination of portable and moving visual cameras and thermal cameras.
  • the multimodal sensing unit 110 comprising a plurality of multimodal cameras, which include a combination of portable and moving visual cameras and thermal cameras.
  • the visual cameras include fixed cameras along the periphery of the ROI (such as visual cameras VC1 and VC2, and thermal cameras TCI and TC2) providing long distance surveillance. Further, depicted are vehicle mounted portable cameras such as visual camera VC3 and thermal camera TC3). As can be understood any appropriate number of multimodal cameras may be used to cover the entire ROI with corresponding field of view of the cameras covering a plurality of subregions within the ROI. Thus, the multimodal cameras are positioned to capture and stream the camera data comprising visual image frames and thermal image frames. The camera data captured by the multimodal cameras provides multiple views of the plurality of subregions comprising local views and global views.
  • the multimodal sensing unit 110 includes a plurality of multimodal sensors positioned to capture the sensor data comprising a plurality of parameters associated with the plurality of subregions.
  • the sensors include doppler radio detection and ranging (radars) such as DR1 and DR2 positioned along the periphery of the ROI to provide long distance surveillance.
  • Doppler radars which can operate for longer distances, essentially provide the distance information of any location of interest within the ROI. For example, a source location of steam flow identified by the system 100, location of the moving vehicles mounted with the cameras and the like.
  • additional sensors such as pressure sensors may be placed across the ROI, which enables the system to capture data providing pressure at various locations across the ROI.
  • sensors such as Geo phones (installed deep inside earth) to measure the seismic response of earth and Light Detection and Ranging (LIDAR) for sensing the 3D information of the scene (sub-region of the ROI) under surveillance can be deployed.
  • LIDAR Light Detection and Ranging
  • RADAR Radio Detection and Ranging
  • sensors used by the system 100 may include using Geographic Information System (GIS) and/or Global Positioning System (GPS) data from satellite for identifying the exact location of the outbursts.
  • GIS Geographic Information System
  • GPS Global Positioning System
  • the one or more hardware processors 104 are configured to preprocessing the camera data and the sensor data to eliminate noise.
  • the camera data is preprocessed using a video stabilization approach to eliminate noise induced in the camera data due to undesired motion of the plurality of multimodal cameras. Any motion captured by images of the camera data, which does not relate a motion due to turbid region flow such as steam flow is termed as undesired motion.
  • the undesired motion may be induced due to dynamic camera motion, say due to strong winds, dynamic object motion, and turbulent motion from the features corresponding to the events, such as outbursts.
  • the motion induced by the cameras is global.
  • the video stabilization approach used herein estimates the global motion between successive frames, this estimated motion can be compensated using known image processing techniques.
  • the motion induced due to turbulent atmosphere is local and the video stabilization approach utilizes non-rigid registration approach to compensate the local motion induced by the turbulent atmosphere.
  • noise in the sensor data Similar to camera data, there is presence of noise in the sensor data as well.
  • the system 100 removes the noise before the sensor data is integrated along with the visual camera features (camera data).
  • noise elimination is critical to extract true or accurate data for processing by the system 100. Thus, any noise induced in the sensor data is eliminated, by using techniques specific to the sensors been used.
  • the one or more hardware processors 104 are configured to derive a flow vector information from the visual image frames and the thermal image frames to extract temporal features, of the turbid regions of the one or more turbid media within every image frame. Deriving the flow vector information comprises steps of :
  • the temporal features derived from the flow vector information include average velocity, volume, area, frequency moments, opacity and the like, which are time varying features.
  • the flow vector information may be obtained by utilizing flow cameras, which further reduces computation related to extraction of flow vector information from visual or thermal camera data.
  • the one or more hardware processors 104 are configured to process the sensor data and the temporal features extracted from the flow vector information to identify a plurality of turbid regions (such as steam regions of FIG.
  • the plurality of turbid regions corresponding to the turbid media of interest are identified based on a set of features uniquely defining the turbid media of interest.
  • the set of features refers to specific values of the temporal features such as the average velocity, the volume, area, the frequency moments, the opacity, specific to the turbid media of interest.
  • the one or more hardware processors 104 are configured to cluster the plurality of turbid regions into a plurality of clusters based on the temporal features and a thermal profile obtained from the thermal image frames. All the steam regions or steam flow are clustered together based on this temporal signature, the thermal profile as well as distance between each of the steam flows or steam regions. All the extracted features can be used for clustering. Distance measures can be used as the criterion for clustering.
  • X and Y represent the set of features extracted for two identified flows then the likelihood of both of them belonging to the same cluster will be high if the distance of Y from X is less.
  • other features like identified source locations and other sensors info like pressure, depth at which the identified turbid media is can be used as a features and proceeded for further clustering.
  • Each cluster among the plurality of clusters comprises one or more turbid regions among the plurality of turbid regions.
  • different clusters are identified by the system 100 since a single source of steam can lead to multiple steam clusters.
  • the clusters can be tracked separately over-time.
  • the source of each of these clusters need to be tracked to identify which of the clusters are generated from the same origin or source.
  • the amount of steam and characteristics or features of the steam at every time instance can be aggregated to calculate the outburst (event).
  • the aggregate steam and its characteristics are used for predicting future outbursts.
  • the one or more hardware processors 104 are configured to fit a convex hull to each turbid region of each cluster to determine a plurality of minimum points of the convex hull fitted to each cluster. Each minimum point is identified by corresponding location coordinates.
  • the disconnected steam regions are also combined using the clustering approach.
  • the one or more hardware processors 104 are configured to group the plurality of minimum points in accordance to corresponding location coordinates to identify a plurality of source locations of a plurality of turbid activities corresponding to the turbid media of interest.
  • Each source location among the plurality of source locations corresponds to one or more minimum points, among the plurality minimum points, which have mapping location coordinates.
  • mapping location coordinates As depicted in FIG. 3, two minimum points of two convex hull fittings to two separate cluster map on each other, indicating a single source location of steam flow. There always exists a possibility that the steam flow(s) or the flow of turbid region at the source location may be discontinuous.
  • stabilizing of an unstable source location, among the plurality of source locations is performed using a moving window approach.
  • the moving window approach used by the system 100 comprises the following steps:
  • the coordinates of the source locations may vary as the minimum of convex hull of each frame is identified as the source location.
  • the actual source location is computed as the weighted average of the source locations at time instant t and as well as source locations identified in the past of corresponding steam flows.
  • the set of frames within a certain duration is termed as window of frames. Such a weighted average always stabilizes the steam flow source location.
  • the one or more hardware processors 104 are configured to monitor and detect temporal variations occurring in the plurality of turbid activities corresponding to the plurality of source locations by performing frame by frame analysis of the temporal features extracted from flow vector information and corresponding thermal profile of ROI obtained from the thermal cameras.
  • the one or more hardware processors 104 are configured to predict one or more source locations among the plurality of source locations as target locations for occurrence of events (for example, sudden outburst of geyser activity in the coal mining region), based on a prediction model.
  • the prediction model utilizes a clustered historical data the plurality of source locations and the detected temporal variations for prediction.
  • the prediction is based on not only features extracted from the current analyzed camera data and the sensor data but also based on a historical data.
  • consideration of historical data adds to accuracy of prediction. For example, after identifying the various sources of steam flow, and if the past data of all those identified source locations is available, then the prediction performed by the system 100 is much more efficient in predicting an outburst well before. The reason being the source locations which had outbursts in the past are more likely to experience the outburst again in the current scenario as well.
  • the predictive analytics approach used by prediction model is explained in conjunction with FIG. 4.
  • the clustered historical data is generated by clustering a historical data corresponding to the ROI.
  • the historical data comprises time series data providing variations of temperature profile in past and temporal signatures of past events. Clustering of the historical data is high order clustering and comprises extracting the temporal signatures of past events from the time series data and clustering multiple turbid media in a single cluster based on the temporal signature of the past events and distance between each turbid
  • this block raises alarm.
  • the source location (target location) of potential event is obtained by a calibrated camera set up along with range information, which is coming through data feed to map the pixel location of the event into the real world location accurately.
  • the system can generate alarms to alert the people working on the ground in the ROI, where there are high probable chances of events such as possible geyser outburst.
  • FIG. 4 is a functional block diagram of the system of FIG. 1 depicting prediction model providing pattern mining for predictive analytics for event prediction in the ROI, in accordance with some embodiments of the present disclosure.
  • an image extraction block of the system 100 receives CCTV feeds or the camera data received from visual cameras and the thermal cameras of the multimodal sensing unit 110 , providing video footage of the one or more turbid media present in the ROI.
  • the image extraction block extracts frames from the received stream of camera data and converts the frames to the appropriate format required for further processing. For example, any format that is supported by opencv libraries for reading videos such as mat format or mp4 format or avi format may be used.
  • the camera data provides a spatio-temporal data of the ROI, A day/night classification block further classifies each frame into day frame or night frame based on light conditions identified in the frames.
  • a filtering block filters high quality images or frames from the classified frames and associates corresponding time stamps to the filtered images or frames.
  • the time stamping provides the temporal information while images provide the spatial information of the turbid media in the ROI.
  • a metadata extraction block extracts other data associated with the images or frames such as size, resolution, day /night information and the like along with the time stamp.
  • a metadata persistence block stores all the extracted metadata along in a database such as the database 108.
  • the extracted images or frames from the camera data are processed by a feature extraction block.
  • This block extracts features, for example around 25 features from the images. These features are a combination of texture based gray level co-occurrence matrix (GLCM) features, frequency domain based feature, motion based feature and haze relevant features and the like.
  • the extracted features are processed by a mapping block, which maps the extracted features with the timestamp, such as date time information at which the data was captured. This helps in forming a spatio- temporal relationship for the extracted features.
  • a qualification block performs qualification and shortlisting of relevant features for pattern mining, by feature reduction method such as Principal Component Analysis (PCA). There is always a chance that the features captured can have correlation with each other.
  • PCA Principal Component Analysis
  • the right features to be used for predictive analysis further can be identified. For example, if the feature Y is highly correlated with X then Y is not adding any extra information that X cannot represent about the turbid media of interest. Thus, removing the redundancy is beneficial for improving the speed of predictive analysis.
  • These qualified relevant features can be visualized via a viewing block through time series graphs.
  • the qualified features are stored in a time series database in the database 108, proving feature persistence.
  • pattern mining is performed on the qualified features.
  • multiple reference signal depicting specific pattern of the turbid media are selected. There reference signals are compared with the features extracted from the images to output the correlation strength. Based on this similarity index similar pattern from the images of the camera data are extracted.
  • Similarity measurement is performed by find correlation between two time series signal.
  • LCSS Local Common Subsequence
  • Each of the features extracted from the camera data is converted to a time series signal and is then compared with the corresponding feature from the reference data.
  • a time series correlation measure signal is generated.
  • a distance measure such as Mahanalobis distance, combines the correlation results corresponding to each feature and is converted to a probability measuring the similarity between the reference data and the data extracted from the live feed.
  • An automatic segregation/clustering block clusters the images using the similarity measure scores. The probability resulting from the similarity measure tells the strength of the similarity between the reference data and the data extracted from the live feed.
  • the image steams (video clips) having high similarity are then extracted and saved to the database. Further, a pattern discovery block combines multiple video clips corresponding to a particular reference signal and classifies as a pattern. The information from these clips are then used for predictive analysis. A modelled template of behavior is matched with CCTV feed and similar patterns are unearthed. [0041] Thus, the system and method disclosed here provides more accurate prediction of events using the historical data and the current data.
  • the hardware device can be any kind of device which can be programmed including e.g. any kind of computer like a server or a personal computer, or the like, or any combination thereof.
  • the device may also include means which could be e.g. hardware means like e.g. an application-specific integrated circuit (ASIC), a field- programmable gate array (FPGA), or a combination of hardware and software means, e.g.
  • ASIC application-specific integrated circuit
  • FPGA field- programmable gate array
  • the means can include both hardware means, and software means.
  • the method embodiments described herein could be implemented in hardware and software.
  • the device may also include software means.
  • the embodiments may be implemented on different hardware devices, e.g. using a plurality of CPUs.
  • the embodiments herein can comprise hardware and software elements.
  • the embodiments that are implemented in software include but are not limited to, firmware, resident software, microcode, etc.
  • the functions performed by various components described herein may be implemented in other components or combinations of other components.
  • a computer-usable or computer readable medium can be any apparatus that can comprise, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device.
  • a computer-readable storage medium refers to any type of physical memory on which information or data readable by a processor may be stored.
  • a computer-readable storage medium may store instructions for execution by one or more processors, including instructions for causing the processor(s) to perform steps or stages consistent with the embodiments described herein.
  • the term “computer-readable medium” should be understood to include tangible items and exclude carrier waves and transient signals, i.e., be non-transitory. Examples include random access memory (RAM), read-only memory (ROM), volatile memory, nonvolatile memory, hard drives, CD ROMs, DVDs, flash drives, disks, and any other known physical storage media.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Image Analysis (AREA)

Abstract

L'invention concerne des systèmes de l'état de la technique utilisant des sources d'énergie active comme des lasers pour capturer des milieux troubles. L'identification de milieux d'intérêt troubles en présence d'autres milieux troubles n'est pas traitée. La prédiction d'événements, tels que des salves de sortie, dans une région d'intérêt (ROI) est obtenue par le recours unique à des données actuelles d'une activité trouble. Le procédé et le système de surveillance de milieux d'intérêt troubles permettant de prédire des événements utilisent des caractéristiques spatio-temporelles extraites à partir des images capturées par des caméras visuelles en fusion avec des caractéristiques extraites à partir d'autres capteurs et caméras de différentes modalités pour identifier correctement des régions troubles. Un ensemble de paramètres définissant des paramètres pertinents pour le milieu d'intérêt trouble permet au système d'identifier correctement le milieu d'intérêt trouble à partir d'une pluralité de milieux troubles présents dans la ROI. Le système prédit avec précision des régions de salves potentielles par l'utilisation de données historiques (données de séries chronologiques fournissant des variations de profil de température et des signatures temporelles d'événements passés) ainsi que de données actuelles.
PCT/IB2021/050725 2020-01-30 2021-01-29 Procédé et système de surveillance de milieux d'intérêt troubles destinés à prédire des événements WO2021152534A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
IN202021004160 2020-01-30
IN202021004160 2020-01-30

Publications (1)

Publication Number Publication Date
WO2021152534A1 true WO2021152534A1 (fr) 2021-08-05

Family

ID=77079574

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2021/050725 WO2021152534A1 (fr) 2020-01-30 2021-01-29 Procédé et système de surveillance de milieux d'intérêt troubles destinés à prédire des événements

Country Status (1)

Country Link
WO (1) WO2021152534A1 (fr)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8124931B2 (en) * 2007-08-10 2012-02-28 Schlumberger Technology Corporation Method and apparatus for oil spill detection
US9476730B2 (en) * 2014-03-18 2016-10-25 Sri International Real-time system for multi-modal 3D geospatial mapping, object recognition, scene annotation and analytics
US20160356665A1 (en) * 2015-06-02 2016-12-08 Umm Al-Qura University Pipeline monitoring systems and methods

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8124931B2 (en) * 2007-08-10 2012-02-28 Schlumberger Technology Corporation Method and apparatus for oil spill detection
US9476730B2 (en) * 2014-03-18 2016-10-25 Sri International Real-time system for multi-modal 3D geospatial mapping, object recognition, scene annotation and analytics
US20160356665A1 (en) * 2015-06-02 2016-12-08 Umm Al-Qura University Pipeline monitoring systems and methods

Similar Documents

Publication Publication Date Title
KR101995107B1 (ko) 딥 러닝을 이용한 인공지능 기반 영상 감시 방법 및 시스템
US20210264217A1 (en) Systems and methods for automatic estimation of object characteristics from digital images
US20220319183A1 (en) System for tracking and visualizing objects and a method therefor
US8958602B1 (en) System for tracking maritime domain targets from full motion video
EP3593324B1 (fr) Détection et cartographie de cible
US11145090B2 (en) Flame finding with automated image analysis
US9292939B2 (en) Information processing system, information processing method and program
US20160165191A1 (en) Time-of-approach rule
US20190019266A1 (en) Method and system for processing image data
US11210529B2 (en) Automated surveillance system and method therefor
CN110796682A (zh) 运动目标的检测识别方法及检测识别系统
CN112673377A (zh) 监测设备以及用于人员落水监测的方法
Refaai et al. An enhanced drone technology for detecting the human object in the dense areas using a deep learning model
Bloisi et al. Integrated visual information for maritime surveillance
WO2021152534A1 (fr) Procédé et système de surveillance de milieux d'intérêt troubles destinés à prédire des événements
CN111753587A (zh) 一种倒地检测方法及装置
US10549853B2 (en) Apparatus, system, and method for determining an object's location in image video data
US20220174240A1 (en) Global tracking system and cloud system thereof
Govada et al. Road deformation detection
SR OBJECT DETECTION, TRACKING AND BEHAVIOURAL ANALYSIS FOR STATIC AND MOVING BACKGROUND.
Karishma et al. Artificial Intelligence in Video Surveillance
Lin et al. Accurate coverage summarization of UAV videos
Mishra et al. Real time landslide monitoring and estimation of its movement velocity
San Miguel et al. A flood detection and warning system based on video content analysis
Sekhar et al. Vehicle Tracking and Speed Estimation Using Deep Sort

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21747146

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21747146

Country of ref document: EP

Kind code of ref document: A1