CN114365505A

CN114365505A - Data-driven object graph for data center monitoring

Info

Publication number: CN114365505A
Application number: CN201980099754.8A
Authority: CN
Inventors: 理栈; 张�浩; 任志星; 王加龙
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2019-11-07
Filing date: 2019-11-07
Publication date: 2022-04-15
Also published as: WO2021087896A1

Abstract

Techniques are disclosed for building and using sensor transaction graphs to correlate sensors based on past events. In one embodiment, a disclosed method includes retrieving raw sensor data collected by a plurality of sensors; identifying a plurality of events based on the raw sensor data; constructing a sensor graph based on the plurality of events, the sensor graph having nodes representing the plurality of sensors and a set of edges connecting the nodes, each edge in the set of edges being associated with a weight calculated based on a number of related events detected in the plurality of events; and querying the sensor map in response to a new event associated with a sensor of the plurality of sensors, the querying including identifying at least one relevant sensor connected to the sensor of the sensor map.

Description

Data-driven object graph for data center monitoring

Copyright notice

This application contains material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent disclosure, as it appears in the patent and trademark office files or records, but otherwise reserves all copyright rights whatsoever.

Background

The disclosed embodiments relate to management of sensors, and more particularly, to techniques for managing sensor data for an Internet Data Center (IDC).

Modern IDCs typically employ one or more monitoring systems for monitoring the output from the various sensors. The data generated by these sensors includes a large amount of raw time series data. In current systems, a human technician or automated system needs to know the information behind these data, especially when there are real-time conditions that require intervention. For example, an increase in temperature sensor readings may be caused by a fan failure of a Computer Room Air Handler (CRAH) or a sudden drop in a weight placed on a server rack.

Current systems do not adequately reveal useful information from the large amounts of raw data generated by such sensors. In fact, many such systems rely on human interpretation of events and trial and error techniques to resolve abnormal sensor readings.

Disclosure of Invention

The disclosed embodiments address these and other problems in IDCs. The disclosed embodiments enable safer and more efficient operation of such IDCs by employing a graph of things (GoT) that can model sensors and events.

In one embodiment, a method is disclosed that includes retrieving raw sensor data collected by a plurality of sensors; identifying a plurality of events based on the raw sensor data; constructing a sensor graph based on the plurality of events, the sensor graph having nodes representing the plurality of sensors and a set of edges connecting the nodes, each edge in the set of edges being associated with a weight calculated based on the number of related events detected in the plurality of events; and querying a sensor map corresponding to a new event associated with a sensor of the plurality of sensors, the querying including identifying at least one relevant sensor connected to the sensor of the sensor map.

In another embodiment, a non-transitory computer readable storage medium for tangibly storing computer program instructions executable by a computer processor, the computer program instructions defining the steps of: retrieving raw sensor data collected by a plurality of sensors; identifying a plurality of events based on the raw sensor data; constructing a sensor graph based on the plurality of events, the sensor graph having nodes representing the plurality of sensors and a set of edges connecting the nodes, each edge in the set of edges being associated with a weight calculated based on the number of related events detected in the plurality of events; and querying a sensor map in response to a new event associated with a sensor of the plurality of sensors, the querying including identifying at least one relevant sensor connected to the sensor of the sensor map.

In another embodiment, an apparatus is disclosed, comprising: a processor; and a storage medium for tangibly storing program logic thereon for execution by the processor, the stored program logic causing the processor to perform the following: the method includes retrieving raw sensor data collected by a plurality of sensors, identifying a plurality of event data based on the raw sensors, constructing a sensor graph based on the plurality of events, the sensor graph having nodes representing the plurality of sensors and a set of edges connecting the nodes, each edge in the set of edges being associated with a weight calculated based on a number of related events detected in the plurality of events, and querying the sensor graph in response to a new event associated with a sensor in the plurality of sensors, the querying including identifying at least one related sensor connected to the sensor in the sensor graph.

Drawings

The foregoing and other objects, features and advantages of the disclosure will be apparent from the following description of embodiments, as illustrated in the accompanying drawings in which reference characters refer to the same parts throughout the different views. The drawings are not necessarily to scale, emphasis instead being placed upon illustrating the principles of the disclosure.

Fig. 1 is a block diagram of a system for generating and analyzing GoT according to some embodiments of the present disclosure.

Fig. 2 is a flow diagram illustrating a method for generating GoT according to some embodiments of the present disclosure.

Fig. 3A through 3F illustrate event diagrams and methods of constructing the same according to some embodiments of the present disclosure.

Fig. 4 illustrates a sensor transaction diagram according to some embodiments of the present disclosure.

Fig. 5A is a flow chart illustrating a method for determining whether a peak or dip event has occurred, according to some embodiments of the present disclosure.

Fig. 5B is a flow chart illustrating a method for determining whether a mean shift event has occurred in accordance with some embodiments of the present disclosure.

Fig. 5C is a flow chart illustrating a method for determining whether a positive or negative trend change event has occurred, according to some embodiments of the present disclosure.

Fig. 5D is a flow diagram illustrating a method for determining whether a variance change event has occurred in accordance with some embodiments of the present disclosure.

Fig. 6 is a flow chart illustrating a method for analyzing GoT according to some embodiments of the present disclosure.

Fig. 7 is a hardware diagram illustrating a device for generating or analyzing GoT according to some embodiments of the present disclosure.

Detailed Description

In the illustrated embodiment, the system (100) includes a plurality of sensors (102a, 102b, 102c, 102 n; collectively 102). Examples of sensors include devices such as temperature sensors, voltage sensors, current sensors, humidity sensors, airflow sensors, humidity or water sensors, smoke sensors, door sensors, video monitoring equipment, power consumption/switch sensors, and other sensors that measure real-world properties of a data center or other environment in which the system (100) is employed. Alternatively, in conjunction with the foregoing, the sensors may include software-based sensing components, such as server load monitoring sensors, log file monitoring sensors, intrusion detection systems, load peak detection software, and various other software for monitoring services provided by, for example, a data center.

A sensor (102) generates time series data. That is, each sensor generates data values at a given time and generates such data over a period of time. Examples of time series data include a continuous temperature monitor (i.e., a periodic temperature reported by a temperature sensor) and the number of inbound connections to the server (e.g., reported by the server) at any given time. In general, any data that can be represented as a function of time can be considered time-series data generated by a sensor (102), and the disclosed embodiments are not limited to a particular sensor or type or format of time-series data.

In the disclosed embodiment, the sensor (102) is primarily described in the context of an IDC, but the sensor (102) may be installed in any type of environment. Furthermore, there is no limitation on the number, type, or location of the sensors (102).

As shown, each of the sensors (102) is connected to a network (118). In one embodiment, each sensor (102) is physically coupled to a network (118) and the network (118) may include a Controller Area Network (CAN) bus, an ethernet, or other type of data communication medium. In other embodiments, the sensor (102) includes a wireless transceiver and may communicate over a wireless medium instead of a physical bus. Examples of such communication networks include Wireless Fidelity (Wi-Fi), cellular, or satellite networks. In some embodiments, the sensor (102) may be directly connected to other devices in the system (e.g., the monitoring system 116 or the pre-processor 104) and may not communicate over the network (118). In some embodiments, the system (100) may employ multiple networks (and directly connected devices). For example, a server employing load monitoring may communicate over an Ethernet network, while a temperature sensor may communicate over a Wi-Fi network. The disclosed embodiments are not limited to any one type (or combination) of networking technology that has been used.

In the illustrated embodiment, the data generated by the sensor (102) is referred to as raw data. The sensor transmits this data to a preprocessor (104) which forms the initial phase of the event extraction phase. This stage is designed to filter raw data into operational events and is described in more detail with respect to various flow diagrams. As shown, the event extraction stage includes a pre-processor (104), a time series data analysis processor (106), and an event detection processor (108). In some embodiments, each processor (104, 106, 108) includes dedicated hardware processing elements. In other embodiments, the event extraction phase may be implemented on one or more hardware devices, and the processors (104, 106, 108) may be implemented as software running on these devices.

In the illustrated embodiment, the preprocessor (104) receives raw data from the sensor (102) over the network (118). As previously described, this raw data includes data values and time values. In the illustrated embodiment, the pre-processor (104) cleans up and smoothes the received data. Details of this operation are provided in the description of fig. 2, particularly in the description of step 204.

After cleaning and smoothing the sensor data, the preprocessor (104) transmits the data to a time series data analysis processor (106). A time series data analysis processor (106) decomposes the cleaned up and smoothed time series data into trend, seasonal and residual components. Details of this operation are provided in the description of fig. 2, particularly in the description of step 206.

The time series data analysis processor (106) then transmits the decomposed components to an event detection processor (108). An event detection processor (108) processes the various components of the time series of data points and identifies actionable events represented by the components. The output of the event detection processor (108) includes a vector including the value of each event type, the time of the sensor data, and the identity of the sensor. The event detection processor (108) transmits this data to the event memory (110). Details of the operation of the event detection processor (108) are provided in the description of fig. 2, and in particular in step 208, and the description of fig. 5A, 5B, and 5C.

In the illustrated embodiment, the event store (110) includes a storage device (physical or logical) that stores events detected during the event fetch stage. In one embodiment, the event store (110) includes a relational database management system (RDBMS) or other type of database. In some embodiments, the event store (110) may include a key-value data store or other less dense data store (e.g., an object store). In some embodiments, the event memory (110) is configured to store events in a chronological order based on the time identified by the event detection processor (108). In this manner, the event memory (110) operates like a queue.

In the illustrated embodiment, a separate drawing phase is shown. In the mapping phase, a graph is constructed based on events stored in an event store (110) and periodically updated. In the illustrated embodiment, the graph builder (112) accesses the event store (110) to retrieve events. In some embodiments, the graph builder (112) actively queries the event store (110). In other embodiments, the graph builder (112) subscribes to the event store (106) and receives events periodically as they are added to the event store (110).

The graph builder (112) also accesses a graph memory (114). The map memory (114) may include a map database or may include an RDBMS. Typically, the graph memory (114) stores a set of nodes, a set of edges, and weights for the edges. In the illustrated embodiment, the node includes a sensor identifier, a point in time, and an event type. The graph builder (112) updates the data stored in the graph memory (114) based on the events received in the event memory (110). In this manner, the graph builder (112) is responsible for updating the event graph captured by the sensors (102) and processed during the event extraction phase. Details of the operation of graph builder (112) and graph memory (114) are provided in the description of fig. 2, and in particular in step 212 and the description of fig. 3A through 3F and 4.

As shown, the system (100) includes a monitoring system (116). In the illustrated embodiment, the monitoring system (116) receives processed event data from the event detection processor (108). Alternatively, in combination with the foregoing, the monitoring system (116) may also receive raw data from the sensors (102). The monitoring system (116) is also communicatively coupled to the graph memory (114). In the illustrated embodiment, the monitoring system (116) may include a hardware device (or devices) responsible for analyzing sensor data and providing operational intelligence to an operator of the device. In other embodiments, the monitoring system (116) may also automatically take action in response to an event (e.g., issue an alarm to a fire department upon detecting a temperature increase).

To provide such insight or take such action, the monitoring system (116) queries the map memory (114) to identify relevant sensors in response to the event and performs root cause analysis. Details of the operation of the monitoring system (116) are provided in the description of fig. 6.

Fig. 2 is a flow chart illustrating a method of generating GoT according to some embodiments of the present disclosure.

In step 202, the method (200) receives sensor data. As described above, the method (200) may receive raw data generated by one or more sensors over a network or other communication medium.

In step 204, the method (200) cleans up and smoothes the data.

In some embodiments, the method (200) performs various cleanup operations on the received data, such as filtering anomalies, removing anomalous sensors, smoothing short term fluctuations in sensor data, and other operations. In some embodiments, cleaning may also include interpolating missing values and/or normalized values. In some embodiments, the method (200) may smooth and clean the data using moving average smoothing. Examples of such moving average methods include Weighted Moving Average (WMA), Exponentially Weighted Moving Average (EWMA), autoregressive integrated moving average (ARIMA) smoothing, and the like. The use of other techniques in addition to (or instead of) these techniques and disclosed embodiments is not intended to limit the particular or single technique used to clean and smooth data from the sensors. Indeed, multiple techniques may be used simultaneously, and the technique for each sensor may be unique based on the type of time series data generated and observations regarding fluctuations in the sensor generated data.

In step 206, the method (200) decomposes the cleaned and smoothed data into trend, seasonal, and residual data.

In the illustrated embodiment, the time series data received in step 202 may be affected by three components: trend, seasonality, and residual components. In short, the seasonal component of a time series refers to a pattern that repeats over a fixed time interval. The trend component refers to the overall change in data over a longer period of time. Finally, residual refers to residual data remaining in the time series after removing trends and seasonal data.

In one embodiment, the time series may be represented by a function y (t), where t represents elapsed time. In this embodiment, y (t) may be represented as a series of vectors y (t) ═ y (1), y (2), … y (n). In one embodiment, the value of y (T) can thus be decomposed into an additive set of trend (T), seasonal (S), and residual (R) functions: y (t) ═ t (t) + s (t) + r (t), where t again denotes the elapsed time. An additive or multiplicative model may be selected based on the underlying sensors.

In one embodiment, the method (200) receives raw time series data and first detects a trend component. In one embodiment, the method (200) may use a centered moving average algorithm to calculate the trend t (t) of the time series y (t), but other algorithms (e.g., fourier transform) may also be used. The method (200) may then remove the trend t (t) from the time series y (t). The method (200) may then analyze the underlying detrended time series (y (t) -t (t)) to determine a period of the resulting data. After determining the period, the method (200) extracts a seasonal component s (t), which leaves a residual or noise component r (t).

In step 208, the method (200) extracts events from the decomposed data.

In one embodiment, the method (200) stores a list of predefined events. Generally, an event includes one or more conditions that utilize a base component (trend, seasonality, residue) as an input. That is, if one or more of the decomposed components satisfy a condition, then it is assumed that an event has occurred. The disclosed embodiments describe four examples of events: peak/dip, mean shift, trend change, and variance shift. However, the disclosed embodiments are not intended to be limited to only these events, other events may be constructed based on the underlying data or requirements of the system implementing the method (200).

Peak/fall refers to a rapid increase or decrease in the value of the remaining component of the time series. As mentioned above, trend refers to gradual changes in a time series, while seasonality refers to periodic (and regular) fluctuations. Thus, peaks or dips in the time series are typically attributable to the remaining components of the time series. The detection of the peak/dip is more fully described in the description of fig. 5A.

Mean shift events refer to the overall change in the moving average of the time series data. In this embodiment, a sliding window is used to calculate the average of the time series at a given time t, and this value is used to determine whether a mean shift has occurred. The detection of mean shift is more fully described in the description of fig. 5B.

A trend change event refers to the time at which the trend component of the time series reverses, from positive (up) to negative (down) or from negative (down) to positive (up). The detection of trend changes is more fully described in the description of FIG. 5C.

A variance shift event refers to a change in variance of the underlying time series data. The detection of variance deviation is more fully described in the description of fig. 5D.

In general, other features may be extracted from the time series, including, but not limited to, fluctuations in mean, variance, skewness, kurtosis, median, mode, quantile, and the like. Information measures such as entropy or autoregressive coefficients may also be included. The disclosed embodiments are not intended to be limited to the specific examples provided in fig. 5A through 5D.

In the illustrated embodiment, the method (200) generates an n-tuple representing the detected event. Thus, using the above three examples, the method (200) will generate a triplet, indicating whether or not an event exists in the time series. Thus, after performing step 208, the method (200) represents the events in the time series as y_j(t_i)＝[e₁(t₁)，e₂(t₁)，e₃(t₁)，e₄(t₁)]_jWherein e is₁、e₂、e₃And e₄Indicating the presence of a peak/dip, mean shift, trend change or variance change event (respectively),

indicates the result of the processing (as described in FIGS. 5A to 5D), j indicates the sensor, t_iRepresents a given time, and y_jRepresenting a time series of sensor j.

In the above manner, the entire time series can thus be defined as Y_j(t)＝[y_j(t_i)，j＝1…k，i＝1…m]＝[[e₁(t_i)，e₂(t_i)，e₃(t_i)]_j，j＝1…k，i＝1…m]Where k denotes k sensors, t_i…t_mRepresenting the time t from₁To t_mThe observation of (2).

In step 210, the method (200) stores the extracted event.

As described above, the output of step 208 includes a data packet that includes a value indicating the presence/absence of each type of event, a time component, and a sensor identifier (referred to as an event data structure). These components are stored in a database. In some embodiments, the event data structure may be stored in an RDBMS or other type of storage medium for future access. As described above, in some embodiments, the event data structure may be stored in a queue type data structure. In other embodiments, the event data structure may be stored in a database that supports a publish/subscribe (pub/sub) schema (e.g., REDIS) such that event streams may be subscribed to by downstream processing devices.

In step 212, the method (200) builds or updates a graph based on the extracted events.

In the illustrated embodiment, the method (200) may first construct an event graph representing events stored in a database. In some embodiments, the graph represents each event type, sensor, and time as nodes in the graph. In the graph, a given sensor node is connected to an event node or time node, while event nodes and time nodes are connected only to sensor nodes (i.e., not to other time nodes or event nodes). In some embodiments, the method (200) stores the map in a map database. In other embodiments, the graph may be converted to a relational form and stored in the RDBMS. An example of constructing a graph from a set of events is provided in the description of fig. 3A through 3F, which are incorporated herein by reference. In some embodiments, the event graph may comprise a separately stored graph. In other embodiments, the event graph may comprise a temporary graph (e.g., stored only in memory). In this embodiment, the event map is used to guide the (things) sensor map described below and in the description of FIG. 4.

From the event graphs, the method (200) further synthesizes a sensor graph (also referred to as a things-sensor graph or simply GoT). In this graph, the nodes contain only sensors, and the edges between the nodes are weighted based on the connectivity of the event graph. In the illustrated embodiment, the sensor map is constructed by analyzing the event map to identify relevant sensors. In one embodiment, sensors are related if they experience events at the same point in time. Thus, as one example, two sensors are correlated if they experience a peak or dip at the same time. In one embodiment, the weight of an edge of the sensor map represents the number of time points at which two sensors experience an event simultaneously.

Fig. 3A through 3F illustrate event graphs and methods of constructing the same according to some embodiments of the present disclosure.

In the illustrated embodiment, the graphs in fig. 3A through 3F may be generated using a set of events generated using historical sensor data. Alternatively, in combination with the foregoing, a graph may be generated using real-time events extracted from a real-time stream of raw sensor data.

In fig. 3A, a first event is analyzed, including a peak or a fall of the sensor 1 occurring at time 1, an event type (peak and fall), a sensor ID (1), and a time point (1) are used as nodes in the graph (300a) and the sensor node is connected to the event type node and the time point node through a non-directional edge. As shown in fig. 3A to 3F, all edges are undirected, and this will not be repeated for the sake of brevity.

In fig. 3B, also at time 1, a second event is processed in which the sensor 3 detects a peak or a drop. In response, the nodes of sensor 3 are added to the graph (300 b). The time 1 node is reused, and the sensor 3 node is connected to the time 1 node through an edge. In addition, the sensor 3 nodes are connected to the existing peak and drop nodes by edges. In some embodiments, new, repeated peaks and drop nodes may be added; however, as shown, event nodes (e.g., time nodes) may be reused.

In fig. 3C, a third event is received at time 2. This event comprises a mean shift type event detected by the sensor 2. In response, a new component is added to the graph (300c), the component including a new sensor 2 node, the sensor 2 node connected to a new time 2 node and a new mean shift node. This is illustrated in fig. 3C, where the graph (300C) consists of two separate components.

In FIG. 3D, a fourth event is received that occurs at time 2. Since the time 2 node has already been created, the node is reused. However, sensor 4 nodes are added as well as new trend change nodes corresponding to the detected trend change events. As will be discussed in more detail, the graph (300D) depicted in FIG. 3D begins to show the correlation between sensors.

In fig. 3E, a fifth event is received at time 3. As shown in fig. 3D, a new time 3 node is added to the graph (300 e). In addition, the sensor 1 node is connected to a new time 3 node. However, since the sensor 1 node is already connected to the peak and drop nodes, the connection is reused. As described above, in some embodiments, new repeated peaks and drop nodes may be added and connected to the sensor 1 node. Alternatively, in some embodiments, the weight of the edges between the sensor 1 node and the peak and down nodes may be increased by one.

In fig. 3F, the sixth (and last) event is received at time 3. Similar to the fifth event, the event in fig. 3F represents a peak and a fall detected at the sensor 3. In this case, all nodes are present (sensor 3, time 3, peak and dip), so no new node is created in the graph (300 f). However, as described above, the event type node may be repeated again. Thus, a new edge between the sensor 3 node and the time 3 node is added in fig. 3F, and the edges between the sensor 3 node and the peak and fall nodes are reused (or in some embodiments, weights are added).

The sensor map may be generated using the map in fig. 3F, as will be described in the following figures.

Fig. 4 illustrates a sensor transaction diagram according to some embodiments of the present disclosure. In the illustrated embodiment, the sensor map (400) is generated based on the event map (300F) shown in FIG. 3F. As shown in fig. 4, the sensor graph (400) differs from the event graph (300f) in that all nodes of the sensor graph (400) include sensor nodes. Furthermore, the edges between sensor nodes are undirected and include only numerical weights. Thus, data regarding the type of event and the time is deleted when the sensor map (400) is generated.

In one embodiment, each sensor node in the event graph (300f) is included in the sensor graph (400). Furthermore, each node may be connected to every other node with zero weight. Alternatively, each node in the sensor graph (400) may initially be unconnected and only connected to other nodes described herein.

In the illustrated embodiment, for a given sensor, the graph generator (as described in FIG. 1) analyzes the event graph (300f) to determine the number of paths between the given sensor and another sensor that is 2 apart and that passes through the time node. To accomplish this, the graph builder may select sensor nodes and identify connected time nodes. The graph builder may then identify, for each time node, any sensor nodes connected to the time node. The resulting sensor nodes are then counted and the count is used as a weight between the given sensor and another sensor. Thus, in graph (300f), sensor 1 is connected to sensor 3 through two time nodes (and vice versa), so the edge between sensor 1 and sensor 3 is added in the sensor graph (400) and the weight is set to 2. Furthermore, the sensors 2 and 4 are connected to each other via a time node (time 2). Therefore, the edge between sensors 2 and 4 is added in the sensor map (400), and the weight is set to 1. As shown, all other edges are set to zero weight or omitted entirely.

In step 502a, the method (500a) receives a remaining component r (t) comprising the remaining part of the given time series calculated as described above.

In step 504, the method (500a) compares the residual component to a peak threshold (Th)_spike) And a falling threshold (Th)_aip) And (6) comparing. In one embodiment, these thresholds include static values above and below the center point. E.g. Th_spikeMay be set to a positive value (e.g., +5), while Th_dipMay be set to a negative value (e.g., -5). The specific values of these thresholds are not intended to be limiting. In one embodiment, Th_spikeMust be greater than zero and Th_dipMust be less than zero.

In step 506a, the method (500a) determines that the value of r (t) exceeds a preconfigured peak threshold Th_spikeAnd marks the time (t) as a peak. In one embodiment, time-stamping the peak includes outputting the event (e.g., e)₁) The value is set to one (1). This value is then passed to and combined with the other outputs generated in fig. 5B through 5DAs described more fully in the description of step 208 of fig. 2.

In step 510a, the method (500a) determines that the value of r (t) is below a preconfigured drop threshold Th_dipAnd time (t) is marked as falling. In one embodiment, time-stamping as falling includes outputting an event (e.g., e)₁) The value is set to minus one (-1). This value is then passed on to and combined with the other outputs generated in fig. 5B through 5D, as described more fully in the description of step 208 of fig. 2.

In step 508a, if the value of R (t) is not greater than Th_spikeAnd is not less than Th_dipThe method (500a) bypasses peak/dip event detection and outputs an event (e.g., e)₁) The value is set to zero (0). Thus, in the illustrated embodiment, the method (500a) utilizes the following rule denoted as e₁The event generation output value of (1):

in some embodiments, the method (500a) may use output values other than-1, 0, and 1. For example, the method (500a) may set the output value to represent a distance of the value of r (t) from the corresponding threshold value.

In one embodiment, two thresholds (Th) are determined using an empirical formula of (t)_spikeAnd Th_dip) So that events are only with small probability

(can be adjusted and can be set to 1%) occurs. This amounts to respecting a priori knowledge, i.e. interesting events are rare. In most cases, the time series is "normal" and does not contain much information. In some embodiments, two thresholds (Th)_spikeAnd Th_dip) Having a threshold value (TH) greater than (>) 0_abs) This may represent a priori knowledge of a human expert. In this embodiment, TH_spikeThe settings were as follows:

TH_spike＝max(TH_spike，TH_abs)，TH_dip＝(TH_dip，-TH_abs)

In step 502b, the method (500b) defines a sliding window for the time series dataset. In some embodiments, the sliding window comprises a fixed duration in which the remaining steps analyze the time series data. The specific duration of this window is not limited and may be set according to the underlying data or needs of the system.

In one embodiment, the method (500b) calculates the mean of the trend components t (t) from two sliding windows and obtains the difference between them. In this embodiment, the left window consists of less recent data:

[T(t-L-R+1)，T(t-L-R+2)，…，T(t-R)]，

and the right window contains the latest data:

[T(t-R+1)，T(t-R+2)，…，T(t)]，

where L and R are the size of the left and right time windows, respectively.

In this example, μ_L(t)、μ_RThe value of (t) is determined by calculating the mean of t (t) in the left and right time windows, respectively, and the difference Δ μ (t) is μ ═ μ_K(t)-μ_L(t) is used as the average sensor time series over the window (denoted as a (t)).

In step 504b, the method (500b) averages the values of the time series of data points (i.e., the values of y (t)) that appear within the window. In some embodiments, the average comprises the average of y (t) over the window.

In step 506b, the method (500b) compares the calculated average for a given window (a) (t) with a mean increase threshold (Th)_increase) And a mean reduction threshold (Th)_decrease) And (6) comparing. In one embodiment, these thresholds include static values above and below the center point. E.g. Th_increaseMay be set to a positive value (e.g., +5), while Th_decreaseCan be set to a negative value (example)Such as, -5). The specific values of these thresholds are not intended to be limiting. However, Th_increaseShould be higher than Th_decreaseTo avoid Th_decrease＞a(t)＞Th_increaseThe case (1).

In step 508b, the method (500b) determines that the value of a (t) exceeds a preconfigured mean increase threshold Th_increaseAnd marks the time (t) as an increase in mean. In one embodiment, time-stamping the mean increase includes outputting the event (e.g., e)₂) The value is set to one (1). This value is then passed on to and combined with the other outputs generated in fig. 5A, 5C, and 5D, as described more fully in the description of step 208 of fig. 2.

In step 512b, the method (500b) determines that the value of a (t) is below a preconfigured mean reduction threshold Th_decreaseAnd time (t) is marked as the mean reduction of the sliding window. In one embodiment, time-stamping as mean reduction includes outputting an event (e.g., e)₂) The value is set to minus one (-1). This value is then passed on to and combined with the other outputs generated in fig. 5A, 5C, and 5D, as described more fully in the description of step 208 of fig. 2.

In step 510b, if the value of a (t) is not greater than Th_increaseAnd is not less than Th_decreaseThe method (500b) bypasses the average increase/decrease event detection and outputs an event (e.g., e)₂) The value is set to zero (0). Thus, in the illustrated embodiment, the method (500b) generates the output value of the event using the following rules, e.g., e₂Shown in the figure:

in some embodiments, the method (500b) may use output values other than-1, 0, and 1. For example, the method (500b) may set the output value to represent the distance of the value of a (t) from the corresponding threshold.

In step 502c, the method (500c) receives a trend component t (t) comprising a trend portion of a given time series calculated as described above.

In step 504c, the method (500c) calculates the time difference between the current trend value (T)) and the previous trend value (T-1)), and then calculates an exponential moving average of Δ T (T) ═ T (T) -T (T-1), which is denoted as d (T).

In step 506c, the method (500c) compares the trend component to a positive trend threshold (Th)_positive) And a negative trend threshold (Th)_negative) And (6) comparing. In one embodiment, these thresholds include static values above and below the center point. E.g. Th_positiveMay be set to a positive value (e.g., +5), while Th_negativeMay be set to a negative value (e.g., -5). The specific values of these thresholds are not intended to be limiting. However, Th_positiveShould be higher than Th_negativeTo avoid occurrence of Th_negative＞D(t)＞Th_positiveThe case (1).

In step 508c, the method (500c) determines that the value of d (t) exceeds a pre-configured positive trend threshold Th_positiveAnd marks time (t) as positive. In one embodiment, time-stamping as being includes outputting an event (e.g., e)₃) The value is set to one (1). This value is then passed on to and combined with the other outputs generated in fig. 5A, 5B, and 5D, as described more fully in the description of step 208 of fig. 2.

In step 512c, the method (500c) determines that the value of D (t) is below a preconfigured negative trend threshold Th_negativeAnd time (t) is marked as a negative value. In one embodiment, time-stamping a negative trend event includes outputting an event (e.g., e)₃) The value is set to minus one (-1). This value is then passed on to and combined with the other outputs generated in fig. 5A, 5B, and 5D, as described more fully in the description of step 208 of fig. 2.

In step 510c, if the value of D (t) is not greater than Th_positiveAnd is not less than Th_negativeThe method (500c) bypasses the positive/negative trending event detection and outputs an event (e.g., e)₃) The value is set to zero (0). Thus, in the illustrated embodiment, the method (500c) generates an output value for an event using the following rules, e.g., e₃Shown in the figure:

in some embodiments, the method (500c) may use output values other than-1, 0, and 1. For example, the method (500c) may set the output value to be a distance representing the value of d (t) from the corresponding threshold value.

In step 502d, the method (500d) receives the calculated standard deviation σ of the left and right sliding window residual components R (t) of the time series data, respectively_L(t) and σ_R(t) of (d). The left and right sliding windows may be calculated as described above and will not be described in detail herein.

In step 504d, the method (500c) calculates the standard deviation between the right and left standard deviations over time, denoted as Δ σ (t).

In step 506d, the method (500d) compares the standard deviation difference to a positive variance threshold (Th)_positive) And a negative variance threshold (Th)_negative) And (6) comparing. In one embodiment, these thresholds include static values above and below the center point. E.g. Th_positiveMay be set to a positive value (e.g., +5), while Th_negativeMay be set to a negative value (e.g., -5). The specific values of these thresholds are not intended to be limiting. However, Th_positiveShould be higher than Th_negativeTo avoid occurrence of Th_negative＞Δσ(t)＞Th_positiveThe case (1).

In step 508d, the method (500d) determines that the value of Δ σ (t) exceeds a pre-configured positive difference threshold Th_positiveAnd marks time (t) as positive. In one embodiment, time-stamping as being includes outputting an event (e.g., e)₄) The value is set to one (1). This value is then passed to and compared with the value generated in FIGS. 5A through 5CThe other outputs are combined as described more fully in the description of step 208 of fig. 2.

In step 512d, the method (500d) determines that the value of Δ σ (t) is below a preconfigured negative variance threshold Th_negativeAnd marks the time (t) as a negative number. In one embodiment, time-stamping a negative variance event includes outputting the event (e.g., e)₄) The value is set to minus one (one 1). This value is then passed on to and combined with other outputs generated in fig. 5A through 5C, as shown in fig. 5A through 5C, as described more fully in the description of step 208 of fig. 2.

In step 510d, if the value of Δ σ (t) is not greater than Th_positiveAnd is not less than Th_negativeThe method (500d) bypasses the positive/negative variance event detection and outputs an event (e.g., e)₄) The value is set to zero (0). Thus, in the illustrated embodiment, the method (500d) generates an output value for an event using the following rules, e.g., e₄Shown in the figure:

in some embodiments, the method (500d) may use output values other than-1, 0, and 1. For example, the method (500d) may set the output value to be a distance representing the value of Δ σ (t) from the corresponding threshold value.

In step 602, the method (600) monitors sensor data. In some embodiments, the method (600) monitors raw sensor data output accessed over a communication bus, as described in the description of fig. 1.

In step 604, the method (600) determines whether an event has occurred.

In one embodiment, the method (600) determines whether an event has occurred by utilizing the processes described in fig. 2, 5A, 5B, and 5C. In some embodiments, these events may be added to the event store (as described in FIG. 2) and processed by the method (600) at the same time. In the illustrated embodiment, the event detected in step 604 includes a sensor identifier (i) that represents the sensor associated with the event. In some embodiments, steps 602 and 604 may be optional. In this embodiment, the method (600) receives the pre-processed event directly from the event extraction stage depicted in fig. 1.

If the method (600) does not detect an event (e.g., no peak/dip, mean shift, trend change, or variance shift is detected), the method (600) continues to monitor the system for events. Alternatively, if the method (600) detects an event, the method (600) proceeds to step 608.

In step 606, the method (600) retrieves the sensor identifier (i) from the event detected in step 606. In some embodiments, the sensor identifier may include a globally unique identifier, an Internet Protocol (IP) address, or other unique identifying information.

In the illustrated embodiment, the method (600) queries a graph store that stores a sensor map using a sensor identifier. That is, the method (600) uses the sensor identifier as a query key to retrieve data from the map memory regarding the identified sensor. In one embodiment, the method (600) requests a first level connection list of nodes associated with a sensor identifier from a sensor graph.

In step 608, the method (600) determines whether the query returns an empty set; that is, if the node associated with the sensor identifier does not have any first level connections. As shown in FIG. 4, a first level connection means that another node in the graph that is connected to the node associated with the sensor identifier has a distance of 1.

In step 610, if the sensor does not have any first level connections, the method (600) flags the event for further analysis. In this scenario, the method (600) cannot identify any sensors that are associated with the sensor identifier extracted in step 606. Thus, when an event occurs (as identified in step 604), it cannot be associated with the sensor map and requires further intervention by, for example, an operator of the monitoring system.

In step 612, if the method (600) determines that one or more first level connections exist, the method (600) retrieves the first level connections from the sensor map. In some embodiments, steps 610 and 612 may be combined. In the illustrated embodiment, the first level connection includes a set of nodes associated with other sensors. Each of these nodes may include one or more events (and times of these events) that were previously recorded for the respective sensor. In some embodiments, further analysis or details may be included for each event.

In step 614, the method (600) analyzes the identified sensor nodes to determine if the same type of event is recorded for any sensor in the first level connection list. In some embodiments, the method (600) limits the analysis in step 614 to concurrent events. In other embodiments, the method (600) may limit the analysis to events that occur within a predefined time from the current event detected in step 604. Alternatively, in conjunction with the foregoing, the method (600) may analyze the shape of the trend of the time series data for each sensor to determine whether the trend is similar to the trend of the sensor identified by the sensor identifier.

In step 616, the method (600) sets a set of relevant sensors equal to those identified in step 614 as having the same or similar events or trends. In the illustrated embodiment, the method (600) stores the identifiers associated with the associated sensors, and then labels the associated sensors for further analysis (previously described in connection with step 610).

In

steps

618, 620, and 622, the method (600) continues to analyze the second level connections of the sensors identified in step 606. As used herein, a second level connection refers to a node in the sensor map that is 2a distance from the sensor identified in step 606.

Steps

618, 620, and 622 are similar to

steps

608, 612, and 614, and the details of these steps are not repeated here.

In general, steps 608, 612, and 614 (and 618, 620, and 622) may be performed for any level of connection of the sensors identified in step 606. That is, in addition to analyzing the first and second level connections (as shown), the method (600) may also analyze nodes that are three or more distant. In some embodiments, the number of levels to be analyzed may be preconfigured by the method (600). In other embodiments, the method (600) may continue searching each level until at least one relevant sensor node is found.

In the illustrated embodiment, the method (600) stops upon identifying at least one relevant sensor in the given level. In other embodiments, the method (600) may continue to analyze other levels even when relevant sensors are found at a given level. In this and other embodiments, the method (600) may rank the relevant sensors based on their distance from the sensor identified in step 606. For example, the relevant first level sensors (step 614) may be ranked higher than the relevant second level sensors (step 622).

The device (700) may include more or fewer components than those shown in fig. 1. The components shown, however, are sufficient to disclose an illustrative embodiment for practicing the present disclosure. The device (700) may represent, for example, the device discussed above in connection with fig. 1.

As shown in fig. 7, the device (700) includes a processing unit (CPU) (702) in communication with a mass storage (730) via a bus (724). The device (700) also includes one or more network interfaces (750), audio interfaces (752), displays (754), keyboards (756), illuminators (758), input/output interfaces (760), and cameras or other optical, thermal, or electromagnetic sensors (762). As understood by those skilled in the art, the device (700) may include one camera/sensor (762) or multiple cameras/sensors (762).

The device (700) may optionally communicate with a base station (not shown), or directly with another computing device. The network interface (750) includes circuitry for coupling the device (700) to one or more networks and is constructed for use with one or more communication protocols and techniques. The network interface (750) is sometimes referred to as a transceiver, transceiving device, or Network Interface Card (NIC).

The audio interface (752) is arranged to generate and receive an audio signal, such as the sound of a human voice. For example, the audio interface (752) may be coupled to a speaker and microphone (not shown) to enable telecommunication with others and generate audio acknowledgements for certain operations. The display (754) may be a Liquid Crystal Display (LCD), gas plasma, Light Emitting Diode (LED), or any other type of display used with computing devices. The display (754) may also include a touch sensitive screen arranged to receive input from an object, such as a stylus or finger from a human hand.

The keyboard (756) may comprise any input device arranged to receive input from a user. For example, the keypad (756) may include a button numeric pad or a keyboard. The keypad (756) may also include command buttons related to selecting and sending images. The illuminator (758) may provide a status indication and provide light. The luminaire (758) may remain active for a particular period of time or in response to an event. For example, when illuminator (758) is active, it may backlight the buttons on keyboard (756) and remain illuminated when the device is powered on. Further, the illuminator (758) may backlight the buttons in various patterns when a particular action is performed (e.g., dialing another device). The illuminator (758) may also cause a light source located within a transparent or translucent housing of the device to illuminate in response to the action.

The device (700) further comprises an input/output interface (760) for communicating with external devices. The input/output interface (760) may use one or more communication technologies, such as USB, infrared, Bluetooth^TMAnd the like.

The mass memory (730) includes RAM (732), ROM (724), and other storage devices. Mass memory (730) illustrates another example of computer storage media for storage of information such as computer readable instructions, data structures, program modules or other data. The mass storage (730) stores a basic input/output system (BIOS) (740) for controlling low-level operation of the device (700). The mass memory may also store an operating system for controlling the operation of the device (700). It will be appreciated that this component may include a general-purpose operating system, such as UNIX or LINUX^TMVersion, or special Client communication operating system, such as Windows Client^TMOr the saiban operating system. The operating system may include, or interface with, a Java virtual machine module that enables control of hardware components and operating system operations via Java application programs.

RAM (732) stores and executes one or more application programs (734). In some embodiments, these applications include software configured to perform one or more operations associated with the aforementioned figures. In some embodiments, the device (700) further includes persistent storage (e.g., hard disk, solid state drive, etc.) storage for storing the application (34) prior to execution in RAM (732).

For purposes of this disclosure, a module is a software, hardware, or firmware (or combination thereof) system, process, or function, or component thereof, that performs or facilitates the processes, features, and functions described herein (with or without human interaction or enhancement). The module may include sub-modules. Software components of the modules may be stored on computer-readable media for execution by the processor. The modules may be integrated into, loaded and executed by, one or more servers. One or more modules may be combined into an engine or application.

Those skilled in the art will recognize that the methods and systems of the present disclosure may be implemented in a variety of ways and, thus, are not limited by the foregoing exemplary embodiments and examples. In other words, functional elements performed by single or multiple components, hardware, and various combinations and subcombinations of software or firmware may be distributed among software applications at either the client-level or server-level, or both. In this regard, any number of the features of the different embodiments described herein may be combined into a single or multiple embodiments, and alternate embodiments having fewer than or more than all of the features described herein are possible.

The functionality may also be distributed, in whole or in part, among multiple components, in manners now known or to become known. Thus, myriad software/hardware/firmware combinations are possible in implementing the functions, features, interfaces, and preferences described herein. Moreover, the scope of the present disclosure encompasses conventionally known ways of performing the described features and functions and interfaces, as well as those variations and modifications that may be made to the hardware or software or firmware components described herein, as would be understood by those skilled in the art.

Furthermore, embodiments of the methods presented and described in this disclosure as flow diagrams are provided by way of example to provide a more complete understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed, and sub-operations described as part of larger operations are performed independently.

While various embodiments are described for purposes of this disclosure, such embodiments should not be considered to limit the teachings of this disclosure to those embodiments. Various changes and modifications may be made to the elements and operations described above to achieve results within the scope of the systems and processes described in this disclosure.

Claims

1. A method, comprising:

retrieving raw sensor data collected by a plurality of sensors;

identifying a plurality of events based on the raw sensor data; and

constructing a sensor graph based on the plurality of events, the sensor graph having nodes representing the plurality of sensors and a set of edges connecting the nodes, each edge in the set of edges being associated with a weight calculated based on a number of related events detected in the plurality of events.

2. The method of claim 1, characterized in that it further comprises:

querying the sensor map in response to a new event associated with a sensor of the plurality of sensors, the querying including identifying at least one relevant sensor connected to the sensor of the sensor map.

3. The method of claim 1, wherein said identifying a plurality of events comprises:

decomposing the raw sensor data into trend, seasonal, and residual components; and

the plurality of events are identified based on one or more of trends, seasonality, and remaining components.

4. The method of claim 3, wherein each event of the plurality of events comprises an event selected from the group consisting of a peak or fall event, a mean shift event, a variance shift, or a trend change event.

5. The method of claim 4, wherein said identifying a plurality of events comprises identifying a peak or a drop event by comparing said residual component to a peak threshold and a drop threshold.

6. The method of claim 4, wherein said identifying a plurality of events comprises identifying mean shift events by:

averaging the raw sensor data over a sliding time window; and

the average is compared to a mean increase and mean decrease threshold.

7. The method of claim 4, wherein identifying a plurality of events comprises identifying a trend change event by comparing the trend component to a positive trend or a negative trend threshold.

8. The method of claim 1, wherein said constructing a sensor map comprises:

constructing an event graph based on the plurality of events, the event graph storing event types, times, and sensor identifiers as nodes; and

constructing the sensor map based on the event map.

9. The method of claim 1, wherein said querying said sensor map comprises:

detecting a second event for a second sensor;

identifying a first level connection sensor in the sensor map for the second sensor; and

using the first level connection sensor as the correlation sensor.

10. A non-transitory computer readable storage medium tangibly storing computer program instructions executable by a computer processor, the computer program instructions defining the steps of:

retrieving raw sensor data collected by a plurality of sensors;

identifying a plurality of events based on the raw sensor data; and

constructing a sensor graph based on the plurality of events, the sensor graph having nodes representing the plurality of sensors and a set of edges connecting the nodes, each edge of the set of edges being associated with a weight calculated based on a number of related events detected in the plurality of events.

11. The non-transitory computer readable storage medium of claim 10 wherein the instructions further define the step of querying the sensor map in response to a new event associated with a sensor of the plurality of sensors, the querying including identifying at least one relevant sensor connected to the sensor of the sensor map.

12. The non-transitory computer readable storage medium of claim 10, wherein said identifying a plurality of events comprises:

13. The non-transitory computer readable storage medium of claim 11, wherein each event of the plurality of events comprises an event selected from the group consisting of a peak or fall event, a mean shift event, a variance shift, or a trend change event.

14. The non-transitory computer readable storage medium of claim 12, wherein the identifying a plurality of events comprises identifying a peak or a drop event by comparing the remaining components to a peak threshold and a drop threshold.

15. The non-transitory computer readable storage medium of claim 12, wherein identifying the plurality of events comprises identifying a mean shift event by:

averaging the raw sensor data over a sliding time window; and

the average is compared to a mean increase and mean decrease threshold.

16. The non-transitory computer readable storage medium of claim 12, wherein the identifying a plurality of events comprises identifying a trend change event by comparing the trend component to a positive trend or a negative trend threshold.

17. The non-transitory computer readable storage medium of claim 10, wherein said constructing a sensor map comprises:

constructing the sensor map based on the event map.

18. The non-transitory computer readable storage medium of claim 10, wherein querying the sensor graph comprises:

detecting a second event for a second sensor;

identifying a first level connection sensor in the sensor map for a second sensor; and

using the first level connection sensor as the correlation sensor.

19. An apparatus, comprising:

a processor; and

a storage medium for tangibly storing program logic thereon for execution by a processor, the stored program logic causing the processor to:

retrieving raw sensor data collected by a plurality of sensors,

identifying a plurality of events based on the raw sensor data, an

20. The apparatus of claim 19, wherein the operations further comprise querying the sensor map in response to a new event associated with a sensor of the plurality of sensors, the querying comprising identifying at least one relevant sensor connected to the sensor of the sensor map.

21. The apparatus of claim 19, wherein said identifying a plurality of events comprises:

a plurality of events are identified based on one or more of the trends, seasonality, and remaining components.

22. The apparatus of claim 19, wherein said constructing a sensor map comprises:

constructing the sensor map based on the event map.

23. The apparatus of claim 19, wherein querying the sensor map comprises:

detecting a second event for a second sensor;

using the first level connection sensor as a correlation sensor.