WO2020227985A1

WO2020227985A1 - Real-time fault detection on network devices and circuits based on traffic volume statistics

Info

Publication number: WO2020227985A1
Application number: PCT/CN2019/087086
Authority: WO
Inventors: Shi ZHAO; Yuehua LIN; Hui Xu; Duncheng SHE; Miao Wang; Hui Liu
Original assignee: Alibaba Group Holding Ltd
Current assignee: Alibaba Group Holding Ltd
Priority date: 2019-05-15
Filing date: 2019-05-15
Publication date: 2020-11-19
Anticipated expiration: 2021-11-15
Also published as: CN113454950A; CN113454950B

Abstract

Systems and methods of detecting abnormity in network switch devices or network circuits in real-time based on traffic volume statistics. In device fault detection, a statistic empirical model of net traffic volume is constructed according to baseline dataset, where each datum corresponds to the net traffic volume accumulated in each interval. In circuit fault detection, a statistic empirical model of link traffic distribution is constructed according to baseline dataset, where each datum corresponds to the link traffic distribution in each interval. For either case, after the initial construction, the model is dynamically updated with qualified and selected new data. Each new datum is evaluated against the updated model to determine whether it is an outlier. Consecutive outliers may trigger a fault alarm.

Description

REAL-TIME FAULT DETECTION ON NETWORK DEVICES AND CIRCUITS BASED ON TRAFFIC VOLUME STATISTICS

TECHNICAL FIELD

Embodiments of the present disclosure relate generally to the field of communication networks, and more specifically, to the field of network fault detection mechanisms.

BACKGROUND OF THE INVENTION

A communication network is composed of links and nodes arranged in a certain topology for transporting Internet traffic. The nodes include network devices interconnected through links, such as servers, switches and routers. Today’s fault detection in commercial networks usually relies on user-defined alarm and rule violations based on measured indexes, which requires detailed knowledge of the characteristics and performance of the hardware and software constituents in a network infrastructure. For a single device or a simple network, fault detection is typically well understood and easy to accomplish.

However, in recent years, network infrastructures have become increasingly complicated due to the exponentially growing number of network devices and links, numerous manufactures of the devices, various software versions running on the devices, and the multiple hierarchies of switches adopted in an infrastructure. Therefore, it is almost impossible to set fault detection rules that can encompass all the possible faults in fast response. The complexity of such a network further comes from some unobservable interactions between devices. For example, two devices are not connected directly but have an indirect path that links them. As a result, it is greatly challenging to define effective rules that can result in fast and reliable fault detection. In addition, the static user-defined rules for fault detection can be quickly outdated as new devices or new software versions are introduced.

SUMMARY OF THE INVENTION

Embodiments of the present disclosure are directed to systems and methods of real-time network fault detection in which traffic volume abnormity of network components are discovered by using dynamic statistics of traffic volume data and yet without the necessity of recognizing the detailed characteristics and operations of the monitored components.

In one aspect, embodiments of the present disclosure provide a mechanism of detecting abnormity in individual switch devices by periodically evaluating net traffic volume of a device against dynamically-updated statistics Net traffic volume corresponds to the difference between ingress and egress traffic volume of a device.

Particularly, for a network device, a statistic empirical model of net traffic volume may be constructed according to a machine learning process. In some embodiments, the model is initially established by using net traffic volume data collected on the device in a plurality of intervals, e.g., consecutive intervals. For example, each datum in baseline dataset of the model corresponds to the net traffic volume accumulated in each interval. For example the model includes a function of a mean value and a standard deviation value of net traffic volume per interval. After the initial model construction, for each interval, it is determined whether a new datum of net traffic volume is qualified and selected to update the model. If yes, the new datum replaces the earliest datum in the dataset and the model is recalculated. Regardless of whether it is used to update the model, the new datum is evaluated against the updated model to determine whether it is an outlier. In responsive to detection of a prescribed number of consecutive outliers, an alarm is generated, which may rigger further automatic or manual diagnostic, troubleshooting and repair actions.

Generally, a circuit includes a group of parallel links sharing the traffic load between two sides of the circuit, each side including one or more devices. The links are functionally equal and the total traffic volume between the two sides can be distributed across the links in stable proportions absent faults. If one link faults, the other links can automatically take over the traffic load that cannot be accomplished by the fault link, and therefore the traffic distribution among the links (or “link traffic distribution” herein) changes. In another aspect of the present disclosure, embodiments provide a mechanism of detecting circuit abnormity by periodically comparing the real-time link traffic distribution in the circuit with a dynamically-updated statistic empirical model. In some embodiments, the model includes an expected link traffic distribution.

An expected link traffic distribution can be obtained by averaging a baseline dataset of link traffic distribution data that are collected in a plurality of intervals, e.g., consecutive intervals. For example, each datum in the dataset corresponds to a set of proportions of traffic that the links assume respectively in one interval. After the initial construction of the model, for each interval, it is determined whether a new datum of link traffic distribution is qualified and selected to update the baseline dataset. If yes, the new datum replaces the earliest datum in the baseline dataset to update the expected link traffic distributions. Regardless of whether it is used to update the model, the new datum is evaluated against the expected distributions to determine whether it is an outlier. In responsive to detection of a prescribed number of consecutive outliers, an alarm is generated, which may rigger further automatic or manual diagnostic, troubleshooting and repair actions.

According to embodiments of the present disclosure, as the net traffic volume or the link traffic distribution is continuously monitored and evaluated in real-time by using simple statistical processing, network abnormity in a device or a circuit can be advantageously captured in fast responses regardless of complexity of the network infrastructures. Since the monitored statistics can be derived simply from traffic volume data, fault detection can be advantageously achieved by using readily-available data in a device or a circuit and an empirical model without requiring comprehensive knowledge of its detailed characteristics and operations.

In addition, because they are updated frequently with collected new data, a statistic model can reflect the most recent data probability distributions which can advantageously enhance effectiveness and accuracy of fault detection. Furthermore, as a model of a device or circuit can be constructed and updated by using the actual empirical data of the monitored device or circuit itself, the model is tailored to the characteristics and operations of the device or circuit. This further contributes to fault detection accuracy.

The foregoing is a summary and thus contains, by necessity, simplifications, generalizations, and omissions of detail; consequently, those skilled in the art will appreciate that the summary is illustrative only and is not intended to be in any way limiting. Other aspects, inventive features, and advantages of the present invention, as defined solely by the claims, will become apparent in the non-limiting detailed description set forth below.

BRIEF DESCRIPTION OF THE DRAWINGS

Embodiments of the present invention will be better understood from a reading of the following detailed description, taken in conjunction with the accompanying figures, in which like reference characters designate like elements.

Fig. 1 illustrates an exemplary communication network with fault detection equipment capable of detecting device faults and circuit faults in real-time based on statistics of traffic volume data in accordance with an embodiment of the present disclosure.

Fig. 2 is a flow chart depicting an exemplary computer implemented process of real-time device fault detection based on traffic volume statistics in accordance with an embodiment of the present disclosure.

Fig. 3 is a flow chart depicting an exemplary computer implemented process of statistic model construction and corresponding fault detection for a device in accordance with an embodiment of the present disclosure.

Fig. 4 illustrates changes in link traffic distribution of an exemplary circuit following a link fault.

Fig. 5 is a flow chart depicting an exemplary computer implemented process of real-time circuit fault detection based on traffic volume statistics in accordance with an embodiment of the present disclosure.

Fig. 6 is a flow chart depicting an exemplary computer implemented process of statistic model construction and corresponding fault detection for a circuit in accordance with an embodiment of the present disclosure.

Fig. 7 is a block diagram illustrating an exemplary computing system capable of real-time device fault detection and link detection based on traffic volume statistics in accordance with an embodiment of the present disclosure.

DETAILED DESCRIPTION

Reference will now be made in detail to the preferred embodiments of the present invention, examples of which are illustrated in the accompanying drawings. While the invention will be described in conjunction with the preferred embodiments, it will be understood that they are not intended to limit the invention to these embodiments. On the contrary, the invention is intended to cover alternatives, modifications, and equivalents which may be included within the spirit and scope of the invention as defined by the appended claims. Furthermore, in the following detailed description of embodiments of the present invention, numerous specific details are set forth in order to provide a thorough understanding of the present invention. However, it will be recognized by one of ordinary skill in the art that the present invention may be practiced without these specific details. In other instances, well-known methods, procedures, components, and circuits have not been described in detail so as not to unnecessarily obscure aspects of the embodiments of the present invention. Although a method may be depicted as a sequence of numbered steps for clarity, the numbering does not necessarily dictate the order of the steps. It should be understood that some of the steps may be skipped, performed in parallel, or performed without the requirement of maintaining a strict order of sequence. The drawings showing embodiments of the invention are semi-diagrammatic and not to scale and, particularly, some of the dimensions are for the clarity of presentation and are shown exaggerated in the Figures. Similarly, although the views in the drawings for the ease of description generally show similar orientations, this depiction in the Figures is arbitrary for the most part. Generally, the invention can be operated in any orientation.

NOTATION AND NOMENCLATURE:

It should be borne in mind, however, that all of these and similar terms are to be associated with the appropriate physical quantities and are merely convenient labels applied to these quantities. Unless specifically stated otherwise as apparent from the following discussions, it is appreciated that throughout the present invention, discussions utilizing terms such as "collecting" "constructing" "processing" or "calculating" or "executing" or "storing" or the like, refer to the action and processes of a computer system, or similar electronic computing device, that manipulates and transforms data represented as physical (electronic) quantities within the computer system's registers and memories and other computer readable media into other data similarly represented as physical quantities within the computer system memories or registers or other such information storage, transmission or display devices. When a component appears in several embodiments, the use of the same reference numeral signifies that the component is the same component as illustrated in the original embodiment.

Embodiments of the present disclosure provide mechanisms of detecting network device or circuit faults based on real-time traffic volume data as well as the data statistics. For a network device, an empirical statistic model can be constructed by using a baseline dataset collected in a plurality of intervals, where the model is representative of a probability distribution of net traffic volume of the device per interval. The model may include a set of statistic metrics or a function thereof, for example the metrics being a mean value and a standard deviation. After the initial model construction, a new datum of net traffic volume of each interval is evaluated against the model to determine whether the datum of the interval is an outlier. Consecutive occurrences of outliers may trigger an alarm of fault. If qualified, the new datum can be randomly selected to update the baseline model. In this manner, the model is updated with the most recent normal data and therefore can accurately reflect the current characteristics and operations of the device.

For a circuit, an empirical statistic model is constructed by using a baseline dataset collected in a plurality of intervals, where the model is representative of the probability distribution of link traffic distribution per interval in the circuit. The model may correspond to an expected link traffic distribution. After the initial model construction, for each interval, a new datum including a set of link traffic volumes or a link traffic distribution is evaluated against the model to determine whether the datum of the interval is an outlier. Consecutive occurrences of outliers may trigger an alarm of fault. If qualified, the new datum can be randomly selected to update the baseline model. In this manner, the model is updated with the most recent normal data and therefore can accurately reflect the current characteristics and operations of the device.

Fig. 1 illustrates an exemplary communication network 100 with

fault detection equipment

121 and 122 capable of detecting device faults and circuit faults in real-time based on statistics of traffic volume data in accordance with an embodiment of the present disclosure. In a simplified form, the network 100 includes a plurality of network switches (e.g., routers) interconnected and arranged in multiple layers and each switch is configured to forward network traffic. The switches belong to a network infrastructure controlled by the Internet Service Provider 110. Terminals (e.g., 131) are coupled to the switches and may be server devices or client devices. It will be appreciated that the present disclosure is not limited to any specific types of network topology or switch devices.

Each switch may be configured to collect traffic volume data of various forms, for example in compliance with the Simple Network Management Protocol (SNMP) . According to embodiments of the present disclosure, the real-time traffic volume data can be used to construct a dynamically-updated statistic model for fault detection on devices and circuits in real-time. In the illustrated embodiments, the model construction and fault detection functions may be implemented in a separate monitoring device (e.g., device 141 or 142) that is coupled to the monitored (e.g., switch 121 or 122) . However, in some other embodiments, the fault detection functions may be integrated in the switches.

As illustrated, during operation, the switch 122 collects its ingress and egress traffic volume periodically for supply to the monitoring device 142. It is assumed that a substantial variation in net traffic volume in a short time may be indicative of abnormity or fault in switch, where the net traffic volume corresponds to a difference between the total ingress traffic volume and the total egress traffic volume. The monitoring device 142 constructs a statistic empirical model of the net traffic volume based on a baseline dataset supplied from the switch 122. The model indicates a probability distribution of per-interval net traffic volume, according to which a normal zone and outlier zones are defined. In some embodiments, the model include as simple as a mean value and a standard deviation of the baseline dataset. However, the present disclosure is not limited to any specific statistic metrics, functions, algorithms or formula related to net traffic volume used in the statistic model. For each interval, the new datum of net traffic volume is evaluated against the model to determine if it falls in an outlier zone. In addition, a qualified new datum may be selected to update model. If consecutive outliers are detected, an alarm may be generated to trigger following manual or automatic fault diagnosis actions.

Further, it is assumed that in a circuit, a substantial change in link traffic distribution in a short time may be indicative of abnormity or fault in links. As illustrated, the

switches

121 and 123 and the several links 151 in between are configured as a circuit. The traffic between the

switches

121 and 123 is proportioned across the links 151 in a set of certain ratios. The switch 123 collects its total ingress or egress traffic volume of each link periodically for supply to the monitoring device 141. The monitoring device 141 constructs a statistic empirical model based on a baseline dataset supplied from the switch 123. The model may indicate an expected link traffic distribution of the circuit. A normal zone and outlier zones are defined based on the model. The present disclosure is not limited to any specific statistic metrics, functions, algorithms or formula related to link traffic volumes used in the statistic model. For each interval, a new datum includes a set of link traffic volumes or a current link traffic distribution, which is evaluated against the expected distribution to determine if the new datum falls in an outlier zone. In addition, a qualified new datum may be selected to update model. If consecutive outliers are detected, an alarm may be generated to trigger following manual or automatic fault diagnosis actions.

According to embodiments of the present disclosure, as net traffic volume or link traffic distribution is continuously monitored and evaluated in real-time by using simple statistical processing, network abnormity in a device or a circuit can be advantageously captured in fast responses despite complexity of the network infrastructures. Since the monitored statistic metrics can be derived from traffic volumes, fault detection can be advantageously achieved by using readily-available data in a device or a circuit and an empirical model without requiring comprehensive knowledge of its detailed characteristics, performance and operations.

In addition, because they are updated frequently with collected new data, the models reflect the most recent data probability distributions which can advantageously enhance effectiveness and accuracy of fault detection. Furthermore, as a model is constructed and updated by using the empirical data collected from the particular device or circuit, the model is still tailored to the monitored device or circuit. This further contributes to fault detection accuracy.

Fig. 2 is a flow chart depicting an exemplary computer implemented process 200 of real-time device fault detection based on traffic volume statistics in accordance with an embodiment of the present disclosure. Process 200 may be performed by a monitoring device that is communicatively coupled to a monitored switch device, or by a monitoring module integrated in a monitored switch device. At 201, a statistic empirical model of net traffic volume per interval is generated based on an initial baseline dataset. For example, the dataset includes net traffic volume data of N consecutive intervals, e.g., each interval being 1 minute and N being 2000. The particular numbers herein are merely exemplary and the present disclosure is not limited thereto. The interval duration and the sample size can be selected by considering factors like data collection noise caused by various engineering limitations, statistic properties of traffic distribution, and adequate representation of the probability distribution.

Each datum in the dataset is a per-interval net traffic volume which corresponds to a difference between the total ingress traffic volume and the total egress traffic volume accumulated in one interval. The ingress and egress total traffic volumes may be sums of traffic across all the ingress ports and egress ports of the device, respectively. The ingress and egress traffic volume data may be collected in real-time at the monitored device and supplied to the monitoring device or the monitoring module for fault detection purposes.

Based on the probability distribution of the net traffic volume data in the N intervals, a normal zone and one or more outlier zones are defined. In one example, the real-life net traffic volume data may follow a normal probability distribution; however, the present disclosure is not limited thereto. In some embodiments, the statistic model involves a mean value and a standard deviation of the baseline dataset, and the outlier zones and the normal zone can be defined by a function of the mean value and the standard deviation, as described in greater detail with reference to Fig. 3 below.

At 202, net traffic volume data of the device is generated periodically, e.g., per minute, in the same manner of generating the baseline dataset at 201. At 203, the statistic model is updated in real-time with new net traffic volume data while maintaining the data count in the baseline dataset. At 204, each new net traffic volume datum is evaluated against the updated statistic model to determine whether it is in an outlier zone thereof. At 205, if M consecutive outliers occur, an alarm is generated which may trigger various further operations such as fault verification, diagnostic operations, and etc. For example, M is predefined as 3.

Fig. 3 is a flow chart depicting an exemplary computer implemented process 300 of statistic model construction and corresponding fault detection for a device in accordance with an embodiment of the present disclosure. At 301, the interval index “i” is set to 1. At 302, a net traffic volume datum Di of the interval Ti is determined based on detected real-time ingress and egress traffic volumes accumulated in the interval. At 303, it is determined whether Di is qualified as a baseline datum for the statistic empirical model. In some embodiments, a datum is qualified on the conditions that (1) both the total ingress volume and the total egress volume in the interval are greater than a particular number, e.g., 1 Mbit per sec (BPS) ; and (2) the previous datum (i-1) is a normal one, as described below. However, various other qualification conditions can be used without departing from the scope of the present disclosure. If it is not a qualified datum, the index i is incremented at 311 to evaluate the next datum.

For a qualified datum, it is determined whether to add it to the baseline dataset of the statistic model. Particularly, at 304, it is determined whether the current baseline dataset has less than 2000 counts. If yes, at 305, the new datum Di is added to the baseline dataset for initial construction of the statistic model, e.g., obtaining the mean value and the standard deviation of the dataset. In some embodiments, the mean value (m) is calculated as

mean = average (log (D1) , …log (Di) , …, log (DN) ) , where N equals 2000, and the standard deviation (sd) is calculated as

sd = sd (log (D1) , …log (Di) , …, log (DN) ) .

It will be appreciated that the various other forms or equations of mean or standard deviation, or other statistic metrics can be used without departing from the scope of the present disclosure. Once the model is updated with Di at 305, the index i is incremented at 311 to evaluate the next datum.

If the current baseline dataset has reached 2000 (as determined at 304) , Di is directly, it is further determined whether Di is an outlier at 306. For example, it is defined that Di is an outlier if (Di-mean) /sd >3. If Di is not an outlier, Di is incorporated to the baseline dataset and replaces the earliest datum in the dataset at 307; and the mean and standard deviation of net traffic volume are updated accordingly at 305. Once the model is updated with Di at 305, the index i is incremented at 311 to evaluate the next datum.

If the current baseline dataset has not reached 2000 and Di is an outlier (as determined at 306) , it is recorded so at 308. At 309, it is further determined whether Di is the 3 ^rd outlier detected in a row. If yes, it means that there have been 3 consecutive outliers and a fault alarm is generated at 310. At 310, the index i is incremented. The foregoing process 302～312 is repeated per interval.

In some other embodiments, Di may be randomly selected to according to a prescribed chance, e.g., 50%. If Di is selected, the earliest datum in the dataset is replaced with Di and thereby the statistic model is updated. For example, Di is incorporated in recalculating the mean value and the standard deviation. If the current baseline dataset has reached 2000, Di is added to the baseline dataset without replacing any datum and used to recalculate the mean value and the standard deviation.

In a basic form, a circuit is composed of a first side A and a second side B as well as several parallel links that are functionally equivalent and can share the traffic load between A and B. Each side has ingress and egress traffic. Any of the traffic volumes A_in, A_out, B_in and B_out can be used to characterize the circuit for fault detection purposes according to embodiments of the present disclosure. The examples described in detail herein may refer to the traffic volume of any combination of side and direction.

When one link faults, its link traffic volume likely decreases significantly and the total traffic between A and B are automatically redistributed across the links. Therefore, a substantial change in link traffic distribution is indicative of a link fault. Fig. 4 illustrates changes in link traffic distribution of an exemplary circuit following a link fault. As shown, in a normal operation condition, the 4 links 401-404 respectively assume 20%, 30%, 40%and 10%of the total traffic volume, e.g., the traffic flowing into\side A. When link 401 faults, its proportion drops to 0%, while the rest become 40%, 40%and 20%.

Fig. 5 is a flow chart depicting an exemplary computer implemented process 500 of real-time circuit fault detection based on traffic volume statistics in accordance with an embodiment of the present disclosure. Process 500 may be performed by a monitoring device that is communicatively coupled to the switches in a monitored circuit, or by a monitoring module in a monitored circuit. At 501, a statistic empirical model representative of link traffic distribution is generated based on an initial baseline dataset. For example, the dataset includes link traffic distribution data of N intervals, e.g., each interval being 1 minute and N being 100. The particular numbers are merely exemplary and the present disclosure is not limited thereto. The interval duration and the sample size can be selected by considering factors like data collection noise caused by various engineering limitations, statistic properties of traffic distribution, and adequate representation of the probability distribution.

Each datum in the dataset corresponds to the respective traffic proportions assumed by all the links in a certain direction (either ingress or egress) on one side of the circuit. The traffic volume data of each link may be collected in each interval and supplied to the monitoring device or the monitoring module for fault detection purposes. The model may correspond to an expected link traffic distribution, which includes a set of expected link traffic proportions. In some embodiments, an expected proportion of a link may be obtained by averaging the traffic proportions of the link over the baseline dataset. A normal zone and one or more outlier zones can be defined as a function of the expected link traffic distribution.

At 502, the traffic volume data of the links are collected and the link traffic distribution data are generated periodically, e.g., per minute, in the same manner of generating the baseline dataset at 501. At 503, the statistic model is updated in real-time with new link traffic distribution data while maintaining the data count in the baseline dataset. At 504, each new link traffic distribution datum is evaluated against the updated statistic model to determine whether it is in an outlier zone thereof. At 505, if M consecutive outliers occur, an alarm is generated which may trigger various further operations such as fault verification, diagnostic operations, and etc. For example, M is predefined as 3.

Fig. 6 is a flow chart depicting an exemplary computer implemented process 600 of statistic model construction and corresponding fault detection for a circuit in accordance with an embodiment of the present disclosure. At 601, the interval index “i” is set to 1. At 602, a link traffic distribution datum Ai of the interval Ti is determined based on detected real-time traffic volumes accumulated in the interval. For example, Ai includes the ingress traffic volumes on side A of all the links in one interval, Ai = (V _{1_i}, V _{2_i}, V _{3_i}, V _{4_i}) . In some other embodiments, Ai may include the ingress link traffic proportions derived from the link traffic volumes or any other suitable variant of link traffic volumes representative of link traffic distribution.

At 603, it is determined whether Ai is a qualified datum, e.g., whether the number of functional links that can provide effective traffic volume data has changed in the last 3 consecutive intervals. If yes, a fault alarm is generated at 604.

For a qualified datum, it is then determined whether to add it to the baseline dataset of the statistic model. Particularly, at 605, it is determined whether the current baseline dataset has less than 100 counts. If yes, the new datum Ai is added to the baseline dataset for initial construction of the statistic model, e.g., obtaining the expected link traffic distribution based on the dataset. At 614, the index i is incremented.

In some embodiments, the expected distribution corresponds to the average distribution over the baseline dataset. It will be appreciated that the various other forms of average or other statistic metrics can be used without departing from the scope of the present disclosure.

If the baseline dataset has reached 100, the distance between the current link traffic distribution and the expected link traffic distribution is evaluated at 607, and the result is then used to judge whether Ai is an outlier at 608. For example, it is defined that Ai is an outlier if

where V _{j_i} fis the ingress traffic volume of link j in interval i; V _all is the total ingress traffic volume across all the links;

is the expected proportion of traffic volume of link j in each interval according to the model; X is a prescribed threshold value.

If Ai is an outlier, it is recorded so at 610. At 611, it is further determined whether Ai is the 3 ^rd outlier detected in a row. If yes, it means that there have been 3 consecutive outliers and a fault alarm is generated at 612. At 614, the index i is incremented. If Ai is not an outlier, the earliest datum in the dataset is replaced with Ai and thereby the statistic model is updated at 606. For example, Ai is incorporated in recalculating the expected link traffic distribution. At 614, the index i is then incremented. The foregoing process 602～614 is repeated per interval.

In some other embodiments, Ai is randomly selected according to a prescribed chance at 606, e.g., 10%. If Ai is selected, the earliest datum in the dataset is replaced with Ai and thereby the statistic is updated.

Fig. 7 is a block diagram illustrating an exemplary computing system 700 capable of real-time device fault detection and link detection based on traffic volume statistics in accordance with an embodiment of the present disclosure. The computing system comprises a main processor (CPU) 701, a system memory 702, a graphics processing unit (GPU) 703, I/O interfaces 704 and network circuits 705, an operating system 706 and application software 710 including real-time fault detection modules 720 and 730 stored in the memory 702. The system 700 is communicatively coupled to a switch device through the network interfaces.

When incorporating traffic volume data from the switch device 750 and executed by the CPU 701, the device fault detection module 720 can detect device faults in real-time based on the traffic volume statistics as described in greater detail with reference to Figs. 1-3. The device fault detection module 720 includes a net traffic data generation module 721, a baseline dataset module 722, a statistic model module 722 and a device fault processing module 724.

The net traffic data generation module 721 is configured to calculate the difference of ingress and egress traffic volumes of the switch device 750 per interval. The baseline dataset module 722 that maintains a fixed count of baseline dataset by selectively admitting qualified new data while removing the earliest ones. The statistic model module 723 can calculate the mean and standard deviation of the baseline dataset, and update these statistic metrics each time the baseline dataset is updated with a new datum. The device fault processing module 724 can determine whether a new datum is an outlier based on the model, generate alarms in responsive to detection consecutive outliers, and perform various other operations of fault detection, verification and diagnosis.

When incorporating traffic volume data from the switch device 750 (or any other type of devices in a circuit) and executed by the CPU 701, the link fault detection module 720 can detect link faults in real-time based on the traffic volume statistics as described in greater detail with reference to Figs. 4-6. The link fault detection module 730 includes a link traffic distribution generation module 731, a baseline dataset module 732, a statistic model module 733 and a link fault processing module 724.

The link traffic distribution generation module 731 is configured to calculate the link traffic proportions in the circuit per interval. The baseline dataset module 732 that maintains a fixed count of baseline dataset by selectively admitting qualified new data while removing the earliest ones. The statistic model module 733 can calculate the expected link traffic distribution, and update the expected distribution when the baseline dataset is updated with a new datum. The link fault processing module 734 can determine whether a new datum is an outlier based on the model, generate alarms in responsive to detection consecutive outliers, and various other operations of fault detection, verification and diagnosis.

As will be appreciated by those with ordinary skills in the art, the fault detection modules 720 and 730 can be implemented in any one or more suitable programming languages that are known to those skilled in the art. In some embodiments, a system includes only one of the fault detection modules 720 and 730.

Although certain preferred embodiments and methods have been disclosed herein, it will be apparent from the foregoing disclosure to those skilled in the art that variations and modifications of such embodiments and methods may be made without departing from the spirit and scope of the invention. It is intended that the invention shall be limited only to the extent required by the appended claims and the rules and principles of applicable law.

Claims

A method of real-time fault detection for network switch devices, the method comprises:

determining a net traffic datum for each interval of a plurality of intervals;

dynamically updating a set of net traffic statistics related to net traffic volume per interval for the switch device based on the net traffic datum;

designating the net traffic datum as an outlier based on the set of net traffic statistics and a prescribed threshold; and

generating a fault alarm responsive to a prescribed number of net traffic data being designated as outliers.
The method of Claim 1, wherein the ingress traffic volume corresponds to a sum of ingress traffic volumes across all ingress ports of the switch device, and wherein the egress traffic volume corresponds to a sum of egress traffic volumes across all egress ports of the switch device.
The method of Claim 1, wherein the set of net traffic statistics comprises an average and a standard deviation of net traffic data over a baseline dataset, and wherein the designating comprises designating the net traffic datum as an outlier based on a distance between the net traffic datum and the average and further based on a ratio between the distance over the standard deviation.
The method of Claim 1 further comprising, determining the set of net traffic statistics of a baseline dataset, wherein the baseline data set comprise net traffic data of a first plurality of consecutive intervals.
The method of Claim 4, wherein the updating comprises:

updating the baseline dataset by adding the net traffic datum of the interval to the baseline dataset and removing the earliest net traffic datum from the baseline dataset; and

recalculating the set of net traffic statistics based on an updated baseline dataset.
The method of Claim 5, wherein the updating the baseline dataset comprises determining that the net traffic datum is a qualified datum for the updating based on a condition that: a preceding net traffic datum is not designated as an outlier; and the ingress traffic and the egress traffic are greater than prescribed thresholds.
A method of real-time fault detection of a network circuit group, the method comprises:

collecting real-time traffic of the network circuit group in each interval of a plurality of intervals, wherein the network circuit group comprises: a first end; a second end; and a plurality of links that are functionally equivalent, wherein the real-time traffic of the interval comprises:

respective link traffic volumes of the plurality of links from the first end to the second end; and

a circuit traffic volume of the network circuit group from the first end to the second end;

dynamically updating an expected link traffic distribution for the network circuit group based on the real-time traffic of the interval;

evaluating the real-time link traffic of the interval against the expected link traffic distribution;

designating the real-time traffic of the interval as an outlier based on the deviation; and

generating a fault alarm responsive to real-time traffic of a prescribed number of intervals being designated as outliers.
The method of Claim 7, further comprising determining a real-time link traffic distribution of the plurality of links that corresponds to ratios of respective link traffic volumes over the circuit traffic volume of the interval, and wherein the evaluating comprises evaluating the real-time link traffic distribution against the expected link traffic distribution.
The method of Claim 7, wherein the evaluating comprises evaluating a distance between a traffic volume proportion of each link and an expected traffic volume proportion of the link.
The method of Claim 7 further comprising, determining the expected link traffic distribution based on a baseline dataset, wherein the updating comprises:

updating the baseline dataset by adding the real-time traffic of the interval to the baseline dataset and removing the earliest real-time traffic from the baseline dataset; and

recalculating the expected link traffic distribution based on an updated baseline dataset.
The method of Claim 10, wherein the updating the baseline dataset comprises:

determining that the real-time traffic of the interval is a qualified datum for updating the baseline dataset; and

randomly selecting the real-time traffic of the interval for updating the baseline dataset according to a selection probability.
The method of Claim 10, wherein the updating the baseline dataset comprises determining that the real-time traffic of the interval is a qualified datum for updating the baseline dataset based on a condition that the number of links in the plurality of links that provide effective real-time traffic remain unchanged from the preceding interval.
A system comprising:

a processor; and

a memory coupled o the processor and storing instructions that, when executed by the professor, implement a method of fault detection of the switch device, wherein the method comprises:

determining a net traffic datum for each interval of a plurality of intervals;

dynamically updating a set of net traffic statistics related to net traffic volume per interval for the switch device based on the net traffic datum;

designating the net traffic datum as an outlier based on the set of net traffic statistics and a prescribed threshold; and

generating a fault alarm responsive to a prescribed number of net traffic data being designated as outliers.
The system of Claim 13, wherein the ingress traffic volume corresponds to a sum of ingress traffic volumes across all ingress ports of the switch device, and wherein the egress traffic volume corresponds to a sum of egress traffic volumes across all egress ports of the switch device.
The system of Claim 13, wherein the set of net traffic statistics comprises an average and a standard deviation of net traffic data over a baseline dataset, and wherein the designating comprises designating the net traffic datum as an outlier based on a distance between the net traffic datum and the average and further based on a ratio between the distance over the standard deviation.
The system of Claim 13, wherein the method further comprises, determining the set of net traffic statistics of a baseline dataset, wherein the baseline data set comprise net traffic data of a first plurality of consecutive intervals.
The system of Claim 16, wherein the updating comprises:

updating the baseline dataset by adding the net traffic datum of the interval to the baseline dataset and removing the earliest net traffic datum from the baseline dataset; and

recalculating the set of net traffic statistics based on an updated baseline dataset.
The system of Claim 17, wherein the updating the baseline dataset comprises:

determining that the net traffic datum is a qualified datum; and

randomly selecting the net traffic datum for updating the baseline dataset according to a selection probability.
The system of Claim 17, wherein the updating the baseline dataset comprises: determining that the net traffic datum is a qualified datum based on a condition that: a preceding net traffic datum is not designated as an outlier; and the ingress traffic and the egress traffic are greater than prescribed thresholds.