CN111740856B - Network communication equipment alarm acquisition abnormity early warning method based on abnormity detection algorithm - Google Patents
Network communication equipment alarm acquisition abnormity early warning method based on abnormity detection algorithm Download PDFInfo
- Publication number
- CN111740856B CN111740856B CN202010377031.4A CN202010377031A CN111740856B CN 111740856 B CN111740856 B CN 111740856B CN 202010377031 A CN202010377031 A CN 202010377031A CN 111740856 B CN111740856 B CN 111740856B
- Authority
- CN
- China
- Prior art keywords
- alarm
- aqmd
- data
- algorithm
- aqifg
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/24323—Tree-organised classifiers
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Abstract
The network communication equipment alarm acquisition abnormity early warning method based on the abnormity detection algorithm can improve early warning certainty, reduce false alarm while finding potential risks, timely find data abnormity and reduce important alarm loss risks through a construction algorithm from alarm quality metadata AQMD to an alarm quality isolated forest cluster AQIFG and an alarm AQMD timing detection program, thereby improving the reliability of a network, shortening the network fault recovery time and indirectly improving the user satisfaction.
Description
Technical Field
The invention relates to a communication equipment alarm management technology, in particular to a network communication equipment alarm acquisition abnormal early warning method based on an abnormal detection algorithm.
Background
Along with the rapid development of networks, communication networks are more and more complex and huge, alarms generated by equipment are more and more, and the reliability of acquiring alarms from equipment is required to be ensured for ensuring the abnormal and timely discovery of the equipment. The monitoring alarm acquisition quality is mainly achieved by monitoring alarm delay time, the existing scheme is mainly achieved by configuring a threshold value mode according to experience of operation and maintenance personnel, and when the fluctuation range exceeds the threshold value, the system generates alarm fluctuation abnormality notification to the operation and maintenance personnel for inspection. The flow is as follows: 1, carrying out manual statistics analysis and induction on historical alarms collected by each alarm data source, and obtaining an alarm delay abnormal experience value; 2, carrying out threshold configuration on each alarm data source according to the experience value, and taking the threshold configuration as an abnormal early warning basis; 3, counting alarm delay at regular time by the system, comparing and analyzing with a pre-configured threshold value, and generating an abnormal notification if the alarm delay exceeds the abnormal value; 4, after receiving the abnormal notification, the operation and maintenance personnel checks the alarm acquisition process to judge whether a problem exists, if so, the operation and maintenance personnel process the problem, and if not, the operation and maintenance personnel directly manually archive the alarm; and 5, periodically analyzing alarm history data and abnormal data, and adjusting a delay abnormal threshold.
At present, in the equipment alarm acquisition process, the early warning of equipment alarm acquisition abnormality is realized, and the problems and defects exist, and have the following points: 1. the equipment alarm quantity is larger, the alarm frequency is higher, and the abnormal fluctuation of alarm acquisition is difficult to identify in a large number of alarms. 2. The traditional alarm acquisition monitoring means often sets a time threshold for each device, but when each device is busy and idle, the thresholds set by different device loads often do not meet the actual conditions of the devices. When the threshold value is larger, the abnormal equipment alarm acquisition can not be found out in time; when the threshold value is smaller, false alarm is caused, so that the operation and maintenance personnel process the alarm and run the life. 3. The traditional alarm acquisition monitoring means is often to set a time threshold for each device, but the alarm quantity and the alarm frequency of each device are inconsistent, when the acquired device quantity is large, the general threshold often cannot correctly reflect the acquisition condition of the alarm of the device, the workload of independently configuring the threshold for each device is huge, and the configured threshold needs to be frequently corrected for the device condition.
Disclosure of Invention
Aiming at the defects or shortcomings in the prior art, the invention provides a network communication equipment alarm acquisition abnormity early warning method based on an abnormity detection algorithm, which can improve early warning certainty, reduce false alarm while finding potential risks, timely find out data abnormity, reduce important alarm loss risks through a construction algorithm from alarm quality metadata AQMD to an alarm quality isolated forest cluster AQIFG and an alarm AQMD timing detection program, thereby improving the reliability of a network, shortening the network fault recovery time and indirectly improving the user satisfaction.
The technical scheme of the invention is as follows:
the network communication equipment alarm acquisition abnormity early warning method based on the abnormity detection algorithm is characterized by comprising a construction algorithm from alarm quality metadata AQMD to an alarm quality isolated forest cluster AQIFG and an alarm AQMD timing detection program, wherein the structure of the AQMD is the following four-tuple: aqmd= (dev, eventID, eventTime, alarmInterval), where dev is the device object, eventID is the unique ID of the alarm, eventTime is the time of occurrence of the device alarm, alarmInterval is the interval between the occurrence of the last alarm and the current alarm; the AQIFG has the following triplet structure: AQIFG= (SampleBeginTime, sampleEndTime, AQ-IFore), where sampleBeginTime is EventTime, sampleEndTime of the earliest AQMD in the sample set and EventTime of the latest AQMD in the sample set, and AQ-IFore is an alert quality island forest created from this sample set.
The method comprises the steps that an AQMD is generated by an equipment object every time of alarming, AQMD data of a plurality of days are taken, eventTime is gathered for each day, busy hours and idle hours of the equipment are distinguished according to the gathering degree, the AQMD gathered data of each day are divided according to EventTime and each hour, 24 alarming collection quality metadata sets AQMDC exist in the same equipment every day, the AQMDCs of a plurality of days of the same equipment are combined in the same period, 24 AQMDC sample sets are generated, an alarming data quality isolation tree AQ-ITrees are constructed based on each sample set, and a piece of alarming data quality forest AQ-IFore is constructed by a plurality of AQ-ITrees.
The construction of the alarm data quality isolation tree AQ-ITrees comprises the following steps: firstly, selecting a sample set to construct an isolated tree, randomly selecting a plurality of sample points from the sample set without replacement to serve as sub-samples, and placing the sub-samples into a root node of the tree; secondly, designating AlarmInterval dimension, and randomly generating a cutting point between the maximum value and the minimum value of the designated dimension in the current node data; and generating a hyperplane by the cutting point, and dividing the current node data into two subspaces. Placing the data smaller than the cutting point in the appointed dimension on the left subtree of the current node, and placing the data larger than or equal to the cutting point on the right subtree of the current node; finally, the above process is recursively repeated in the leaf nodes, and new leaf nodes are continuously constructed until the height of the current node exceeds the threshold set by the algorithm, or the current subtree contains only one leaf node, or all the attributes of all the node values of the current subtree are completely consistent.
When constructing AQ-ITrees, 256 is set as the size of the root node sample set so that the isolated tree can cut and divide the outlier and the normal point for multiple times during construction, and the inundation and the masking effect caused by a large data volume are avoided.
In constructing AQ-ITrees, the root node sampling number is set to 256, and the tree height is limited to 8, so that the algorithm time and space consumption is reduced.
Constructing the AQ-IFore comprises the following steps: firstly, defining an empty AQ-IFore, and defining an upper limit of the quantity of AQ-ITrees in a forest; secondly, selecting a sample set, applying an AQ-ITree construction algorithm to construct an AQ-ITree, and adding the AQ-ITree into a forest; thirdly, judging the quantity of AQMD-ITrees in the AQ-IFore, and if the quantity does not reach the set upper limit, continuing constructing the AQ-ITrees; finally, when the number of AQ-ITrees in the AQ-IFore reaches an upper limit, construction of the AQ-IFore is terminated. Outputting the constructed AQ-IFore.
The AQ-IFore comprises 100 AQ-ITrees.
The construction algorithm of the alarm quality isolated forest cluster AQIFG comprises a sliding window updating algorithm, and the sliding window updating algorithm comprises the following steps: firstly, deleting the AQMD of the oldest day in a sample set at fixed time of each day, importing the AQMD newly added in the current day into an alarm acquisition quality metadata set AQMDC, and dividing again to form a latest sample set; secondly, iteratively constructing an alarm data quality isolation tree AQ-ITree and an AQ-IFore by using the latest sample set; again, aggregating AQ-iforst into a new AQIFG completes the AQIFG sliding window update.
And detecting alarm data quality abnormality by using the constructed AQIFG, wherein the alarm data quality abnormality detection comprises the following steps: firstly, an alarm AQMD timing detection program is utilized, the input of the program is alarm acquisition data of all accessed devices, the program uses two kinds of data to judge the alarm data quality, one is real-time AQMD data of the current device, the other is virtual I-AQMD data generated by timing detection, the I-AQMD data is different from real AQMD data, the real EventID does not exist, the EventTime value is the current time, and AlarmInterval is the interval between the last AQMD and the current I-AQMD; then, the two data are input into AQIFG in real time, an abnormal point is evaluated according to an AQIFG abnormal detection algorithm, and the abnormal probability of the abnormal point is higher as the evaluation result is closer to 1.
And detecting alarm data quality abnormality by using the constructed AQIFG, wherein the alarm data quality abnormality detection comprises the following steps: step 1, an AQMD timing detection system detects the AQMD timing of equipment; step 2, judging whether the AQMD is updated, if so, acquiring the AQMD, and if not, constructing the I-AQMD; step 3, inputting AQIFG; step 4, selecting a corresponding AQ-IFore according to EventTime; each of which is input AQ-ITree in parallel; step 5, respectively detecting outlier characteristics; step 6, integral evaluation of outliers; and 7, outputting an evaluation result.
The invention has the following technical effects: the network communication equipment alarm acquisition abnormity early warning method based on the abnormity detection algorithm can analyze alarm delay characteristics of different network elements and different time periods in real time and in quasi-real time by adopting an artificial intelligence method. The sliding threshold value of the method is more in line with the actual scene, and is more accurate and intelligent than the existing scheme. The invention can improve early warning certainty, reduce false alarm while finding potential risk, and timely find data abnormality, reduce serious alarm loss risk, thereby improving network reliability, shortening network fault recovery time, and indirectly improving user satisfaction. Alarm acquisition is a stable process, the probability of occurrence of problems is low, which means that abnormal points in the acquisition process are quite few, and the characteristics of the acquired abnormal points are different from those of normal points. According to the characteristics, the invention selects an isolated forest algorithm in the abnormality detection algorithm to detect the alarm abnormality in real time. In particular, according to the characteristics of alarm acquisition, the invention optimizes the original isolated forest algorithm so as to be more in line with a specific service model. Samples conforming to the service model need to be selected according to the alarm metadata before the alarm data quality isolation tree is created. The invention here makes the selection of samples according to busy and idle times of the device. When the equipment has heavy service, the alarm is easy to generate, otherwise, the probability of generating the alarm is reduced. Through the division, the modeling can be carried out on the isolated tree more scientifically, so that the accuracy of the anomaly detection algorithm is improved. In summary, the invention has the following features: 1. the entire construction algorithm from AQMD to AQIFG; alarming an AQMD timing detection program; and (5) an integral analysis algorithm for alarm acquisition anomaly detection.
Drawings
Fig. 1 is a schematic diagram of a network communication device alarm acquisition abnormality early warning method based on an abnormality detection algorithm in the implementation of the invention. AQMD (Alarm Quality Meta Data) in fig. 1 is alarm quality metadata, and alarm quality metadata aqmd=quadruple (device object dev, alarm event identification EventID, alarm event time EventTime, alarm occurrence interval AlarmInterval), that is aqmd= (dev, eventID, eventTime, alarmInterval). The I-AQMD is virtual alarm quality metadata, namely virtual AQMD. AQIFG (Alarm Quality IForest Group) is a collection of alert quality isolated forest clusters, aqifg=triples (earliest sample time sampleengittime, latest sample time SampleEndTime, alert quality isolated forest AQ-iforst), i.e. aqifg= (SampleBeginTime, sampleEndTime, AQ-iforst). AQ-ITrees are alarm data quality isolation trees, and a plurality of alarm data quality isolation trees AQ-ITrees are constructed into an alarm quality isolation forest AQ-IFore. Several AQMDs constitute AQMDC (Alarm Quality Meta Data Cluster), an alert collection quality metadata set. The detection flow of the abnormal early warning in the warning of the network communication equipment in fig. 1 is as follows: step 1, an AQMD timing detection system detects the AQMD timing of equipment; step 2, judging whether the AQMD is updated, if so, acquiring the AQMD, and if not, constructing the I-AQMD; step 3, inputting AQIFG; step 4, selecting a corresponding AQ-IFore according to EventTime; each of which is input AQ-ITree in parallel; step 5, respectively detecting outlier characteristics; step 6, integral evaluation of outliers; and 7, outputting an evaluation result.
Detailed Description
The invention will be described with reference to the accompanying drawings (fig. 1).
Fig. 1 is a schematic diagram of a network communication device alarm acquisition abnormality early warning method based on an abnormality detection algorithm in the implementation of the invention. Referring to fig. 1, the network communication equipment alarm acquisition abnormality early warning method based on the abnormality detection algorithm includes a construction algorithm from alarm quality metadata AQMD to an alarm quality isolated forest cluster AQIFG and an alarm AQMD timing detection program, where the structure of the AQMD is the following four elements: aqmd= (dev, eventID, eventTime, alarmInterval), where dev is the device object, eventID is the unique ID of the alarm, eventTime is the time of occurrence of the device alarm, alarmInterval is the interval between the occurrence of the last alarm and the current alarm; the AQIFG has the following triplet structure: AQIFG= (SampleBeginTime, sampleEndTime, AQ-IFore), where sampleBeginTime is EventTime, sampleEndTime of the earliest AQMD in the sample set and EventTime of the latest AQMD in the sample set, and AQ-IFore is an alert quality island forest created from this sample set. The following is described in two steps:
the first step: the data sources are clear, and the basic data sources required by the scheme uniformly collect metadata generated in the process of equipment alarm collection by the platform: 1. a device object; 2. a device alert unique ID; 3. the equipment alarm occurrence time; 4. time interval of device alarms. Metadata of the device alarms are composed according to the existing data. The definition is as follows: alarm acquisition quality metadata (Alarm Quality Meta Data), the structure of the AQMD alarm acquisition quality metadata is a quadruple,
AQMD＝(dev,EventID,EventTime,AlarmInterval)
wherein: dev is the device object; eventID is the unique ID of the alarm; eventTime is the time of occurrence of the device alert; alarmInterval is the interval between the occurrence of the last alarm and the current alarm. Each device generates a number of AQMD data per day based on the number of device alarms.
And a second step of: alarm anomaly analysis algorithm. Definition of alarm anomaly detection algorithm: alarm acquisition is a stable process, the probability of occurrence of problems is low, which means that abnormal points in the acquisition process are quite few, and the characteristics of the acquired abnormal points are different from those of normal points. According to the characteristics, the alarm abnormality is detected in real time by using an isolated forest algorithm in an abnormality detection algorithm. In particular, according to the characteristics of alarm acquisition, we optimize the original isolated forest algorithm so as to be more in line with our business model. An isolated tree construction algorithm for alarm data quality: samples conforming to the service model need to be selected according to the alarm metadata before the alarm data quality isolation tree is created. Here we make the selection of samples based on busy and idle times of the device. When the equipment has heavy service, the alarm is easy to generate, otherwise, the probability of generating the alarm is reduced. Through the division, the modeling can be carried out on the isolated tree more scientifically, so that the accuracy of our anomaly detection algorithm is improved.
Definition of alarm acquisition quality metadata set (Alarm Quality Meta Data Cluster) AQMDC: the alarm acquisition quality metadata AQMDC is a set, and the set includes a plurality of AQMD. AQMD in a collection are metadata with the same characteristics, aggregated according to different types, and divided according to time. For the same equipment dev, AQMD data of a plurality of days are taken, eventTime is gathered for each day, and busy hours and idle hours of the equipment are distinguished according to the gathering degree. For the data aggregated by the AQMD per day, the data is divided according to EventTime, and 24 AQMDCs are arranged in the same device per day. With AQMDC, multiple days of AQMDC for the same device are combined as a sample set for our algorithm. After combining, 24 sample sets are generated.
Based on each sample set, we can build an alarm data quality isolation tree. The construction algorithm of the quality isolation tree AQ-ITrees about the alarm data comprises the following steps:
firstly, selecting a sample set to construct an isolated tree, randomly selecting a plurality of sample points from the sample set without replacement to serve as sub-samples, and placing the sub-samples into a root node of the tree.
Next, an AlarmInterval dimension is specified, and a cut point located between the maximum value and the minimum value of the specified dimension in the current node data is randomly generated in the current node data.
And generating a hyperplane by the cutting point, and dividing the current node data into two subspaces. Data smaller than the cutting point in the appointed dimension is placed in the left subtree of the current node, and data larger than or equal to the cutting point is placed in the right subtree of the current node.
Finally, recursively repeating the above processes in the leaf nodes, and continuously constructing new leaf nodes until the height of the current node exceeds the threshold value set by the algorithm; or the current subtree contains only one leaf node; or all attributes of all node values of the current subtree are completely identical.
In particular, 256 is selected as the size of the root node sample set when constructing AQ-ITrees, so that the isolation tree can be cut for multiple times to divide outliers and normal points during construction, and flooding and masking effects caused by large data volume can be avoided. And when the root node sampling number is 256(s), the tree height is limited to 8 (layers), so that the algorithm time and space consumption is less.
An isolated forest construction algorithm for alarm data quality: when only one AQ-ITree exists, the algorithm has high contingency and cannot be used for detecting abnormal points of the quality of alarm data. Thus, we can iterate the AQ-ITrees construction algorithm described above to construct the isolated forest of data quality AQ-IFore.
The forest AQ-IFore construction algorithm for the alarm data quality comprises the following steps:
first, a null AQ-IFore is defined and an upper limit on the number of AQ-ITrees in the forest is defined.
Secondly, selecting a sample set, constructing an AQ-ITreee by applying the AQ-ITreee construction algorithm, and adding the AQ-ITreee into a forest.
And judging the quantity of the AQMD-ITrees in the AQ-IFore, and if the quantity does not reach the set upper limit, continuing constructing the AQ-ITrees.
Finally, when the number of AQ-ITrees in the AQ-IFore reaches an upper limit, construction of the AQ-IFore is terminated. Outputting the constructed AQ-IFore.
In particular, when the AQ-IFore comprises 100 AQ-ITrees, the algorithm can be ensured to construct a certain number of effective isolation trees, the time and space consumption can be ensured to be in a small range, and meanwhile, the combination characteristic of the integration method is embodied.
Regarding an alarm data quality isolated forest cluster construction algorithm, the alarm quality isolated forest cluster (Alarm Quality IForest Group) AQIFG is a set of AQ-IFore constructed for all sample sets, the set is a triplet:
AQIFG＝(SampleBeginTime,SampleEndTime,AQ-IForest)
wherein: sampleBeginTime is the EventTime of the earliest AQMD in the sample set. SampleEndTime is the EventTime of the latest AQMD in the sample set. AQ-iforst is an alert quality isolated forest created from this sample set. We use AQIFG to check the quality of the alert data.
The alarm conditions of the devices change over time. Our sample set should also be updated accordingly, otherwise, too old samples cannot well reflect changes in equipment traffic. The reliability of our algorithm will be reduced and the false alarm rate will be increased. The sliding window algorithm is adopted to update the sample set, and after the sample set is updated, new AQIFG is reconstructed according to the previous algorithm in sequence.
The sliding window updating algorithm for AQIFG comprises:
first, at a fixed time of each day, the AQMD of the oldest day in the sample set is deleted. And the AQMD newly added in the same day is imported into AQMDC and divided again to form a sample set.
Next, AQ-ITrees, AQ-IFore are iteratively constructed using the most current sample set.
Again, AQ-iforst is aggregated into a new AQIFG.
So far, the AQIFG sliding window is updated. The AQIFG constructed by the algorithm can ensure that our algorithm can well accord with our business mode.
Regarding alarm anomaly algorithm application: we use the AQIFG, which has been built, to perform anomaly detection of the alert data quality.
Firstly, an alarm AQMD timing detection program needs to be introduced, the input of the program is alarm acquisition data of all accessed devices, and the program uses two kinds of data to judge the data quality of alarms. One is real-time AQMD data of the current device, and the other is virtual AQMD data (I-AQMD) generated by timing detection, wherein the I-AQMD data is different from real AQMD data, the real AQMD data does not have real EventID, the EventTime value is the current time, and AlarmInterval is the interval between the last AQMD and the current I-AQMD. We input these two real-time data into AQIFG, and evaluate the outliers according to AQIFG outlier detection algorithm. The higher the anomaly probability of this piece of data, the result of the evaluation is approximately close to 1. I.e. the higher the probability of anomalies resulting from the alert collection.
The detection process is shown in fig. 1, and the detection flow of the abnormal early warning in the warning of the network communication equipment is as follows: step 1, an AQMD timing detection system detects the AQMD timing of equipment; step 2, judging whether the AQMD is updated, if so, acquiring the AQMD, and if not, constructing the I-AQMD; step 3, inputting AQIFG; step 4, selecting a corresponding AQ-IFore according to EventTime; each of which is input AQ-ITree in parallel; step 5, respectively detecting outlier characteristics; step 6, integral evaluation of outliers; and 7, outputting an evaluation result.
It is noted that the above description is helpful for a person skilled in the art to understand the present invention, but does not limit the scope of the present invention. Any and all such equivalent substitutions, modifications and/or deletions as may be made without departing from the spirit and scope of the invention.
Claims (4)
1. The network communication equipment alarm acquisition abnormity early warning method based on the abnormity detection algorithm is characterized by comprising a construction algorithm from alarm quality metadata AQMD to an alarm quality isolated forest cluster AQIFG and an alarm AQMD timing detection program, wherein the structure of the AQMD is the following four-tuple: aqmd= (dev, eventID, eventTime, alarmInterval), where dev is the device object, eventID is the unique ID of the alarm, eventTime is the time of occurrence of the device alarm, alarmInterval is the interval between the occurrence of the last alarm and the current alarm; the AQIFG has the following triplet structure: aqifg= (SampleBeginTime, sampleEndTime, AQ-iforst), where SampleBeginTime is EventTime, sampleEndTime of the earliest AQMD in the sample set and EventTime of the latest AQMD in the sample set, AQ-iforst is an alert quality island forest created from this sample set;
the method comprises the steps that an AQMD is generated by an equipment object every time of alarming, AQMD data of a plurality of days are taken, eventTime is gathered for each day respectively, busy hours and idle hours of the equipment are distinguished according to the gathering degree, the AQMD gathered data of each day are divided according to EventTime and each hour, 24 alarming collection quality metadata sets AQMDC exist in the same equipment every day, the AQMDCs of a plurality of days of the same equipment are combined in the same period to generate 24 AQMDC sample sets, an alarming data quality isolation tree AQ-ITrees are constructed based on each sample set, and a piece of alarming data quality forest AQ-IFore is constructed by a plurality of AQ-ITrees;
the construction of the alarm data quality isolation tree AQ-ITrees comprises the following steps: firstly, selecting a sample set to construct an isolated tree, randomly selecting a plurality of sample points from the sample set without replacement to serve as sub-samples, and placing the sub-samples into a root node of the tree; secondly, designating AlarmInterval dimension, and randomly generating a cutting point between the maximum value and the minimum value of the designated dimension in the current node data; thirdly, generating a hyperplane by the cutting point, dividing the current node data into two subspaces, placing the data smaller than the cutting point in the designated dimension on the left subtree of the current node, and placing the data larger than or equal to the cutting point on the right subtree of the current node; finally, recursively repeating the above processes in the leaf nodes, and continuously constructing new leaf nodes until the height of the current node exceeds the threshold value set by the algorithm, or the current subtree only comprises one leaf node, or all the attributes of all the node values of the current subtree are completely consistent;
constructing the AQ-IFore comprises the following steps: firstly, defining an empty AQ-IFore, and defining an upper limit of the quantity of AQ-ITrees in a forest; secondly, selecting a sample set, applying an AQ-ITree construction algorithm to construct an AQ-ITree, and adding the AQ-ITree into a forest; thirdly, judging the quantity of AQMD-ITrees in the AQ-IFore, and if the quantity does not reach the set upper limit, continuing constructing the AQ-ITrees; finally, when the quantity of AQ-ITrees in the AQ-IFore reaches the upper limit, terminating the construction of the AQ-IFore, and outputting the constructed AQ-IFore;
the construction algorithm of the alarm quality isolated forest cluster AQIFG comprises a sliding window updating algorithm, and the sliding window updating algorithm comprises the following steps: firstly, deleting the AQMD of the oldest day in a sample set at fixed time of each day, importing the AQMD newly added in the current day into an alarm acquisition quality metadata set AQMDC, and dividing again to form a latest sample set; secondly, iteratively constructing an alarm data quality isolation tree AQ-ITree and an AQ-IFore by using the latest sample set; thirdly, aggregating the AQ-IFore into a new AQIFG to finish the updating of the AQIFG sliding window;
and detecting alarm data quality abnormality by using the constructed AQIFG, wherein the alarm data quality abnormality detection comprises the following first flow step or second flow step, and the first flow step comprises the following steps: step 1, an AQMD timing detection system detects the AQMD timing of equipment; step 2, judging whether the AQMD is updated, if so, acquiring the AQMD, and if not, constructing the I-AQMD; step 3, inputting AQIFG; step 4, selecting a corresponding AQ-IFore according to EventTime; each of which is input AQ-ITree in parallel; step 5, respectively detecting outlier characteristics; step 6, integral evaluation of outliers; step 7, outputting an evaluation result;
the second process step includes: firstly, an alarm AQMD timing detection program is utilized, the input of the program is alarm acquisition data of all accessed devices, the program uses two kinds of data to judge the alarm data quality, one is real-time AQMD data of the current device, the other is virtual I-AQMD data generated by timing detection, the I-AQMD data is different from real AQMD data, the real EventID does not exist, the EventTime value is the current time, and AlarmInterval is the interval between the last AQMD and the current I-AQMD; then, the two data are input into AQIFG in real time, an abnormal point is evaluated according to an AQIFG abnormal detection algorithm, and the abnormal probability of the abnormal point is higher as the evaluation result is closer to 1.
2. The method for warning and collecting abnormal early warning of network communication equipment based on an abnormal detection algorithm according to claim 1, wherein 256 is set as the size of a root node sample set when constructing AQ-ITrees, so that an isolated tree can be cut for a plurality of times to divide an outlier and a normal point when constructing, and flooding and masking effects caused by a large amount of data are avoided.
3. The anomaly detection algorithm-based network communication device alarm acquisition anomaly early warning method of claim 1, wherein the root node sampling number is set to 256 and the tree height is limited to 8 when constructing AQ-ITree, so as to reduce algorithm time and space consumption.
4. The anomaly detection algorithm-based network communication device alarm acquisition anomaly pre-warning method of claim 1, wherein the AQ-iforst comprises 100 AQ-ITree.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010377031.4A CN111740856B (en) | 2020-05-07 | 2020-05-07 | Network communication equipment alarm acquisition abnormity early warning method based on abnormity detection algorithm |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010377031.4A CN111740856B (en) | 2020-05-07 | 2020-05-07 | Network communication equipment alarm acquisition abnormity early warning method based on abnormity detection algorithm |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111740856A CN111740856A (en) | 2020-10-02 |
| CN111740856B true CN111740856B (en) | 2023-04-28 |
Family
ID=72646988
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010377031.4A Active CN111740856B (en) | 2020-05-07 | 2020-05-07 | Network communication equipment alarm acquisition abnormity early warning method based on abnormity detection algorithm |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111740856B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN113645181B (en) * | 2021-06-21 | 2023-07-28 | 上海电力大学 | Distributed protocol attack detection method and system based on isolated forest |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108777873A (en) * | 2018-06-04 | 2018-11-09 | 江南大学 | The wireless sensor network abnormal deviation data examination method of forest is isolated based on weighted blend |
| CN109543765A (en) * | 2018-08-23 | 2019-03-29 | 江苏海平面数据科技有限公司 | A kind of industrial data denoising method based on improvement IForest |
| CN110046665A (en) * | 2019-04-17 | 2019-07-23 | 成都信息工程大学 | Based on isolated two abnormal classification point detecting method of forest, information data processing terminal |
| CN110334085A (en) * | 2019-05-30 | 2019-10-15 | 广州供电局有限公司 | Power distribution network data monitoring and modification method, device, computer and storage medium |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP6782679B2 (en) * | 2016-12-06 | 2020-11-11 | パナソニック インテレクチュアル プロパティ コーポレーション オブ アメリカＰａｎａｓｏｎｉｃ Ｉｎｔｅｌｌｅｃｔｕａｌ Ｐｒｏｐｅｒｔｙ Ｃｏｒｐｏｒａｔｉｏｎ ｏｆ Ａｍｅｒｉｃａ | Information processing equipment, information processing methods and programs |
| US10706701B2 (en) * | 2018-01-12 | 2020-07-07 | Qognify Ltd. | System and method for dynamically ordering video channels according to rank of abnormal detection |
-
2020
- 2020-05-07 CN CN202010377031.4A patent/CN111740856B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108777873A (en) * | 2018-06-04 | 2018-11-09 | 江南大学 | The wireless sensor network abnormal deviation data examination method of forest is isolated based on weighted blend |
| WO2019233189A1 (en) * | 2018-06-04 | 2019-12-12 | 江南大学 | Method for detecting sensor network abnormal data |
| CN109543765A (en) * | 2018-08-23 | 2019-03-29 | 江苏海平面数据科技有限公司 | A kind of industrial data denoising method based on improvement IForest |
| CN110046665A (en) * | 2019-04-17 | 2019-07-23 | 成都信息工程大学 | Based on isolated two abnormal classification point detecting method of forest, information data processing terminal |
| CN110334085A (en) * | 2019-05-30 | 2019-10-15 | 广州供电局有限公司 | Power distribution network data monitoring and modification method, device, computer and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| "校园网络设备接入状态评估和推演系统的研究与实现";姚植元;《中国优秀硕士学位论文全文数据库 信息科技辑》;20190115;全文 * |
| Simin Luo ; Le Luan ; Yiping Cui ; Xueyan Chai ; Zhuzhu Wang ; Yiming."An Attribute Associated Isolation Forest Algorithm for Detecting Anomalous Electro-data".《2019 Chinese Control Conference (CCC)》.2019, * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111740856A (en) | 2020-10-02 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN108415789B (en) | Node fault prediction system and method for large-scale hybrid heterogeneous storage system | |
| CN110659173B (en) | Operation and maintenance system and method | |
| CN111047082A (en) | Early warning method and device for equipment, storage medium and electronic device | |
| CN109189736B (en) | Method and device for generating alarm association rule | |
| CN110399347A (en) | Alarm log compression method, apparatus and system, storage medium | |
| CN109887242B (en) | Method, device and system for sending alarm information to user terminal | |
| Khatib et al. | Self-healing in mobile networks with big data | |
| CN111125268B (en) | Network alarm analysis model creation method, alarm analysis method and device | |
| US11348023B2 (en) | Identifying locations and causes of network faults | |
| US10977152B2 (en) | Rule-based continuous diagnosing and alerting from application logs | |
| CN111865407B (en) | Intelligent early warning method, device, equipment and storage medium for optical channel performance degradation | |
| CN112769605B (en) | Heterogeneous multi-cloud operation and maintenance management method and hybrid cloud platform | |
| CN108509309A (en) | A kind of system and method carrying out performance monitoring based on access log | |
| CN110135603B (en) | Power network alarm space characteristic analysis method based on improved entropy weight method | |
| CN117176560B (en) | Monitoring equipment supervision system and method based on Internet of things | |
| CN111740856B (en) | Network communication equipment alarm acquisition abnormity early warning method based on abnormity detection algorithm | |
| CN117273550B (en) | Information management method of intelligent laboratory for food detection | |
| CN104881436B (en) | A kind of electric power communication device method for analyzing performance and device based on big data | |
| CN108446202B (en) | Method for judging safety condition of machine room equipment | |
| CN117041312A (en) | Enterprise-level information technology monitoring system based on Internet of things | |
| CN113836203A (en) | Network data diagnosis detection analysis system | |
| CN114860543A (en) | Anomaly detection method, device, equipment and computer readable storage medium | |
| EP3840453B1 (en) | Method for detecting anomalies in mobile telecommunication networks | |
| CN117527527B (en) | Multi-source alarm processing method and system | |
| CN116861204B (en) | Intelligent manufacturing equipment data management system based on digital twinning |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |