CN115333922B - Operation and maintenance support network alarm data mining method, system and storage medium - Google Patents
Operation and maintenance support network alarm data mining method, system and storage medium Download PDFInfo
- Publication number
- CN115333922B CN115333922B CN202211256954.XA CN202211256954A CN115333922B CN 115333922 B CN115333922 B CN 115333922B CN 202211256954 A CN202211256954 A CN 202211256954A CN 115333922 B CN115333922 B CN 115333922B
- Authority
- CN
- China
- Prior art keywords
- data
- alarm
- mining
- transaction data
- transaction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0631—Management of faults, events, alarms or notifications using root cause analysis; using analysis of correlation between notifications, alarms or events based on decision criteria, e.g. hierarchy, tree or time analysis
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
- G06F16/2465—Query processing support for facilitating data mining operations in structured databases
Abstract
The invention provides an operation and maintenance support network alarm data mining method, which comprises the following steps: acquiring alarm data, wherein the alarm data comprises an alarm sequence generated according to the equipment alarm rule or the program alarm rule; preprocessing the alarm data, extracting the relevance of the alarm sequence, and judging the alarm data according to the relevance so as to generate transaction data of the alarm sequence, obtain a plurality of transaction data and form an alarm event set; performing weight value calculation on the transaction data, and extracting the weight value of each transaction data to generate weight value list data; and mining the transaction data and the alarm event set according to the weight value list data based on stream type calculation, and outputting a mining result.
Description
Technical Field
The invention belongs to the technical field of data analysis, and particularly relates to an operation and maintenance support network alarm data mining method, system and storage medium.
Background
In recent years, the development speed of the communication industry is changing day by day, and the competition among operators is also gradually intensified; the operation and maintenance support network is used as a network for maintaining and supporting stable operation of an operator, and in order to improve efficiency and standard of operation and maintenance work, it is necessary to reasonably manage the operation and maintenance support network, wherein fault management is the most critical part in network management of the operator, hundreds of alarm data can be caused by a small fault, and it is obviously unlikely that the data are analyzed and processed by using a manual mode.
Disclosure of Invention
In order to overcome the defects of the prior art, the invention provides an operation and maintenance support network alarm data mining method, system and storage medium, so as to solve the problems in the prior art.
One embodiment of the invention provides an operation and maintenance support network alarm data mining method, which comprises the following steps:
establishing a network security zone, and connecting with communication equipment through the network security zone;
the network safety area comprises a safety connection area, a safety conveying area and a safety management area;
issuing a preset instruction through the safety management area, wherein the safety management area is provided with a data comparison database, the data comparison database stores a data source table, a data matching table and an alarm mining table, and the data matching table stores alarm rule data aiming at the preset instruction;
the safety connection area receives the preset instruction and obtains the operation data of the communication equipment in real time according to the preset instruction; analyzing the operating data according to the preset instruction to generate first target data;
the safe conveying area receives the first target data, analyzes the first target data according to the preset instruction and generates second target data;
the data source table acquires the operating data, the first target data and the second target data in real time;
performing matching judgment on the data source table and the data matching table to generate alarm data;
mining the alarm data by adopting stream type calculation, outputting a mining result and storing the mining result into the alarm mining table;
the alarm rule data comprises equipment alarm rules and program alarm rules;
the streaming computation comprises the following steps:
the acceleration processing layer is used for processing the real-time stream transaction data according to the weighted value list data and generating real-time calculation data;
the batch processing layer is used for storing and managing an alarm event set; and/or processing the alarm event set according to the weight value list data and generating offline calculation data; and
the service layer is used for responding to a query request of a user, combining the real-time calculation data and the off-line calculation data according to the query request of the user and returning the combined data to the user;
the service layer is used for responding to the query request of the user and comprises the following steps:
the service layer comprises a processing module and an inquiry interface, and a batch processing program and a stream processing program are arranged in the processing module;
when the quantity of the transaction data processed by the service layer reaches a preset value, the batch processing program starts a parallel computing mode of the accelerated processing layer and outputs respective computing results;
and/or, in the processing module, initializing the alarm event set to obtain an alarm event database
Wherein G represents a conditional attribute set cardinality,
representing characteristic parameters of the transaction data, and R representing a feasible solution dimension;
mining transaction data of Tth generation
The calculation formula is as follows:
wherein r is 1 ,r 2 ,r 3 E {1,2, \8230;, G } all represent transaction data parameters,
the amount of the base compound is expressed,
representing the difference component, h represents the stream processing scaling factor,
is the alarm data of the T-th generation,
mining the alarm data of the T +1 th generation;
the query interface is used for merging and outputting the results of the query requests of the users;
the batch processing program starts a parallel computing mode of the accelerated processing layer and outputs respective computing results, and the method comprises the following steps:
the accelerated processing layer acquires real-time stream transaction data through a Spark frame, and performs stream processing and discrete processing on the real-time stream transaction data through the Spark frame, wherein the Spark frame is provided with a parallel computing program for performing parallel computing on the transaction data to realize analysis and computation on the transaction data;
the Spark framework comprises the following processing flow steps:
acquiring transaction data;
the real-time batch processing of the transaction data is realized through the data distributor;
distributing the batch processed data to different loading modules;
transmitting the loaded data to a data receiving terminal, and storing the data in a database;
in Spark framework, dividing the alarm event set into k alarm data clusters U 1 ,U 2 ,…,U k The corresponding central data are u 1 ,u 2 ,…,u k In which
g j Representing a data cluster U j The number of medium alarm events; the objective function of the transaction data is the following formula:
wherein d is ij (s i ,u j ) Representing a sequence of alarms s i And alarm center data u j The Euclidean distance between the two clusters is obtained by performing iterative calculation on the target function of the formula, and the average value of all alarm data clusters is the following formula:
wherein, itr represents the index of iterative computation, c (itr) represents the number of alarm data clusters after the iterative computation of the ith time, i represents the ith alarm center data, j represents the jth alarm event, and u ij The ith alarm center data in the jth alarm event;
when the iter calculation is carried out for the third time, the threshold value of the alarm data cluster is defined as the following formula:
MCT(itr)＝ζC avg (itr)；
where ζ represents the Spark framework parallel computation threshold parameter.
In one embodiment, the mining the alarm data by using streaming calculation, outputting a mining result, and storing the mining result in the alarm mining table includes:
acquiring alarm data, wherein the alarm data comprises an alarm sequence generated according to the equipment alarm rule or the program alarm rule;
preprocessing the alarm data, extracting the relevance of the alarm sequence, judging the alarm data according to the relevance, generating transaction data of the alarm sequence, acquiring a plurality of transaction data, and forming an alarm event set;
performing weight value calculation on the transaction data, and extracting the weight value of each transaction data to generate weight value list data;
and mining the transaction data according to the weight value list data based on stream type calculation, and outputting a mining result.
In one embodiment, the alarm data is preprocessed, the relevance of the alarm sequence is extracted, and the alarm data is determined according to the relevance; the method comprises the following steps:
presetting the width of a time window and the sliding step length for a sliding window time algorithm;
and converting the alarm data into the transaction data by using a sliding window time algorithm.
In one embodiment, the performing weight value calculation on the transaction data and extracting the weight value of each transaction data includes:
establishing a judgment matrix, and carrying out layered processing on the transaction data through the judgment matrix;
product K for each line in the decision function i (i =1,2, \8230;, n) was calculated;
one embodiment of the present invention further provides a system, comprising:
a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program when executed by the processor implementing the steps of the operation and maintenance support network alarm data mining method as described in any one of the above.
One embodiment of the present invention further provides a computer-readable storage medium, on which a computer program is stored, where the computer program, when executed by a processor, implements the steps of the operation and maintenance support network alarm data mining method according to any one of the above.
The method, the system and the storage medium for mining the alarm data of the operation and maintenance support network provided by the embodiment have the following beneficial effects:
1. preprocessing the acquired alarm data and calculating a weight value to eliminate non-fault data in the alarm data; the data parallel program in the Spark framework is utilized to carry out batch processing on the data, so that the overall calculation efficiency of the algorithm is improved; and finally, obtaining an ideal mining result by continuously adjusting the threshold value of the alarm event database.
2. In one embodiment, the alarm data is converted into an alarm event set suitable for mining through a sliding window time algorithm, and occurrence time attribute values of the alarm event set are unified, so that accurate mining of subsequent data is guaranteed.
3. In one embodiment, the weight value of the alarm data of the operation and maintenance support network is obtained through a weight value calculation process, and then the alarm data of the operation and maintenance support network with a higher weight value is mined by using a stream type calculation method, so that the mining accuracy of the alarm data of the operation and maintenance support network is further improved.
4. In one embodiment, by using a Spark framework in the streaming computation, when processing a large amount of data, a parallel computation mode is adopted, so that the overall computation efficiency of the algorithm is improved.
Drawings
In order to more clearly illustrate the embodiments or technical solutions of the present invention, the drawings used in the embodiments or technical solutions of the prior art will be briefly described below, it is obvious that the drawings in the following description are only some embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the structures shown in the drawings without creative efforts.
Fig. 1 is a schematic workflow diagram of an operation and maintenance support network alarm data mining method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a part of a working flow of the alarm data mining method for the operation and maintenance support network in FIG. 1;
FIG. 3 is a schematic diagram of a flow-type computing workflow of the operation and maintenance support network alarm data mining method of FIG. 1;
FIG. 4 is a schematic diagram illustrating a workflow of calculating a weight value according to the flow chart of FIG. 2;
fig. 5 is a schematic workflow diagram of Spark framework processing in the streaming computing in fig. 2;
FIG. 6 is a schematic diagram illustrating the operation of the streaming computing system of FIG. 2;
FIG. 7 is a schematic diagram of the working principle of the Spark frame in FIG. 5;
FIG. 8 shows the situation of memory usage of the alarm data set for three algorithms;
FIG. 9 is a comparison of data mining times for three algorithms;
FIG. 10 shows the results of comparison of three algorithms F-measure;
figure 11 is the AUC value comparison results of the three algorithms.
Detailed Description
The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
It should be noted that if directional indications (such as up, down, left, right, front, back, 8230) \8230;) are involved in the embodiment of the present invention, the directional indications are only used to explain the relative positional relationship between the components in a specific posture, the motion situation, etc., and if the specific posture is changed, the directional indications are correspondingly changed.
In addition, if there is a description of "first", "second", etc. in an embodiment of the present invention, the description of "first", "second", etc. is for descriptive purposes only and is not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one such feature. In addition, if the meaning of "and/or" and/or "appears throughout, the meaning includes three parallel schemes, for example," A and/or B "includes scheme A, or scheme B, or a scheme satisfying both schemes A and B. In addition, technical solutions between various embodiments may be combined with each other, but must be realized by a person skilled in the art, and when the technical solutions are contradictory or cannot be realized, such a combination should not be considered to exist, and is not within the protection scope of the present invention.
One embodiment of the invention provides an operation and maintenance support network alarm data mining method, which comprises the following steps:
establishing a network security zone, and connecting with communication equipment through the network security zone;
the network safety area comprises a safety connection area, a safety conveying area and a safety management area;
issuing a preset instruction through the safety management area, wherein the safety management area is provided with a data comparison library, the data comparison library stores a data source table, a data matching table and an alarm mining table, and the data matching table stores alarm rule data aiming at the preset instruction;
the safety connection area receives the preset instruction and acquires the running data of the communication equipment in real time according to the preset instruction; analyzing the operating data according to the preset instruction to generate first target data;
the safe conveying area receives the first target data, analyzes the first target data according to the preset instruction and generates second target data;
the data source table acquires the operating data, the first target data and the second target data in real time;
performing matching judgment on the data source table and the data matching table to generate alarm data;
mining the alarm data by adopting stream type calculation, outputting a mining result and storing the mining result into the alarm mining table;
wherein the alarm rule data comprises an equipment alarm rule and a program alarm rule.
In this embodiment, a forward isolation device and a reverse isolation device are disposed between the safety connection area, the safety transportation area, and the safety management area, so that data intercommunication among the safety connection area, the safety transportation area, and the safety management area is realized, and data real-time acquisition or transmission of the unidirectional communication link is realized.
According to needs, the safety conveying areas can be arranged in a plurality of modes, corresponding setting is carried out according to working requirements, and a firewall is arranged among the safety conveying areas so as to guarantee bidirectional data transmission of communication links among the safety conveying areas.
The running data, the first target data or the second target data comprise equipment running state data and program running state data, the equipment running state data comprise equipment normal running state data and equipment abnormal running state data, and the program running state data comprise program normal running state data and program abnormal running state data;
the equipment alarm rule is used for judging the equipment running state data, and the equipment alarm rule comprises a normal running value or an abnormal running value of the communication equipment in a real-time running state; specifically, when the operation data, the first target data or the second target data is out of the range of the normal operation value, alarm data corresponding to an abnormal operation state of the device is generated, or when the operation data, the first target data or the second target data is in the range of the abnormal operation value, alarm data corresponding to an abnormal operation state of the device is generated, where the abnormal operation state of the device includes a device failure abnormality, a memory usage abnormality, a CPU utilization abnormality and/or a network speed abnormality.
According to the requirement, the normal operation value or the abnormal operation value is set according to the working requirement, for example, the normal operation value of the memory utilization rate is set to be 60% or less, or the abnormal operation value of the memory utilization rate is set to be more than 60%, and when the memory utilization rate exceeds the preset normal operation value by 60% or is more than 60% of the preset abnormal operation value, the alarm data is generated.
According to the requirement, the program alarm rule is used for judging the program running state data, the program alarm rule comprises reference data corresponding to the preset instruction, the reference data are standard data generated by the communication equipment, the safety connection area and/or the safety conveying area based on the preset instruction, the reference data are distributed in a partition mode, are respectively a running data area, a first target data area and a second target data area, and are correspondingly matched with the running data, the first target data and/or the second target data; specifically, when the running data, the first target data and/or the second target data are not matched with the reference data, alarm data corresponding to abnormal running states of the program are generated, wherein the abnormal running states of the program comprise abnormal quantity of the program, abnormal data of the program and/or abnormal distributed programs.
Specifically, the abnormal quantity of the program is that the quantity of the running data, the first target data and/or the second target data is inconsistent with the quantity of the reference data; the program data exception is that the running data, the first target data and/or the second target data are inconsistent with the reference data; the distributed program exception is that the distribution positions of the running data, the first target data and/or the second target data are inconsistent with the distribution positions of the reference data.
In this embodiment, the alarm rule data is correspondingly set according to the use requirement, the operating environment and other scenes of the communication system by the staff, the system monitors the device operating state, the program operating state or the operating indexes of the communication device and the network security area within a certain time period according to the alarm rule data, and when the state change of the index data or the state data is compared with historical data or is compared with set alarm rule data, an alarm event is generated immediately, or when the operating state or the program operating state is not within the set alarm rule data, alarm data is generated and a corresponding alarm event is sent.
The method comprises the steps of monitoring hardware equipment and program data of communication equipment and a network security region through alarm rule data stored in a data matching table, realizing virtual-real bidirectional monitoring, improving the accuracy of alarm data mining, realizing hierarchical management and control of communication links by establishing the network security region, ensuring the mining accuracy of hierarchical alarm data mining, sending preset instructions through a security management region, acquiring data information of each hierarchy in real time, and combining a data source table, a data matching table and an alarm mining table for use, thereby realizing efficient, ordered and normative integration processing of the data, and accurately mining the alarm data by adopting a stream type calculation algorithm, so as to ensure that the alarm data of an operation and maintenance support network are mined efficiently and accurately.
The number of the communication equipment can be one or more, the operation data of the communication equipment is obtained in real time through the network security zone, the communication equipment is accurately analyzed and processed through the combination of the flow calculation and the network security zone and the real-time data and the off-line data, and the alarm data is accurately mined.
According to the requirement, a network safety area is established in the server, and the server is in wireless connection or line connection with the communication equipment, so that the server or the network safety area can acquire the operation data of the communication equipment in real time and analyze and process the operation data, and real-time monitoring and alarm data mining of the communication equipment are realized.
In one embodiment, the mining the alarm data by using streaming computation, outputting a mining result, and storing the mining result in the alarm mining table includes:
acquiring alarm data, wherein the alarm data comprises an alarm sequence generated according to the equipment alarm rule or the program alarm rule;
preprocessing the alarm data, extracting the relevance of the alarm sequence, and judging the alarm data according to the relevance so as to generate transaction data of the alarm sequence, obtain a plurality of transaction data and form an alarm event set;
performing weight value calculation on the transaction data, and extracting the weight value of each transaction data to generate weight value list data;
and mining the transaction data according to the weight value list data based on stream type calculation, and outputting a mining result.
The alarm event set is composed of transaction data of a plurality of the alarm sequences, and the alarm event set is generated based on the alarm data.
In this embodiment, preprocessing the acquired alarm data and calculating a weight value, eliminating non-fault data therein, and generating transaction data of the alarm data or transaction data of an alarm sequence, where the transaction data and the alarm transaction are data convenient for mining; the data parallel program in the Spark framework is utilized to carry out batch processing on the transaction data, and the alarm data is accurately mined through parallel calculation and off-umbrella processing, so that the overall calculation efficiency of the algorithm is improved; finally, obtaining an ideal mining result by continuously adjusting the threshold value of the alarm event database; and performing off-line calculation and real-time calculation on the transaction data by using stream type calculation, integrating the calculation results to obtain a mining result, and outputting the mining result through an inquiry interface.
Specifically, the communication system is maintained and supported through an operation and maintenance support network; and the operation and maintenance support network interacts with the communication system, and when the communication system breaks down and generates alarm data, the operation and maintenance support network acquires the alarm data in real time, converts the alarm data into transaction data, and then performs calculation, analysis and mining.
In one embodiment, the alarm data is preprocessed, the relevance of the alarm sequence is extracted, and the alarm data is determined according to the relevance; the method comprises the following steps:
converting alarm data into the transaction data by using a sliding window time algorithm;
the time window width and the sliding step length are preset for the sliding window time algorithm.
According to the requirement, each communication system has fixed propelling speed, and the propelling speed is different from each other, so even faults generated by the same system can cause inconsistent sequence of alarm data due to different occurrence time. That is, the occurrence time attribute values of all alarm data are different, which is not beneficial to subsequent data mining. Therefore, the alarm data of the operation and maintenance support network is converted into alarm affair or affair data suitable for mining through a sliding window time algorithm, specifically, the alarm data is formatted to obtain alarm data with uniform attributes, and the formatted alarm data is converted into the alarm affair or affair data through the time window algorithm.
In this embodiment, the relevance of the alarm sequence is found by presetting the time window width and the sliding step length and presetting the maximum interval and the minimum interval; in the execution process, the following two situations exist in the minimum interval: within the same time window, when a scene within an alarm sequence is given, the scene appears within a minimum interval; and/or, no scene in any sub-interval in the minimum interval is in the minimum interval;
in the same time window, the longest time distance between two scenes is the maximum interval;
the scenario includes at least two alarm events.
Filtering or converting the alarm data by using a sliding window time algorithm to form transaction data;
in the embodiment, the collected original alarm data is preprocessed through real-time calculation, so that data irrelevant to faults in the original alarm data are removed, the calculated data amount is reduced, and the algorithm is ensured to have an ideal mining result.
In one embodiment, the presetting of the time window width and the sliding step size for the sliding window time algorithm includes:
the alarm sequence consists of a plurality of alarm events, and the alarm events are label values generated based on alarm rule data for abnormal operation states of the communication equipment;
the alarm sequence formula is as follows: s = { S, T s ,T e }; wherein S represents an alarm sequence, T s Indicating the start time, T, of the sequence of alarm events e Represents the end time of the alarm event sequence, s = (e) 1 ,t 1 ),(e 2 ,t 2 ),…,(e n ,t n ) Incrementing a sequence for the alarm;
optionally selecting a subsequence S in S w ＝{w,t s ,t e }；
Wherein, t s >T s ，t e <T e ，
t e -t s = W, W being the time window;
setting the sliding step length of the time window as r, giving a time window arbitrarily, and setting the start time and the end time of the alarm sequence in the time window as T respectively i And T m ，T m -T i = W, a start and end time are obtained after the time window is subjected to sliding step length r distanceIs T i + r and T m + r new time window.
In the present embodiment, t e -t s Represents the end time of the alarm subsequence minus the start time of the alarm subsequence, and T e And T s Correspondingly, w is an alarm subsequence, the alarm data sequences are distributed in the operation and maintenance support network in an uneven mode, and only when the relevance among the data is large enough, the alarm data sequences can be tightly gathered in a time period. Therefore, the mining of the alarm data is to find out a tight time period, set the width of a time window and the sliding step length according to the requirements of working requirements, running environments and the like, and make basic preparation work for subsequent data mining.
In one embodiment, the performing weight value calculation on the transaction data and extracting the weight value of each transaction data includes:
establishing a judgment matrix, and carrying out layered processing on the transaction data through the judgment matrix;
product K for each line in the decision function i (i =1,2, \8230;, n) was calculated;
calculating K i Root of cubic (n times)
In this embodiment, the transaction data is subjected to weight value calculation by an analytic hierarchy process to obtain the weight value of each transaction data, and the transaction data is sequentially arranged from large to small, specifically, the alarm event of the transaction data is subjected to hierarchical processing by the judgment matrix.
In one embodiment, the establishing a judgment matrix, and performing hierarchical processing on the transaction data through the judgment matrix includes:
the alarm event set is set as E, the alarm event set comprises transaction data of n alarm sequences, and the transaction data Z of each alarm sequence is used for i (i = 1...., n) is classified;
product K of each line in the pair judgment function i (i =1,2, \8230;, n) performing calculations including: while lovel(Z i ) When the element belongs to {1,2,3,4,5}, substituting the element into the formula (1) and the formula (2) to obtain the formula (3):
z ij ＝1+2(level(Z i )-level(Z j )) (1)
wherein z is ij Transaction data Z representing a sequence of alarms i And Z j The comparison result of relative importance under the attribute rule; imparting z ij Specific numerical values 1, 3, 5, 7, 9,1 represent Z i And Z j Equally important, 9 denotes Z i Ratio Z j Extremely important, obtaining an importance degree grade comparison matrix shown in formula (2);
the calculation K i Root of cubic (n times)
The method comprises the following steps: k was calculated using the following formula (4) i Root of square root of
Wherein, K i Representing the product, Z, of each line in the decision function i Indicating the ith alarm data, Z j And representing the jth alarm data, acquiring the weighted value of the alarm data of the operation and maintenance support network according to the weighted value calculation process, and mining the alarm data of the operation and maintenance support network with higher weighted value by using a stream type calculation method, thereby further improving the mining accuracy of the alarm data of the operation and maintenance support network.
In one embodiment, the streaming computation comprises:
the acceleration processing layer is used for processing the real-time stream transaction data according to the weighted value list data and generating real-time calculation data;
the batch processing layer is used for storing and managing an alarm event set; and/or processing the alarm event set according to the weight value list data and generating offline calculation data; and
the service layer is used for responding to a query request of a user, combining the real-time calculation data and the off-line calculation data according to the query request of the user and returning the combined data to the user;
in the streaming calculation, when the alarm event set comprises real-time calculation data, discarding the real-time calculation data generated by the accelerated processing layer, and the batch processing layer retreats the alarm event set and generates new offline calculation data.
In this embodiment, the modules in the streaming computation are all written by using a program with complex logic but low time delay, and when the alarm data to be processed is large, the batch processing program starts a parallel computation mode, outputs respective computation results, and merges at the query interface.
The batch processing layer comprises a data storage module and a data preprocessing module; the service layer comprises an alarm data event sequence module, a processing module and an inquiry interface; the accelerated processing layer includes a stream processing module and a parallel computing module.
In one embodiment, the service layer, configured to respond to a query request of a user, includes:
the service layer comprises a processing module and an inquiry interface, and a batch processing program and a stream processing program are arranged in the processing module;
when the quantity of the transaction data processed by the service layer reaches a preset value, the batch processing program starts a parallel computing mode of the accelerated processing layer and outputs respective computing results;
in the processing module, the alarm event set is initialized to obtain an alarm event database
Wherein G represents a conditional attribute set cardinality,
representing characteristic parameters of the transaction data, and R representing a feasible solution dimension;
mining transaction data of generation T
The calculation formula is formula (8):
wherein r is 1 ,r 2 ,r 3 The E {1,2, \8230;, G } all represent transaction data parameters,
the amount of the base component is expressed,
representing a difference component, h representing a stream processing scaling factor;
the query interface is used for merging and outputting the results of the query requests of the users.
In the present embodiment, it is preferred that,
is the alarm data of the Tth generation,
and for the alarm data mining result of the T +1 th generation, the streaming computation adopts a Spark frame to compute the transaction data, the Spark frame comprises a processing module and a parallel computation module, the transaction data is computed in parallel through a batch processing program and a streaming processing program in the processing module, and the optimal threshold value is confirmed through continuously adjusting the threshold value of an alarm event database, so that the alarm data mining of the operation and maintenance support network is realized.
In one embodiment, the batch program starts a parallel computing mode of the accelerated processing layer and outputs respective computing results, including:
the accelerated processing layer acquires real-time stream transaction data through a Spark frame, and performs stream processing and discrete processing on the real-time stream transaction data through the Spark frame, wherein the Spark frame is provided with a parallel computing program used for performing parallel computing on the transaction data and realizing analysis and computation on the transaction data;
the Spark framework comprises the following processing flow steps:
acquiring transaction data;
the real-time batch processing of the transaction data is realized through the data distributor;
distributing the batched data to different loading modules;
and transmitting the loaded data to a data receiving terminal and storing the data in a database.
In this embodiment, the Spark framework in the streaming computation is a big data real-time parallel computation framework, and an API (Application Programming Interface) and an efficient engine based on a memory are configured inside the Spark framework, and are interactively combined with the processing module and the interactive query Interface to jointly act on the operation and maintenance support network alarm data.
In one embodiment, the alarm event set is divided into k alarm data clusters U in Spark framework 1 ,U 2 ,…,U k The corresponding central data are u 1 ,u 2 ,…,u k In which
g j Representing a data cluster U j The number of medium alarm events; the objective function of the transaction data is as follows:
wherein d is ij (s i ,u j ) Representing a sequence of alarms s i And alarm center data u j Performing iterative calculation on the above formula (5) to obtain the average value of all alarm data clusters as formula (6):
wherein, itr represents an index of iterative computation, and c (itr) represents the number of alarm data clusters after the iterative computation of the itr time;
when the iter calculation is performed for the third time, the threshold of the alarm data cluster is defined as formula (7):
MCT(itr)＝ζC avg (itr) (7)；
where ζ represents the Spark framework parallel computation threshold parameter.
In this embodiment, i represents the ith alarm center data, j represents the jth alarm event, u represents the ith alarm center data ij The method has the advantages that the algorithm can be used for directly calculating and analyzing on the premise of not storing data, is not influenced by the size of data volume, integrates offline calculation and real-time calculation together, and meets the requirements of low time delay, high fault tolerance and the like of alarm data mining of a communication system.
In the method, a Spark framework in the streaming computation is used, when a large amount of data is processed, a parallel computation mode is adopted, the overall computation efficiency of the algorithm is improved, and the alarm event database threshold is continuously adjusted, so that an excellent mining effect is obtained, preferably, in the embodiment, ζ is set to be 0.5.
There is also provided, in one embodiment of the present invention, a system, including:
a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program when executed by the processor implementing the steps of the operation and maintenance support network alarm data mining method as described above.
A computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the operation and maintenance support network alarm data mining method as described above.
Carrying out experimental analysis on the operation and maintenance support network alarm data mining method:
by carrying out an expansion comparison experiment with a weighting method and a K-Means clustering algorithm,
the method comprises the following steps: preparing experimental data;
the operation and maintenance support network is a support network for maintaining and supporting an operator network, so that data used in the experiment are all from alarm records in a real telecommunication system, and 25954 alarm data of one week of 3 months and 6 days to 3 months and 12 days in 2021 are selected from the alarm records. The method comprises the steps of preprocessing original alarm data before the experiment begins, and extracting alarm equipment, alarm occurrence positions, alarm types and alarm event 4 attribute information by using SPSS software to serve as judgment bases of alarm information.
Step two: setting an experiment index;
(1) Testing the memory occupation: the size of the memory occupied by the data volume of the alarm data set by the three algorithms is tested, and the smaller the occupied memory is, the lower the requirement of the algorithm on hardware is, and the applicable range is wide.
(2) Alarm data mining efficiency: the running time mined by the three algorithms is tested, and the smaller the running time is, the higher the overall calculation efficiency of the algorithms is.
(3) Excavating effect: this part of the test was evaluated by two indices of F-measure value (F value) and AUC (area size under ROC curve).
The evaluation index is established on the basis of the confusion matrix (table 1).
TABLE 1 confusion matrix
Wherein, TP (True Positive) is True: indicating that the positive class is predicted as a positive class number,
TN (True Negative) is True Negative: indicating that a negative class is predicted as a negative class number,
FP (False Positive) is False Positive: indicating that the negative class was predicted as a positive class number false positive (Type I error),
FN (False Negative) is False Negative: indicates that the positive class is predicted as a negative class number → false negative (Type II error);
the measure value (magnitude) is used as an algorithm mining performance evaluation index and is obtained by calculating the average value of Recall and Precision. The larger the F-measure value is, the better the mining performance of the algorithm is. The calculation formula is formula (8):
in the formula (8), the reaction mixture is,
AUC is a judging index for measuring the superior and inferior performance of the learner, the value range is [0,1], and when the result is closer to 1, the performance of the algorithm is better. The calculation formula is formula (9):
in the formula (9), the accuracy of the positive type mining
Negative type mining accuracy
Step three: results and analysis; on the basis of the experimental data preparation in the first step, in 25954 alarm records, according to the alarm device, the alarm occurrence position, the alarm type and 4 attributes of the alarm event, the number of the alarm data sets is randomly selected and selected to be 2000, and the test is performed in MATLAB software based on the indexes in the second step, wherein the specific test result is a memory occupation test as shown in FIG. 8; the memory occupation sizes of the three algorithms are tested, and the test results are shown in fig. 8: as can be seen from FIG. 8, with the increasing of the number of alarm data sets, the memory usage curve of the method increases slowly, and only reaches about 20MB at most, while the weighting method and the K-Means (K-Means clustering algorithm) clustering algorithm both reach 100MB. In contrast, the method occupies the least memory when mining the alarm data set, and has more applicable hardware equipment. The method is characterized in that the collected original alarm data is preprocessed, data irrelevant to faults in the original alarm data are removed, the amount of calculated data is reduced, and an ideal mining result of an algorithm is ensured.
Testing the mining efficiency of the alarm data; the alarm data mining time of the three algorithms is tested, and the result is shown in fig. 9: as can be seen by observing Table 3, the running time of algorithm mining is different due to the difference of the number of alarm data sets. The method has small increase when the number of the data sets is the first 600, and then the stability is restored; the running time curves of the weighting method and the K-Means clustering algorithm always fluctuate greatly, and both show a vertical rising trend. In contrast, it follows that the method herein takes the least amount of mining time. The method utilizes a Spark framework in the streaming calculation, and adopts a parallel calculation mode when processing data with larger quantity, thereby improving the overall calculation efficiency of the algorithm.
Testing the excavation efficiency; FIGS. 10 and 11 are the comparison results of the three algorithms on the two evaluation indexes of F-measure value and AUC, respectively: as can be seen by observing FIGS. 10 and 11, the results obtained by the methods herein are highest in all three algorithms, both for the F-measure value and the AUC value. Therefore, the method has ideal alarm data mining effect;
when a communication system fails, a large amount of alarm data can be generated, the traditional manual mode is low in efficiency, and the mining effect is not ideal. Therefore, an operation and maintenance support network alarm data mining algorithm based on stream computing is provided, after alarm data are preprocessed and weighted value computing is conducted, parallel computing is conducted through a Spark frame in the stream computing, and accurate mining of the alarm data is achieved through setting a reasonable threshold value for the algorithm. Compared with other methods, the method has the advantages that the result of the contrast experiment is shown, the method can finish accurate data mining in the shortest time, the operation and maintenance efficiency of a communication system is improved, and a data basis is provided for stable communication of an operation and maintenance network.
The above description is only a preferred embodiment of the present invention, and is not intended to limit the scope of the present invention, and all modifications and equivalents of the present invention, which are made by the contents of the present specification and the accompanying drawings, or directly/indirectly applied to other related technical fields, are included in the scope of the present invention.
Claims (6)
1. A method for mining alarm data of an operation and maintenance support network is characterized by comprising the following steps:
establishing a network security zone, and connecting with communication equipment through the network security zone;
the network safety area comprises a safety connection area, a safety conveying area and a safety management area;
issuing a preset instruction through the safety management area, wherein the safety management area is provided with a data comparison database, the data comparison database stores a data source table, a data matching table and an alarm mining table, and the data matching table stores alarm rule data aiming at the preset instruction;
the safety connection area receives the preset instruction and obtains the operation data of the communication equipment in real time according to the preset instruction; analyzing the operating data according to the preset instruction to generate first target data;
the safe conveying area receives the first target data, analyzes the first target data according to the preset instruction and generates second target data;
the data source table acquires the operating data, the first target data and the second target data in real time;
performing matching judgment on the data source table and the data matching table to generate alarm data;
mining the alarm data by adopting stream type calculation, outputting a mining result and storing the mining result into the alarm mining table;
the alarm rule data comprises equipment alarm rules and program alarm rules;
the streaming computation comprises the following steps:
the acceleration processing layer is used for processing the real-time stream transaction data according to the weighted value list data and generating real-time calculation data;
the batch processing layer is used for storing and managing an alarm event set; and/or processing the alarm event set according to the weight value list data and generating offline calculation data; and
the service layer is used for responding to a query request of a user, combining the real-time calculation data and the off-line calculation data according to the query request of the user and returning the combined data to the user;
the service layer is used for responding to the query request of the user and comprises the following steps:
the service layer comprises a processing module and an inquiry interface, wherein a batch processing program and a stream processing program are arranged in the processing module;
when the quantity of the transaction data processed by the service layer reaches a preset value, the batch processing program starts a parallel computing mode of the accelerated processing layer and outputs respective computing results;
and/or, in the processing module, initializing the alarm event set to obtain an alarm event database
Wherein G represents a conditional attribute set cardinality,
representing characteristic parameters of the transaction data, and R representing a feasible solution dimension;
mining transaction data of Tth generation
The calculation formula is as follows:
wherein r is 1 ,r 2 ,r 3 The E {1,2, \8230;, G } all represent transaction data parameters,
the amount of the base compound is expressed,
representing the difference component, h represents the stream processing scaling factor,
is the alarm data of the Tth generation,
mining the alarm data of the T +1 th generation;
the query interface is used for merging and outputting the results of the query requests of the users;
the batch processing program starts a parallel computing mode of the accelerated processing layer and outputs respective computing results, and the method comprises the following steps:
the accelerated processing layer acquires real-time stream transaction data through a Spark frame, and performs stream processing and discrete processing on the real-time stream transaction data through the Spark frame, wherein the Spark frame is provided with a parallel computing program for performing parallel computing on the transaction data to realize analysis and computation on the transaction data;
the Spark framework comprises the following processing flow steps:
acquiring transaction data;
the real-time batch processing of the transaction data is realized through the data distributor;
distributing the batch processed data to different loading modules;
transmitting the loaded data to a data receiving terminal and storing the data in a database;
dividing the alarm event set into k alarm data clusters U in Spark framework 1 ,U 2 ,…,U k The corresponding central data are u 1 ,u 2 ,…,u k In which
g j Representing a data cluster U j The number of medium alarm events; the objective function of the transaction data is the following formula:
wherein d is ij (s i ,u j ) Representing a sequence of alarms s i And alarm center data u j The Euclidean distance between the alarm data clusters is obtained by carrying out iterative computation on the target function of the formula, and the average value of all the alarm data clusters is the following formula:
wherein, itr represents the index of iterative computation, c (itr) represents the number of alarm data clusters after the iterative computation of the ith time, i represents the ith alarm center data, j represents the jth alarm event, and u ij The ith alarm center data in the jth alarm event;
when the iter time is calculated, defining the threshold value of the alarm data cluster as the following formula:
MCT(itr)＝ζC avg (itr)；
where ζ represents the Spark framework parallel computation threshold parameter.
2. The method for mining alarm data of an operation and maintenance support network according to claim 1, wherein the mining the alarm data by stream computing, outputting a mining result, and storing the mining result in the alarm mining table includes:
acquiring alarm data, wherein the alarm data comprises an alarm sequence generated according to the equipment alarm rule or the program alarm rule;
preprocessing the alarm data, extracting the relevance of the alarm sequence, and judging the alarm data according to the relevance so as to generate transaction data of the alarm sequence, obtain a plurality of transaction data and form an alarm event set;
performing weight value calculation on the transaction data, and extracting the weight value of each transaction data so as to generate weight value list data;
and mining the transaction data according to the weight value list data based on stream type calculation, and outputting a mining result.
3. The method according to claim 2, wherein the alarm data is preprocessed, the relevance of the alarm sequence is extracted, and the alarm data is determined according to the relevance; the method comprises the following steps:
presetting the width of a time window and the sliding step length for a sliding window time algorithm;
and converting the alarm data into the transaction data by using a sliding window time algorithm.
4. The method for mining alarm data of an operation and maintenance support network according to claim 2, wherein the performing weight value calculation on the transaction data and extracting the weight value of each transaction data includes:
establishing a judgment matrix, and carrying out layered processing on the transaction data through the judgment matrix;
product K for each line in the judgment function i (i =1,2, \8230;, n) was calculated;
calculating K i Root of cubic (n times)
5. A communication system, comprising:
a memory, a processor and a computer program stored on the memory and executable on the processor, the computer program when executed by the processor implementing the steps of the operation and maintenance support network alarm data mining method of any of claims 1-4.
6. A computer-readable storage medium, characterized in that the computer-readable storage medium has stored thereon a computer program, which when executed by a processor, implements the steps of the operation and maintenance support network alarm data mining method according to any one of claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211256954.XA CN115333922B (en) | 2022-10-13 | 2022-10-13 | Operation and maintenance support network alarm data mining method, system and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211256954.XA CN115333922B (en) | 2022-10-13 | 2022-10-13 | Operation and maintenance support network alarm data mining method, system and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115333922A CN115333922A (en) | 2022-11-11 |
| CN115333922B true CN115333922B (en) | 2023-01-06 |
Family
ID=83914389
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211256954.XA Active CN115333922B (en) | 2022-10-13 | 2022-10-13 | Operation and maintenance support network alarm data mining method, system and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115333922B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN116132257A (en) * | 2022-11-25 | 2023-05-16 | 浪潮通信信息系统有限公司 | Derived alarm determining method and device based on stream computing |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107203199A (en) * | 2017-06-12 | 2017-09-26 | 北京匡恩网络科技有限责任公司 | A kind of industry control network safe early warning method and system |
| WO2021016978A1 (en) * | 2019-08-01 | 2021-02-04 | 中国科学院深圳先进技术研究院 | Telecommunication network alarm prediction method and system |
| WO2021254244A1 (en) * | 2020-06-16 | 2021-12-23 | 中兴通讯股份有限公司 | Alarm mining model determination method and apparatus, device and storage medium |
| WO2022042152A1 (en) * | 2020-08-31 | 2022-03-03 | 中兴通讯股份有限公司 | Method and device for analyzing association rule of multi-dimensional network indexes, and storage medium |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| EP3436951B1 (en) * | 2016-03-29 | 2024-03-20 | Anritsu Company | Systems and methods for measuring effective customer impact of network problems in real-time using streaming analytics |
-
2022
- 2022-10-13 CN CN202211256954.XA patent/CN115333922B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN107203199A (en) * | 2017-06-12 | 2017-09-26 | 北京匡恩网络科技有限责任公司 | A kind of industry control network safe early warning method and system |
| WO2021016978A1 (en) * | 2019-08-01 | 2021-02-04 | 中国科学院深圳先进技术研究院 | Telecommunication network alarm prediction method and system |
| WO2021254244A1 (en) * | 2020-06-16 | 2021-12-23 | 中兴通讯股份有限公司 | Alarm mining model determination method and apparatus, device and storage medium |
| WO2022042152A1 (en) * | 2020-08-31 | 2022-03-03 | 中兴通讯股份有限公司 | Method and device for analyzing association rule of multi-dimensional network indexes, and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| 基于大数据挖掘技术的智能变电站故障追踪架构;王磊等;《电力系统自动化》;20171222(第03期);全文 * |
| 智能电网监控运行大数据分析系统总体设计;冷喜武等;《电力系统自动化》;20180514(第12期);全文 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115333922A (en) | 2022-11-11 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111694879B (en) | Multielement time sequence abnormal mode prediction method and data acquisition monitoring device | |
| CN105608144B (en) | A kind of big data analysis stage apparatus and method based on multilayered model iteration | |
| CN106844161B (en) | Abnormity monitoring and predicting method and system in calculation system with state flow | |
| US8180914B2 (en) | Deleting data stream overload | |
| CN111984499A (en) | Fault detection method and device for big data cluster | |
| CN115333922B (en) | Operation and maintenance support network alarm data mining method, system and storage medium | |
| CN112860695B (en) | Monitoring data query method, device, equipment, storage medium and program product | |
| CN105979532B (en) | Performance capacity analysis early warning method and device of service processing system | |
| US20050038769A1 (en) | Methods and apparatus for clustering evolving data streams through online and offline components | |
| CN108074022A (en) | A kind of hardware resource analysis and appraisal procedure based on concentration O＆M | |
| CN106383830B (en) | Data retrieval method and equipment | |
| CN110084326A (en) | A kind of industrial equipment method for detecting abnormality based on fuzzy set | |
| CN114860462B (en) | Intelligent computing resource distribution system and method for double-path rack-mounted server | |
| CN114896121A (en) | Monitoring method and device of distributed processing system | |
| CN114707834A (en) | Alarm reminding method and device and storage medium | |
| CN112383427A (en) | 5G network slice deployment method and system based on IOTIPS fault early warning | |
| CN114565325B (en) | Big data analysis method and system of power Internet of things | |
| CN116089012A (en) | Self-adaptive container anomaly detection method based on container resource index | |
| US20200034406A1 (en) | Real-time data aggregation | |
| CN111476316B (en) | Method and system for clustering mean value of power load characteristic data based on cloud computing | |
| CN111628901B (en) | Index anomaly detection method and related device | |
| CN114912638A (en) | Digital cable for reporting effective operation parameters | |
| Romdhani et al. | QoS-based trust evaluation for data services as a black box | |
| CN111813542A (en) | Load balancing method and device for parallel processing of large-scale graph analysis tasks | |
| Huang et al. | Lightpro: Lightweight probabilistic workload prediction framework for database-as-a-service |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |