CN118075090A - Network fault prediction method based on machine learning - Google Patents
Network fault prediction method based on machine learning Download PDFInfo
- Publication number
- CN118075090A CN118075090A CN202410148841.0A CN202410148841A CN118075090A CN 118075090 A CN118075090 A CN 118075090A CN 202410148841 A CN202410148841 A CN 202410148841A CN 118075090 A CN118075090 A CN 118075090A
- Authority
- CN
- China
- Prior art keywords
- data
- network
- alarm log
- model
- alarm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 23
- 238000010801 machine learning Methods 0.000 title claims abstract description 12
- 238000012549 training Methods 0.000 claims abstract description 17
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 16
- 238000004140 cleaning Methods 0.000 claims abstract description 10
- 238000000605 extraction Methods 0.000 claims abstract description 4
- 238000007781 pre-processing Methods 0.000 claims abstract description 4
- 230000002159 abnormal effect Effects 0.000 claims abstract 2
- 238000012795 verification Methods 0.000 claims description 7
- 230000006870 function Effects 0.000 claims description 6
- 238000012544 monitoring process Methods 0.000 claims description 6
- 238000012360 testing method Methods 0.000 claims description 4
- 239000000758 substrate Substances 0.000 claims 3
- 239000000523 sample Substances 0.000 claims 2
- 238000011156 evaluation Methods 0.000 claims 1
- 230000009466 transformation Effects 0.000 claims 1
- 238000010200 validation analysis Methods 0.000 claims 1
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000010586 diagram Methods 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 1
- 238000013527 convolutional neural network Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012545 processing Methods 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
Landscapes
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The invention provides a network fault prediction method based on machine learning, which belongs to the technical field of networks and collects data in the modes of SNMP protocol, netFlow protocol and the like; and preprocessing the collected data, including data cleaning, feature extraction and other operations. The data cleaning can remove interference factors such as abnormal values, missing values and the like, and the feature extraction can extract the features related to network faults from the data; model training is carried out on the preprocessed data by using a machine learning algorithm, and a network fault prediction model is obtained; the network is monitored and optimized according to the prediction result, and when a certain device is predicted to possibly fail, the device can be maintained or replaced in advance.
Description
Technical Field
The invention relates to the technical field of networks, in particular to a network fault prediction method based on machine learning.
Background
With the rapid development of network technology, the scale and complexity of the network are continuously increased, and network faults are also increased. The network fault prediction is used as an effective technical means, so that potential faults can be found in advance, and the occurrence of the faults is avoided or the influence of the faults is reduced. However, the current network fault prediction method has some problems, such as low prediction accuracy, poor real-time performance, and the like, and cannot meet the requirements of practical application. Therefore, a method capable of accurately predicting network failure in real time is needed.
Disclosure of Invention
In order to solve the technical problems, the invention provides a network fault prediction method based on machine learning, which is used for solving the problems of low prediction accuracy and poor real-time performance existing in the existing network fault prediction method.
The technical scheme of the invention is as follows:
a machine learning based network failure prediction method, comprising:
Step 1) the acquisition and cleaning of data,
Step 2) training and verification of the model,
Step 3) network fault real-time prediction and monitoring
Further, the method specifically comprises the following steps:
collecting network operation data including alarm log data of network equipment and the like;
Preprocessing the collected data, including data cleaning, feature extraction and other operations;
model training is carried out on the preprocessed data by using a machine learning algorithm, and a network fault prediction model is obtained;
inputting network operation data acquired in real time into a trained network fault prediction model to obtain a prediction result;
And monitoring and optimizing the network according to the prediction result, finding potential faults in time and taking corresponding measures.
Still further, the processing includes:
(01) Collecting data by SNMP protocol and NetFlow protocol;
(02) The collected data comprises alarm log data;
(03) Extracting and reading each field in the collected original alarm log data;
(04) Preliminary pretreatment of original alarm log data:
(05) The pretreatment comprises the following steps:
(06) Denoising: removing data which cannot be interpreted in the system;
(07) Filling up missing values: and deducing whether the information has a missing condition according to the corresponding format of the alarm log data field. If the field is missing, filling is needed to be carried out manually, and a complete alarm log is formed;
(08) The original alarm log after preliminary pretreatment passes through a rule interpretation module;
(09) Inquiring alarm lists of all versions in an alarm log database;
(10) Resolving the corresponding field of the extracted original alarm log;
(11) The alarm log data with high readability is converted into alarm log data;
(12) Designing and building a network fault prediction model; the method comprises the following steps:
(13) Collecting an original alarm log;
(14) Cleaning dirty data after analyzing the log;
(15) Clustering the initial data set to obtain clustering features and extracting features according to the attribute of the alarm log data;
(16) Taking an alarm title, a manufacturer alarm level and alarm interpretation as judging factors;
(17) Defining the network fault risk degree;
(18) Feature conversion: converting the multi-vector into a new feature set;
(19) Dividing alarm log data into a training set, a verification set and a test set;
(20) Setting model parameters such as weight w and bias b;
(21) Training a training set by using different algorithm models;
(22) Taking the cross entropy loss function as a standard for evaluating the performance of the model;
(23) Adjusting model parameters;
(24) Until the loss function is minimum, finding out the most suitable w and b;
(25) Obtaining model parameters of each algorithm;
(26) Verifying the performances of the different models through a verification set;
(27) Evaluating the performance of each algorithm according to the indexes of the accuracy rate, the recall rate and the F value;
(28) And determining a network fault prediction algorithm model.
The invention has the beneficial effects that
The network operation data is modeled and predicted through a machine learning algorithm, so that potential network faults can be accurately found in real time. Meanwhile, the invention can also carry out customized model training according to actual requirements so as to meet the requirements of different scenes. In addition, the invention can also apply the prediction result to the monitoring and optimization of the network, and improve the reliability and performance of the network.
Drawings
FIG. 1 is a schematic workflow diagram of the present invention;
FIG. 2 is a training flow diagram of the predictive model of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments, and all other embodiments obtained by those skilled in the art without making any inventive effort based on the embodiments of the present invention are within the scope of protection of the present invention.
The invention provides a network fault prediction method based on machine learning, which is realized by the following technical scheme:
(1) Collecting data through SNMP protocol;
(2) The collected data is specifically alarm log data;
(3) Extracting and reading each field in the collected original alarm log data;
(4) Preliminary pretreatment of original alarm log data:
(5) The pretreatment comprises the following steps:
(6) Denoising: removing data which cannot be interpreted in the system;
(7) Filling up missing values: and deducing whether the information has a missing condition according to the corresponding format of the alarm log data field. If the field is missing, filling is needed to be carried out manually, and a complete alarm log is formed;
(8) The original alarm log after preliminary pretreatment passes through a rule interpretation module;
(9) Inquiring alarm lists of all versions in an alarm log database;
(10) Resolving the corresponding field of the extracted original alarm log;
(11) The alarm log data with high readability is converted into alarm log data;
(12) Designing a network fault prediction model; the method comprises the following steps:
(13) Collecting an original alarm log;
(14) Cleaning dirty data after analyzing the log;
(15) Clustering the initial data set by using a K-means method to obtain clustering features and extracting features according to the attribute of the alarm log data;
(16) Taking an alarm title, a manufacturer alarm level and alarm interpretation as judging factors;
(17) Defining the network fault risk degree (fault) as 0, 1 and 2;
(18) fault is 0, and the corresponding network system has no fault; fault is 1, 2 corresponding to the network system having failed;
(19) Feature conversion: converting the multi-vector into a new feature set through an RTE algorithm;
(20) Dividing alarm log data into a training set, a testing set and a verification set according to a ratio of 6:2:2;
(21) 60% of log samples are training sets, 20% of log samples are verification sets, and 20% of log samples are test sets;
(22) Setting model parameters such as weight w and bias b;
(23) Selecting XGBoost, random forests, bayes and convolutional neural network algorithm models, and utilizing training sets to respectively model parameters;
(24) Taking the cross entropy loss function as a standard for evaluating the performance of the model;
(25) Adjusting model parameters;
(26) Until the loss function is minimum, finding out the most suitable w and b of the model;
(27) Obtaining model parameters of each algorithm;
(28) Verifying the performances of the different models through a verification set;
(29) Evaluating the performance of each algorithm according to the indexes of the accuracy rate, the recall rate and the F value;
(30) Determining a network fault prediction algorithm model;
(31) Network operation data acquired in real time are input into a trained network fault prediction model;
(32) Obtaining a network fault prediction result;
(33) Monitoring and optimizing the network according to the prediction result;
(34) Finding a potential fault;
(35) And adopting corresponding measures to solve the faults.
The foregoing description is only illustrative of the preferred embodiments of the present invention, and is not intended to limit the scope of the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention are included in the protection scope of the present invention.
Claims (8)
1. A network fault prediction method based on machine learning is characterized in that,
Comprising
Step 1) data acquisition and cleaning;
Step 2) training and verifying a model;
And 3) predicting and monitoring network faults in real time.
2. The method of claim 1, wherein the step of determining the position of the substrate comprises,
Collecting network operation data, including alarm log data of network equipment;
preprocessing the collected data, including data cleaning and feature extraction;
model training is carried out on the preprocessed data by using a machine learning algorithm, and a network fault prediction model is obtained;
inputting network operation data acquired in real time into a trained network fault prediction model to obtain a prediction result;
And monitoring and optimizing the network according to the prediction result, finding potential faults in time and taking corresponding measures.
3. The method of claim 2, wherein the step of determining the position of the substrate comprises,
Data are collected through SNMP protocol and NetFlow protocol modes.
4. The method of claim 3, wherein the step of,
The data is cleaned to remove interference factors of abnormal values and missing value classes, and the characteristics are extracted to extract the characteristics related to network faults from the data.
5. A method according to claim 2 or 3, characterized in that,
Data acquisition and cleaning) specifically includes the following steps:
(1-1) extracting and interpreting each field in the collected original alarm log data;
(1-2) preliminary preprocessing of original alarm log data:
(1-3) the original alarm log after preliminary pretreatment passes through a rule interpretation module;
(1-4) querying an alarm list of each version in an alarm log database;
(1-5) resolving the corresponding field of the extracted original alarm log;
(1-6) converting the alarm log data into alarm log data with high readability.
6. The method of claim 5, wherein the step of determining the position of the probe is performed,
The preliminary pretreatment includes the following steps:
Denoising: removing data which cannot be interpreted in the system;
Filling up missing values: deducing whether the information has a missing condition according to the corresponding format of the alarm log data field; if the field is missing, the field needs to be filled manually to form a complete alarm log.
7. The method of claim 2, wherein the step of determining the position of the substrate comprises,
Step 2) training and verifying a model, which specifically comprises the following steps:
(2-1) collecting an original alarm log;
(2-2) cleaning dirty data after analyzing the log;
(2-3) clustering the initial data set to obtain clustering features and extracting features according to the attribute of the alarm log data;
(2-4) taking an alarm title, a manufacturer alarm level and an alarm interpretation as judging factors;
(2-5) defining a network failure risk level;
(2-6) feature transformation: converting the multi-vector into a new feature set;
(2-7) dividing the alarm log data into a training set, a verification set and a test set;
(2-8) setting model parameters such as weight w and bias b;
(2-9) training the training set using different algorithmic models;
(2-10) taking the cross entropy loss function as a standard of model performance evaluation;
(2-11) adjusting model parameters;
(2-12) until the loss function is minimum, finding out the most suitable w and b;
(2-13) obtaining model parameters of each algorithm;
(2-14) validating the performance of the different models by the validation set;
(2-15) evaluating the performance of each algorithm based on the precision, recall and F-value multiple indicators; (2-16) determining a network failure prediction algorithm model.
8. The method of claim 7, wherein the step of determining the position of the probe is performed,
The initial dataset was clustered using the K-means method.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410148841.0A CN118075090A (en) | 2024-02-02 | 2024-02-02 | Network fault prediction method based on machine learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410148841.0A CN118075090A (en) | 2024-02-02 | 2024-02-02 | Network fault prediction method based on machine learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN118075090A true CN118075090A (en) | 2024-05-24 |
Family
ID=91098303
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410148841.0A Pending CN118075090A (en) | 2024-02-02 | 2024-02-02 | Network fault prediction method based on machine learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN118075090A (en) |
-
2024
- 2024-02-02 CN CN202410148841.0A patent/CN118075090A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111459700B (en) | Equipment fault diagnosis method, diagnosis device, diagnosis equipment and storage medium | |
| CN111259947A (en) | Power system fault early warning method and system based on multi-mode learning | |
| CN116450399B (en) | Fault diagnosis and root cause positioning method for micro service system | |
| CN110837866A (en) | XGboost-based electric power secondary equipment defect degree evaluation method | |
| CN116559598B (en) | Smart distribution network fault positioning method and system | |
| CN116148679B (en) | Battery health state prediction method and related device | |
| CN111078513A (en) | Log processing method, device, equipment, storage medium and log alarm system | |
| CN116739829B (en) | Big data-based power data analysis method, system and medium | |
| CN111666978B (en) | Intelligent fault early warning system for IT system operation and maintenance big data | |
| CN115858794B (en) | Abnormal log data identification method for network operation safety monitoring | |
| CN111984442A (en) | Method and device for detecting abnormality of computer cluster system, and storage medium | |
| CN110580492A (en) | Track circuit fault precursor discovery method based on small fluctuation detection | |
| CN117170915A (en) | Data center equipment fault prediction method and device and computer equipment | |
| CN117368651B (en) | Comprehensive analysis system and method for faults of power distribution network | |
| CN117560300B (en) | Intelligent internet of things flow prediction and optimization system | |
| CN115114124A (en) | Host risk assessment method and device | |
| CN116520068A (en) | Diagnostic method, device, equipment and storage medium for electric power data | |
| CN116578436A (en) | Real-time online detection method based on asynchronous multielement time sequence data | |
| CN118075090A (en) | Network fault prediction method based on machine learning | |
| CN115659271A (en) | Sensor abnormality detection method, model training method, system, device, and medium | |
| CN116126807A (en) | Log analysis method and related device | |
| CN112579429A (en) | Problem positioning method and device | |
| CN111626508A (en) | Rail transit vehicle-mounted data prediction method based on xgboost model | |
| CN117271350B (en) | Software quality assessment system and method based on log analysis | |
| CN114637793B (en) | Equipment fault frequent region positioning method based on big data analysis |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |