CN116738347A - Fault analysis positioning method and system based on machine learning - Google Patents
Fault analysis positioning method and system based on machine learning Download PDFInfo
- Publication number
- CN116738347A CN116738347A CN202310568431.7A CN202310568431A CN116738347A CN 116738347 A CN116738347 A CN 116738347A CN 202310568431 A CN202310568431 A CN 202310568431A CN 116738347 A CN116738347 A CN 116738347A
- Authority
- CN
- China
- Prior art keywords
- data
- index data
- early warning
- index
- alarm
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000004458 analytical method Methods 0.000 title claims abstract description 51
- 238000000034 method Methods 0.000 title claims abstract description 40
- 238000010801 machine learning Methods 0.000 title claims abstract description 27
- 230000002159 abnormal effect Effects 0.000 claims abstract description 28
- 238000013145 classification model Methods 0.000 claims abstract description 22
- 238000012549 training Methods 0.000 claims abstract description 21
- 238000012545 processing Methods 0.000 claims abstract description 17
- 238000000605 extraction Methods 0.000 claims abstract description 12
- 238000012423 maintenance Methods 0.000 claims abstract description 12
- 238000002372 labelling Methods 0.000 claims abstract description 5
- 238000001514 detection method Methods 0.000 claims description 16
- 230000004807 localization Effects 0.000 claims description 11
- 238000006243 chemical reaction Methods 0.000 claims description 8
- 238000003745 diagnosis Methods 0.000 claims description 7
- 238000001914 filtration Methods 0.000 claims description 7
- 230000008569 process Effects 0.000 claims description 7
- 238000011897 real-time detection Methods 0.000 claims description 7
- 238000004590 computer program Methods 0.000 claims description 6
- 238000004422 calculation algorithm Methods 0.000 claims description 5
- 238000013507 mapping Methods 0.000 claims description 5
- 238000010606 normalization Methods 0.000 claims description 4
- 230000002776 aggregation Effects 0.000 claims description 2
- 238000004220 aggregation Methods 0.000 claims description 2
- 230000006835 compression Effects 0.000 claims description 2
- 238000007906 compression Methods 0.000 claims description 2
- 238000003860 storage Methods 0.000 claims description 2
- 230000000873 masking effect Effects 0.000 claims 4
- 230000000737 periodic effect Effects 0.000 description 10
- 238000013528 artificial neural network Methods 0.000 description 9
- 238000010586 diagram Methods 0.000 description 9
- 238000013527 convolutional neural network Methods 0.000 description 8
- 230000005856 abnormality Effects 0.000 description 6
- 238000013473 artificial intelligence Methods 0.000 description 3
- 238000007726 management method Methods 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 125000004122 cyclic group Chemical group 0.000 description 2
- 230000000306 recurrent effect Effects 0.000 description 2
- 230000008439 repair process Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 241001123248 Arma Species 0.000 description 1
- 241000282414 Homo sapiens Species 0.000 description 1
- 238000012300 Sequence Analysis Methods 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000005540 biological transmission Effects 0.000 description 1
- 230000001419 dependent effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 230000006870 function Effects 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000001788 irregular Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- YHXISWVBGDMDLQ-UHFFFAOYSA-N moclobemide Chemical compound C1=CC(Cl)=CC=C1C(=O)NCCN1CCOCC1 YHXISWVBGDMDLQ-UHFFFAOYSA-N 0.000 description 1
- 238000003032 molecular docking Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 239000002243 precursor Substances 0.000 description 1
- 238000003672 processing method Methods 0.000 description 1
- 238000011895 specific detection Methods 0.000 description 1
- 238000000547 structure data Methods 0.000 description 1
- 238000013024 troubleshooting Methods 0.000 description 1
- 238000012795 verification Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/24—Querying
- G06F16/245—Query processing
- G06F16/2458—Special types of queries, e.g. statistical queries, fuzzy queries or distributed queries
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/28—Databases characterised by their database models, e.g. relational or object models
- G06F16/284—Relational databases
- G06F16/285—Clustering or classification
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/10—Pre-processing; Data cleansing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/213—Feature extraction, e.g. by transforming the feature space; Summarisation; Mappings, e.g. subspace methods
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/042—Knowledge-based neural networks; Logical representations of neural networks
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/044—Recurrent networks, e.g. Hopfield networks
- G06N3/0442—Recurrent networks, e.g. Hopfield networks characterised by memory or gating, e.g. long short-term memory [LSTM] or gated recurrent units [GRU]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/04—Architecture, e.g. interconnection topology
- G06N3/0464—Convolutional networks [CNN, ConvNet]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- Databases & Information Systems (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Bioinformatics & Computational Biology (AREA)
- Evolutionary Computation (AREA)
- Evolutionary Biology (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Artificial Intelligence (AREA)
- Life Sciences & Earth Sciences (AREA)
- Fuzzy Systems (AREA)
- Mathematical Physics (AREA)
- Probability & Statistics with Applications (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Test And Diagnosis Of Digital Computers (AREA)
Abstract
The invention discloses a fault analysis and positioning method and system based on machine learning, wherein the method comprises the following steps: acquiring alarm data and index data; carrying out standardized processing on the acquired alarm data and index data; shielding the processed alarm data and index data through a shielding rule to obtain clean alarm data and index data; adding associated information for clean alarm data and index data to obtain sample data; sample labeling and feature extraction are carried out on the sample data, so that a feature data set is obtained; training the abnormal classification model by adopting a characteristic data set to obtain a trained abnormal classification model; and inputting the index data acquired in real time into a trained abnormal classification model to obtain a classification result, and performing early warning analysis on the index data with the classification result not being alarmed to obtain an early warning result. The invention can assist in analyzing and positioning faults, building a fault early warning automatic matching knowledge base and improving the overall operation and maintenance level.
Description
Technical Field
The invention relates to an information technology, in particular to a fault analysis positioning method and system based on machine learning.
Background
In a broad sense, machine learning is a method that can be given the ability to machine learning to allow it to perform functions that direct programming cannot do. In a practical sense, however, machine learning is a method of training a model by using data and then using model predictions. It is generally known that a processing system and algorithm for Machine Learning (ML) are mainly used for finding hidden patterns in data to make a predicted recognition pattern, which is an important sub-field of artificial intelligence (Artificial Intelligence, AI) and is also a core of artificial intelligence, and it is mainly used for researching how a computer simulates or implements Learning behaviors of human beings to acquire new knowledge or skills, and helping the computer reorganize existing knowledge structures to continuously improve the performance of the computer.
In the prior art, the problems of difficult fault positioning, inaccurate analysis and diagnosis results and the like exist, and a new fault analysis and positioning method and system are needed to improve the overall operation and maintenance level.
Disclosure of Invention
The invention aims to: the invention aims to provide a fault analysis positioning method and system based on machine learning, so that the problems of difficult fault positioning, inaccurate analysis and diagnosis results and the like are solved, and the horizontal improvement of the overall operation and maintenance is realized.
The technical scheme is as follows: the invention discloses a fault analysis and positioning method based on machine learning, which comprises the following steps:
step 1, acquiring alarm data and index data;
step 2, carrying out standardized processing on the acquired alarm data and index data;
step 3, shielding the standardized alarm data and index data through a shielding rule to obtain clean alarm data and index data;
step 4, adding relevant information to clean alarm data and index data by matching, converting, mapping and extracting by using a configuration management database CMDB to obtain sample data;
step 5, sample labeling and feature extraction are carried out on the sample data to obtain a feature data set;
step 6, training the abnormal classification model by adopting a characteristic data set to obtain a trained abnormal classification model;
and 7, inputting the index data acquired in real time into a trained abnormal classification model to obtain a classification result, and performing early warning analysis on the index data with the non-alarming classification result to obtain an early warning result.
The normalization process in step 2 includes format conversion, data filtering, data deduplication, data splitting, data merging, condition judgment, field addition, and field removal. The above standardized processing methods are not performed sequentially, but are performed according to service requirements, for example: and converting the format of a specific field in the currently acquired data into a format conforming to the algorithm requirement or removing redundant field information, thereby reducing analysis and processing cost. The data processing provides a technical means and capability to process the source data into a format used for meeting the service requirements of the fault analysis scene, for example: when predicting the index trend, only three fields of the index name, date, value in the source data are needed for prediction.
The shielding rule in the step 3 refers to automatic aggregation, compression and grading of the same management object alarms according to machine shielding, time shielding, grade shielding.
The step 4 specifically comprises the following steps:
step 4.1, picking the server addresses in the alarm data and the index data, namely IP information, through the steps of data splitting and data deduplication;
step 4.2, obtaining topology architecture information of the whole system through butting the cmdb;
step 4.3, configuring names of all levels according to the system topology architecture, and dividing corresponding fault analysis diagnosis system levels according to modules, types, machines, examples and indexes:
a. the module comprises a service cluster, a micro-service, a task environment and load balancing;
b. the type comprises a host, a database, middleware and interface detection;
c. the machine refers to the machine information actually acquired in the step 4.1, and is distributed through a checking form;
d. the examples refer to the specific running examples of the machine;
e. the index refers to automatic division through an algorithm after data splitting and data duplication removal according to types and machine addresses;
and 4.4, after the business state is abnormal, analyzing the related hierarchical performance indexes through fault analysis diagnosis capability and a system hierarchical structure, giving out modules, types, machines, examples and indexes related to the abnormal state, presenting operation data and alarm data information related to the abnormal state, reducing the fault removal range, improving the efficiency and assisting operation and maintenance personnel to quickly solve the faults.
The early warning analysis in the step 7 comprises the following steps:
forming an index data fitting curve by using index data with the classification result of no alarm;
determining an early warning interval by a 3sigma criterion or a quantile criterion;
and fitting a curve and an early warning interval according to the index data to obtain an early warning result.
A machine learning based fault analysis and localization system comprising the following modules:
and a data acquisition module: the method is used for acquiring alarm data and index data;
and a data processing module: the method comprises the steps of carrying out standardized processing on acquired alarm data and index data, shielding the alarm data and the index data after the standardized processing through shielding rules to obtain clean alarm data and index data, adding associated information to the clean alarm data and the index data through matching, conversion, mapping and extraction by utilizing a configuration management database CMDB to obtain sample data, and carrying out sample labeling and feature extraction on the sample data to obtain feature data;
model training module: the method comprises the steps of training an abnormal classification model by adopting a characteristic data set to obtain a trained abnormal classification model;
and the real-time detection module is used for: the method comprises the steps of inputting index data acquired in real time into a trained abnormal classification model to obtain a classification result;
early warning analysis module: and the early warning analysis is used for carrying out early warning analysis on the index data with the classified result of no warning, so as to obtain an early warning result.
The normalization processing comprises format conversion, data filtering, data de-duplication, data splitting, data merging, condition judgment, field addition and field removal.
The early warning analysis module comprises the following submodules:
the index data fitting curve generating module: the method comprises the steps of using the classified result as index data without warning to form an index data fitting curve;
the early warning interval determining module: the early warning interval is determined by a 3sigma criterion or a quantile criterion;
the early warning result output module: and the method is used for fitting a curve and an early warning interval according to the index data to obtain an early warning result.
A computer storage medium having stored thereon a computer program which, when executed by a processor, implements a machine learning based fault analysis localization method as described above.
A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, the processor implementing a machine learning based fault analysis localization method as described above when executing the computer program.
The beneficial effects are that: compared with the prior art, the invention has the following advantages: 1. according to the invention, expert experience, intelligent analysis results and deep analysis on fault cases found in a production environment are converted into knowledge tools which can be directly used in the technical modes of machine learning and the like, expert knowledge is directly converted into capabilities, power-assisted fault analysis and positioning are performed, a fault and early warning automatic matching knowledge base is built, and the overall operation and maintenance level is improved; 2. according to the invention, by constructing a comprehensive visual and observable platform, the island of the operation and maintenance data is broken, and the multi-type data is fused, so that the value of the operation and maintenance data is realized; 3. the invention can find the source of fault occurrence, guide operation and maintenance personnel to carry out targeted repair, improve efficiency and quality and save cost
Drawings
FIG. 1 is a data source type selection interface;
FIG. 2 is a newly added data source;
FIG. 3 is a field selection;
FIG. 4 is an association data set;
FIG. 5 is a SQL conditional predicate data set;
FIG. 6 is an application of supervised learning in classifying problems;
FIG. 7 is a model detection and real-time detection flow;
FIG. 8 is an anomaly training model classification result;
FIG. 9 is a periodic anomaly detection model detection result;
FIG. 10 is a block diagram of a gated recurrent neural network;
FIG. 11 is a block diagram of a convolutional neural network;
FIG. 12 is a schematic network architecture diagram;
FIG. 13 is a diagram of a multi-layer perceptron;
FIG. 14 is a flow chart of a modular structure;
fig. 15 is a schematic diagram of fault reporting.
Detailed Description
The technical scheme of the invention is further described below with reference to the accompanying drawings.
As shown in fig. 1-15, a fault analysis positioning method based on machine learning includes the following steps:
step 1, defining acquisition and access modes according to different data source types, and acquiring data such as alarms and indexes; specifically, through the graphical page, according to the types of Kafka, mysql, interfaces, file transmission and the like (shown in fig. 1), corresponding acquisition and access modes (shown in fig. 2) are set, and corresponding access rules, access fields and other information are set, so that data acquisition (shown in fig. 3) is realized.
And 2, based on the data set, carrying out standardized processing (shown in fig. 4 and 5) on various data acquired in the step 1, realizing data processing such as format conversion, data filtering, data deduplication, data splitting, data converging, condition judgment, field addition, field removal and the like, rapidly completing data processing and access work such as pipelined alarm, index and the like, and providing a usable data base for constructing a fault intelligent analysis scene.
And 3, filtering and shielding the data such as alarms, indexes and the like through various shielding rules. The data conforming to the shielding rule is shielded, and no notification is sent and stored in the database, so that shielding and filtering of the data conforming to the conditions such as known faults, equipment maintenance, service peaks and the like are realized, and the interference of the data to maintenance personnel is reduced.
And 4, based on the acquired data such as the alarm and the index, simultaneously docking with the CMDB, adding the associated information for the alarm and the index through operations such as matching, conversion, mapping, extraction and the like, and enriching the data content to improve the readability of the alarm information.
Step 5, marking and extracting the characteristics of the data processed in the step 4 to obtain a characteristic data set;
step 6, training on a specified learning system by using a characteristic data set in a supervised learning mode to obtain an abnormal classification model; the overall thought of supervised learning is as follows: (X1, Y1), (X2, Y2), …, (xn, yn) is the training data set, the learning system learns a classifier P (y|x) or y=f (X) from the training data, the classification system classifies the new input instance xn+1 by the learned classifier, and predicts its output class yn+1.
And 7, inputting the index data acquired in real time into a trained abnormal classification model to obtain a classification result, and performing early warning analysis on the index data with the non-alarming classification result to obtain an early warning result. The early warning analysis here is: and detecting and analyzing the index data, timely finding out the index abnormality to reveal the risk in advance, realizing early finding and early processing, and winning precious time for subsequent fault diagnosis and repair. The index data fitting curve may be determined by fbprophet, arma, arima, exponential smoothing, etc., the interval may be determined by a 3sigma criterion, or a quantile criterion, and the anomaly may be determined outside the interval.
The abnormal classification model of the embodiment is composed of a GRU network, a CNN network, a GAT network and an MLP; the output of the GRU network is input into the CNN network, the CNN network outputs fault codes, the GAT network takes the fault codes, the dependent fault diagrams and other fault codes on the diagrams as inputs, the output of the GAT network is the fault codes with the diagram information, the MLP takes the fault codes with the diagram information as inputs, and the root cause coefficients are output. The above-mentioned network will now be described.
The GRU network is a gating cyclic neural network (gated recurrent neural network), is a common gating cyclic neural network, and mainly comprises an update gate and a reset gate, and is used for better capturing a dependency relationship with larger interval in time sequence data:
the CNN network is a convolutional neural network (Convolutional Neural Networks, CNN) which is a type of feedforward neural network (Feedforward Neural Networks) comprising convolutional calculations and having a deep structure.
GAT networks are graph-annotation networks (GATs), a new type of neural network architecture that operates on graph structure data, utilizing a masked self-attention layer to address the shortcomings of previous approaches based on graph convolution or its approximation. By stacking layers whose nodes can participate in their neighborhood characteristics, different weights can be (implicitly) assigned to different nodes in the neighborhood without any type of expensive matrix operations (e.g., inversion) or reliance on knowledge of the structure precursors of the graph.
The MLP is a multi-layer perceptron (MLP, multilayer Perceptron), also called artificial neural network (ANN, artificial Neural Network), which may have multiple hidden layers in between, except for input and output layers, the simplest MLP having a structure of only one hidden layer, i.e. three layers.
Example 2:
the technical scheme of the invention will be further described by taking time sequence data as an example.
The time series data are classified into three types of periodic type, stationary type and irregular wave type. Among the three types, the periodic type is most common, the duty ratio is more than 30%, and most service indexes are included, wherein core indexes such as service request quantity, order quantity and the like are periodic, for a large amount of time sequence data, abnormality judgment through rules cannot be met, a general solution is needed, abnormality detection can be carried out on all periodic indexes, and a set of completely independent strategies is not adopted as an index, so that a machine learning method is preferred.
The automatic anomaly detection method for the periodic indexes mainly comprises two parts, namely offline model training and real-time detection, wherein the model training mainly generates a classification model according to sample data set training, and the real-time detection utilizes the classification model to carry out real-time anomaly detection. The specific process is as follows:
offline model training: based on the marked sample data set, performing feature extraction by using a designed feature extractor, generating a feature data set, training by Xgboost, obtaining a classification model, and storing.
And (3) real-time detection: when in online real-time detection, the time sequence data is pre-detected (the probability of entering a feature extraction link is reduced, the calculation pressure is reduced), then feature extraction is carried out according to the designed feature engineering, and then an offline trained model is loaded for abnormal classification.
And (3) data feedback: if it is determined to be abnormal, an alarm is issued.
The user can feed back the alarm according to the actual situation, and the feedback result is added into the sample data set and used for updating the detection model at regular time.
The training flow of the classification model is arranged according to the results explored on the periodic indexes, the number of samples used for training and testing is preset to be 28000, wherein the proportion for training is 75%, the proportion for verification is 25%, the accuracy rate on the test set is 94%, and the recall rate is 89%. Meanwhile, the corresponding execution flow is arranged, and the detection flow mainly comprises logic such as null filling, pre-detection, feature extraction, classification judgment, low peak period judgment, offset fluctuation judgment and the like besides abnormal classification, and the application range of the execution flow is a periodic and stable index. In addition, the process optimization capability is also provided, each algorithm in the detection process can expose the super-parameters, and for a specific index, a group of better super-parameters in the process can be obtained through training of sample data of the index, so that the recall rate of abnormality detection of the index is improved.
After the abnormality detection flow is applied to indexes of a production environment, a specific detection effect related scheme is shown in fig. 9, and for periodic indexes, abnormality can be timely and accurately found, feedback is carried out on abnormal points, and the accuracy rate reaches more than 90%. In addition, deformation analysis anomaly detection is also compared, and for 4 cases which cannot be found by three deformation analysis in a production environment, 3 cases can be found by a periodic index anomaly detection flow, and the performance is superior to that of the deformation analysis.
By a time sequence analysis method, historical occurrence characteristics of alarms at the moment of faults are described, and time sequence characteristic labels are added to the alarms to assist operation and maintenance personnel in identifying noise alarms and real fault alarms in the alarms by combining occurrence conditions of the alarms at the moment of faults, so that the dynamic level of the alarms at the moment of faults is judged. The suspicious degree of the faults is ordered by combining the alarm characteristics, IT architecture characteristics and actual running conditions during the faults, so that first-line operators are assisted to judge the overall fault conditions of the system and conduct fault troubleshooting more orderly.
Claims (10)
1. The fault analysis and positioning method based on machine learning is characterized by comprising the following steps of:
step 1, acquiring alarm data and index data;
step 2, carrying out standardized processing on the acquired alarm data and index data;
step 3, shielding the standardized alarm data and index data through a shielding rule to obtain clean alarm data and index data;
step 4, adding relevant information to clean alarm data and index data by matching, converting, mapping and extracting by using a configuration management database CMDB to obtain sample data;
step 5, sample labeling and feature extraction are carried out on the sample data to obtain a feature data set;
step 6, training the abnormal classification model by adopting a characteristic data set to obtain a trained abnormal classification model;
and 7, inputting the index data acquired in real time into a trained abnormal classification model to obtain a classification result, and performing early warning analysis on the index data with the non-alarming classification result to obtain an early warning result.
2. The machine learning based fault analysis localization method of claim 1, wherein the normalization process in step 2 comprises format conversion, data filtering, data deduplication, data offloading, data merging, condition judgment, field addition, and field removal.
3. The machine learning based fault analysis positioning method according to claim 1, wherein the masking rule in step 3 refers to automatic aggregation, compression and classification of alarms of the same management object according to machine masking, time masking, level masking.
4. The machine learning-based fault analysis and localization method according to claim 1, wherein the step 4 specifically comprises:
step 4.1, picking the server addresses in the alarm data and the index data, namely IP information, through the steps of data splitting and data deduplication;
step 4.2, obtaining topology architecture information of the whole system through butting the cmdb;
step 4.3, configuring names of all levels according to the system topology architecture, and dividing corresponding fault analysis diagnosis system levels according to modules, types, machines, examples and indexes;
a. the module comprises a service cluster, a micro-service, a task environment and load balancing;
b. the type comprises a host, a database, middleware and interface detection;
c. the machine refers to the machine information actually acquired in the step 4.1, and is distributed through a checking form;
d. the examples refer to the specific running examples of the machine;
e. the index refers to automatic division through an algorithm after data splitting and data duplication removal according to types and machine addresses;
and 4.4, after the business state is abnormal, analyzing the related hierarchical performance indexes through fault analysis diagnosis capability and a system hierarchical structure, giving out modules, types, machines, examples and indexes related to the abnormal state, presenting operation data and alarm data information related to the abnormal state, reducing the fault removal range, improving the efficiency and assisting operation and maintenance personnel to quickly solve the faults.
5. The machine learning based fault analysis and localization method of claim 1, wherein the pre-warning analysis in step 7 comprises the steps of:
forming an index data fitting curve by using index data with the classification result of no alarm;
determining an early warning interval by a 3sigma criterion or a quantile criterion;
and fitting a curve and an early warning interval according to the index data to obtain an early warning result.
6. A machine learning based fault analysis and localization system, comprising the following modules:
and a data acquisition module: the method is used for acquiring alarm data and index data;
and a data processing module: the method comprises the steps of carrying out standardized processing on acquired alarm data and index data, shielding the alarm data and the index data after the standardized processing through shielding rules to obtain clean alarm data and index data, adding associated information to the clean alarm data and the index data through matching, conversion, mapping and extraction by utilizing a configuration management database CMDB to obtain sample data, and carrying out sample labeling and feature extraction on the sample data to obtain feature data;
model training module: the method comprises the steps of training an abnormal classification model by adopting a characteristic data set to obtain a trained abnormal classification model;
and the real-time detection module is used for: the method comprises the steps of inputting index data acquired in real time into a trained abnormal classification model to obtain a classification result;
early warning analysis module: and the early warning analysis is used for carrying out early warning analysis on the index data with the classified result of no warning, so as to obtain an early warning result.
7. The machine learning based fault analysis localization system of claim 6, wherein the normalization process comprises format conversion, data filtering, data deduplication, data splitting, data merging, condition determination, field addition, and field removal.
8. The machine learning based fault analysis and localization system of claim 6, wherein the pre-alarm analysis module comprises the following sub-modules:
the index data fitting curve generating module: the method comprises the steps of using the classified result as index data without warning to form an index data fitting curve;
the early warning interval determining module: the early warning interval is determined by a 3sigma criterion or a quantile criterion;
the early warning result output module: and the method is used for fitting a curve and an early warning interval according to the index data to obtain an early warning result.
9. A computer storage medium having stored thereon a computer program which, when executed by a processor, implements a machine learning based fault analysis localization method as claimed in any one of claims 1 to 4.
10. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, wherein the processor implements a machine learning based fault analysis localization method as claimed in any one of claims 1 to 4 when the computer program is executed by the processor.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310568431.7A CN116738347A (en) | 2023-05-19 | 2023-05-19 | Fault analysis positioning method and system based on machine learning |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202310568431.7A CN116738347A (en) | 2023-05-19 | 2023-05-19 | Fault analysis positioning method and system based on machine learning |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116738347A true CN116738347A (en) | 2023-09-12 |
Family
ID=87900244
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202310568431.7A Pending CN116738347A (en) | 2023-05-19 | 2023-05-19 | Fault analysis positioning method and system based on machine learning |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116738347A (en) |
-
2023
- 2023-05-19 CN CN202310568431.7A patent/CN116738347A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111209131B (en) | Method and system for determining faults of heterogeneous system based on machine learning | |
| CN111124840B (en) | Method and device for predicting alarm in business operation and maintenance and electronic equipment | |
| CN109800127A (en) | A kind of system fault diagnosis intelligence O&M method and system based on machine learning | |
| CN110097037A (en) | Intelligent monitoring method, device, storage medium and electronic equipment | |
| CN113822421B (en) | Neural network-based anomaly locating method, system, equipment and storage medium | |
| KR20180108446A (en) | System and method for management of ict infra | |
| CN112101431A (en) | Electronic equipment fault diagnosis system | |
| CN110757510B (en) | Method and system for predicting remaining life of robot | |
| CN112083244A (en) | Integrated avionics equipment fault intelligent diagnosis system | |
| CN109871002B (en) | Concurrent abnormal state identification and positioning system based on tensor label learning | |
| CN117393076B (en) | Intelligent monitoring method and system for heat-resistant epoxy resin production process | |
| CN117034143B (en) | Distributed system fault diagnosis method and device based on machine learning | |
| CN114139589A (en) | Fault diagnosis method, device, equipment and computer readable storage medium | |
| CN117406689A (en) | Data driving and knowledge guiding fault diagnosis method and system | |
| CN117114420B (en) | Image recognition-based industrial and trade safety accident risk management and control system and method | |
| Wang et al. | Software defect fault intelligent location and identification method based on data mining | |
| CN113093695A (en) | Data-driven SDN controller fault diagnosis system | |
| Bond et al. | A hybrid learning approach to prognostics and health management applied to military ground vehicles using time-series and maintenance event data | |
| CN115718472A (en) | Fault scanning and diagnosing method for hydroelectric generating set | |
| CN116738347A (en) | Fault analysis positioning method and system based on machine learning | |
| CN113296994A (en) | Fault diagnosis system and method based on domestic computing platform | |
| CN110728310A (en) | Target detection model fusion method and system based on hyper-parameter optimization | |
| Wang et al. | Complex equipment diagnostic reasoning based on neural network algorithm | |
| Saddem et al. | Machine learning-based approach for online fault Diagnosis of Discrete Event System | |
| CN116771576B (en) | Comprehensive fault diagnosis method for hydroelectric generating set |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |