CN116244646A - Machine learning-based big data platform fault diagnosis method and system - Google Patents
Machine learning-based big data platform fault diagnosis method and system Download PDFInfo
- Publication number
- CN116244646A CN116244646A CN202211705102.4A CN202211705102A CN116244646A CN 116244646 A CN116244646 A CN 116244646A CN 202211705102 A CN202211705102 A CN 202211705102A CN 116244646 A CN116244646 A CN 116244646A
- Authority
- CN
- China
- Prior art keywords
- model
- fault diagnosis
- data platform
- fault
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
- G06N3/00—Computing arrangements based on biological models
- G06N3/02—Neural networks
- G06N3/08—Learning methods
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- General Health & Medical Sciences (AREA)
- Biomedical Technology (AREA)
- Biophysics (AREA)
- Computational Linguistics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Evolutionary Computation (AREA)
- Artificial Intelligence (AREA)
- Molecular Biology (AREA)
- Computing Systems (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Mathematical Physics (AREA)
- Software Systems (AREA)
- Health & Medical Sciences (AREA)
- Test And Diagnosis Of Digital Computers (AREA)
Abstract
The invention discloses a machine learning-based fault diagnosis method and system for a big data platform, belonging to the field of machine learning; modeling is carried out by the method, and a fault diagnosis model is established; the method is used for rapidly processing the operation problem of the big data platform, solving the problem of difficult investigation of the big data platform assembly, having long investigation period, improving the capability of solving the problem, providing a beneficial decision for intelligent maintenance of the big data platform, and predicting the residual service life of the equipment based on the beneficial decision, thereby making active equipment maintenance and guarantee measures. The invention can automatically diagnose the system faults, and the maintenance is simpler.
Description
Technical Field
The invention discloses a machine learning-based fault diagnosis method and system for a big data platform, and relates to the technical field of machine learning.
Background
With the opening of the new industrial revolution, technologies such as internet of things, industrial internet, artificial intelligence and the like become the most attractive new star on the stage.
Bayesian Networks (BN) are a graphical network model for describing the causal relationships of uncertainty between variables, used for uncertainty system modeling and reasoning, and deal with problems related to predictive intelligent reasoning, diagnosis, decision risk and reliability analysis.
The big data platform is particularly referred to as hadoop platform, which relates to various technical components, and has various professional technical types required for maintenance, and has high technical level requirements for technicians. To improve the problem-solving efficiency, prevent the occurrence of problems, and failure diagnosis based on machine learning has been developed.
Early fault diagnosis typically relies on the experience of the technician. For example, a professional engineer may determine the problem of operation by looking at the cause of a fault in the status diagnostic platform service of a component in the big data platform, or by analyzing the error information through the operation log. Obviously, the method relying on manual judgment has larger problems in terms of accuracy, expansibility, instantaneity and the like. In recent years, the establishment of a fault classifier by using a machine learning model is the most popular field of fault diagnosis research. The data-driven diagnosis method can automatically diagnose the fault type of the equipment according to the running state data of the equipment, and has more advantages than the manual method in the aspects of accuracy, expansibility, instantaneity and the like.
Therefore, the invention discloses a machine learning-based fault diagnosis method and system for a big data platform, which are used for solving the problems.
Disclosure of Invention
Aiming at the problems in the prior art, the invention provides a machine learning-based large data platform fault diagnosis method and a machine learning-based large data platform fault diagnosis system, and the adopted technical scheme is as follows: a big data platform fault diagnosis method based on machine learning, the method models and establishes a fault diagnosis model;
the method is in the training stage of the fault diagnosis model: obtaining a training sample, inputting the marked training sample into a machine learning model, judging the fault type of the current big data platform, carrying out normalization processing on training sample data, and optimizing model parameters by minimizing model classification errors so as to obtain an optimal fault diagnosis model;
the method is in the working phase of a fault diagnosis model: and inputting the sample without the mark into a fault diagnosis model, and calculating the fault type of the current large data platform.
The method is in the training stage of the fault diagnosis model, and the specific method is as follows:
s11, data processing, namely collecting system information and monitoring information of software services in a big data platform, retrieving a system operation log, and enabling network information in a local area network to comprise traffic and network connectivity;
s12, preprocessing the characteristics, namely preprocessing the acquired frequency domain data, time domain data and time frequency data so as to be suitable for machine learning and model learning;
and S13, modeling a fault diagnosis model, adopting a deep convolutional neural network model, completing fault classification through a softmax function, extracting features by convolution, removing redundancy information in the features by pooling, and reducing the dimension of the features.
In the working stage of the fault diagnosis model, the method utilizes a TANd classifier to carry out fault diagnosis on the big data platform product, and the specific method is as follows:
s21, combining the preprocessed fault data and the conventional data to form a training set, and generating a TANd classifier model according to the training set;
s22, performing fault diagnosis on the big data platform product by using the generated TANd classifier model.
And the TANd classifier model calculates condition information functions among different attribute qualities, and establishes a directed acyclic graph in a condition arc among the attribute.
A big data platform fault diagnosis system based on machine learning, the system models and establishes a fault diagnosis model;
the system is in the training phase of the fault diagnosis model: obtaining a training sample, inputting the marked training sample into a machine learning model, judging the fault type of the current big data platform, carrying out normalization processing on training sample data, and optimizing model parameters by minimizing model classification errors so as to obtain an optimal fault diagnosis model;
the system is in the working phase of a fault diagnosis model: and inputting the sample without the mark into a fault diagnosis model, and calculating the fault type of the current large data platform.
The system specifically comprises the following steps in a training stage of a fault diagnosis model:
the data processing module is used for collecting system information and monitoring information of software services in the big data platform, retrieving a system operation log and network information in the local area network, wherein the network information comprises flow and network connection conditions;
the characteristic preprocessing module is used for preprocessing the acquired frequency domain data, time domain data and time frequency data so as to be suitable for machine learning and model learning;
the fault diagnosis model modeling module adopts a deep convolutional neural network model, completes fault classification through a softmax function, extracts features by convolution, removes redundancy information in the features by pooling, and reduces the dimension of the features.
In the working stage of the fault diagnosis model, the system utilizes the TANd classifier to carry out fault diagnosis on the big data platform product, and specifically comprises the following steps:
and a data combination module: combining the preprocessed fault data and the conventional data to form a training set, and generating a TANd classifier model according to the training set;
the diagnosis processing module: and performing fault diagnosis on the big data platform product by using the generated TANd classifier model.
And the TANd classifier model calculates condition information functions among different attribute qualities, and establishes a directed acyclic graph in a condition arc among the attribute.
The beneficial effects of the invention are as follows: the method is used for rapidly processing the operation problem of the big data platform, solves the problem of difficult investigation of the big data platform assembly, has long investigation period, improves the capability of solving the problem, provides beneficial decisions for intelligent maintenance of the big data platform, predicts the residual service life of the equipment based on the beneficial decisions, and thereby establishes active equipment maintenance and guarantee measures. The invention can automatically diagnose the system faults, and the maintenance is simpler.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions of the prior art, the drawings that are needed in the embodiments or the description of the prior art will be briefly described below, it will be obvious that the drawings in the following description are some embodiments of the present invention, and that other drawings can be obtained according to these drawings without inventive effort to a person skilled in the art.
FIG. 1 is a two-stage schematic diagram of machine learning based fault diagnosis of an embodiment of the method of the present invention;
FIG. 2 is a schematic of the basic flow of an embodiment of the method of the present invention;
FIG. 3 is a schematic diagram of a fault diagnosis model of a deep convolutional neural network in an embodiment of the method of the present invention.
Detailed Description
The present invention will be further described with reference to the accompanying drawings and specific examples, which are not intended to be limiting, so that those skilled in the art will better understand the invention and practice it.
Embodiment one:
a big data platform fault diagnosis method based on machine learning, the method models and establishes a fault diagnosis model;
machine learning based fault diagnosis methods are typically modeled as a supervised multi-classification problem, as shown in fig. 1. The method comprises the following two stages:
the method is in the training stage of the fault diagnosis model: in the training stage, training samples are acquired, and the sample data comprise system operation parameter data, monitoring data, operation log data, network data and the like of a big data platform. Inputting the marked training sample into a machine learning model, judging the fault type of the current big data platform, carrying out normalization processing on the training sample data, and optimizing model parameters by minimizing model classification errors, thereby obtaining an optimal fault diagnosis model.
The method is in the working phase of a fault diagnosis model: in the diagnosis stage, the sample without the mark is input into a fault diagnosis model, and the fault type of the current large data platform is calculated. The key points are mainly focused on two aspects: the first key point is how to construct the sample, i.e. which state parameters of the big data platform are taken as sample features; the second key point is how to design the structure of the machine learning model to obtain the most accurate classification result.
Further, in the training stage of the fault diagnosis model, the basic flow of establishing the multi-classifier based on the labeled sample set is shown in fig. 2, and the method comprises the following 3 steps of data collection, feature preprocessing and modeling:
s11, data processing, namely collecting system information and monitoring information of software services in a big data platform, retrieving a system operation log, and enabling network information in a local area network to comprise traffic and network connectivity;
s12, preprocessing the characteristics, namely collecting system information in a big data platform and monitoring information of various software services, retrieving a system operation log, and obtaining network information in a local area network, wherein the network information comprises traffic and network connectivity.
S13, modeling of a fault diagnosis model, and with the deep neural network, great success is achieved in natural language processing and image recognition. Deep neural networks are also introduced in intelligent diagnosis of equipment failures. The model of this scheme employs a deep convolutional neural network model, which is a deep feed-forward neural network with local connections and weight sharing, as shown in fig. 3. It includes several convolution layers and pooling layers, and finally, fault classification is accomplished by a softmax function. The convolution is used for extracting the features, and the pooling can remove redundant information in the features and reduce the dimension of the features.
The calculation principle adopted by the invention is Bayesian theorem. Let the sample space of test E be S, the event of a being E, B1, B2, …, bn be a partition of S, and P (a) >0, P (Bi) >0 (i=1, 2, …, n) be the bayesian formula
For a given sample x, P (x) is independent of class labels, P (c) is referred to as class prior probability, and P (x|c) is referred to as class conditional probability.
Prior probability: based on past experience and analysis.
Posterior probability: the posterior probability is a probability estimate closer to the actual situation obtained by correcting the original prior probability based on the new information.
For the class prior probability p (c), p (c) is the proportion of various samples in the sample space, and according to the big theorem (when the samples are enough, the frequency tends to be stable and equal to the probability thereof), when the training samples are enough, the p (c) can be replaced by the frequency of various occurrences. Therefore, only class conditional probability p (x|c) remains, which means the probability of x occurring in class c, which involves the problem of joint probability of attributes, and if only one discrete attribute is good, it is very difficult to estimate the frequency when the attributes are many, so that the maximum likelihood method is generally used for estimation.
It is clear that the biggest problem of the original bayesian classifier is that the estimation of the joint probability density function firstly needs to be based on experience to assume the joint probability distribution, and secondly when the attributes are many, the training samples are often not covered enough, and great deviation occurs in the estimation of the parameters. To avoid this problem, a naive bayes classifier (naive bayes classifier) employs an "attribute condition independence assumption", i.e., all attributes of sample data are independent of each other. Such a class conditional probability p (x|c) can be rewritten as:
in this way, estimating the class conditional probability for each sample becomes estimating the class conditional probability for each attribute of each sample.
In the model modeling process, a discrimination class condition Bayesian network model is combined with a TAN classifier (the TAN classifier is an improved model of a naive Bayesian classifier), parameters of each node are obtained by maximizing a conditional log likelihood function, and a quantum particle swarm optimization algorithm is used for maximizing an objective function. This classifier is denoted as a tadd classifier. The advantages of classifying the TANd classifier are not obvious enough when the data size is smaller, but when the scale of the data size is increased and the independence assumption condition among the attributes of each parameter is not satisfied, the classification accuracy of the common Bayesian network classifier is greatly reduced, the classification effect of the TANd classifier can still keep high classification accuracy, and the classification problem is more efficient.
Furthermore, in the working stage of the fault diagnosis model, the method utilizes the TANd classifier to carry out fault diagnosis on the big data platform product, and the specific method is as follows:
s21, combining the preprocessed fault data and the conventional data to form a training set, and generating a TANd classifier model according to the training set;
s22, performing fault diagnosis on the big data platform product by using the generated TANd classifier model.
Still further, the TANd classifier model calculates the condition information function between different attributes, and establishes a directed acyclic graph in the condition arc between the attributes.
Because the parameters of the big data platform are not completely independent of each other, certain relation exists between the attribute parameters. The TANd classifier calculates the condition mutual information function among different attributes, adds arcs among the attributes, and establishes a directed acyclic graph. And classifying the training set according to the Bayesian theory and the maximum posterior probability principle, and judging whether the training set belongs to fault data or which type of faults.
Data preprocessing: and selecting 4 groups of fault data of hadoop as NameNode abnormality, HDFS disk abnormality, HBase abnormality and resource manager abnormality. In order to improve the classification precision and the optimization speed, the corresponding parameters of the partial components which are easy to break down are selected as the final attribute parameters participating in classification. To improve the classification and optimization efficiency, the data can be normalized by the following formula:
in the formula, X ij The value of the jth attribute, X, for the ith sample j Mean value of j-th attribute in data sample
The training set is adopted as a complete data training set, and comprises data of Normal operation and 4 kinds of fault data, and 5 kinds of class variables are taken as a total, so that the total is { NameNode abnormality, HDFS disk abnormality, HBase abnormality, resource manager abnormality, normal } and is marked as { Fautl1, fautl2, fautl3, fautl4 and Normal }.
The data results are as follows
Where TPRate represents the positive sample rate predicted to be positive by the model, FPRate represents the negative sample rate predicted to be positive by the model, precision is Precision, recall is, F-Measure is the weighted harmonic mean of Precision and Recall, and the closer the value is to 1, the more efficient the diagnostic method is.
The adoption of the QPSO optimization algorithm is independent of the speed of the particles, the position of the particles and the probability density generated by the wave function. Under the condition that the particle positions are bounded, the algorithm meets the global convergence condition and can converge to the global optimal solution according to the probability, so that the parameters of the Bayesian formula are optimized, and the training is more accurate.
Embodiment two:
a big data platform fault diagnosis system based on machine learning, the system models and establishes a fault diagnosis model;
the system is in the training phase of the fault diagnosis model: obtaining a training sample, inputting the marked training sample into a machine learning model, judging the fault type of the current big data platform, carrying out normalization processing on training sample data, and optimizing model parameters by minimizing model classification errors so as to obtain an optimal fault diagnosis model;
the system is in the working phase of a fault diagnosis model: and inputting the sample without the mark into a fault diagnosis model, and calculating the fault type of the current large data platform.
Further, the system specifically includes, in a training stage of the fault diagnosis model:
the data processing module is used for collecting system information and monitoring information of software services in the big data platform, retrieving a system operation log and network information in the local area network, wherein the network information comprises flow and network connection conditions;
the characteristic preprocessing module is used for preprocessing the acquired frequency domain data, time domain data and time frequency data so as to be suitable for machine learning and model learning;
the fault diagnosis model modeling module adopts a deep convolutional neural network model, completes fault classification through a softmax function, extracts features by convolution, removes redundancy information in the features by pooling, and reduces the dimension of the features.
Further, in the working stage of the fault diagnosis model, the system performs fault diagnosis on the big data platform product by using a TANd classifier, and specifically comprises the following steps:
and a data combination module: combining the preprocessed fault data and the conventional data to form a training set, and generating a TANd classifier model according to the training set;
the diagnosis processing module: and performing fault diagnosis on the big data platform product by using the generated TANd classifier model.
Still further, the TANd classifier model calculates the condition information function between different attributes, and establishes a directed acyclic graph in the condition arc between the attributes.
Finally, it should be noted that: the above embodiments are only for illustrating the technical solution of the present invention, and are not limiting; although the invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical scheme described in the foregoing embodiments can be modified or some technical features thereof can be replaced by equivalents; such modifications and substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims (8)
1. A large data platform fault diagnosis method based on machine learning is characterized in that the method is used for modeling and establishing a fault diagnosis model;
the method is in the training stage of the fault diagnosis model: obtaining a training sample, inputting the marked training sample into a machine learning model, judging the fault type of the current big data platform, carrying out normalization processing on training sample data, and optimizing model parameters by minimizing model classification errors so as to obtain an optimal fault diagnosis model;
the method is in the working phase of a fault diagnosis model: and inputting the sample without the mark into a fault diagnosis model, and calculating the fault type of the current large data platform.
2. The method according to claim 1, characterized in that it is in a training phase of a fault diagnosis model, and comprises the following steps:
s11, data processing, namely collecting system information and monitoring information of software services in a big data platform, retrieving a system operation log, and enabling network information in a local area network to comprise traffic and network connectivity;
s12, preprocessing the characteristics, namely preprocessing the acquired frequency domain data, time domain data and time frequency data so as to be suitable for machine learning and model learning;
and S13, modeling a fault diagnosis model, adopting a deep convolutional neural network model, completing fault classification through a softmax function, extracting features by convolution, removing redundancy information in the features by pooling, and reducing the dimension of the features.
3. The method of claim 2, wherein the method performs fault diagnosis on the big data platform product by using a tad classifier in the working phase of the fault diagnosis model, and the specific method is as follows:
s21, combining the preprocessed fault data and the conventional data to form a training set, and generating a TANd classifier model according to the training set;
s22, performing fault diagnosis on the big data platform product by using the generated TANd classifier model.
4. A method according to claim 3, wherein the TANd classifier model computes a conditional information function between different attributes, and creates a directed acyclic graph from conditional arcs between attributes.
5. A large data platform fault diagnosis system based on machine learning is characterized in that the system is used for modeling and establishing a fault diagnosis model;
the system is in the training phase of the fault diagnosis model: obtaining a training sample, inputting the marked training sample into a machine learning model, judging the fault type of the current big data platform, carrying out normalization processing on training sample data, and optimizing model parameters by minimizing model classification errors so as to obtain an optimal fault diagnosis model;
the system is in the working phase of a fault diagnosis model: and inputting the sample without the mark into a fault diagnosis model, and calculating the fault type of the current large data platform.
6. The system according to claim 5, wherein the system is in a training phase of a fault diagnosis model, and specifically comprises:
the data processing module is used for collecting system information and monitoring information of software services in the big data platform, retrieving a system operation log and network information in the local area network, wherein the network information comprises flow and network connection conditions;
the characteristic preprocessing module is used for preprocessing the acquired frequency domain data, time domain data and time frequency data so as to be suitable for machine learning and model learning;
the fault diagnosis model modeling module adopts a deep convolutional neural network model, completes fault classification through a softmax function, extracts features by convolution, removes redundancy information in the features by pooling, and reduces the dimension of the features.
7. The system according to claim 6, wherein the system performs fault diagnosis on the big data platform product by using a tad classifier in the working phase of the fault diagnosis model, and specifically comprises:
and a data combination module: combining the preprocessed fault data and the conventional data to form a training set, and generating a TANd classifier model according to the training set;
the diagnosis processing module: and performing fault diagnosis on the big data platform product by using the generated TANd classifier model.
8. The system of claim 7, wherein the TANd classifier model computes a conditional information function between different attributes, and creates a directed acyclic graph from conditional arcs between attributes.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211705102.4A CN116244646A (en) | 2022-12-29 | 2022-12-29 | Machine learning-based big data platform fault diagnosis method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211705102.4A CN116244646A (en) | 2022-12-29 | 2022-12-29 | Machine learning-based big data platform fault diagnosis method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116244646A true CN116244646A (en) | 2023-06-09 |
Family
ID=86634084
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211705102.4A Pending CN116244646A (en) | 2022-12-29 | 2022-12-29 | Machine learning-based big data platform fault diagnosis method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116244646A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117232577A (en) * | 2023-09-18 | 2023-12-15 | 杭州奥克光电设备有限公司 | Optical cable distributing box bearing interior monitoring method and system and optical cable distributing box |
-
2022
- 2022-12-29 CN CN202211705102.4A patent/CN116244646A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117232577A (en) * | 2023-09-18 | 2023-12-15 | 杭州奥克光电设备有限公司 | Optical cable distributing box bearing interior monitoring method and system and optical cable distributing box |
| CN117232577B (en) * | 2023-09-18 | 2024-04-05 | 杭州奥克光电设备有限公司 | Optical cable distributing box bearing interior monitoring method and system and optical cable distributing box |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR101984730B1 (en) | Automatic predicting system for server failure and automatic predicting method for server failure | |
| CN113496262B (en) | Data-driven active power distribution network abnormal state sensing method and system | |
| CN105391579B (en) | Power communication network fault positioning method based on crucial alarm collection and supervised classification | |
| US20100030521A1 (en) | Method for analyzing and classifying process data | |
| CN110335168B (en) | Method and system for optimizing power utilization information acquisition terminal fault prediction model based on GRU | |
| CN112217674B (en) | Alarm root cause identification method based on causal network mining and graph attention network | |
| CN112906775B (en) | Equipment fault prediction method and system | |
| CN116244646A (en) | Machine learning-based big data platform fault diagnosis method and system | |
| CN110851654A (en) | Industrial equipment fault detection and classification method based on tensor data dimension reduction | |
| CN114266301A (en) | Intelligent power equipment fault prediction method based on graph convolution neural network | |
| CN111327480A (en) | Method for monitoring multiple QoS of Web service under mobile edge environment | |
| CN113884807A (en) | Power distribution network fault prediction method based on random forest and multi-layer architecture clustering | |
| CN115791174B (en) | Rolling bearing abnormality diagnosis method, system, electronic equipment and storage medium | |
| CN115438190B (en) | Power distribution network fault auxiliary decision knowledge extraction method and system | |
| JP2002169611A (en) | Fault diagnosis system and automated design system therefor | |
| CN112613227B (en) | Model for predicting remaining service life of aero-engine based on hybrid machine learning | |
| Qiu et al. | A fusion of bayesian and fuzzy analysis for print faults diagnosis. | |
| KR20230075150A (en) | Method for managing system health | |
| CN113807704A (en) | Intelligent algorithm platform construction method for urban rail transit data | |
| Mbuli et al. | Root causes analysis and fault prediction in intelligent transportation systems: coupling unsupervised and supervised learning techniques | |
| CN113049249A (en) | Motor bearing fault diagnosis method and system | |
| Balaram et al. | Software Fault Detection using Multi-Distinguished-Features Sampling with Ensemble Random Forest Classifier. | |
| Yin et al. | Cascaded fault detection system of error back-propagation network based on node association degree | |
| Wang et al. | Operating Condition Identification of Complete Wind Turbine using DBN and Improved DDPG-SOM | |
| Din et al. | GNN Approach for Software Reliability |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |