CN110958136A - Deep learning-based log analysis early warning method - Google Patents
Deep learning-based log analysis early warning method Download PDFInfo
- Publication number
- CN110958136A CN110958136A CN201911093214.7A CN201911093214A CN110958136A CN 110958136 A CN110958136 A CN 110958136A CN 201911093214 A CN201911093214 A CN 201911093214A CN 110958136 A CN110958136 A CN 110958136A
- Authority
- CN
- China
- Prior art keywords
- log
- early warning
- logs
- deep learning
- clustering
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/069—Management of faults, events, alarms or notifications using logs of notifications; Post-processing of notifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/06—Management of faults, events, alarms or notifications
- H04L41/0677—Localisation of faults
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
- H04L41/147—Network analysis or design for predicting network behaviour
Abstract
The invention discloses a log analysis early warning method based on deep learning, which comprises the following steps: preprocessing the acquired logs of different types in the target system; performing log analysis on the preprocessed logs by using a clustering-based method; encoding the analyzed log events into digital feature vectors; learning the encoded log by using a neural network based on LSTM and a clustering method based on LogCollect to form early warning information; tracing the source of the early warning information to the component server corresponding to the load, and judging a fault point. The method and the device realize early warning and positioning of possible faults of the application system, provide corresponding solutions, further remove system risks in advance and improve the safety condition of the system.
Description
Technical Field
The invention relates to the technical field of computer communication, in particular to a log analysis early warning method based on deep learning.
Background
At present, for power transformation equipment, correlation analysis is performed on routing inspection data of various power transformation equipment by using Pearson correlation analysis, single-factor variance analysis, statistical comparison and the like, and then a Logistic regression fault prediction model is established to realize advanced early warning on the equipment state; and then, a classified deep learning neural network model is established by utilizing deep learning, so that the possible fault types of the early warning equipment can be accurately judged. And finally, visually displaying the fault early warning model of the transformer equipment by building an intelligent operation and maintenance aid decision big data platform of the transformer substation.
With the development of the technology, the application of the deep learning technology in anomaly detection is more and more extensive, and it is hoped that the early warning, positioning and solution display of the fault can be finally formed by analyzing and integrating the log content of the application system and other information by means of the deep learning technology and other machine learning methods.
Disclosure of Invention
The invention aims to provide a log analysis early warning method based on deep learning, which can be used for early warning and positioning faults possibly occurring in an application system and providing a corresponding solution, thereby eliminating system risks in advance and improving the safety condition of the system.
In order to achieve the purpose, the invention adopts the following technical scheme:
a log analysis early warning method based on deep learning comprises the following steps:
preprocessing the acquired logs of different types in the target system;
performing log analysis on the preprocessed logs by using a clustering-based method;
encoding the analyzed log events into digital feature vectors;
learning the encoded log by using a neural network based on LSTM and a clustering method based on LogCollect to form early warning information;
tracing the source of the early warning information to the component server corresponding to the load, and judging a fault point.
Further, the preprocessing the acquired logs of different types in the target system specifically includes:
configuring all paths for generating logs in the server in the Agent, and collecting different types of logs into a log cleaning server through the Agent;
decomposing and formatting the collected logs through a log cleaning server, and writing the logs into a database;
and performing secondary processing on the log data which is stored in the database and exceeds the set time threshold, and storing only the abnormal log into the HDFS.
Further, before the step of performing log parsing on the preprocessed log by using a clustering-based method, the method further includes:
and connecting each component log in the target system according to the historical standing book information and the IP address recorded in the log.
Further, the log analysis of the preprocessed log by using a clustering-based method specifically includes:
calculating the distance between the logs, and grouping the logs into different clusters by using a clustering method;
a set of event templates is extracted from each cluster so that an original log can be constructed.
Further, the encoding the analyzed log event into a digital feature vector specifically includes:
dividing an original log into a group of log sequences by using different grouping technologies, wherein the log sequences comprise a fixed window, a sliding window and a session window;
an event count vector is generated for each log sequence that represents the number of times each event occurred.
Further, learning the encoded log by using a neural network based on the LSTM and a clustering method based on the LogCollect to form early warning information, specifically comprising:
inputting the offline data into an LSTM-based neural network model for mode evaluation and optimization;
generating normal clusters and abnormal clusters by using historical log data as an initialization stage knowledge base;
adding a new log for learning, and adjusting a knowledge base cluster constructed in an initialization stage;
and predicting whether the target system gives an alarm or not in the next time window by using the trained neural network model, detecting the abnormality by using LogCollect, summarizing prediction results, and filtering by rules to form early warning information.
Further, the generating of normal clusters and abnormal clusters by using the historical log data as an initialization stage knowledge base specifically includes:
the log sequence is vectorized into an event count vector;
the LogCluster respectively clusters the normal event counting vector and the abnormal event counting vector, and generates two groups of vector clusters as a knowledge base in a clustering hierarchy clustering mode;
a representative vector is selected by calculating the centroid of each cluster.
Further, adding a new log for learning, and adjusting a knowledge base cluster constructed in an initialization stage, specifically comprising:
calculating the distance between a given one of the event count vectors and the existing representative vector, and if the minimum distance is less than a threshold, adding the event count vector to the nearest cluster and the representative vector of that cluster to be updated; otherwise, LogCluster will create a new cluster using this event count vector.
Further, the detecting the abnormality by using LogCollect specifically includes:
the distance of the log sequence to a representative vector in the knowledge base is calculated. If the minimum distance is greater than the threshold, the log sequence is reported as anomalous. Otherwise, if the most recent cluster is a normal/abnormal cluster, the log sequence will be reported as normal/abnormal.
Further, tracing to the source of the early warning information to the component server corresponding to the load, judging the fault point, specifically including:
and tracing to a server of the component corresponding to the load according to the association rule of the historical account information and the historical data and the IP address and the component information in the source log record, and judging the specific component of the specific target system corresponding to the fault information.
The effect provided in the summary of the invention is only the effect of the embodiment, not all the effects of the invention, and one of the above technical solutions has the following advantages or beneficial effects:
the invention realizes the collection, cleaning, processing and arrangement of system logs, and the mining and analysis of data by means of deep learning, machine learning algorithm and the like, realizes the early warning and fault positioning of possible faults of an application system, and provides corresponding solutions according to corresponding faults. The logs of the business information system are collected, cleaned, processed and analyzed, and faults possibly occurring in the application system are early warned and positioned by means of deep learning and machine learning algorithms, so that corresponding solutions are provided, system risks are eliminated in advance, and the safety condition of the system is improved.
Drawings
FIG. 1 is a flowchart of a deep learning-based log analysis early warning method of the present invention;
FIG. 2 is a flowchart of a step S1 according to an embodiment of the present invention;
FIG. 3 is a flowchart of a step S2 according to an embodiment of the present invention;
FIG. 4 is a flowchart of a step S3 according to an embodiment of the present invention;
FIG. 5 is a schematic diagram of supervised learning in an embodiment;
FIG. 6 is a schematic diagram of unsupervised learning in an embodiment;
FIG. 7 is a schematic diagram of knowledge graph establishment in an embodiment.
Detailed Description
In order to clearly explain the technical features of the present invention, the following detailed description of the present invention is provided with reference to the accompanying drawings. The following disclosure provides many different embodiments, or examples, for implementing different features of the invention. To simplify the disclosure of the present invention, the components and arrangements of specific examples are described below. Furthermore, the present invention may repeat reference numerals and/or letters in the various examples. This repetition is for the purpose of simplicity and clarity and does not in itself dictate a relationship between the various embodiments and/or configurations discussed. It should be noted that the components illustrated in the figures are not necessarily drawn to scale. Descriptions of well-known components and processing techniques and procedures are omitted so as to not unnecessarily limit the invention.
As shown in fig. 1, a deep learning based log analysis and early warning method includes the following steps:
s1, preprocessing the acquired logs of different types in the target system;
s2, analyzing the preprocessed log by a clustering method;
s3, encoding the analyzed log events into digital feature vectors;
s4, learning the encoded log by using a neural network based on LSTM and a clustering method based on LogCollect to form early warning information;
and S5, tracing the source of the early warning information to a component server corresponding to the load, and judging a fault point.
As shown in fig. 2, as an embodiment of the present invention, in step S1, the preprocessing is performed on the obtained logs of different types in the target system, and specifically includes:
s11, all paths for generating the logs in the server are configured in the Agent, and different types of logs are collected into the log cleaning server through the Agent.
Agents covering different operating system versions in the current environment are deployed in the target system, and different types of agents including windows, Red Hat, CentoS, Sun Solaris, IBMAIX and the like are covered. Configuring all paths for generating logs in the server in the installed Agent, including logs generated by an operating system of the server, logs of a middleware such as Tomcat, Weblogic and the like, running records, operation records and the like running on the server, logs of a database such as Oracle, Mysql and the like, and logs of other components. So that different types of logs of the components are collected into the log cleaning server through the Agent.
And S12, decomposing and formatting the collected log through the log cleaning server, and writing the log into the database.
And S13, performing secondary processing on the log data which is stored in the database and exceeds the set time threshold, and storing only the abnormal log in the HDFS.
Because the collected logs are unstructured text files, different rules need to be formulated according to different types of log files, and the logs are decomposed, formatted and written into an elastic search by means of a big data framework such as MapReduce. Because the log generation amount of the target system is large (the daily log production amount exceeds 300GB), in order to ensure the retrieval efficiency of the ElasticSearch library, only log data of the latest 7 days are set to be reserved in the ElasticSearch, the data exceeding 7 days need to be processed again, only abnormal logs in the logs are reserved, and the abnormal logs are written into the HDFS.
As an embodiment of the present invention, before the step of performing the log parsing step on the preprocessed log by using a clustering-based method in step S2, the method further includes:
and connecting each component log in the target system according to the historical standing book information and the IP address recorded in the log. Because the log formats, contents and extracted information of different components are not only the same, but also the extracted information is stored according to the component types and cannot directly reflect the state of the system, the logs of each component in the system need to be connected in a regular manner according to the ledger and the IP addresses recorded in the logs.
As shown in fig. 3, as an embodiment of the present invention, in step S2, performing log analysis on the preprocessed log by using a clustering-based method specifically includes:
s21, calculating the distance between the logs, and grouping the logs into different clusters by using a clustering method;
s22, extracting a set of event templates from each cluster, so that an original log can be constructed.
A log is a plain text composed of fixed and variable parts that may differ in different events. For example, for a given log "Connection from 10.10.34.12closed" and "Connection from 10.10.34.13closed", the word Connection from closed is considered to be a constant part because they always remain the same, while the rest of the parts are considered to be variable parts because they are always changing. Developers pre-define the constant part in the source code and the variable part is usually dynamically generated (e.g. port number, IP address), which is not well utilized in anomaly detection. The purpose of log parsing is to separate the constant part from the variable part and form a mature log event (i.e., "Connection from closed" in the example).
As shown in fig. 4, as an embodiment of the present invention, in step S3, encoding the parsed log event into a digital feature vector specifically includes:
s31, using different grouping techniques, the original log is split into a set of log sequences, including a fixed window, a sliding window, and a session window.
S32, an event count vector is generated for each log sequence indicating the number of occurrences of each event.
After parsing the logs into individual events, they need to be further encoded into digital feature vectors so that the machine learning model can be applied. To do this, the original log is first split into a set of log sequences, including a fixed window, a sliding window, and a session window, using different grouping techniques. Then, for each log sequence, we generate a feature vector (event count vector) that represents the number of occurrences of each event. All the feature vectors together may form a feature matrix, i.e. an event count matrix. Through experiments, the embodiment chooses to use a sliding window to divide log data. This may also take into account the subsequent impact of the preamble operation on the target system.
Unlike fixed windows, sliding windows consist of two attributes, window size and step size, e.g., the window slides every five minutes every hour. Typically, the step size is smaller than the window size, thus resulting in an overlap of different windows. The number of sliding windows is usually more than the number of data records, depending mainly on the window size and the step size. Logs that occur in the same sliding window are also grouped into log sequences, which may be repeated in multiple sliding windows due to overlap.
As shown in fig. 5 and 6, as an embodiment of the present invention, in step S4, learning the encoded log by using a LSTM-based neural network and a LogCollect-based clustering method to form early warning information, specifically including:
the fault early warning is embodied in log data as various errors and anomalies in the log, and for the processing of the abnormal states, two methods, namely supervised learning and unsupervised learning, can be generally adopted. The embodiment selects a mode of combining supervised learning and unsupervised learning, and forms an alarm by filtering the result through the rule.
Supervised learning the present embodiment chooses the way to use LSTM-based neural networks. The problem of gradient disappearance exists in a Recurrent Neural Network (RNN) model, so that Long-sequence data is difficult to process, many scientific researchers improve the RNN to obtain a special case LSTM (Long Short-Term Memory) of the RNN, the special case LSTM can avoid the gradient disappearance of the conventional RNN, and the subsequent influence of the previous event can be saved, so that the method is widely applied in the industry. By training the LSTM neural network, it forms an "impression" of the target system runtime, state, system loading conditions, etc. Based on this "impression" we can predict whether the target system will generate an alarm in the next or even multiple time windows.
And adopting a clustering method based on LogCollect in an unsupervised mode: LogCluster requires two training phases, namely a knowledge base initialization phase and an online learning phase. Thus, the training instance is divided into two parts of these two phases, respectively.
The initialization stage of the knowledge base comprises three steps of log vectorization, log clustering and representative vector extraction. First, the log sequence is vectorized into an event count vector and further modified by Inverse Document Frequency (IDF) [41] and normalization. Secondly, the LogCluster clusters the normal and abnormal event counting vectors respectively, and generates two groups of vector clusters (namely normal clusters and abnormal clusters) as a knowledge base in a clustering hierarchy clustering mode. Finally, we select one representative vector by computing the centroid of each cluster. And the online learning stage is used for further adjusting the clusters constructed in the knowledge base initialization stage.
In the online learning phase, event count vectors are added one by one to the knowledge base. Given an event count vector, the distance between it and the existing representative vector is calculated. If the minimum distance is less than the threshold, the event count vector will be added to the nearest cluster and the representative vector for that cluster will be updated. Otherwise, LogCluster will create a new cluster using this event count vector.
After the knowledge base is constructed and the online learning process is completed, LogCollect can be used to detect anomalies. In particular, to determine the state of a new log sequence, we compute its distance to a representative vector in the knowledge base. If the minimum distance is greater than the threshold, the log sequence is reported as anomalous. Otherwise, if the most recent cluster is a normal/abnormal cluster, the log sequence will be reported as normal/abnormal.
And summarizing results through a neural network based on the LSTM and a clustering mode based on the LogCollect, and filtering through rules to form final alarm information. And reminding operation and maintenance personnel to maintain.
As an embodiment of the present application, in step S5, tracing to the early warning information to the component server corresponding to the load, and determining the fault point specifically includes:
and tracing to a server of the component corresponding to the load according to the association rule of the historical account information and the historical data and the IP address and the component information in the source log record, and judging the specific component of the specific target system corresponding to the fault information.
As shown in fig. 7, a system-component-failure-solution knowledge graph is constructed by means of the graph database Neo4j, based on the history library in the HDFS and the knowledge library in the LogCollect clustering method. Neo4j is a high-performance, NOSQL graph database that stores structured data on a network rather than in tables. It is an embedded, disk-based Java persistence engine with full transactional properties, but it stores structured data on the network (called a graph mathematically) instead of in tables. And writing the operation and maintenance experience and the method which are arranged in advance into a database, and searching according to the estimated fault. Thereby showing possible solutions and methods for resolving the fault.
The invention uses MapReduce, Spark, Yarn, HDFS and other big data components, ElasticSearch and other NoSQL databases, TensorFlow, Spark MLlib and other frameworks for deep learning and machine learning, and is developed by using Java, Python, JavaScript and other languages. The method realizes the collection, cleaning, processing and arrangement of system logs, and carries out data mining and analysis by means of deep learning, machine learning algorithm and the like, thereby realizing the early warning and fault positioning of possible faults of the application system, and providing a corresponding solution according to the corresponding faults. The method has the advantages that a good effect is achieved on the public data set, the accuracy rate can reach more than 60% on average under the condition of multiple evaluations of different types of logs, and the recall rate is 80% on average. That is, we need to ensure that the predicted alarm will have a fault in each case with a certain accuracy.
Although the embodiments of the present invention have been described with reference to the accompanying drawings, it is not intended to limit the scope of the present invention, and it should be understood by those skilled in the art that various modifications and variations can be made without inventive efforts by those skilled in the art based on the technical solution of the present invention.
Claims (10)
1. A log analysis early warning method based on deep learning is characterized by comprising the following steps:
preprocessing the acquired logs of different types in the target system;
performing log analysis on the preprocessed logs by using a clustering-based method;
encoding the analyzed log events into digital feature vectors;
learning the encoded log by using a neural network based on LSTM and a clustering method based on LogCollect to form early warning information;
tracing the source of the early warning information to the component server corresponding to the load, and judging a fault point.
2. The deep learning-based log analysis and early warning method as claimed in claim 1, wherein the preprocessing of the acquired logs of different types in the target system specifically comprises:
configuring all paths for generating logs in the server in the Agent, and collecting different types of logs into a log cleaning server through the Agent;
decomposing and formatting the collected logs through a log cleaning server, and writing the logs into a database;
and performing secondary processing on the log data which is stored in the database and exceeds the set time threshold, and storing only the abnormal log into the HDFS.
3. The deep learning-based log analysis and early warning method as claimed in claim 2, wherein before the log parsing step of the preprocessed log by using a clustering-based method, the method further comprises:
and connecting each component log in the target system according to the historical standing book information and the IP address recorded in the log.
4. The deep learning-based log analysis and early warning method as claimed in claim 3, wherein the log analysis of the preprocessed log by using a clustering-based method specifically comprises:
calculating the distance between the logs, and grouping the logs into different clusters by using a clustering method;
a set of event templates is extracted from each cluster so that an original log can be constructed.
5. The deep learning-based log analysis and early warning method as claimed in claim 4, wherein the encoding of the parsed log events into digital feature vectors specifically comprises:
dividing an original log into a group of log sequences by using different grouping technologies, wherein the log sequences comprise a fixed window, a sliding window and a session window;
an event count vector is generated for each log sequence that represents the number of times each event occurred.
6. The deep learning-based log analysis and early warning method as claimed in claim 5, wherein learning the encoded log by using a neural network based on LSTM and a clustering method based on Logillect to form early warning information, specifically comprises:
inputting the offline data into an LSTM-based neural network model for mode evaluation and optimization;
generating normal clusters and abnormal clusters by using historical log data as an initialization stage knowledge base;
adding a new log for learning, and adjusting a knowledge base cluster constructed in an initialization stage;
and predicting whether the target system gives an alarm or not in the next time window by using the trained neural network model, detecting the abnormality by using LogCollect, summarizing prediction results, and filtering by rules to form early warning information.
7. The deep learning-based log analysis and early warning method as claimed in claim 6, wherein the using of the historical log data to generate normal clusters and abnormal clusters as the initialization stage knowledge base specifically comprises:
the log sequence is vectorized into an event count vector;
the LogCluster respectively clusters the normal event counting vector and the abnormal event counting vector, and generates two groups of vector clusters as a knowledge base in a clustering hierarchy clustering mode;
a representative vector is selected by calculating the centroid of each cluster.
8. The deep learning-based log analysis and early warning method as claimed in claim 7, wherein a new log is added for learning, and a knowledge base cluster constructed in an initialization stage is adjusted, specifically comprising:
calculating the distance between a given one of the event count vectors and the existing representative vector, and if the minimum distance is less than a threshold, adding the event count vector to the nearest cluster and the representative vector of that cluster to be updated; otherwise, LogCluster will create a new cluster using this event count vector.
9. The deep learning-based log analysis and early warning method as claimed in claim 8, wherein the detecting abnormality using LogCollect specifically comprises:
the distance of the log sequence to a representative vector in the knowledge base is calculated. If the minimum distance is greater than the threshold, the log sequence is reported as anomalous. Otherwise, if the most recent cluster is a normal/abnormal cluster, the log sequence will be reported as normal/abnormal.
10. The deep learning-based log analysis and early warning method as claimed in claim 9, wherein tracing the early warning information to the component server corresponding to the load and determining the fault point specifically comprises:
and tracing to a server of the component corresponding to the load according to the association rule of the historical account information and the historical data and the IP address and the component information in the source log record, and judging the specific component of the specific target system corresponding to the fault information.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911093214.7A CN110958136A (en) | 2019-11-11 | 2019-11-11 | Deep learning-based log analysis early warning method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911093214.7A CN110958136A (en) | 2019-11-11 | 2019-11-11 | Deep learning-based log analysis early warning method |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110958136A true CN110958136A (en) | 2020-04-03 |
Family
ID=69977119
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911093214.7A Pending CN110958136A (en) | 2019-11-11 | 2019-11-11 | Deep learning-based log analysis early warning method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110958136A (en) |
Cited By (22)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111552609A (en) * | 2020-04-12 | 2020-08-18 | 西安电子科技大学 | Abnormal state detection method, system, storage medium, program and server |
| CN111585799A (en) * | 2020-04-29 | 2020-08-25 | 杭州迪普科技股份有限公司 | Network fault prediction model establishing method and device |
| CN111639497A (en) * | 2020-05-27 | 2020-09-08 | 北京东方通科技股份有限公司 | Abnormal behavior discovery method based on big data machine learning |
| CN112015705A (en) * | 2020-06-05 | 2020-12-01 | 浙商银行股份有限公司 | Block chain log monitoring method based on deep learning |
| CN112434245A (en) * | 2020-11-23 | 2021-03-02 | 北京八分量信息科技有限公司 | Method and device for judging abnormal behavior event based on UEBA (unified extensible architecture), and related product |
| CN112685215A (en) * | 2021-01-22 | 2021-04-20 | 浪潮云信息技术股份公司 | Cloud platform abnormal log analysis method |
| CN112988440A (en) * | 2021-02-23 | 2021-06-18 | 山东英信计算机技术有限公司 | System fault prediction method and device, electronic equipment and storage medium |
| CN113254255A (en) * | 2021-07-15 | 2021-08-13 | 苏州浪潮智能科技有限公司 | Cloud platform log analysis method, system, device and medium |
| CN113269368A (en) * | 2021-06-07 | 2021-08-17 | 上海航空工业(集团) 有限公司 | Civil aircraft safety trend prediction method based on data driving |
| CN113485878A (en) * | 2021-07-06 | 2021-10-08 | 国网江苏省电力有限公司信息通信分公司 | Multi-data center fault detection method |
| CN113553245A (en) * | 2021-05-20 | 2021-10-26 | 中国海洋大学 | Log anomaly detection method combining bidirectional slice GRU and gate control attention mechanism |
| WO2021212409A1 (en) * | 2020-04-23 | 2021-10-28 | Citrix Systems, Inc. | Identification of log events for computing systems |
| CN113839852A (en) * | 2020-06-23 | 2021-12-24 | 中国科学院计算机网络信息中心 | Mail account abnormity detection method, device and storage medium |
| CN114064196A (en) * | 2020-07-31 | 2022-02-18 | 慧与发展有限责任合伙企业 | System and method for predictive assurance |
| WO2022057425A1 (en) * | 2020-09-21 | 2022-03-24 | International Business Machines Corporation | Identifying siem event types |
| CN114417817A (en) * | 2021-12-30 | 2022-04-29 | 中国电信股份有限公司 | Session information cutting method and device |
| CN114546706A (en) * | 2022-03-08 | 2022-05-27 | 哈尔滨禹晖文化传媒有限公司 | Application program defect analysis method applied to deep learning and server |
| CN115065556A (en) * | 2022-07-28 | 2022-09-16 | 国网浙江省电力有限公司 | Log malicious behavior detection method and system based on graph contrast learning |
| CN115277235A (en) * | 2022-08-01 | 2022-11-01 | 石家庄铁道大学 | Software anomaly monitoring method based on block chain |
| CN115357469A (en) * | 2022-10-21 | 2022-11-18 | 北京国电通网络技术有限公司 | Abnormal alarm log analysis method and device, electronic equipment and computer medium |
| CN116346590A (en) * | 2023-05-30 | 2023-06-27 | 国网汇通金财（北京）信息科技有限公司 | Positioning system for full link fault |
| CN116595425A (en) * | 2023-07-13 | 2023-08-15 | 浙江大有实业有限公司杭州科技发展分公司 | Defect identification method based on power grid dispatching multi-source data fusion |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105653427A (en) * | 2016-03-04 | 2016-06-08 | 上海交通大学 | Log monitoring method based on abnormal behavior detection |
| CN106790008A (en) * | 2016-12-13 | 2017-05-31 | 浙江中都信息技术有限公司 | Machine learning system for detecting abnormal host in enterprise network |
| CN107885642A (en) * | 2017-11-29 | 2018-04-06 | 小花互联网金融服务（深圳）有限公司 | Business monitoring method and system based on machine learning |
| CN109062763A (en) * | 2018-07-31 | 2018-12-21 | 云南大学 | One kind dynamic realtime from SVN log event stream excavates the movable method of software process |
| WO2019060327A1 (en) * | 2017-09-20 | 2019-03-28 | University Of Utah Research Foundation | Online detection of anomalies within a log using machine learning |
| CN110288004A (en) * | 2019-05-30 | 2019-09-27 | 武汉大学 | A kind of diagnosis method for system fault and device excavated based on log semanteme |
-
2019
- 2019-11-11 CN CN201911093214.7A patent/CN110958136A/en active Pending
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN105653427A (en) * | 2016-03-04 | 2016-06-08 | 上海交通大学 | Log monitoring method based on abnormal behavior detection |
| CN106790008A (en) * | 2016-12-13 | 2017-05-31 | 浙江中都信息技术有限公司 | Machine learning system for detecting abnormal host in enterprise network |
| WO2019060327A1 (en) * | 2017-09-20 | 2019-03-28 | University Of Utah Research Foundation | Online detection of anomalies within a log using machine learning |
| CN107885642A (en) * | 2017-11-29 | 2018-04-06 | 小花互联网金融服务（深圳）有限公司 | Business monitoring method and system based on machine learning |
| CN109062763A (en) * | 2018-07-31 | 2018-12-21 | 云南大学 | One kind dynamic realtime from SVN log event stream excavates the movable method of software process |
| CN110288004A (en) * | 2019-05-30 | 2019-09-27 | 武汉大学 | A kind of diagnosis method for system fault and device excavated based on log semanteme |
Non-Patent Citations (6)
| Title |
|---|
| SHILIN HE，ET AL.: "Experience Report: System Log Analysis for Anomaly Detection", 《2016 IEEE 27TH INTERNATIONAL SYMPOSIUM ON SOFTWARE RELIABILITY ENGINEERING》 * |
| ZHANG K,XU J,MIN M R,ET AL: "Automated IT system failure prediction:A deep learning approach", 《2016IEEE INTERNATIONAL CONFERENCE ON BIG DATA》 * |
| 仲思超等: "基于深度学习的Web安全事件分析系统及其实现", 《江苏通信》 * |
| 任明等: "基于深度学习的云计算系统异常检测方法", 《计算机技术与发展》 * |
| 杨瑞朋等: "日志异常检测技术研究", 《信息工程大学学报》 * |
| 钟雅等: "基于机器学习的日志解析系统设计与实现", 《计算机应用》 * |
Cited By (37)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111552609A (en) * | 2020-04-12 | 2020-08-18 | 西安电子科技大学 | Abnormal state detection method, system, storage medium, program and server |
| CN111552609B (en) * | 2020-04-12 | 2022-03-11 | 西安电子科技大学 | Abnormal state detection method, system, storage medium, program and server |
| WO2021212409A1 (en) * | 2020-04-23 | 2021-10-28 | Citrix Systems, Inc. | Identification of log events for computing systems |
| CN111585799A (en) * | 2020-04-29 | 2020-08-25 | 杭州迪普科技股份有限公司 | Network fault prediction model establishing method and device |
| CN111639497A (en) * | 2020-05-27 | 2020-09-08 | 北京东方通科技股份有限公司 | Abnormal behavior discovery method based on big data machine learning |
| CN111639497B (en) * | 2020-05-27 | 2021-01-15 | 北京东方通科技股份有限公司 | Abnormal behavior discovery method based on big data machine learning |
| CN112015705A (en) * | 2020-06-05 | 2020-12-01 | 浙商银行股份有限公司 | Block chain log monitoring method based on deep learning |
| CN113839852B (en) * | 2020-06-23 | 2023-03-24 | 中国科学院计算机网络信息中心 | Mail account abnormity detection method, device and storage medium |
| CN113839852A (en) * | 2020-06-23 | 2021-12-24 | 中国科学院计算机网络信息中心 | Mail account abnormity detection method, device and storage medium |
| US11805005B2 (en) | 2020-07-31 | 2023-10-31 | Hewlett Packard Enterprise Development Lp | Systems and methods for predictive assurance |
| CN114064196A (en) * | 2020-07-31 | 2022-02-18 | 慧与发展有限责任合伙企业 | System and method for predictive assurance |
| US11503055B2 (en) | 2020-09-21 | 2022-11-15 | International Business Machines Corporation | Identifying siem event types |
| GB2618216A (en) * | 2020-09-21 | 2023-11-01 | Ibm | Identifying SIEM event types |
| WO2022057425A1 (en) * | 2020-09-21 | 2022-03-24 | International Business Machines Corporation | Identifying siem event types |
| CN112434245A (en) * | 2020-11-23 | 2021-03-02 | 北京八分量信息科技有限公司 | Method and device for judging abnormal behavior event based on UEBA (unified extensible architecture), and related product |
| CN112685215A (en) * | 2021-01-22 | 2021-04-20 | 浪潮云信息技术股份公司 | Cloud platform abnormal log analysis method |
| CN112988440A (en) * | 2021-02-23 | 2021-06-18 | 山东英信计算机技术有限公司 | System fault prediction method and device, electronic equipment and storage medium |
| CN112988440B (en) * | 2021-02-23 | 2023-08-01 | 山东英信计算机技术有限公司 | System fault prediction method and device, electronic equipment and storage medium |
| CN113553245A (en) * | 2021-05-20 | 2021-10-26 | 中国海洋大学 | Log anomaly detection method combining bidirectional slice GRU and gate control attention mechanism |
| CN113553245B (en) * | 2021-05-20 | 2022-04-19 | 中国海洋大学 | Log anomaly detection method combining bidirectional slice GRU and gate control attention mechanism |
| CN113269368A (en) * | 2021-06-07 | 2021-08-17 | 上海航空工业(集团) 有限公司 | Civil aircraft safety trend prediction method based on data driving |
| CN113269368B (en) * | 2021-06-07 | 2023-06-30 | 上海航空工业(集团)有限公司 | Civil aircraft safety trend prediction method based on data driving |
| CN113485878A (en) * | 2021-07-06 | 2021-10-08 | 国网江苏省电力有限公司信息通信分公司 | Multi-data center fault detection method |
| CN113254255A (en) * | 2021-07-15 | 2021-08-13 | 苏州浪潮智能科技有限公司 | Cloud platform log analysis method, system, device and medium |
| CN114417817B (en) * | 2021-12-30 | 2023-05-16 | 中国电信股份有限公司 | Session information cutting method and device |
| CN114417817A (en) * | 2021-12-30 | 2022-04-29 | 中国电信股份有限公司 | Session information cutting method and device |
| CN114546706B (en) * | 2022-03-08 | 2022-11-11 | 极客邦控股(北京)有限公司 | Application program defect analysis method applied to deep learning and server |
| CN114546706A (en) * | 2022-03-08 | 2022-05-27 | 哈尔滨禹晖文化传媒有限公司 | Application program defect analysis method applied to deep learning and server |
| CN115065556A (en) * | 2022-07-28 | 2022-09-16 | 国网浙江省电力有限公司 | Log malicious behavior detection method and system based on graph contrast learning |
| CN115065556B (en) * | 2022-07-28 | 2022-11-22 | 国网浙江省电力有限公司 | Log malicious behavior detection method and system based on graph contrast learning |
| CN115277235B (en) * | 2022-08-01 | 2023-08-22 | 石家庄铁道大学 | Software anomaly monitoring method based on blockchain |
| CN115277235A (en) * | 2022-08-01 | 2022-11-01 | 石家庄铁道大学 | Software anomaly monitoring method based on block chain |
| CN115357469B (en) * | 2022-10-21 | 2022-12-30 | 北京国电通网络技术有限公司 | Abnormal alarm log analysis method and device, electronic equipment and computer medium |
| CN115357469A (en) * | 2022-10-21 | 2022-11-18 | 北京国电通网络技术有限公司 | Abnormal alarm log analysis method and device, electronic equipment and computer medium |
| CN116346590A (en) * | 2023-05-30 | 2023-06-27 | 国网汇通金财（北京）信息科技有限公司 | Positioning system for full link fault |
| CN116595425A (en) * | 2023-07-13 | 2023-08-15 | 浙江大有实业有限公司杭州科技发展分公司 | Defect identification method based on power grid dispatching multi-source data fusion |
| CN116595425B (en) * | 2023-07-13 | 2023-11-10 | 浙江大有实业有限公司杭州科技发展分公司 | Defect identification method based on power grid dispatching multi-source data fusion |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110958136A (en) | Deep learning-based log analysis early warning method | |
| Li et al. | Swisslog: Robust and unified deep learning based log anomaly detection for diverse faults | |
| CN108415789B (en) | Node fault prediction system and method for large-scale hybrid heterogeneous storage system | |
| CN111435366A (en) | Equipment fault diagnosis method and device and electronic equipment | |
| CN113282461B (en) | Alarm identification method and device for transmission network | |
| CN103761173A (en) | Log based computer system fault diagnosis method and device | |
| Wang et al. | Loguad: log unsupervised anomaly detection based on word2vec | |
| CN114465874B (en) | Fault prediction method, device, electronic equipment and storage medium | |
| CN111949480B (en) | Log anomaly detection method based on component perception | |
| CN112308126A (en) | Fault recognition model training method, fault recognition device and electronic equipment | |
| CN111585799A (en) | Network fault prediction model establishing method and device | |
| CN113064873B (en) | Log anomaly detection method with high recall rate | |
| CN115269314A (en) | Transaction abnormity detection method based on log | |
| CN112306787A (en) | Error log processing method and device, electronic equipment and intelligent sound box | |
| CN111581056B (en) | Software engineering database maintenance and early warning system based on artificial intelligence | |
| CN112951311A (en) | Hard disk fault prediction method and system based on variable weight random forest | |
| CN115344414A (en) | Log anomaly detection method and system based on LSTM-Transformer | |
| CN112433874A (en) | Fault positioning method, system, electronic equipment and storage medium | |
| CN115544519A (en) | Method for carrying out security association analysis on threat information of metering automation system | |
| An et al. | Real-time Statistical Log Anomaly Detection with Continuous AIOps Learning. | |
| CN110399278B (en) | Alarm fusion system and method based on data center anomaly monitoring | |
| Li et al. | Improving performance of log anomaly detection with semantic and time features based on bilstm-attention | |
| CN116167370A (en) | Log space-time characteristic analysis-based distributed system anomaly detection method | |
| CN112712443B (en) | Event analysis method and analysis device for converter station | |
| CN114816962A (en) | ATTENTION-LSTM-based network fault prediction method |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200403 |
|
| RJ01 | Rejection of invention patent application after publication |