WO2025113711A1 - Abnormal log processing method and device - Google Patents

Abstract

An abnormal log processing method, comprising: collecting abnormal logs generated in the operation process of a service system; preprocessing the abnormal logs; on the basis of a pre-trained log classification model, classifying the preprocessed abnormal logs to obtain corresponding fault types; and searching a pre-created solution library to query solutions matching the fault types, and pushing optimal solutions to operations and maintenance workers. Thus, the operations and maintenance workers can quickly solve or process faults occurring in the operation process of the system, thereby improving the fault solving efficiency of the operations and maintenance workers. Further provided is an abnormal log processing device.

Description

Abnormal log processing method and device Technical Field

The present application relates to the field of computer technology, and in particular to an abnormal log processing method and device.

Background Art

With the rapid development of information technology, systems and applications have been widely used. During the operation of system applications, massive amounts of log data will be generated, including a large number of abnormal logs. By analyzing the abnormal logs, the fault location and cause of the system can be located.

However, when operation and maintenance personnel analyze exception logs to determine the fault location, they usually adopt the following three solutions: one is to try various solutions directly in the system based on the exception log; the second is to ask other experienced operation and maintenance personnel; the third is to search the operation and maintenance logs or the Internet to find solutions; no matter which method is chosen, there is a problem of low fault location efficiency.

Summary of the invention

In view of this, the purpose of this application is to provide an abnormal log processing method and device to solve at least part of the above technical problems. The technical solution provided is as follows:

In a first aspect, an embodiment of the present application provides a method for processing abnormal logs, including:

Collect abnormal logs generated during the operation of business systems;

Preprocess the abnormal logs;

Based on the pre-trained log classification model, the pre-processed abnormal logs are classified to obtain the corresponding fault types;

A pre-created solution library is searched to query a solution matching the fault type, wherein the solution library includes the fault type and an association relationship between the fault type and the corresponding solution.

In a second aspect, the embodiment of the present application further provides an abnormal log processing device, including:

The log collection module is used to collect abnormal logs generated during the operation of the business system;

A log management module, used to pre-process the abnormal logs, and call the log classification model to classify the pre-processed abnormal logs to obtain corresponding fault categories;

The solution recommendation module is used to search a pre-created solution library to query for solutions matching the fault category.

Compared with the prior art, the above technical solution provided by the present application has the following advantages: the abnormal log processing method provided by the present application can quickly classify the abnormal logs by collecting the abnormal logs during the operation of the system (microservice or application) and using the pre-trained log classification model. Further, a solution matching the classification category of the abnormal log will be searched from the pre-created solution library, and the optimal solution will be pushed to the operation and maintenance personnel, so that the operation and maintenance personnel can quickly solve or handle the faults that occur during the operation of the system, thereby improving the efficiency of the operation and maintenance personnel in solving faults.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

FIG1 is a flow chart of an abnormal log processing method provided by an embodiment of the present application;

FIG2 is a flow chart of another abnormal log processing method provided in an embodiment of the present application;

FIG3 is a schematic diagram of the structure of an abnormal log processing device provided in an embodiment of the present application;

FIG4 is a schematic diagram of the structure of another abnormal log processing device provided in an embodiment of the present application.

DETAILED DESCRIPTION

In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

In order to make the description of the following embodiments clear and concise, a brief introduction to the related technology is first given:

Natural Language Processing (NLP): NLP technology can convert text data into a form that computers can understand and process. In log classification, NLP can be applied to tasks such as text segmentation, part-of-speech tagging, and named entity recognition to extract useful feature information.

Machine Learning: Machine learning algorithms can be used to identify and classify different types of logs by training models. Common machine learning algorithms include Naive Bayes classifier, Support Vector Machine (SVM), Decision Tree, Random Forest, etc. These algorithms can learn the characteristic patterns of log text and infer the category to which the log belongs.

Feature Engineering: Feature engineering refers to building effective feature representations from raw data to improve the performance of classification algorithms. In log classification, feature engineering can include word frequency statistics, TF-IDF (term frequency-inverse document frequency) calculation, word embedding and other technologies.

Deep learning: Deep learning techniques such as convolutional neural networks (CNN) or recurrent neural networks (RNN) are also used in log classification. These models can automatically learn features from data and perform classification and prediction through multi-layer neural networks.

Incremental learning: Since the amount of log data is huge and growing, the use of incremental learning algorithms can achieve rapid classification of new logs and update models without retraining the entire model.

Please refer to FIG1 , which shows a flow chart of an abnormal log processing method provided by an embodiment of the present application. The method is applied to an operation and maintenance system and may include the following steps:

S101, collecting abnormal logs generated during the operation of the business system.

Collect exception logs generated during the operation of each system (microservice or application). The collection object is mainly exception logs, including logs of different types, levels, and events. For example, the levels of exception logs may include:

Debug: Records debugging-related information for development and troubleshooting.

Warning: Records warning information, indicating possible problems or potential errors.

Error: Records error information, indicating that a specific error or failure has occurred.

Critical: A serious error message is recorded, indicating that a very serious failure has occurred, which may cause the system to crash or fail to work properly.

Fatal: Records fatal error information, indicating that an unrecoverable error has occurred and the system cannot continue to run.

In an exemplary embodiment, a corresponding Agent collection program can be deployed according to the framework structure of the system or microservice log recording. The collected log level can be configured, such as warning, error, severe and fatal levels. When the log of the system or microservice reaches the configured collection level, an abnormal log collection action will be triggered, for example, when there is a log with an error mark, the log is collected.

S102, performing data preprocessing on the abnormal log.

The preprocessing of logs is crucial to the effectiveness of the classification algorithm. The preprocessing step can include removing stop words, standardizing the text format (i.e. log formatting), processing missing values and outliers, etc. to reduce noise interference and improve classification accuracy.

In the embodiment of the present application, the purpose of log formatting is to perform data standardization processing on the collected logs, and the original log information is sorted and organized in a certain format to facilitate subsequent processing, storage and analysis.

In an exemplary embodiment, the method of formatting log data may include the following:

(1) Timestamp

Add timestamp information to the log to record the specific time when the event occurred. The timestamp can be in a standard date and time format, for example, "2023-06-21T10:30:00Z" or other custom formats.

(2) Log level

Assign a level to each log entry to indicate its importance or severity. Common log levels include debug, warning, error, critical, fatal, etc.

(3) Log source

Record the name or identifier of the module, component, or application that generates the log to facilitate tracing the source of the log.

(4) Message content

Record the specific content of the log, including descriptive text information, error stack trace, exception information, etc. Make sure the message content is concise and can clearly convey the required information.

(5) Contextual Information

Add contextual information related to the log as needed. This information may include user ID, device ID, request parameters, response code, etc., to better understand and analyze the log.

(6) Separators and format specifications

Use appropriate delimiters and format specifications to ensure readability and easy parsing of log data. For example, you can use delimiters such as commas and tabs, or use common log formats such as Apache log format or Json format.

(7) Visualization

As needed, the formatted log data can be displayed in graphical or tabular form to provide more intuitive observation and analysis methods.

S103, extracting the feature vector of the preprocessed abnormal log and inputting it into a pre-trained log classification model for classification to obtain a fault category.

The log classification model is a model based on the machine learning algorithm. The log classification model is designed and trained with training data so that the model automatically learns the log classification method from the training data, and finally a trained log classification model is obtained.

In one embodiment, the process of obtaining log classification using the log classification model may include:

(1) Feature extraction

Useful features can be extracted from the formatted exception log data, such as timestamp, log level, component name, keyword, etc. In addition, for text features, operations such as word segmentation, stop word removal, and stem extraction can be performed.

For example, the log data "10.20.20.10; [2018-07-16 13:12:57]; GET/online/sample HTTP/1.1; 200" is segmented based on punctuation or special symbols such as; :<>[]{}/\n\t\r, etc., and the log is segmented as follows: 10.20.20.10 2018-07-16 13 12 57GET online sample HTTP 1.1 200. After segmentation, stem extraction can be performed, and the stem extracted from the log is: get online sample http.

(2) Feature vectorization

Convert the extracted features into numerical representations. For example, vector conversion tools such as one-hot encoding, bag of words, TF-IDF (Term Frequency-Inverse Document Frequency) can be used to convert text data into sparse vectors or dense vectors.

(3) The vector corresponding to the log data is input into the log classification model, and the category label corresponding to the log is output through the prediction function of the model.

S104, searching a pre-established solution library to obtain an optimal solution corresponding to the fault category, and pushing the optimal solution.

In one embodiment, an association relationship between faults and solutions is established based on a rule matching model to form a solution library. For example, the solution library may be in the form of the following Table 1:

Table 1

Based on the category labels of the exception logs obtained by the log classification model analysis, the existing solution library is searched to query solutions that match the fault manifestations and causes of the current exception logs. For example, the search can be performed based on the category labels, keywords or fault descriptions of the exception logs to quickly find relevant solutions.

For example, if it is determined from the exception log that the current fault is a database fault and is caused by a lock conflict, the category of the exception log is "database fault-lock conflict", and a solution matching "database fault-lock conflict" is further searched from the solution library.

Furthermore, after searching for a solution that matches the abnormal log from the solution library, it is possible to evaluate whether the searched solution matches the fault. For example, it can be determined based on the confidence between the two. The higher the confidence, the more the solution matches the fault, thereby ensuring that the steps and suggestions in the searched solution are applicable to the current fault situation, that is, improving the accuracy of the searched solution. For example, the confidence between the solution and the fault can be calculated by entropy value, which can be obtained by certain calculation logic. The smaller the entropy value, the higher the confidence.

In an exemplary embodiment, the optimal solution may be sent to a terminal device used by an operation and maintenance personnel, or the optimal solution may be displayed on a client of the operation and maintenance system. This application does not specifically limit the specific method of pushing the optimal solution.

In addition, in other embodiments of the present application, after the searched solution is pushed to the operation and maintenance personnel, the operation and maintenance personnel will provide feedback on the solution, that is, whether the pushed solution can solve the fault problem corresponding to the abnormal log, and the log classification model will be further strengthened and trained based on this feedback result.

In addition, during the actual application process, the solution library is continuously optimized and updated to include the latest and most applicable solutions, thereby improving the accuracy and applicability of the solution library.

The abnormal log processing method provided in this embodiment collects abnormal logs during the operation of the system (microservice or application) and uses a pre-trained log classification model to quickly classify the abnormal logs. Further, a solution matching the classification category of the abnormal log is searched from a pre-created solution library, and the optimal solution is pushed to the operation and maintenance personnel, so that the operation and maintenance personnel can quickly solve or handle the faults that occur during the operation of the system, thereby improving the efficiency of the operation and maintenance personnel in solving faults.

Please refer to Figure 2, which shows a flowchart of another abnormal log processing method provided by an embodiment of the present application. This embodiment will elaborate on the abnormal log processing process and focus on the training process of the log classification model. As shown in Figure 2, the method may include the following steps:

S201, collecting abnormal logs generated during the operation of the business system.

S202: Pre-process the abnormal log.

The implementation process of S201 to S202 is the same as that of S101 to S102, and will not be repeated here.

S203, classify the preprocessed exception logs using the pre-trained log classification model; if the model can automatically classify the exception logs, execute S204; if the model cannot automatically classify the exception logs, execute S205, that is, use the exception log data that cannot be automatically classified to conduct targeted training on the log classification model, thereby improving the applicability of the log classification model.

The feature vector is extracted from the abnormal log and input into the trained log classification model. The model predicts the category label of the abnormal log, obtains the predicted category labels, and sorts them from high to low according to the confidence of each predicted category label. Finally, the category label with the highest confidence is determined as the fault category of the abnormal log.

For the process of extracting the feature vector of the abnormal log, please refer to the relevant content of S103 in the embodiment shown in FIG. 1 , which will not be described in detail here.

S204, searching a pre-created solution library, searching for solutions that match the fault category of the exception log, and pushing them.

S205, construct a fault classification system.

The fault classification system is to classify faults according to certain rules and standards in order to better manage and maintain the system. The classification system is constructed from multiple dimensions, including but not limited to abnormal classification, performance classification, configuration problem classification, etc.

Log data may be classified in different ways. In some embodiments, the log data classification methods may include the following:

(1) Classification by log level

Classify logs according to their importance or severity, such as warning, error, severe, and fatal. This classification method can help operation and maintenance personnel (or developers) better understand the meaning and importance of logs and quickly locate problems.

(2) Classification by log type

Logs are classified according to their functions or purposes, such as application logs, system logs, security logs, performance logs, access logs, etc. Each type of log records different aspects of information, which helps to analyze and troubleshoot specific problems.

(3) Classification by log source

Categorize logs based on the module, component, or application that generates them, such as database logs, network logs, server logs, application logs, etc. This classification method can help track and filter related logs in complex systems.

(4) Classification by time period

Classify logs according to time periods, such as daily, weekly, monthly, or yearly. This classification method helps in the implementation of log storage and archiving strategies.

(5) Classification by log format

Log data is classified according to its format, such as text log, JSON log, XML log, etc. Different formats are suitable for different application scenarios and analysis requirements.

(6) Classification by keywords

Classification by keywords: Classify logs according to keywords or specific content in the logs, such as error logs, warning logs, exception logs, information logs, etc.

In practical applications, at least two of the above classification methods can be selected, and different classification methods can be selected for different application scenarios.

S206, annotating historical abnormal log data to obtain fault sample data.

In some embodiments, the fault sample data is manually labeled, for example, the fault type and labeling content of the fault sample data (ie, historical abnormal log data) can be determined according to the fault classification system constructed in the previous step to obtain the labeled data.

S207, design a log classification model.

Select a classification model, such as a naive Bayes model, a support vector machine (SVM) model, a decision tree model, a random forest, etc. Depending on the specific situation, a deep learning model such as a convolutional neural network (CNN) module or a recurrent neural network (RNN) model can also be used.

S208, training a log classification model.

The labeled fault sample data is divided into a training set and a test set. Appropriate evaluation indicators are selected, and the established log classification model is trained using the training set. The specific training process may include: extracting the feature vector of the fault sample data and inputting it into the log classification model for classification to obtain the category prediction result. Then, based on the loss function, the accuracy, precision, and recall rate between the category prediction result and the labeled fault type corresponding to the fault sample data are calculated. If the indicator is not within the allowable range, the log classification model parameters are adjusted as needed until each indicator is within the allowable range to obtain a usable log classification model.

In addition, the trained log classification model is tested using the test data set, the feature vector of the test data set is extracted, and input into the log classification model to be tested, and the category label corresponding to each test data is output through the prediction function of the model. Then, indicators such as accuracy, precision, and recall are calculated. If all indicators are within the allowable range, it is ensured that the accuracy and generalization performance of the trained log classification model meet the application requirements.

The collected abnormal log data is classified using the trained log classification model to obtain a category label, that is, the trained log classification model can be used to execute S203.

The abnormal log processing method provided in this embodiment obtains fault sample data by annotating historical abnormal log data, further extracts the feature vector of the fault sample data, and inputs the feature vector into the initial log classification model, and uses the initial log classification model to classify the fault sample data to obtain the corresponding category label. Based on the category obtained by the model and the category annotated by the fault sample data, the accuracy rate, recall rate and other indicators are calculated, and finally the model parameters are adjusted according to the indicator data until each indicator meets the requirements. The model training process is simple and fast, and the accuracy and efficiency of the log classification model are superior to the log classification method based on rules and keywords. Therefore, the efficiency of abnormal log classification can be improved by using the abnormal log processing method. Moreover, the method can also directly search for solutions that match the fault type of the abnormal log, and push the optimal solution to the operation and maintenance personnel, so that the operation and maintenance personnel can quickly solve or handle the faults that occur during the operation of the system, thereby improving the efficiency of the operation and maintenance staff in solving faults.

Please refer to FIG3 , which shows a schematic diagram of the structure of an abnormal log processing device provided in an embodiment of the present application. As shown in FIG3 , the device may include:

The log collection module 101 is used to collect abnormal logs generated during the operation of the business system.

The log management module 102 manages the collected abnormal logs, such as mainly including log formatting and log classification.

The purpose of log formatting is to standardize the collected logs. The original log information will be sorted and organized in a certain format to facilitate subsequent processing, storage and analysis.

Log classification is to classify log data according to different classification methods. The log classification of this module will be quickly classified by calling the log classification algorithm.

The log classification model 103 mainly uses the training data to train the classification model, automatically learns and generates a model for log classification from the training data, and applies the learned model to unknown log data for classification, thereby realizing rapid log classification.

The solution library 104 is a knowledge base for organizing and storing methods, techniques, best practices and experiences for solving problems. Solutions will be classified and categorized according to multiple dimensions such as problem type, technical field, product module, etc. This classification will intersect and correlate with the log classification, thereby providing a basis for solution recommendation.

The solution recommendation module 105 is used to quickly search for the optimal solution that matches the fault type of the abnormal log from the existing solutions in the solution library 104 and push it to the operation and maintenance personnel.

The abnormal log processing device provided in this embodiment collects the abnormal logs during the operation of the system (microservice or application) through the log collection module, and sends them to the log management module for further processing. After the log management module pre-processes the abnormal logs, the abnormal logs can be quickly classified based on the log classification model. The solution recommendation module further searches for solutions that match the classification category of the abnormal log from the pre-created solution library, and pushes the optimal solution to the operation and maintenance personnel, so that the operation and maintenance personnel can quickly solve or handle the faults that occur during the operation of the system, thereby improving the efficiency of the operation and maintenance staff in solving faults.

Please refer to FIG. 4, which shows a schematic diagram of the structure of another abnormal log processing device provided in an embodiment of the present application. As shown in FIG. 4, the device may further include, based on the embodiment shown in FIG. 3:

The fault classification system building module 201 is used to build a fault classification system.

The fault classification system is to classify faults according to certain rules and standards in order to better manage and maintain the system. The classification system is constructed from multiple dimensions, including but not limited to abnormal classification, performance classification, configuration problem classification, etc.

The data annotation module 202 is used to annotate historical abnormal log data to obtain fault sample data.

The fault type and annotation content of the fault sample data (ie, historical abnormal log data) can be determined according to the constructed fault classification system to obtain the annotation data.

The classification model training module 203 is used to train the initial log classification model using the fault sample data.

The labeled fault sample data is divided into a training set and a test set. Appropriate evaluation indicators are selected, and the established log classification model is trained using the training set. The specific training process may include: extracting the feature vector of the fault sample data and inputting it into the log classification model for classification to obtain the category prediction result. Then, based on the loss function, the accuracy, precision, and recall rate between the category prediction result and the labeled fault type corresponding to the fault sample data are calculated. If the indicator is not within the allowable range, the log classification model parameters are adjusted as needed until each indicator is within the allowable range to obtain a usable log classification model.

The abnormal log processing device provided in this embodiment obtains fault sample data by annotating historical abnormal log data, further extracts the feature vector of the fault sample data, and inputs the feature vector into the initial log classification model, and uses the initial log classification model to classify the fault sample data to obtain the corresponding category label. Based on the category obtained by the model and the category annotated by the fault sample data, the accuracy rate, recall rate and other indicators are calculated, and finally the model parameters are adjusted according to the indicator data until each indicator meets the requirements. The model training process is simple and fast, and the accuracy and efficiency of the log classification model are superior to the log classification method based on rules and keywords. Therefore, the efficiency of abnormal log classification can be improved by using this abnormal log processing method. Moreover, the method can also directly search for solutions that match the fault type of the abnormal log, and push the optimal solution to the operation and maintenance personnel, so that the operation and maintenance personnel can quickly solve or handle the faults that occur during the operation of the system, thereby improving the efficiency of the operation and maintenance staff in solving faults.

The present application provides a computing device, which includes a processor and a memory, wherein the memory stores a program that can be run on the processor. When the processor runs the program stored in the memory, the above-mentioned abnormal log processing method embodiment is implemented.

The present application also provides a storage medium executable by a computing device, in which a program is stored. When the program is executed by the computing device, the above-mentioned abnormal log processing method is implemented.

It should be noted that the technical features recorded in the various embodiments in this specification can be replaced or combined with each other, and can also be adjusted, merged and deleted in order according to actual needs. Each embodiment focuses on the differences from other embodiments, and the same and similar parts between the various embodiments can be referred to each other. For the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

The modules or submodules described as separate components may or may not be physically separated, and the components of the modules or submodules may or may not be physical modules or submodules, that is, they may be located in one place, or they may be distributed on multiple network modules or submodules. Some or all of the modules or submodules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional module or submodule in each embodiment of the present application may be integrated into one processing module, or each module or submodule may exist physically separately, or two or more modules or submodules may be integrated into one module. The above-mentioned integrated modules or submodules may be implemented in the form of hardware or in the form of software functional modules or submodules.

Finally, it should be noted that, in this article, relational terms such as first and second, etc. are merely used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations.

The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims (10)

A method for processing abnormal logs, characterized by comprising: Collect abnormal logs generated during the operation of business systems; Preprocess the abnormal logs; Based on the pre-trained log classification model, the pre-processed abnormal logs are classified to obtain the corresponding fault types; A pre-created solution library is searched to query a solution matching the fault type, wherein the solution library includes the fault type and an association relationship between the fault type and the corresponding solution. The method according to claim 1 is characterized in that the pre-processed abnormal logs are classified based on the pre-trained log classification model to obtain corresponding fault types, including: Extract the feature vector of the preprocessed abnormal log; The feature vector is input into a pre-trained log classification model for classification to obtain the fault type corresponding to the abnormal log. The method according to claim 1, characterized in that the method further comprises: If the log classification model cannot obtain the fault type corresponding to the abnormal log, the log classification model is strengthened and trained based on the abnormal log. The method according to any one of claims 1 to 3 is characterized in that the method further comprises: pushing the searched solution to operation and maintenance personnel. The method according to claim 4, characterized in that the method further comprises: Receive feedback from operation and maintenance personnel on the pushed solutions; If the feedback result is that the solution cannot solve the fault problem of the abnormal log, the log classification model is trained based on the abnormal log. The method according to claim 1, characterized in that the preprocessing of the abnormal log comprises: Stop words, missing values and abnormal values in the abnormal log are removed, and the abnormal log is formatted to obtain a standard format log. The method according to any one of claims 1 to 3, characterized in that the training process of the log classification model comprises: Build a fault classification system; Annotate historical abnormal logs to obtain fault sample data; Extracting a feature vector of the fault sample data, and inputting the feature vector into an initial log classification model for classification to obtain a fault category prediction result; Based on the fault category prediction result and the labeled category corresponding to the historical abnormal log, obtaining the performance index of the initial log classification model; If the performance index is not within the allowable range, the model parameters of the initial log classification model are adjusted, and the historical abnormal log data are classified using the adjusted log classification model to obtain a fault category prediction result, until the performance index is within the allowable range, and the log classification model is obtained. An abnormal log processing device, characterized by comprising: The log collection module is used to collect abnormal logs generated during the operation of the business system; A log management module, used to pre-process the abnormal logs, and call the log classification model to classify the pre-processed abnormal logs to obtain corresponding fault categories; The solution recommendation module is used to search a pre-created solution library to query for solutions matching the fault category. The device according to claim 8 is characterized in that the solution recommendation module is also used to: push the searched solution to operation and maintenance personnel. The device according to claim 8 or 9 is characterized in that it also includes: a log classification model retraining module, which is used to receive feedback results from operation and maintenance personnel on the pushed solution; if the feedback result is that the solution cannot solve the fault problem of the abnormal log, the log classification model is trained based on the abnormal log.