CN116010151A - Method and system for detecting all-journaling feature abnormality without journal analysis - Google Patents

Abstract

The invention discloses a method and a system for detecting full-log feature abnormality without log analysis, wherein the method comprises the following steps: grouping the logs to obtain a log sequence; performing word segmentation conversion on the log sequence, and constructing model input; training the neural network model to obtain a trained neural network model; obtaining logs to be tested, converting the logs into model input, and transmitting the model input to a trained neural network model to obtain characteristic information; and inputting the characteristic information into a classifier to obtain a log abnormality detection result. The system comprises: the system comprises a log grouping module, an input construction module, a learning training module, a characteristic extraction module and an abnormality detection module. By using the method and the device, log analysis and preprocessing operations are not needed, all the characteristics in all the information of the log are fully utilized, and the most comprehensive detection of log abnormality is realized. The method and the system for detecting the abnormality of the full log features without log analysis can be widely applied to the field of log audit.

Description

Method and system for detecting all-journaling feature abnormality without journal analysis

Technical Field

The invention relates to the field of log audit, in particular to a method and a system for detecting full log feature abnormality without log analysis.

Background

With the advent of cloud computing and internet of things, modern large-scale services have become more complex and flexible, with most continuous operations generating large amounts of log data. Any downtime or anomaly of these services can result in significant losses. And the accurate and timely abnormality detection can quickly reduce loss. Thus, anomaly detection is critical to the construction of reliable services. As one of the most valuable data sources for anomaly detection, journaling has been widely used in a variety of services and systems to record runtime states and critical events. Thus, detection of anomalies through log-based is critical to service maintenance and reliability assurance.

The existing log anomaly detection methods all adopt log analysis as the first step of work, however, the log analysis errors often reduce the accuracy of the whole log anomaly detection method, and the reasons are as follows: (1) the number of the log templates obtained by the log analysis method is usually far more than the number of the correct templates, so that the accuracy of the unsupervised log anomaly detection method which relies on the log template numbers for classification is greatly reduced; (2) the log analysis method can identify keywords in a plurality of log templates as log parameters or identify log parameter errors as part of the log templates, so that the error log templates are obtained, and the accuracy of log abnormality detection is affected.

Moreover, most existing log anomaly monitoring methods only use log template information or other single information to discard all other information, including time stamps, log levels, components, process Identifications (PIDs) and log parameter information, during training. However, some anomalies are closely related to discarded information, for example, a timestamp can be used to extract detailed information such as time series, PID is often used to monitor malicious behaviors, log parameter information reflects an execution path, etc., and if the information which is helpful for anomaly detection is discarded, the corresponding log anomaly detection method cannot detect anomalies related to the information.

Disclosure of Invention

In order to solve the technical problems, the invention aims to provide a full-log feature abnormality detection method and system without log analysis, which can fully reserve all information of a log and realize the most comprehensive detection of log abnormality by utilizing all features in all information of the log.

The first technical scheme adopted by the invention is as follows: a full-journaling feature anomaly detection method without journal analysis comprises the following steps:

grouping the logs to obtain a log sequence;

performing word segmentation conversion on the log sequence, and constructing model input;

the model input is conveyed to a neural network model for training, and a trained neural network model is obtained;

obtaining logs to be tested, converting the logs into model input, and transmitting the model input to a trained neural network model to obtain characteristic information;

and inputting the characteristic information into a classifier to obtain a log abnormality detection result.

Further, the step of grouping the logs to obtain a log sequence specifically includes:

traversing the log information to determine a grouping mode;

window division is carried out on the logs according to the grouping mode, and window information is obtained;

and grouping the logs according to the window information to obtain a log sequence.

Through the preferred step, the grouping mode of the optimal log can be selected, and when the window size is set to be 1, the log can be detected piece by piece.

Further, the step of word segmentation and conversion of the log sequence and model input construction specifically includes:

performing word segmentation on each log in each log sequence to obtain log word segmentation of each log in the log sequence;

taking an ID number corresponding to the log word segmentation as an input ID;

taking the log sequence number in the log sequence as a word segmentation class;

taking the attention value of the attention mechanism to the filling mark as an attention mask;

fitting the input ID, word segmentation class, and attention mask, results in a model input.

By this preferred step, a deliberate input is constructed for each log, retaining all log information, enabling the model to learn both sequence information and information for each log at the same time.

Further, the step of obtaining the log to be tested and converting the log to be tested into a model input and transmitting the model input to the trained neural network model to obtain the characteristic information specifically comprises the following steps:

vector conversion is respectively carried out on model input to be detected based on a vector embedding layer of the trained neural network model, and an embedding vector is obtained;

superposing the embedded vectors to obtain input vectors;

learning the input vector by utilizing a multi-head attention layer in the trained neural network model to obtain a multi-head attention output vector;

residual connection and layer normalization are carried out on the multi-head attention output vector to obtain a normalized vector;

the normalized vector is learned by utilizing a feedforward neural network layer in the trained neural network model, and the learned vector is obtained;

and carrying out residual connection and layer normalization on the learned vector to obtain characteristic information.

Through the optimization step, the trained neural network model can extract normal and abnormal characteristics input by the model to be detected, and preparation is made for judging whether the log is abnormal or not through a classifier later.

Further, the step of inputting the feature information into a classifier to obtain a log abnormality detection result specifically includes:

inputting the characteristic information into a pooling layer in the classifier to carry out pooling operation to obtain pooling information;

inputting the pooling information into a full-connection layer in the classifier for full-connection to obtain integrated information;

and inputting the integrated information into a Softmax function in a classifier to classify, so as to obtain a log abnormality detection result.

Through the preferred step, it is determined whether the input log sequence or single log is abnormal.

The second technical scheme adopted by the invention is as follows: a full log feature anomaly detection system without log parsing, comprising:

the log grouping module is used for grouping logs to obtain a log sequence;

the input construction module is used for carrying out word segmentation conversion on the log sequence and constructing model input;

the learning training module is used for conveying the model input to the neural network model for training to obtain a trained neural network model;

the feature extraction module is used for acquiring logs to be detected, converting the logs into model input and transmitting the model input to the trained neural network model to obtain feature information;

the abnormality detection module is used for inputting the characteristic information into the classifier to obtain a log abnormality detection result.

The method and the system have the beneficial effects that: according to the method, log analysis and preprocessing operations are not needed, so that not only is the reduction of accuracy caused by log analysis errors avoided, but also all information of the log is reserved to the greatest extent; by constructing model input for the log sequence which reserves all log information, the log sequence information and the information of each log are simultaneously learned and detected, the utilization of all the characteristics in all the log information is realized, and the most comprehensive detection of log abnormality is realized.

Drawings

FIG. 1 is a flowchart illustrating steps of a method for detecting anomalies in full log features without log parsing according to the present invention;

FIG. 2 is a block diagram of a system for detecting anomalies in full log features without log parsing in accordance with the present invention;

fig. 3 is a schematic diagram of a neural network model of a full log feature anomaly detection method without log parsing according to the present invention.

Detailed Description

The invention will now be described in further detail with reference to the drawings and to specific examples. The step numbers in the following embodiments are set for convenience of illustration only, and the order between the steps is not limited in any way, and the execution order of the steps in the embodiments may be adaptively adjusted according to the understanding of those skilled in the art.

Referring to fig. 1, the present invention provides a full log feature anomaly detection method without log parsing, comprising the steps of:

s1, grouping the logs to obtain a log sequence.

Traversing the log information to determine a grouping mode. Traversing log information, and if a session identifier exists in the log, adopting a session window mode; if the session identifier does not exist in the log, a sliding window mode or a fixed window mode can be selected according to actual conditions, wherein the sliding window is composed of two attributes of window size and step length, the fixed window has only window size, and the fixed window is grouped according to log frequency.

And carrying out window division on the log according to the grouping mode to obtain window information. When grouping log information in a session window mode, based on the type of session identification in the log, the logs with the same identifier belong to the logs in the same execution path and are divided into the same group, wherein the window size is the number of the logs with the same identifier; when the log information is grouped in a sliding window mode, setting a window size and a step length, wherein the window size can be a time span or the log number in a log sequence, and the step length is a window moving distance; when the log information is grouped in a fixed window manner, the logs are grouped according to the log frequency, which is defined by the time stamp of the log message or the number of times it occurs in the data set, each window having a fixed window size, i.e., the time span or the number of logs. In addition, the window size may be set to 1 or more, and when the window size is set to 1, the model may detect whether the log is abnormal one by one.

Grouping each log according to the window information, wherein all logs in each group form a log sequence.

S2, word segmentation conversion is carried out on the log sequence, and model input is constructed.

Performing word segmentation on each log in each log sequence, and sequentially performing ID (identification) numbering on segmented parts to obtain log word segmentation of each log in the log sequence;

and performing word segmentation conversion on the log sequence, and constructing model input. Firstly, taking an ID number corresponding to log word segmentation as an input ID, taking a log sequence number in a log sequence as a word segmentation class to mark which log in the log sequence the word segmentation belongs to, taking the attention value of an attention mechanism to a filling mark as an attention mask, taking the attention value of the attention mechanism to the filling mark as 0 or 1, wherein 0 represents no attention, 1 represents attention, and finally fitting the input ID, the word segmentation class and the attention mask to obtain model input.

And S3, inputting and conveying the model into a neural network model for training, and obtaining the trained neural network model. The normal and abnormal characteristic information of the log can be extracted through learning the trained neural network model.

S4, acquiring logs to be tested, converting the logs into model input, and conveying the model input into a trained neural network model to obtain characteristic information;

as shown in fig. 3, the neural network model includes an embedded vector superposition layer capable of superposing three embedded vectors, a multi-head attention layer, a feedforward neural network layer and two normalization processing layers, and after the input of the model, vector transformation is performed on the input of the model to be detected based on the trained vector embedding layer of the neural network model to obtain the embedded vectors; then, the embedded vectors are overlapped to obtain input vectors; then, the multi-head attention layer in the trained neural network model is utilized to learn the input vector, and a multi-head attention output vector is obtained; carrying out residual connection and layer normalization on the multi-head attention output vector to obtain a normalized vector; then, the normalized vector is learned by utilizing a feedforward neural network layer in the trained neural network model, and the learned vector is obtained; and finally, carrying out residual connection and layer normalization on the learned vector to obtain characteristic information.

S5, inputting the characteristic information into a classifier to obtain a log abnormality detection result.

The classifier consists of a pooling layer, a full connection layer and a Softmax function, and characteristic information is input into the pooling layer in the classifier to carry out pooling operation to obtain pooling information; then inputting the pooling information into a full-connection layer in the classifier for full connection to obtain integrated information; and finally, inputting the integrated information into a Softmax function in a classifier to classify, and obtaining a log abnormality detection result.

As shown in fig. 2, a full log feature anomaly detection system without log parsing includes:

the log grouping module is used for grouping logs to obtain a log sequence;

the input construction module is used for carrying out word segmentation conversion on the log sequence and constructing model input;

the learning training module is used for conveying the model input to the neural network model for training to obtain a trained neural network model;

the feature extraction module is used for acquiring logs to be detected, converting the logs into model input and transmitting the model input to the trained neural network model to obtain feature information;

the abnormality detection module is used for inputting the characteristic information into the classifier to obtain a log abnormality detection result

The content in the method embodiment is applicable to the system embodiment, the functions specifically realized by the system embodiment are the same as those of the method embodiment, and the achieved beneficial effects are the same as those of the method embodiment.

While the preferred embodiment of the present invention has been described in detail, the invention is not limited to the embodiment, and various equivalent modifications and substitutions can be made by those skilled in the art without departing from the spirit of the invention, and these modifications and substitutions are intended to be included in the scope of the present invention as defined in the appended claims.

Claims (7)

1. The full-log feature anomaly detection method without log analysis is characterized by comprising the following steps of:

grouping the logs to obtain a log sequence;

performing word segmentation conversion on the log sequence, and constructing model input;

the model input is conveyed to a neural network model for training, and a trained neural network model is obtained;

obtaining logs to be tested, converting the logs into model input, and transmitting the model input to a trained neural network model to obtain characteristic information;

and inputting the characteristic information into a classifier to obtain a log abnormality detection result.

2. The method for detecting the abnormal full log features without log parsing according to claim 1, wherein the step of grouping the logs based on the window to obtain the log sequence specifically comprises the steps of:

traversing the log information to determine a grouping mode;

window division is carried out on the logs according to the grouping mode, and window information is obtained;

and grouping the logs according to the window information to obtain a log sequence.

3. The method for detecting the abnormality of the full log feature without log parsing according to claim 1, wherein the step of performing word segmentation and transformation on the log sequence and constructing a model input specifically comprises the steps of:

performing word segmentation on each log in each log sequence to obtain log word segmentation of each log in the log sequence;

and converting and constructing the log word segmentation to obtain model input.

4. The method for detecting the abnormal full-log features without log analysis according to claim 1, wherein the step of obtaining the log to be detected and converting the log to a model input and transmitting the model input to a trained neural network model to obtain the feature information comprises the following steps:

constructing a model input to be tested according to the log to be tested;

vector conversion is respectively carried out on the input of the model to be tested based on the vector embedding layer of the trained neural network model, and an embedded vector is obtained;

superposing the embedded vectors to obtain input vectors;

learning the input vector by utilizing the trained neural network model through a multi-head attention layer in the network model to obtain a multi-head attention output vector;

residual connection and layer normalization are carried out on the multi-head attention output vector to obtain a normalized vector;

the normalized vector is learned by utilizing the trained neural network model through a feedforward neural network layer in the network model, and the learned vector is obtained;

and carrying out residual connection and layer normalization on the learned vector to obtain characteristic information.

5. The method for detecting the abnormal features of the whole log without log parsing according to claim 1, wherein the step of inputting the feature information to the classifier to obtain the detection result of the abnormal features of the log specifically comprises the steps of:

inputting the characteristic information into a pooling layer in the classifier to carry out pooling operation to obtain pooling information;

inputting the pooling information into a full-connection layer in the classifier for full-connection to obtain integrated information;

and inputting the integrated information into a Softmax function in a classifier to classify, so as to obtain a log abnormality detection result.

6. The method for detecting the abnormality of the full log feature without log parsing according to claim 3, wherein the step of converting and constructing the log word to obtain the model input specifically comprises the steps of:

taking an ID number corresponding to the log word segmentation as an input ID;

taking the log sequence number in the log sequence as a word segmentation class;

taking the attention value of the attention mechanism to the filling mark as an attention mask;

fitting the input ID, word segmentation class, and attention mask, results in a model input.

7. A full-log feature anomaly detection system without log parsing, comprising:

the log grouping module is used for grouping logs to obtain a log sequence;

the input construction module is used for carrying out word segmentation conversion on the log sequence and constructing model input;

the learning training module is used for conveying the model input to the neural network model for training to obtain a trained neural network model;

the feature extraction module is used for acquiring logs to be detected, converting the logs into model input and transmitting the model input to the trained neural network model to obtain feature information;

the abnormality detection module is used for inputting the characteristic information into the classifier to obtain a log abnormality detection result.

Images