CN115587007A - Robertta-based weblog security detection method and system - Google Patents

Abstract

The invention discloses a RoBERTa-based network log security detection method and system. The method includes: acquiring tagged network log data sets of all network devices; preprocessing tagged network log data; constructing a RoBERTa model and passing the tagged network The log data set trains it, and the RoBERTa model uses a bidirectional Transformer network structure as an encoder, and uses a Softmax classifier to obtain the probability that the log is at risk; the optimal model is screened through the dropout function; the labeled network log data is input to the optimal The RoBERTa model obtains the probability that the log is at risk. The invention can process logs of various unknown types and formats, and improves the accuracy of network log security detection.

Description

Network log security detection method and system based on RoBERTa

technical field

The invention relates to the technical field of network security, in particular to a RoBERTa-based network log security detection method and system.

Background technique

Network log data is very important to network administrators because it contains information on every event that occurs in the network, including system errors, alerts, and packet sending status. Efficiently analyzing large volumes of disparate log data brings opportunities to identify problems before they become a problem and prevent future cyber attacks; however, processing of disparate NetFlow data presents challenges such as volume, speed, and accuracy of log data. The invention can simplify the advanced network attack detection model through the RoBERTa model. The characteristics of various cyber attacks can be learned from this model by understanding the cyber attack behavior and cross-validating with a log analysis system.

Network logs include messages of various types, from critical failures to normal console logs. Log messages typically consist of three components: a timestamp, a host identifier (such as an IP address), and the message. The format of log messages depends on the provider or service, and there is no uniform description rule. This is why it is time consuming to describe regular expressions and define new alert rules for each message.

At present, the industry usually uses the syslog protocol as the recording standard for transmitting messages in the Internet protocol. This protocol is mainly used for network information management and security auditing. The format of the Syslog message has a certain structure, and the log server can directly receive the syslog message and analyze its content to realize the simple judgment of the time.

The current syslog log component has many shortcomings. For example, there is no strict format control, and operation and maintenance engineers need to learn a lot of professional knowledge; there is no unified standard for log warning level classification, and effective correlation analysis cannot be performed. Therefore, for network operation and maintenance engineers, there is an extremely urgent need for a simple and easy-to-operate log processing method that requires less knowledge reserve.

Logs have become an important information resource generated by current information systems. Log-based anomaly detection technology can effectively discover security problems in the system and explore potential security threats, which has become a hot spot of current inventions. With the development and popularization of artificial intelligence technology, more and more related inventions have been applied to log-based anomaly detection. In the log-based anomaly detection method, steps such as log collection, log parsing, feature extraction, and anomaly detection are included. Among them, log parsing and anomaly detection are the core parts, which are also the focus of this patent.

At present, the invention of log parsing has developed from the traditional definition of regular expressions to automated methods, mainly including code analysis, machine learning, and natural language processing. In the log-based anomaly detection method, it is mainly divided into supervised learning, unsupervised learning and deep learning. Most anomaly detection methods perform offline analysis for specific scenarios and datasets, lacking practical methods with versatility and high accuracy. When the number of samples is small, the model often cannot produce the best detection effect. To obtain the ideal model effect, a large number of labeled data sets are required for multiple iterations of training, which consumes a lot of manpower and material resources. Moreover, the current attacks are becoming more and more hidden, and the attack steps are becoming more and more cumbersome, and the joint analysis of related device logs can effectively discover potential attacks.

To sum up, in order to solve these problems, the model should not only focus on a single log source, but also combine different events and different devices for log analysis, and then perform anomaly detection, etc. In addition, related inventions using machine learning will be further applied to online detection , it becomes very important to construct a general and effective online log-based anomaly detection method and apply it to practice.

Contents of the invention

The object of the present invention is to provide a RoBERTa-based network log security detection method and system. The present invention can process logs of various unknown types and formats, and improves the accuracy of network log security detection.

The technical solution that realizes the object of the present invention is: a kind of network log security detection method based on RoBERTa, comprises steps:

Obtain a labeled network log dataset of all network devices;

Preprocessing of labeled web log data;

Construct a RoBERTa model and train it through a labeled network log data set. The RoBERTa model uses a bidirectional Transformer network structure as an encoder, and uses a Softmax classifier to obtain the probability that the log is at risk;

Filter the optimal model through the dropout function;

Input the labeled network log data into the optimal RoBERTa model to obtain the probability that the log is at risk.

Further, the RoBERTa model converts the input log data into a 768-dimensional high-dimensional vector.

Further, the BiLSTM of the RoBERTa model includes a forward LSTM and a backward LSTM.

Further, the Transformer block includes multiple sublayers, each sublayer includes a multi-head self-attention mechanism and a fully connected feedforward network, and a residual connection module and a normalization module are added between every two sublayers.

Further, the multi-head self-attention mechanism performs multiple sets of linear transformations on the Query vector, Key vector and Value vector of each character, performs self-attention calculations respectively, and then splices all calculation results.

Further, the lengths of the Query vector, the Key vector and the Value vector are all 64.

Further, the multi-head self-attention mechanism uses a scaling factor for correction.

Further, the RoBERTa model adds [CLS][SEP] characters to the input log text data, and divides the log text data into individual characters, and then stores the individual characters as a vocabulary, and each character corresponds to a unique identifier.

Further, adding [CLS][SEP] characters to the log text data is specifically: the first vector of each log text data is a [CLS] sign, which is used for downstream network log classification tasks, and the sentence end vector is [SEP ] mark, which is used as a separator for different logs, and the log text data input by the RoBERTa model only uses one sentence vector.

A network log security detection system based on RoBERTa, comprising a data collection module, a log word segmentation module, a network log security detection module, a training module and a database, the data collection module is used to collect device information and log files thereof in a network environment, And the collected data is saved to the database; the log word segmentation module is used for data preprocessing; the network log security detection module is based on the RoBERTa model, and the RoBERTa model adopts a bidirectional Transformer network structure as an encoder, and uses a Softmax classifier to obtain logs There is a probability of risk; the training module is used to train and update the network log security detection module, and the optimal model is screened through the dropout function; the database is used to save log data.

Compared with the prior art, the beneficial effect of the present invention is that: the present invention not only focuses on a single log source, but also combines different events and different devices for log analysis, and performs efficient and accurate detection through the constructed RoBERTa; the present invention handles various unknown types and format logs, which solves the weakness of the previous template-based method for log analysis of unknown definitions, and improves the usability of the system and the ease of operation for users; the invention has the advantages of low cost, low consumption, and high execution efficiency.

Description of drawings

Figure 1 is a framework diagram of log-based anomaly detection.

Figure 2 is a flowchart of training the RoBERTa model.

Figure 3 is the overall architecture diagram of the model.

Figure 4 is a Transformer structure diagram.

detailed description

The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only some of the embodiments of the present invention, not all of them. Based on the embodiments of the present invention, all other embodiments obtained by persons of ordinary skill in the art without making creative efforts belong to the protection scope of the present invention.

Terms used in the embodiments of the present invention are only for the purpose of describing specific embodiments, and are not intended to limit the present invention. As used in the embodiments of the present invention and the appended claims, the singular forms "a", "said" and "the" are also intended to include the plural forms unless the context clearly indicates otherwise.

It should be understood that the term "and/or" used in this patent is just a same field describing associated objects, indicating that there may be three relationships, for example, A and/or B, which may mean: A exists alone, and at the same time A and B, there are three cases of B alone. In addition, the character "/" in this patent generally indicates that the contextual objects are an "or" relationship.

The invention can simplify the advanced network attack detection model through the BERT model. The characteristics of various cyber attacks can be learned from this model by understanding the cyber attack behavior and cross-validating with a log analysis system.

A RoBERTa-based network log security detection system provided in this embodiment includes a natural language processing component and a database; the natural language processing component includes a word segmentation system and a RoBERTa algorithm; the database includes a thesaurus and a large number of different types of networks The device exports logs, and each log is accompanied by a corresponding label; the database is associated with the natural language processing component; the word segmentation system adopts the split function, and the word segmentation work is performed with empty intervals.

The natural processing component is used to summarize and classify the syslog source data and log files of the device, analyze and determine the meaning contained in the log statement, and obtain a certain number of statements to train the language processing model of the database. The natural language processing component is based on The preset word meaning database acquires valid fields of the sentence for training and learning to generate several training words as keywords, generates analytical information for the keywords, and generates a language processing model based on the keywords and corresponding analytical information, and the language processing model adopts RoBERTa model plus bidirectional Transformer structure.

Preferably, the natural language processing component also includes a data acquisition module, a log word segmentation module, and an analysis module: the acquisition module is used to receive basic information of the device source or cache the training sentences in the database, and convert the sentences into 768 words through the RoBERTa model Dimension vector, according to the predefined rules for classification processing.

Preferably, the collection module includes a device determination module, a log collection module and an association analysis module, and the device determination module uses device discovery technology to obtain device information in the network environment, and stores the basic information of the device in the word segmentation database of the database , the log collection module collects network log files from the network equipment monitored by the syslog log server as a data source for RoBERTa model training, and collects logs that need to be analyzed simultaneously.

Preferably, the system further includes a training module, the training module is used to update the RoBERTa model with the analysis information obtained from the analysis module, so as to update the model of the corresponding equipment.

Preferably, the devices include but not limited to switches, servers, gateways, routers and network security devices.

Preferably, the basic information of the device includes but not limited to device name, device type, device IP and manufacturer name.

The invention provides a network log processing method based on RoBERTa, the specific steps are as follows: the acquisition module obtains the equipment information in the network environment, records its basic information and uses it as the basic statement for statement collection, and simultaneously acquires the network information of the equipment in the syslog log server log; then import the log and device information into the trained RoBERTa model for analysis; finally output the judgment result of whether there is a security risk in the log.

A kind of network log security detection method based on RoBERTa, its feature comprises:

The model building module is used to obtain the network log data set, and constructs a log anomaly detection network model according to the mapping vectors in all network logs in the network log data set:

A model analysis module, configured to analyze the abnormal logs in the network log data set according to the network access characteristics of each user in the user behavior network model, and the access status of each node and path in the user behavior network model identify.

Example 1

With reference to Fig. 1, a kind of abnormal user detection method based on network log that the embodiment of the present invention provides comprises:

Step 1, first obtain the labeled network log data sets of various network devices.

Step 2. After obtaining the network log data set composed of a large number of network logs, since the data of the original log is a dictionary type, it mainly includes fields such as user IP, request time, request method, request size, status code, and request UL . So requesting UL can be expressed as a set of request resource path and optional request parameters. According to the network access characteristics of each user in the pre-trained network model, and the access status of each node and path in the user behavior network model, the abnormal users in the network log data set are identified. At this time, the collected network logs are preprocessed such as word segmentation and stop words removal, and the required parameters are extracted from the original logs, and processed into a data format that can be directly processed.

Step 3, use the pre-trained RoBERTa model to process the processed log data set, and extract the corresponding feature representation vector after the data set passes through the bidirectional Transformer encoder network structure. Introduce RoBERTa as a vectorized feature representation method for pre-trained text, which uses a bidirectional Transformer network structure as an encoder.

Step 4, then normalize and calculate the safety polarity probability through the Softmax function, and finally output the probability that the log is at risk. The extracted [CLS] feature vector is processed through the Softmax classifier function, and finally compared with the probability of the safe model to infer whether there is an abnormality. The logs with the same IP in each log are regarded as logs generated by the same user. After obtaining the preprocessed log data, the user behavior network model can be generated through the access relationship between paths. For each user, a smaller behavioral network model N1 belonging to the user can be constructed according to the log access conditions of the user, and then a larger pre-training model N2 can be constructed using the log access conditions of all users.

Specifically, after constructing the anomaly detection model, according to the network access characteristics of a single user in the model, and the access status of each node and path in the user behavior network model, each user in the user behavior network model is analyzed as an analysis index And nodes are analyzed to detect abnormal users and abnormal nodes.

Through this method, after the original log data is preprocessed, the required data is extracted, and then a user behavior network model is established based on these data. During the log analysis, the user network access characteristics and each The status of the node is used as an analysis index, and abnormal log detection can be carried out quantitatively based on the log data. Through the method of cosine similarity detection, the network log can be analyzed quickly, efficiently and scientifically.

With reference to Fig. 2, described RoBERTa model training procedure comprises:

Step 5, artificially mark the risk level for the collected data set.

Step 6. Preprocess the labeled data set and delete irrelevant sequences.

Step 7: Divide the data set according to the proportion of 80% of the training set and 20% of the test set, while ensuring that the data volumes of different types of risks in the training set and the test set are consistent.

Step 8, add [CLS] and [SEP] characters in the dataset. The first vector of each log is the [CLS] flag, which can be used for downstream network log classification tasks. The sentence-end vector [SEP] flag is used as a separator for different logs. Since the log is a sentence-level classification problem, the input is a sentence, so only one sentence vector is used.

Step 9: Convert the log into a 768-dimensional high-dimensional vector through the pre-trained RoBERTa model, where each log contains multiple word embedding vectors, and the word embedding vector is the static code of each word.

Step 10, the high-dimensional vector vector is passed into the BiLSTM structure network for training, and the representation vector is output.

Step 11, the [CLS] vector is subjected to the Softmax normalization function to calculate the probability of different types.

Step 12, filter the optimal model through the dropout function.

Referring to Figure 3, the RoBERTa model uses the Transformer Encoder block for connection, because it is a typical two-way encoding model. In the Transformer block, the data first passes through the multi-head attention module to obtain a weighted feature vector. In the attention mechanism, each character has 3 different vectors, namely Query vector (Q), Key vector (K) and Value vector (V), all of length 64. Concretely include word segmentation input layer 19, division layer 18, Transformer encoder block 17, vector output layer 16, forward LSTM 15, backward LSTM 14 and hidden layer 13 connected in sequence;

The hidden layer 13 of the RoBERTa model is different from the simple weight combination model of RoBERTa and BiLSTM. The present invention uses the character vector generated by RoBERTa as the character embedding layer in the upstream part, and uses BiLSTM as the feature extractor in the downstream part to model and analyze network logs. to dig. RoBERTa is used to dynamically construct representations of character vectors, while BiLSTM is used to integrate textual information and sequential features of sentences. The combination of the two can obtain more complex semantic features and build a more accurate semantic representation.

The backward LSTM 14 of the RoBERTa model, the backward LSTM 14 can easily capture the sequential information of the text due to its linear structure, but it cannot encode the information from the back to the front, while BiLSTM can.

Forward LSTM 15 of the RoBERTa model, BiLSTM is a combination of forward LSTM 15 and backward LSTM 14. BiLSTM is adopted as a language model with long-term sequence information to generate accident report mining results, where the result of the character vectorization representation of the RoBERTa output layer is used as the input of the BiLSTM layer.

The vector output layer 16 of the RoBERTa model outputs high-dimensional representation vectors.

The Transformer encoder block 17 of the RoBERTa model has a specific structure shown in FIG. 4 . Transformer is the core structure of RoBERTa. Each Trm of the RoBERTa structure corresponds to a Transformer block on the right. Each Transformer block consists of two sublayers, a multi-head self-attention mechanism and a fully connected feedforward network. Each sublayer adds residual connections and normalization. Transformer is a new model proposed by Google to solve the Seq2Seq problem. It uses the Attention structure to completely replace the LSTM structure, brings the idea of the attention mechanism to the extreme, and has made considerable progress in the field of machine translation. Because the Transformer model does not use the LSTM structure to model temporal information, a position embedding layer is added in the encoding process to make up for the inability of the self-attention mechanism to capture temporal information. The Transformer encoder block 17 includes an input module 25, a weighting module 24, a multi-head attention mechanism module 23, a second residual module 22, an Encoder module 21 and a first residual module 20 connected step by step; specifically:

The first residual module 20 of the Transformer encoder block adds residuals. The introduction of residuals is to solve the network degradation caused by depth. In fact, many experiments have shown that the residuals here contribute greatly to network performance. Norm is a normalization module, which refers to normalizing the output value.

The Encoder module 21 of the Transformer model is partially composed of sublayers with the same N-layer structure. Each sublayer is a combination of two network structures, namely Multi-Head Self Attention Network (MSAN) and fully connected Feed Forward Network (FFN). Moreover, the output of each layer is normalized and residually connected to deal with the degradation problem in the process of multi-layer network stacking.

The second residual module 22 is the same as the first residual module 20 .

The starting point of the multi-head attention mechanism module 23 is to expand the feature representation space by performing attention calculations on the input in different subspaces, so as to achieve the purpose of deep modeling of the text. The implementation method is to perform multiple sets of linear transformations on Q, K, and V, perform self-attention calculations separately, and then stitch all the results. The multi-head self-attention network used in the encoder adds a scaling factor and a multi-head attention mechanism to the traditional self-attention mechanism. The scaling factor refers to the correction of the traditional attention calculation method to prevent the problem that the dimension is too high and the result is too large and the gradient is too small.

Weighting module 24, in the Transformer block, the data first passes through the multi-head attention module to obtain a weighted feature vector. In the attention mechanism, each character has 3 different vectors, namely Query vector (Q), Key vector (K) and Value vector (V), all of length 64.

The input module 25 implements input word embedding.

After word embedding, position embedding, and segment embedding, the high-dimensional vector is divided into layer 18 to segment characters, and the input text data is divided into individual characters (E1, E2, ..., En), and then the individual characters are stored as a vocabulary, namely Each character has a corresponding unique identifier (denoted as [character: ID]).

After preprocessing such as word segmentation, the log data after adding [CLS][SEP] characters through the word segmentation input layer 19.

A network log security detection system based on RoBERTa, comprising a data collection module, a log word segmentation module, a network log security detection module, a training module and a database, the data collection module is used to collect device information and log files thereof in a network environment, And the collected data is saved to the database; the log word segmentation module is used for data preprocessing; the network log security detection module is based on the RoBERTa model, and the RoBERTa model adopts a bidirectional Transformer network structure as an encoder, and uses a Softmax classifier to obtain logs There is a probability of risk; the training module is used to train and update the network log security detection module, and the optimal model is screened through the dropout function; the database is used to save log data. The system includes all technical features of the abnormal user detection method.

It should be pointed out that since RoBERTa is a well-known model, this embodiment only elaborates on the improvements and innovations in detail, and the common knowledge in the field that is not elaborated in detail will not be repeated here.

Claims (10)

1. a network log safety detection method based on RoBERTa, is characterized in that, comprises steps: Obtain a labeled network log dataset of all network devices; Preprocessing of labeled web log data; Construct a RoBERTa model and train it through a labeled network log data set. The RoBERTa model uses a bidirectional Transformer network structure as an encoder, and uses a Softmax classifier to obtain the probability that the log is at risk; Filter the optimal model through the dropout function; Input the labeled network log data into the optimal RoBERTa model to obtain the probability that the log is at risk. 2. The network log security detection method according to claim 1, wherein the RoBERTa model converts the input log data into a high-dimensional vector of 768 dimensions. 3. the network log security detection method according to claim 1, is characterized in that, the BiLSTM of described RoBERTa model comprises forward LSTM and backward LSTM. 4. the network log security detection method according to claim 1, is characterized in that, described Transformer block comprises a plurality of sub-layers, and each sub-layer comprises multi-head self-attention mechanism and fully connected feed-forward network, between every two sub-layers A residual connection module and a normalization module have been added. 5. the network log security detection method according to claim 4, is characterized in that, described multi-head self-attention mechanism carries out multiple groups of linear transformations to the Query vector, Key vector and Value vector of each character, carries out self-attention respectively Calculate, and then splicing all the calculation results. 6. The network log security detection method according to claim 1, wherein the lengths of the Query vector, the Key vector and the Value vector are all 64. 7. The network log security detection method according to claim 1, wherein the multi-head self-attention mechanism adopts a scaling factor for correction. 8. the network log security detection method according to claim 1, is characterized in that, described RoBERTa model adds [CLS] [SEP] character to the log text data of input, and is divided into single character through log text data, and then Individual characters are stored as a vocabulary, with each character corresponding to a unique identifier. 9. The network log security detection method according to claim 1, wherein adding [CLS][SEP] characters to the log text data is specifically: the first vector of each log text data is a [CLS] sign , used for downstream network log classification tasks, the sentence end vector is the [SEP] flag, used as a separator for different logs, and the log text data input by the RoBERTa model only uses one sentence vector. 10. A network log security detection system based on RoBERTa, it is characterized in that, comprises data collection module, log segmentation module, network log security detection module, training module and database, and described data collection module is used for collecting the equipment in the network environment information and its log files, and save the collected data to the database; the log word segmentation module is used for data preprocessing; the network log security detection module is based on the RoBERTa model, and the RoBERTa model uses a bidirectional Transformer network structure as an encoder, A Softmax classifier is used to obtain the probability that the log is at risk; the training module is used to train and update the network log security detection module, and the optimal model is screened through a dropout function; the database is used to save log data.

Images