CN110011990B

CN110011990B - Intelligent analysis method for intranet security threats

Info

Publication number: CN110011990B
Application number: CN201910219705.5A
Authority: CN
Inventors: 王志; 肖旭航; 谢学说; 李涛
Original assignee: Nankai University
Current assignee: Nankai University
Priority date: 2019-03-22
Filing date: 2019-03-22
Publication date: 2022-03-04
Anticipated expiration: 2039-03-22
Also published as: CN110011990A

Abstract

The invention provides an intelligent analysis method for intranet security threats, which is applied to the field of network security. The method is realized by the following steps: 1, calculating a multi-model inconsistency score, comprising: step 1.1, generating a log template set; step 1.2, calculating an inconsistency score; the intelligent analysis method for the intranet security threat comprises the following steps: step 2.1, calculating P-Value; and 2.2, predicting the log stream to be tested based on statistical learning. The invention utilizes artificial intelligence analysis instead of manual analysis to realize the analysis of the log; changing a storage mode of storing the logs in a local area, and then adopting a block chain to perform safe storage, and realizing the merging use of the logs generated by different devices in the intranet; the method supports various log analysis models and realizes multi-model cooperation; by using the statistical learning method, the detection capability of the abnormal log is improved.

Description

Intelligent analysis method for intranet security threats

Technical Field

The present invention belongs to the field of computer network security.

Background

With the development of networks, the amount of logs generated by equipment is larger and larger, and the logs are difficult to analyze manually; due to the development of the technology, the logs originally stored in the local area face a huge risk of being tampered; at present, in an intranet, a device cannot inform other devices in the same network of an attack behavior encountered by the device; the single model can be degraded along with time, so that a comprehensive and accurate detection result cannot be obtained. Therefore, a model which can analyze the logs by using artificial intelligence, can store the logs more safely by using a block chain, can detect abnormal logs by using the advantages of an intranet and can defend through multi-model cooperation is required to be constructed to discover threats.

Disclosure of Invention

The invention aims to solve the problems that under the condition that a large number of logs are generated in an intranet, the logs are easy to tamper and cannot be combined for use, and a model is degraded, so that a comprehensive and accurate result cannot be obtained in prediction, and provides an intranet security threat intelligent analysis method based on statistical learning. The method utilizes artificial intelligence analysis instead of manual analysis to realize the analysis of the log; changing a storage mode of storing the logs in a local area, and then adopting a block chain to perform safe storage, and realizing the merging use of the logs generated by different devices in the intranet; the method supports various log analysis models and realizes multi-model cooperation; and the statistical learning algorithm is utilized to improve the identification capability of the abnormal log.

Technical scheme of the invention

The intelligent analysis method for the safety threat of the intranet comprises the following steps:

basic concept:

(1) logging: recording text of valuable application and system running information, wherein the content comprises a time stamp and the like;

(2) log stream: the log with the standard format can be used as the input of a log analysis algorithm, and is different from the log in that each line only has one log stream, but a plurality of lines of logs can exist;

(3) a log template: the log stream is only composed of constants in the log stream and can represent a group of log streams;

(4) inconsistency metric function: evaluating the similarity of the log stream to be tested and the known log stream set through the score;

describing the similarity between a log stream and a group of known log streams, inputting a log template and a log stream to be tested, outputting a numerical value, namely an inconsistency score, wherein the higher the score is, the more similar the log stream to be tested is to the group of log streams; the lower the score is, the more dissimilar the log stream to be tested is to the group of log streams;

(5) P-Value: the statistical quantity of the significance of the log stream to be measured in the known log stream set is measured and used for comparing the credibility of the multi-model prediction result;

1, calculating a multi-model inconsistency score, which comprises the following steps:

step 1.1, generating a log template set

1.1.1, processing the log stream set by using a log analysis algorithm f to obtain a log template set T;

1.1.2, input of log analysis: original log stream set X, log parsing algorithm set G:

original log stream set X: containing n log streams x_j,j∈{1,2,...,n},X＝{x₁,...,x_n}；

Log analysis algorithm set G: comprising m log parsing algorithms f_k,k∈{1,2,...,m},G＝{f₁,...,f_m}；

1.1.3, output of log analysis: comprising a set of m log templates T_k,k∈{1,2,...,m},T＝{T₁,...,T_m}; wherein each log template set T_kContaining q log templates t_kb，b∈{1,2,...,q},T_k＝{t_k1,...,t_kb}。

Step 1.2, calculate the inconsistency score

1.2.1, each heterogeneous log analysis algorithm, for the log stream y to be tested_iCan be based on the log template t_kbCalculating an inconsistency score alpha by utilizing an inconsistency measurement function g; inconsistency scores given by the heterogeneous model have no comparability, and the quality of a model prediction result cannot be directly compared according to the inconsistency scores;

1.2.2, input of inconsistency metric: a log template set T, a log stream set Y to be tested, and an inconsistency measurement function g:

log template set T: comprising a set of m log templates T_k,k∈{1,2,...,m},T＝{T₁,...,

T_m}; wherein each log template set T_kContaining q log templates t_kb，b∈{1,2,...,

q},T_k＝{t_k1,...,t_kb}；

Secondly, a log stream set Y to be tested: contains w logs y_i,i∈{1,2,...,w},Y＝{y₁,...,y_w}；

Thirdly, an inconsistency measurement function g; the return value is a real number called the inconsistency score, and the input to the function is the log stream and the log template t_kbThe return value is a real number which indicates the inconsistency between the log stream to be tested and the log stream set;

1.2.3, output of inconsistency metric: a set of inconsistency scores;

1.2.4, algorithm flow:

2, an intelligent analysis method for intranet security threats, comprising the following steps:

step 2.1, calculating P-Value

2.1.1, log template t_kbIn particular, for the original log stream x_jCarrying out inconsistency measurement to obtain a corresponding inconsistency score set { a_{kb_1},a_{kb_2},...,a_{kb_j}}；

2.1.2 Log stream y to be tested_iDiscordance score of a_{kb_i}Put into the inconsistency score set of the class, and the P-Value of the class is more than or equal to the log stream y to be tested_iDiscordance score of a_{kb_i}The ratio of the number of log streams to the total number of log streams;

the larger the Value of the P-Value is, the larger the Value of the 2.1.3 indicates that the log stream y to be tested is_iThe higher the significance in this class;

2.1.4, input: a set of inconsistency scores for the log stream;

2.1.5, output: log stream y to be tested_iP-Value of (1);

2.1.6, algorithm flow:

step 2.2, predicting the log stream to be tested based on statistical learning

2.2.1, selecting P-Value values larger than the maximum error probability epsilon from all the P-Value sets obtained in the previous step to obtain template class sets corresponding to the P-Value values, and predicting the log stream y to be tested if an empty set exists_iIs an abnormal log, otherwise, the log stream y to be tested is predicted_iThe log is a normal log;

2.2.2, input: log stream y to be tested_iA set of P-Value values of; acceptable maximum error probability ε, provided by the user, indicates that the user is able toAn accepted maximum error probability;

2.2.3, output: predicting the result;

2.2.4, algorithm flow:

the invention has the advantages and positive effects that:

the invention provides an intranet security threat intelligent analysis method based on statistical learning. The method utilizes artificial intelligence analysis instead of manual analysis to realize the analysis of the log, thereby improving the analysis efficiency; the method changes the storage mode of the log stored in the local area, and adopts the block chain to perform safe storage, thereby improving the safety; the method realizes the merging use of the logs generated by different equipment of the internal network, and improves the efficiency of detecting the abnormity; the method supports various heterogeneous log analysis models, and achieves multi-model cooperative defense by using a statistical method.

Drawings

Fig. 1 is a flow chart of an intranet security threat intelligent analysis method based on statistical learning.

FIG. 2 is a portion of an industrial control log stream collection.

FIG. 3 is a log template set obtained by using IPLoM algorithm for industrial control logs.

FIG. 4 is a log template set obtained by using a Drain algorithm for an industrial control log.

FIG. 5 is a log template set obtained by using the LogSig algorithm for the industrial control log.

FIG. 6 is a P-Value set obtained by an IPLoM algorithm in an industrial control log.

FIG. 7 is a P-Value set obtained by using a Drain algorithm on an industrial control log.

Detailed Description

The invention specifically describes the detection of abnormal logs as an example, any log analysis algorithm for obtaining a log template set by inputting an original log stream set can be used for the method, the flow of the method is shown in fig. 1, and four log analysis algorithms of IPLoM, Drain V1 and LogSig are exemplified in the embodiment, and the specific description is as follows:

IPLoM is a log resolution algorithm. The algorithm is divided into four steps when the log is analyzed, and all the log is input at the beginning. The first step is that all original logs are divided into different groups according to the length; the second step is to continue grouping the original logs in the same log length group. All words in the same position of all log records are counted, the position with the least unique words is found, and the words are classified according to the unique words. Raw logs with the same unique word are grouped into a group. The third step will operate on each group obtained in the previous step. And dividing the original log into different groups according to the corresponding relation between words. And step four, processing each finally obtained group in the step. And if the words at the same position are different, replacing the words with wildcards, and directly writing down the same words to obtain the log template.

Drain is a log parsing algorithm. The algorithm is divided into three steps when the log is analyzed, and one log is input at a time. The purpose of the first step is to group original logs with the same length into a group; the second step is to classify the logs with the same token and the same length into a group, wherein the token in the method can be the first word of the log, the last word of the log and a wildcard; the third step is to gather the similar logs in the original logs with the same length and the same token together to obtain a log template which can represent the set.

Drain V1 is a log parsing algorithm. The algorithm is divided into three steps when the log is analyzed, and one log is input at a time. The purpose of the first step is to group original logs with the same length into a group; the second step is to classify logs with the same token and the same length into a group, wherein the token in the method can be the first word of the log and can also be a wildcard; the third step is to gather the similar logs in the original logs with the same length and the same token together to obtain a log template which can represent the set.

LogSig is a log parsing algorithm. The algorithm is divided into three steps when the log is analyzed, and all the log is input at the beginning. The first step is that each log stream is cut into a plurality of word pairs to form word pair groups; the second step is that all logs are randomly divided into groups with fixed number, and then each log is transferred to a proper group; the third step is to obtain a suitable log template for each group obtained in the previous step.

Each log analysis algorithm can process the original log set to obtain a template set T capable of representing the log set_k. For the log y to be tested_iCan collect T according to the obtained log template_kAnd calculating an inconsistency score by utilizing an inconsistency metric function g. The inconsistency scores given by the heterogeneous models are not comparable, and the quality of the model prediction result cannot be directly compared according to the inconsistency scores, so that the logs to be tested need to be predicted based on statistical learning after the inconsistency scores are obtained by the heterogeneous models.

1. Preprocessing raw logs

In this embodiment, the original log set is changed into a log stream set. The automatic script is used for processing the original textual logs with different formats to form a log stream set with a standard format suitable for being input as a log analysis algorithm. The set of industrial control log streams obtained after preprocessing is shown in fig. 2.

2. Storing and sharing log streams

The embodiment utilizes the block chain to realize safe and effective storage of the log stream and merging use of the log streams generated by various devices. Each block of the block chain is composed of a block header and a block body. The chunks are in one-to-one correspondence with the hashes. When encrypting using a chunk chain, the hash value of the current chunk is encrypted with the content of the current chunk header, which includes the hash of the previous chunk and the hash of the current chunk. This ensures that the use of the blockchain can achieve secure storage of the log stream; when data is written into the block chain, all nodes need to synchronize the data, so that each device on the intranet can use logs generated by all devices, and the log streams generated by various devices can be combined for use.

3. Get a set of log templates

In this embodiment, two thousand log streams are processed by four log analysis algorithms, i.e., IPLoM, Drain, DrainV1, and LogSig, to obtain a log template set. The input is a set of log streams stored on the blockchain and the output is a log template. And generating a log event template by using the constant in the unstructured content in the log stream. The IPLoM algorithm combines the log length, the location of the word tokens and the mapping between word tokens to generate a log event template, as shown in FIG. 3. The Drain algorithm utilizes a directed acyclic graph that can be automatically generated and updated to generate a log time template, which is used primarily in online and distributed systems, as shown in FIG. 4. The LogSig algorithm first converts each log into a plurality of word pairs including two words and a position between them, then performs a grouping operation on them, and finally obtains a log event template, as shown in fig. 5.

4. Calculating an inconsistency score

In this embodiment, the obtained log template set is used to calculate an inconsistency score for the log stream set according to the inconsistency metric function, and each log analysis algorithm obtains each log template set to obtain an inconsistency score set. For each log analysis algorithm, we can finally get the same number of inconsistency score sets as the log templates.

And then, carrying out inconsistency measurement on the log stream x to be measured by utilizing the templates obtained by four log analysis algorithms of IPLoM, Drain V1 and LogSig, wherein each template corresponds to an inconsistency score. Inconsistency scores given by four log analysis algorithms of IPLoM, Drain V1 and LogSig are not comparable, and the quality of a model prediction result cannot be directly compared according to the inconsistency scores.

5. Calculating P-Value

In this embodiment, one of the templates t is used_kbFor example, the corresponding inconsistency is recorded as diversity a_{kb_1}，a_{kb_2},...,a_{kb_j}}, log stream y to be tested_iThe inconsistency score corresponding to this template is marked as a_{kb_i}. To-be-tested log stream y_iDiscordance score of a_{kb_i}Put into the corresponding inconsistency score set { a of the template_{kb_1},a_{kb_2},...,a_{kb_j}In the description, the P-Value corresponding to this template is the set { a }_{kb_1},a_{kb_2},...,a_{kb_j}In } the inconsistency score is greater than or equal to a_{kb_i}The ratio of the number of (c) to the total number. And performing the same calculation on all the templates obtained by the same log analysis algorithm to obtain a P-Value set corresponding to each log analysis algorithm. The P-Value sets corresponding to the two log resolution algorithms, i.e., IPLoM and Drain, are used herein as examples, as shown in fig. 6 and 7.

6. Detecting log streams based on statistical learning

And selecting the P-Value values larger than the maximum error probability epsilon from all the P-Value sets obtained in the previous step, wherein the maximum error probability epsilon is provided by a user, and obtaining the template class sets corresponding to the P-Value values. If the empty set exists, predicting the log stream y to be tested_iIs an abnormal log, otherwise, the log stream y to be tested is predicted_iIs a normal log.

7. General algorithm flow

(1) Inputting: a log stream set X, a log Y to be tested, a log analysis algorithm set G, an inconsistency measurement function G and an acceptable maximum error probability epsilon:

Third, log analysis algorithm set G: comprising m log parsing algorithms f_k,k∈{1,2,...,m},G＝{f₁,...,f_m}；

Fourthly, an inconsistency measurement function g; the return value is a real number called the inconsistency score, and the input to the function is the log stream and the log template t_kbThe return value is a real number indicating the value to be measuredInconsistency of log streams with the log stream set;

acceptable maximum error probability epsilon provided by the user, indicating the maximum error probability acceptable by the user.

(2) And (3) outputting:

prediction results, i.e. logs to be tested y_iNormal or abnormal.

(3) The algorithm flow is as follows:

Claims

1. an intelligent analysis method for intranet security threats is characterized by comprising the following steps:

1, calculating the inconsistency scores of the log template of the multiple models and the log stream to be tested, which comprises the following steps:

step 1.1, generating a log template set;

step 1.2, calculating the inconsistency score of the log template and the log stream to be tested;

the intelligent analysis method for the intranet security threat comprises the following steps:

step 2.1, calculating P-Value;

step 2.2, predicting the log stream to be tested based on statistical learning;

wherein

The step 1.1 comprises the following steps: processing the original log stream set by using a log analysis algorithm f to obtain a log template set T;

the step 1.2 comprises the following steps: each heterogeneous log analysis algorithm is used for the log stream y to be tested_iI ∈ {1, 2., w }, which can be based on the log template t_kbK represents one log template set in the log template sets, b represents one log template in the log template sets, and the inconsistency measuring function g is utilized to calculate the log templateAnd the inconsistency score a of the log stream to be tested_{kb_i}；

The step 2.1 comprises the following steps:

for log template t_kbIn particular, for the original log stream x_jAnd j belongs to {1, 2.. once, n }, carrying out inconsistency measurement to obtain a corresponding inconsistency score set { a }_{kb_1},a_{kb_2},...,a_{kb_j}}；

To-be-tested log stream y_iDiscordance score of a_{kb_i}Put into original log stream x_jAnd a log template t_kbIn the obtained inconsistency score set, the log stream y to be tested is related to_iAnd a log template t_kbThe P-Value of (A) is_{kb_j}Greater than or equal to the log stream y to be tested_iDiscordance score of a_{kb_i}The larger the P-Value is, the larger the ratio of the number of the log streams to the total number of the log streams is, the larger the P-Value is, the log stream y to be tested is_iThe higher the significance in the set of log templates.

The step 2.2 comprises the following steps:

selecting P-Value values larger than the maximum error probability epsilon from the P-Value sets of all log templates obtained in the previous step to obtain a set formed by the log templates corresponding to the P-Value values, and predicting the log stream y to be tested if an empty set exists_iIs an abnormal log, otherwise, the log stream y to be tested is predicted_iThe log is a normal log;

2. the intelligent analysis method for intranet security threats according to claim 1, wherein a specific method for generating a log template set comprises the following steps:

input of log analysis: original log stream set X, log analysis algorithm set G:

A log analysis algorithm set G: comprising m log parsing algorithms f_k,k∈{1,2,...,m},G＝{f₁,...,f_m}；

Output of log parsing: comprising a set of m log templates T_k,k∈{1,2,...,m},T＝{T₁,...,T_m}; wherein each log template set T_kContaining q log templates t_kb，b∈{1,2,...,q},T_k＝{t_k1,...,t_kb}。

3. The intelligent analysis method for the intranet security threats according to claim 1, wherein a specific method for calculating the inconsistency scores of the log template and the log stream to be tested comprises the following steps:

input of inconsistency metric: a log template set T, a log stream set Y to be tested, and an inconsistency measurement function g:

log template set T: comprising a set of m log templates T_k,k∈{1,2,...,m},T＝{T₁,...,T_m}; wherein each log template set T_kContaining q log templates t_kb，b∈{1,2,...,q},T_k＝{t_k1,...,t_kb}；

Thirdly, an inconsistency measurement function g; the return value is a real number called the inconsistency score, and the input to the function is the log stream and the log template t_kbThe return value is a real number indicating the log stream to be tested and the log template t_kbInconsistency of (2);

output of inconsistency metric: a set of inconsistency scores.