CN111885059A - A method for detecting and locating anomaly in industrial network traffic - Google Patents

Abstract

The invention discloses an industrial network abnormality detection algorithm, which includes step 1: deploying switches at nodes of industrial network traffic exchange; step 2: reading traffic data through a network interface, and transmitting the traffic data to a protocol analysis algorithm for real-time layered protocol Analyze and extract network behavior features; step 3, process the missing features, process the data features into digital form, and select appropriate feature combinations from the data features for model training; step 4, establish a network behavior model for each protocol , to judge whether there is an abnormality; Step 5, use normal traffic data to establish a network behavior model for the protocol characteristics of each layer of the OSI network model, input the abnormal traffic into the network behavior model for further abnormal analysis, and output the abnormal location result of the traffic; Step 6. At regular intervals, update the training data of the network behavior model and replace the original model; the anomaly detection algorithm uses machine learning algorithms to discriminate, completes the detection of unknown anomalies, and solves the problem that traditional methods cannot identify new types of anomalies. disadvantages.

Description

A method for detecting and locating anomaly in industrial network traffic

technical field

The invention relates to the field of industrial network security, in particular to an abnormal detection and location method based on network traffic analysis.

Background technique

With the development of network technology, the Internet has begun to combine with the production of various industries, playing an increasingly important role. However, while the industrial Internet brings convenience to production, it also adds many security risks and threats. my country has suffered from serious and complex overseas network attacks, the security of industrial control systems has been threatened significantly, and the security incidents of malicious programs in accordance with the law have also appeared frequently.

The industrial network intrusion detection method is mainly aimed at the network composed of industrial control equipment, and it is an algorithm to detect behaviors that endanger the security of computer systems, such as collecting vulnerability information, causing access denial, and trying to obtain illegal system control rights.

Anomaly detection methods can be divided into two categories according to detection methods: one is misuse detection, and the other is anomaly detection. Misuse detection is a detection algorithm based on string matching of abnormal features, which is detected by comparing with existing abnormal features in the database. Misuse detection has a high accuracy rate and a low false positive rate. However, it is very labor-intensive to extract features from abnormal information and write them into the database in a specific form. As the size of the database increases, the detection time increases significantly, resulting in poor real-time performance.

The anomaly detection method abandons the database comparison method, establishes a normal behavior model for the characteristic values of network behavior traffic data, or combines the machine learning algorithms that have emerged in recent years to judge the traffic characteristics to be detected through model detection, which will obviously deviate from the normal model. The data is determined to be abnormal traffic. This method greatly improves the detection rate, but sometimes results in a certain degree of false alarm rate. At the same time, the anomaly detection method can detect new anomalies without manual maintenance of the feature pattern library.

However, the current detection methods all have shortcomings: the misuse detection method cannot detect new anomalies that do not exist in the signature database due to the detection method that relies on database comparison. Anomaly detection methods mostly use machine learning algorithms. Although they have a high detection accuracy, due to the lack of interpretability, the detection results can only report the occurrence of anomalies, but cannot accurately locate the location of the anomaly and the cause of the anomaly.

SUMMARY OF THE INVENTION

In order to solve the drawbacks of the existing industrial network intrusion detection methods, the present invention proposes a real-time anomaly detection and location method, which can quickly detect anomalies and locate the cause of the anomalies while the data stream is input.

The technical scheme of the present invention provides an industrial network anomaly detection algorithm, which includes the following steps:

Step 1. Deploy a switch at the node of industrial network traffic exchange, collect the data passing through the switch, and transfer it to the network interface of the server in a mirror image;

Step 2, read the traffic data through the network interface, pass the traffic data to the protocol analysis algorithm for real-time layered protocol analysis, and extract network behavior characteristics;

The protocol parsing algorithm includes parsing the traffic data according to the TCP/IP five-layer model, and extracting network behavior characteristics from the data link layer, network layer and transport layer according to the protocol;

When extracting network behavior features, extract public behavior feature keywords and representative behavior feature keywords in various formats for data in different formats of a certain protocol at the application layer, and use public keywords and representative keywords as certain Keyword of the agreement;

Step 3. Process the situation of missing features, process the data features into digital form, and select a suitable feature combination from the data features for model training;

Step 4. Establish a network behavior model for each protocol to determine whether there is an abnormality; perform alarm processing for the traffic whose abnormality score is greater than the threshold;

Step 5, using normal traffic data to establish a network behavior model for the protocol characteristics of each layer of the OSI network model, input abnormal traffic into the network behavior model for further abnormal analysis, and output the abnormal location result of the traffic;

Step 6. At regular intervals, update the training data of the network behavior model and replace the original model;

Set the time window t, and in each time window, pass the data through the foreground detection module, the background training module, and the positioning module in turn, generate a detection report, and complete the training of the detection sub-model and the positioning sub-model corresponding to the new time window, replacing the original one. There are network behavior models.

Further, in step 3, for the COTP protocol, if the 5th bit of the payload data is not 2, the purpose reference is the number represented by the 7-8 bits of the payload data, otherwise there is no purpose to reference this field, so it is marked as 0;

For the URL field of the HTTP protocol, convert it to ASCII code and handle it as a numeric field.

Further, in step 4, the IForest algorithm is used to establish a network behavior model, the subsequent traffic is detected, and the anomaly score formula of the feature groups in the isolated forest for the feature values of all the network behavior traffic eigenvalues to be detected is calculated as follows:

where E(h(x)) represents the mean path length of the data feature in the isolated forest, ψ represents the number of training samples of a single isolated tree, and C(ψ) represents the binary isolation tree constructed with ψ pieces of data. The average path length, C(ψ), is calculated as follows:

H(ψ-1)=ln(ψ-1)+Euler, where Euler is Euler's constant.

Further, step 5 specifically includes:

Step 5.1. Determine whether the same type of traffic data has the same network behavior characteristic field. When the value of the field fluctuates widely, it is considered that the field is abnormal, and the coefficient of variation is used to measure the abnormal degree of the traffic data:

x_i是第i个字段的数据，μ为某字段的平均值，CV为标准差与均值的比值；in,

x _i is the data of the ith field, μ is the average value of a field, and CV is the ratio of the standard deviation to the average value;

Step 5.2. Construct the data processing matrix, and use the normal data model of the first 10 minutes as the input matrix A of PCA:

Among them, m is the number of historical data, n is the number of fields, and the column represents the data of the field;

Step 5.3, standardize the original input matrix, and use the standardized matrix to obtain eigenvalues and eigenvectors;

Since the value range of each field is different, the original input matrix A needs to be standardized and transformed to obtain the standardized matrix Z;

in,

Next, find the covariance matrix Σ of matrix Z:

cov(Z_i,Z_j)＝E((Z_i-E(Z_i))(Z_j-E(Z_j)))；最后，求得协方差矩阵的特征值λ₁,λ₂,…,λ_n和特征向量μ₁,μ₂,…,μ_n；in,

cov(Z _i ,Z _j )=E((Z _i -E(Z _i ))(Z _j -E(Z _j ))); finally, find the eigenvalues of the covariance matrix λ ₁ ,λ ₂ ,... ,λ _n and eigenvectors μ ₁ , μ ₂ ,…,μ _n ;

β为设定的阈值；Step 5.4, sort the eigenvalues λ ₁ , λ ₂ ,...,λ _n in descending order, and then calculate the variance percentage of the principal components, so that

β is the set threshold;

Step 5.5, sort the selected principal components, select the principal component with the largest amount of information, and use the field represented by it as the root cause field candidate set;

Step 5.6. Define the root cause field candidate set of k elements as the current abnormal pattern, and compare it with the known abnormal pattern; if there is an abnormal pattern of the same type, report the abnormal pattern and its detailed information; otherwise, update the abnormal pattern Pattern library and report exceptions.

The beneficial effects of the present invention are:

(1) High efficiency: By separating the training and detection of the model, that is, after training the detection algorithm offline, the model is used for online traffic detection to achieve efficient anomaly detection. It can not only detect static data, but also ensure the detection of data flow.

(2) High accuracy rate: The anomaly detection algorithm uses machine learning algorithms to discriminate, completes the detection of unknown anomalies, and solves the drawback that traditional methods cannot identify new types of anomalies.

(3) Interpretability: After the abnormality detection is completed, the abnormal data will be further analyzed to determine the cause of the abnormality and generate a detection report. Through the automatic abnormal location, the location and cause of the abnormality can be clearly seen, and there is no need to allocate manpower to locate the abnormality warning.

(4) Automatic update: In order to adapt to the detection of streaming data, the detection algorithm can be automatically updated with the change of real-time data characteristics, and the updated model will replace the original detection model to detect anomalies in subsequent data.

Description of drawings

Fig. 1 is the system data flow diagram of this method;

Figure 2 is a flowchart of the method for sub-protocol and layered detection of industrial control protocols.

Detailed ways

All features disclosed in this specification, or steps in all methods or processes disclosed, may be combined in any way except mutually exclusive features and/or steps. Any feature disclosed in this specification, unless expressly stated otherwise, may be replaced by other equivalent or alternative features serving a similar purpose. That is, unless expressly stated otherwise, each feature is but one example of a series of equivalent or similar features.

The preferred embodiments of the invention will be described in further detail below.

As shown in FIG. 1 , this embodiment provides an industrial network anomaly detection algorithm of an embodiment, including the following steps:

Step 1. Deploy a switch at a node where industrial network traffic is exchanged, collect data passing through the switch, mirror it and transmit it to the network interface of the server.

As shown in Figure 1, the collected data includes the flow data exchanged between each PLC device and the flow data transmitted by the host computer to the PLC device.

Step 2: Read the traffic data through the network interface, and transmit the traffic data to the protocol analysis algorithm for real-time layered protocol analysis, and extract network behavior characteristics.

Among them, as shown in Figure 1, the protocol parsing algorithm includes parsing the traffic data according to the TCP/IP five-layer model, and extracting network behavior characteristics from the data link layer, network layer, and transport layer according to the protocol.

When extracting network behavior characteristics, due to the complexity of the application layer protocol format, the public network behavior keywords and the representative network behavior keywords of various formats are extracted for data in different formats of a certain protocol in the application layer, and the public key words and representative keywords as keywords for a certain protocol.

For example, for all traffic data of the HTTP protocol, the public keywords include the source MAC address and destination MAC address of the link layer, the source IP address and destination IP address of the network layer, and the source port, destination port, and window size of the transport layer. . Since the traffic data of the HTTP protocol is divided into request packets and response packets, it is necessary to extract representative keywords from the two types of packets. The representative keywords of the HTTP protocol include HTTP message types, status codes, methods, and URLs.

Step 3: Process the missing feature, process the data feature into a digital form, and select an appropriate feature combination from the data feature for model training.

For example: for COTP protocol, if the 5th bit of the payload data is not 2, the destination reference is the number represented by the 7-8 bits of the payload data, otherwise there is no destination reference to this field, so it is marked as 0. For the URL field of the HTTP protocol, this is a string-type field. In order to facilitate the input of the algorithm, it is converted into ASCII code and processed as a number-type field. However, due to the large number of fields, in order to improve the efficiency of the algorithm, several main fields can be extracted as input, such as discarding one of the source IP and source MAC fields with the same information, or using PCA, tSNE and other data dimension reduction algorithms to choose.

As shown in Figure 1, the processing data includes filling in the case where the value of some fields does not exist, according to the specific distribution of the field value, such as filling 0 or -1, etc.; hashing or encoding the character string in the keyword, etc. and other digitization operations, so that all eigenvalues are represented in digital form.

Step 4: Establish a network behavior model for each protocol to determine whether there is an abnormality; perform alarm processing for traffic whose abnormality score is greater than a threshold.

As shown in Figure 2, the IForest algorithm is used to establish a network behavior model to detect subsequent traffic.

The detection method is as follows: Calculate the anomaly score of the feature group in the isolated forest for all the feature values of the network behavior traffic to be detected. The formula is as follows:

where E(h(x)) represents the mean path length of the data feature in the isolated forest, ψ represents the number of training samples of a single isolated tree, and C(ψ) represents the binary isolation tree constructed with ψ pieces of data. The average path length, C(ψ), is calculated as follows:

H(ψ-1)=ln(ψ-1)+Euler, where Euler is Euler's constant.

For example, for the traffic data of the HTTP protocol, select historical sample data, select the source MAC, destination MAC, source port, destination port, TCP flags, HTTP protocol type, status code and other fields as the input of the IForest algorithm. As a result, the threshold of the abnormal score is preset and applied to the detection of subsequent stream data.

Step 5: Use normal traffic data to establish a network behavior model for the protocol features of each layer of the OSI network model, input abnormal traffic into the network behavior model for further abnormal analysis, and output the abnormal location result of the traffic.

Step 5.1. Determine whether the same type of traffic data has the same network behavior characteristic fields, such as HTTP data packets, Modbus data packets, etc. The value of each field is relatively stable. If the value of the field fluctuates widely, it is considered that the field is abnormal. The coefficient of variation is used to measure the abnormal degree of traffic data.

x_i是第i个字段的数据，μ为某字段的平均值，CV为标准差与均值的比值。in,

x _i is the data of the ith field, μ is the average value of a field, and CV is the ratio of the standard deviation to the mean.

If the coefficient of variation CV is large, it indicates that the traffic packet is abnormal in this time period, and further positioning analysis is performed. In multivariate statistical analysis, when there are more than 10 variables, most of the discarded dimensions are redundant with other dimensions, so principal component analysis (PCA) is used to reduce the n-dimensional data into p principal components (p<n ), to represent the original data information, and at the same time, it can effectively reduce the data dimension and narrow the scope of the problem.

Step 5.2. Construct the data processing matrix, and use the normal data model of the first 10 minutes as the input matrix A of PCA:

Among them, m is the number of historical data, n is the number of fields, and the column represents the data of the field.

Step 5.3: Standardize the original input matrix, and use the standardized matrix to obtain eigenvalues and eigenvectors.

Since the value range of each field is different, the original input matrix A needs to be standardized and transformed to obtain the standardized matrix Z.

接着，求矩阵Z的协方差矩阵Σ：in,

Next, find the covariance matrix Σ of matrix Z:

cov(Z_i,Z_j)＝E((Z_i-E(Z_i))(Z_j-E(Z_j)))。最后，求得协方差矩阵的特征值λ₁,λ₂,…,λ_n和特征向量μ₁,μ₂,…,μ_n。in,

cov(Z _i , Z _j )=E((Z _i -E(Z _i ))(Z _j -E(Z _j ))). Finally, the eigenvalues λ ₁ , λ ₂ ,…,λ _n of the covariance matrix and the eigenvectors μ ₁ , μ ₂ ,…,μ _n are obtained.

β为设定的阈值。Step 5.4, sort the eigenvalues λ ₁ , λ ₂ ,...,λ _n in descending order, and then calculate the variance percentage of the principal components, so that

β is the set threshold.

The principal component is actually a linear combination of the original dimensions, and the coefficient vector is the corresponding feature vector. The coefficient represents the correlation between the principal component and the original data dimension. The larger the coefficient, the greater the contribution of the dimension to the principal component, that is, the method corresponding to the dimension is the main field that causes anomalies.

Step 5.5, sort the selected principal components, select the principal component with the largest amount of information, and take the field represented by it as the root cause field candidate set.

The anomaly localization algorithm first selects k principal components p ₁ , p ₂ ,...,p _n , and sorts them according to the corresponding eigenvalues λ ₁ , λ ₂ ,..., λ _k in descending order. The more significant; then, select the maximum coefficient of the principal component and the corresponding field in turn, and calculate the weight factor of the method to obtain k fields and weight factors; secondly, sort the selected k fields in reverse order according to the weight factors, as the abnormal location. Root cause field.

Step 5.6: Define the root cause field candidate set of k elements as the current abnormal pattern, and compare it with the known abnormal pattern. If an abnormal pattern of the same type already exists, report the abnormal pattern and its detailed information; otherwise, update the abnormal pattern library and report the exception.

Step 6. Update the training data of the network behavior model and replace the original model at regular intervals.

Set the time window t, and in each time window, pass the data through the foreground detection module, the background training module, and the positioning module in turn, generate a detection report, and complete the training of the detection sub-model and the positioning sub-model corresponding to the new time window, replacing the original one. There are network behavior models.

Although the present invention has been disclosed in detail with reference to the accompanying drawings, it is to be understood that these descriptions are exemplary only and are not intended to limit the application of the present invention. The protection scope of the present invention is defined by the appended claims, and may include various modifications, alterations and equivalent solutions made to the invention without departing from the protection scope and spirit of the present invention.

Claims (4)

1. An industrial network anomaly detection algorithm, comprising the steps of:

step 1, deploying a switch at a node of industrial network flow exchange, collecting data passing through the switch and transmitting the data to a network interface of a server in a mirror image manner;

step 2, reading the flow data through a network interface, transmitting the flow data to a protocol analysis algorithm to analyze a real-time layered protocol, and extracting network behavior characteristics;

the protocol analysis algorithm comprises analyzing the flow data according to a TCP/IP five-layer model, and extracting network behavior characteristics from a data link layer, a network layer and a transmission layer according to a protocol;

when network behavior characteristics are extracted, common behavior characteristic keywords and representative behavior characteristic keywords in various formats are extracted from data in different formats of a certain protocol of an application layer, and the common keywords and the representative keywords are used as keywords of the certain protocol;

step 3, processing the condition of feature missing, processing the data features into a digital form, and selecting a proper feature combination from the data features for model training;

step 4, establishing a network behavior model for each protocol to judge whether the abnormality occurs; alarming the flow with the abnormal score larger than the threshold value;

step 5, establishing a network behavior model for each layer of protocol characteristics of the OSI network model by using normal traffic data, inputting abnormal traffic into the network behavior model for further abnormal analysis, and outputting abnormal positioning results of the traffic;

step 6, updating training data of the network behavior model at intervals and replacing the original model;

and setting time windows t, sequentially passing the data through a foreground detection module, a background training module and a positioning module in each time window, generating a detection report, finishing the training of a detection sub-model and a positioning sub-model corresponding to the new time window, and replacing the original network behavior model.

2. The industrial network anomaly detection algorithm of claim 1, wherein:

in step 3, for the COTP protocol, if the 5 th bit of the payload data is not 2, the destination reference is a number represented by the 7-8 bits of the payload data, otherwise, no destination reference is made to this field, and therefore, the number is marked as 0;

for the URL field of the HTTP protocol, it is converted into ASCII code and processed as a digital type field.

3. The industrial network anomaly detection algorithm of claim 1, wherein:

in step 4, a network behavior model is established by using an IForest algorithm, subsequent flow is detected, and an abnormal component formula of a characteristic group of all to-be-detected network behavior flow characteristic values in an soliton forest is calculated as follows:

wherein E (h (x)) represents the mean of the path lengths of the data features in the solitary forest, ψ represents the number of samples of training samples of a single solitary tree, C (ψ) represents the mean path length of a binary solitary tree constructed with ψ pieces of data, and the calculation formula of C (ψ) is as follows:

h (ψ -1) ═ ln (ψ -1) + Euler, Euler being the Euler constant.

4. The industrial network anomaly detection algorithm of claim 1, wherein: the step 5 specifically comprises the following steps:

step 5.1, judging whether the same-class flow data has the same network behavior characteristic field, when the field value fluctuates in a large range, considering that the field is abnormal, and measuring the abnormal degree of the flow data by adopting a variation coefficient:

wherein,

x_iis the data of the ith field, mu is the average value of a certain field, and CV is the ratio of the standard deviation to the average value;

step 5.2, constructing a data processing matrix, and taking the normal data model of the previous 10 minutes as an input matrix A of the PCA:

wherein m is the number of historical data, n is the number of fields, and the columns represent the data of the fields;

step 5.3, carrying out standardization processing on the original input matrix, and obtaining eigenvalues and eigenvectors by using the standardized matrix;

because the value ranges of all fields are different, the original input matrix A needs to be subjected to standardized conversion to obtain a standardized matrix Z;

wherein,

next, a covariance matrix Σ of the matrix Z is obtained:

wherein,

finally, the eigenvalue lambda of the covariance matrix is solved₁,λ₂,…,λ_nAnd a feature vector mu₁,μ₂,…,μ_n；

Step 5.4, the characteristic value lambda is subjected to₁,λ₂,…,λ_nSorted in descending order, and then the principal component variance percentages are calculated such that

Beta is a set threshold value;

step 5.5, sorting the selected principal components, selecting the principal component with the largest information amount, and taking the field represented by the principal component as a root cause field candidate set;

step 5.6, defining the root cause field candidate set of k elements as a current abnormal mode, and comparing the current abnormal mode with a known abnormal mode; if the same type of abnormal mode exists, reporting the abnormal mode and detailed information thereof; otherwise, updating the abnormal mode library and reporting the abnormality.

Images