CN110825545A - Anomaly detection method and system for cloud service platform - Google Patents

Abstract

The invention discloses a cloud service platform anomaly detection method and system. The method includes the following steps: 1) real-time collection of system metric data when a cloud platform host is working normally, and calculating a system operating environment vector according to the system metric data; 2) Using the normal system operating environment vector, combined with the maximum average deviation algorithm MMD and the support vector data description algorithm SVDD to train to obtain an anomaly detection model; 3) When receiving new host system metric data, use the hypersphere of the cluster where the host is located. Host system metrics are classified to detect host anomalies. The invention combines MMD and SVDD algorithm training to obtain an abnormality detection model, effectively solves the problem of extreme imbalance between normal and abnormal samples in the cloud service platform, enables it to detect unknown system abnormalities in the cloud service platform, and no longer needs An anomaly detection model is built for each host, which greatly reduces the time and system resource consumption of anomaly modeling.

Description

Anomaly detection method and system for cloud service platform

technical field

The invention relates to cloud computing security technology, in particular to a cloud service platform anomaly detection method and system.

Background technique

The cloud service platform is an open public platform that provides various application services for a large number of users. The reliability of these application services is critical to their consumers. The presence of anomalies in a cloud service platform can bring its reliability into question. Due to the scale and complexity, cloud service platforms will generate a large number of system exceptions, which are mainly caused by cloud platform administrators' operational errors, over/under-provisioning of resources, hardware/software failures, and network attacks. Therefore, it is of great significance to perform real-time anomaly detection on the system running state of the cloud service platform.

The basic principle of anomaly detection is to take the behavior of the system, user, process or network as the corresponding contour model on the basis of system monitoring. When the system running state deviates from the normal contour model, it can be judged as abnormal. At present, there are related anomaly detection methods and their corresponding anomaly detection systems. It is mainly divided into statistical-based anomaly detection algorithms and machine learning-based anomaly detection algorithms.

The statistics-based anomaly detection method firstly mines the characteristics of the performance data by using the statistical learning method, and then calculates the anomaly score of the sample data based on the distribution characteristics of the data, and issues an alarm message if it exceeds the specified threshold. This approach typically requires knowledge of the time-series distribution of system performance data for cloud service platform hosts, or may not be well suited for expanding clusters.

Machine learning-based methods first need to learn and model from a large number of data samples, and then can detect new data samples to determine whether they are abnormal. Hyunjoo Kim et al. proposed a machine learning-based approach to cyber threat detection, which first uses random forests to select salient features, generates clusters by applying K-means and DBSCAN to unlabeled data collected from cloud platforms, and uses new -Kyoto-2006+ dataset to label clustering results. Elham Besharati et al. used a combination of three different classifiers: neural network, decision tree and linear discrimination to identify various attacks. This method has high accuracy, but this method requires building a neural network and a neural network for each host. Decision tree model, the cost is high.

The existing technical difficulties mainly include: (1) the cloud service platform is in normal operation most of the time, and the abnormal data samples are far less than the normal data samples; (2) the methods based on machine learning usually require Building an anomaly detection model, and the training process of anomaly detection model requires a lot of time and system resources, so these methods are difficult to apply to large-scale cloud service platforms.

SUMMARY OF THE INVENTION

The technical problem to be solved by the present invention is to provide a cloud service platform anomaly detection method and system aiming at the defects in the prior art.

The technical solution adopted by the present invention to solve the technical problem is: a cloud service platform abnormality detection method, comprising the following steps:

1) Collect real-time system metric data when the cloud platform host is working normally, and calculate the system operating environment vector according to the system metric data;

2) Using the normal system operating environment vector, combined with the maximum average deviation algorithm MMD and the support vector data description algorithm SVDD (support vector domain description) training to obtain the anomaly detection model;

In the step 2), the process of obtaining the anomaly detection model in combination with the training of the MMD and SVDD algorithms is as follows:

Step 2.1) use MMD to cluster the normal operating environment vector of the system host, thereby dividing the cloud platform host into multiple clusters according to the similarity of the system operating environment;

Step 2.2) According to the normal system operating environment vector, use SVDD to construct a hypersphere for each host cluster, which is used to describe the data distribution when each host is running normally, so as to detect abnormal data;

3) When receiving new host system metric data, use the hypersphere of the cluster where the host is located to classify the host system metric data, so as to detect host abnormality.

According to the above scheme, the step 3) is specifically:

Step 3.1) when receiving the new host system measurement data, calculate the system operating environment vector according to the system measurement data;

Step 3.2) According to the cluster where the host is located, select the hypersphere corresponding to the cluster as the anomaly detection model;

Step 3.3) When the host system operating environment vector falls on the hypersphere or on the hypersphere, it is determined that the host is currently in a normal state;

Step 3.4) When the operating environment vector of the host system falls outside the hypersphere, it is determined that the host is currently in an abnormal state.

According to the above scheme, the construction process of the hypersphere in the step 2.2) is as follows:

2.2.1) Considering the operating environment vector set, where N is the number of hosts in the cluster, the construction process of the hypersphere can be expressed by the following formula:

At the same time, the above formula satisfies the following constraints:

( _xi -a) ^T ( _xi -a)≤R ² +ξ _i ,ξ _i ≥0

Among them, R is the radius of the hypersphere, a is the center of the hypersphere, the above formula shows that if the data point is inside or on the surface of the hypersphere, ξ _i = 0, otherwise ξ _i > 0, C is a constant, if C takes If C is very large, when the model is trained, it will try to find a larger circle to include as many points as possible. If C is relatively small, it will prefer to find a small circle;

2.2.2) Set the size of the constant C, and ignore the sample points whose distance from the origin exceeds the set value during the construction process;

2.2.3) Through continuous fitting, the radius of the hypersphere and the center of the hypersphere are finally calculated. The fitting process can be expressed by the following formula:

According to the above solution, the system metric data in step 1) includes CPU, memory, disk, and network-related system metric data.

A cloud service platform anomaly detection system, comprising:

The acquisition module is used to collect real-time system metric data when the cloud platform host is working normally, and calculate the system operating environment vector according to the system metric data;

The model building module is used to use the normal system operating environment vector, combined with the maximum mean deviation algorithm MMD and the support vector data description algorithm SVDD (support vector domain description) training to obtain an anomaly detection model;

The process of obtaining the anomaly detection model by combining the training of the MMD and SVDD algorithms in the model building module is as follows:

1) Use MMD to cluster the normal operating environment vector of the system host, thereby dividing the cloud platform host into multiple clusters according to the similarity of the system operating environment;

2) According to the normal system operating environment vector, use SVDD to construct a hypersphere for each host cluster, which is used to describe the data distribution of each host during normal operation, so as to detect abnormal data;

The detection module is used for classifying the metric data of the host system by using the hypersphere of the cluster where the host is located, so as to detect the abnormality of the host when receiving new host system metric data.

According to the above scheme, the detection module is specifically:

1) When receiving the new host system metric data, calculate the system operating environment vector according to the system metric data;

2) According to the cluster where the host is located, select the hypersphere corresponding to the cluster as the anomaly detection model;

3) When the operating environment vector of the host system falls on the hypersphere or on the hypersphere, it is determined that the host is currently in a normal state;

4) When the operating environment vector of the host system falls outside the hypersphere, it is determined that the host is currently in an abnormal state.

According to the above scheme, the construction process of the hypersphere in the model building module is as follows:

1) Considering the operating environment vector set, where N is the number of hosts in the cluster, the construction process of the hypersphere can be expressed by the following formula:

At the same time, the above formula satisfies the following constraints:

( _xi -a) ^T ( _xi -a)≤R ² +ξ _i ,ξ _i ≥0

Among them, R is the radius of the hypersphere, a is the center of the hypersphere, the above formula shows that if the data point is inside or on the surface of the hypersphere, ξ _i = 0, otherwise ξ _i > 0, C is a constant, if C takes If the value is very large, the model will try to find a larger circle to include as many points as possible during training. If C is relatively small, it will prefer to find a small circle.

2) Set the size of the constant C, and ignore the sample points whose distance from the origin exceeds the set value during the construction process;

3) Through continuous fitting, the radius of the hypersphere and the center of the hypersphere are finally calculated. The fitting process can be expressed by the following formula:

According to the above solution, the system metric data in the collection module includes CPU, memory, disk, and network-related system metric data.

The beneficial effects that the present invention produces are:

1. Combine MMD and SVDD algorithm training to obtain anomaly detection model, which effectively solves the problem of extreme imbalance between normal and abnormal samples in the cloud service platform, enabling it to detect unknown system anomalies in the cloud service platform. An anomaly detection model is built for each host, which greatly reduces the time and system resource consumption of anomaly modeling.

2. Through the built model, classify the host system measurement data, realize the online abnormal detection of the cloud service platform host, timely discover the abnormal behavior of the system in the cloud service platform, and improve the reliability of the cloud service platform.

Description of drawings

The present invention will be further described below in conjunction with the accompanying drawings and embodiments, in which:

Fig. 1 is the method flow chart of the embodiment of the present invention;

Fig. 2 is a pseudo-code schematic diagram of a clustering process according to an embodiment of the present invention;

FIG. 3 is a flowchart of a construction process of a hypersphere according to an embodiment of the present invention.

Detailed ways

In order to make the objectives, technical solutions and advantages of the present invention clearer, the present invention will be further described in detail below with reference to the embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention, but not to limit the present invention.

As shown in Figure 1, a cloud service platform anomaly detection method, the method includes:

Step 1: Collect system metric data related to the CPU/memory/disk/network of the cloud platform host in real time, and calculate the system operating environment vector according to the system metric data. The calculation process is as follows:

1) Consider CPU measurement data, through CPU system time (SYS), CPU user time (USER), CPU disk IO wait time (IO_WAIT), CPU hard interrupt event (IRQ), CPU soft interrupt event (SOFT_IRQ), CPU idle time (IDLE) Calculate the usage rate of the CPU Usage _cpu , the calculation formula is as follows:

2) Considering the memory measurement data, the memory usage is calculated by the total memory size (TOTAL), the actual memory size (ACTUAL), and the memory size in the cache (CACHE+BUFFERS). The calculation formula is as follows:

3) Considering the disk metric data, the disk IO frequency is calculated by the number of disk reads (READ_COUNT), the number of disk writes (WRITE_COUNT) and the maximum number of disk IOs (MAX_IO_COUNT). The calculation formula is as follows:

4) Considering the network measurement data, the network load is calculated by the network inbound traffic size (IN_SIEZ), the network outbound traffic size (OUT_SIEZ) and the network bandwidth (MAX_SIZE). The calculation formula is as follows:

5) Obtain the system operating environment vector according to the above calculation results, and the system operating environment vector is expressed as follows:

RE=(Usage _cpu ,Usage _mem ,Freq _disk ,Load _net )

Step 2: Using the normal system operating environment vector, combined with MMD and SVDD algorithm training to obtain anomaly detection model, so that it can not only detect unknown anomalies in the absence of anomalous samples, but also detect multiple computers with similar operating environments at the same time. host exceptions, thereby greatly reducing modeling time and system resource consumption;

The training process combining the two algorithms of MMD and SVDD is as follows:

Step 2.1: Use MMD to cluster the operating environment vectors of the system hosts, and divide the hosts with similar operating environments into one cluster, while hosts in different operating environment clusters have large differences in their operating environments, and the clustering process The pseudo code is shown in Figure 2, and the clustering process is as follows:

1) According to the principle of minimum distance, select the point closest to the origin from the system operating environment vector set as the first cluster center, and use the Euclidean distance to calculate the distance;

2) Select the point farthest from the first point from the system operating environment vector set as the second cluster center;

3) According to the principle of minimum distance, the sample points are divided into the nearest cluster centers, and the cluster centers are updated. If the distance between the sample points and all the cluster centers is greater than the set threshold, the sample point is regarded as a new one. cluster center;

4) Repeat process 3) until all sample points are divided.

Step 2.2: Use the SVDD algorithm to train the data samples in each cluster, and construct a hypersphere for each cluster. These hyperspheres describe the data distribution of the operating environment of the hosts in each cluster under normal operating conditions. The construction process is shown in Figure 3, and the construction process is described as follows:

1) Considering the operating environment vector set, where N is the number of hosts in the cluster, the construction process of the hypersphere can be expressed by the following formula:

At the same time, the above formula satisfies the following constraints:

( _xi -a) ^T ( _xi -a)≤R ² +ξ _i ,ξ _i ≥0

Among them, R is the radius of the hypersphere, a is the center of the hypersphere, the above formula shows that if the data point is inside or on the surface of the hypersphere, ξ _i = 0, otherwise ξ _i > 0, C is a constant, if C takes If the value is very large, the model will try to find a larger circle to include as many points as possible during training. If C is relatively small, it will prefer to find a small circle.

2) Set the size of the constant C, and ignore the sample points whose distance from the origin exceeds the set value during the construction process;

3) Through continuous fitting, the radius of the hypersphere and the center of the hypersphere are finally calculated. The fitting process can be expressed by the following formula:

Step 3: When receiving the new host system metric data, use the hypersphere of the host cluster to classify the host system metric data. The classification formula is as follows:

where Ω represents a hypersphere. When the operating environment vector of the host system falls within the hypersphere or on the hypersphere, it is determined that the host is currently in a normal state; when the operating environment vector of the host system falls outside the hypersphere, it is determined that the host is currently in an abnormal state, and an alarm is sent by email.

A cloud platform anomaly detection method proposed in this implementation has the following beneficial effects:

1) Combine MMD and SVDD algorithm training to obtain anomaly detection model, which effectively solves the problem of extreme imbalance between normal and abnormal samples in the cloud service platform, enabling it to detect unknown system anomalies in the cloud service platform. An anomaly detection model is built for each host, which greatly reduces the time and system resource consumption of anomaly modeling.

2) Through the constructed model, the host system measurement data is classified, the online abnormal detection of the cloud service platform host is realized, the abnormal behavior of the system in the cloud service platform is detected in time, and the reliability of the cloud service platform is improved.

The present invention further provides a cloud service platform abnormality detection system. The cloud service platform abnormality detection system proposed by the present invention includes a collection module, a communication module, a modeling module and a detection module:

The collection module runs in the cloud platform host, collects the CPU/memory/disk/network related system resource measurement data of the cloud service platform host in real time, and submits the collected data to the client sub-module of the communication module.

After the client sub-module of the communication module obtains the data submitted by the acquisition module, it encapsulates the data, adds the local Mac address to the data packet header to distinguish the measurement data of different host systems, and pushes the encapsulated data packet to the Kafka message in the SYS_METRICS topic of the queue.

The communication module service terminal module regularly pulls and parses data from the SYS_METRICS topic of the Kafka message queue, and saves the parsed data to the database in a certain format.

The modeling module analyzes and models the system metric data under normal conditions (that is, has not been attacked, and generates an anomaly detection model). The specific steps are:

Step 1: Extract incremental data from the HBase database, and calculate the system operating environment vector according to the system measurement data;

Step 2: Using the normal system operating environment vector, combined with MMD and SVDD algorithm training to obtain anomaly detection model, so that it can not only detect unknown anomalies in the absence of anomalous samples, but also detect multiple computers with similar operating environments at the same time. The exception of the host, thereby greatly reducing the modeling time and system resource consumption, the modeling process is as follows:

Step 2.1: Use MMD to cluster all the calculated system operating environment vectors, so that the cloud service platform hosts are divided into multiple clusters according to the similarity of the system operating environment;

Step 2.2: According to the normal operating environment vector of the system, use SVDD to construct a hypersphere for each host cluster, which is used to describe the data distribution when the host is running normally.

The detection module loads the hyperspheres corresponding to all host clusters. When receiving new host system metric data, first calculates the host system operating environment vector according to the host system metric data; The hypersphere classifies the system metric data; when the host system operating environment vector falls within the hypersphere or on the hypersphere, it is determined that the host is currently in a normal state; when the host system operating environment vector falls outside the hypersphere, it is determined that the host is currently in a normal state. It is in an abnormal state and will be alerted by email.

It should be understood that, for those skilled in the art, improvements or changes can be made according to the above description, and all these improvements and changes should fall within the protection scope of the appended claims of the present invention.

Claims (8)

1. a cloud service platform anomaly detection method, is characterized in that, comprises the following steps: 1) Collect real-time system metric data when the cloud platform host is working normally, and calculate the system operating environment vector according to the normal and abnormal system metric data; 2) Using the normal system operating environment vector, combined with the maximum average deviation algorithm MMD and the support vector data description algorithm SVDD training to obtain anomaly detection model; In the step 2), the process of obtaining the anomaly detection model in combination with the training of the MMD and SVDD algorithms is as follows: Step 2.1) use MMD to cluster the normal operating environment vector of the system host, thereby dividing the cloud platform host into multiple clusters according to the similarity of the system operating environment; Step 2.2) According to the normal system operating environment vector, use SVDD to construct a hypersphere for each host cluster, which is used to describe the data distribution when each host is running normally, so as to detect abnormal data; 3) When receiving new host system metric data, use the hypersphere of the cluster where the host is located to classify the host system metric data, so as to detect host abnormality. 2. cloud service platform anomaly detection method according to claim 1, is characterized in that, described step 3) is specifically: Step 3.1) when receiving the new host system measurement data, calculate the system operating environment vector according to the system measurement data; Step 3.2) According to the cluster where the host is located, select the hypersphere corresponding to the cluster as the anomaly detection model; Step 3.3) When the host system operating environment vector falls on the hypersphere or on the hypersphere, it is determined that the host is currently in a normal state; Step 3.4) When the operating environment vector of the host system falls outside the hypersphere, it is determined that the host is currently in an abnormal state. 3. cloud service platform anomaly detection method according to claim 1, is characterized in that, the construction process of hypersphere in described step 2.2) is as follows: 2.2.1) Considering the operating environment vector set, where N is the number of hosts in the cluster, the construction process of the hypersphere can be expressed by the following formula:

At the same time, the above formula satisfies the following constraints: ( _xi -a) ^T ( _xi -a)≤R ² +ξ _i ,ξ _i ≥0 Among them, R is the radius of the hypersphere, a is the center of the hypersphere, the above formula shows that if the data point is inside or on the surface of the hypersphere, ξ _i = 0, otherwise ξ _i > 0, C is a constant; 2.2.2) Set the size of the constant C, and ignore the sample points whose distance from the origin exceeds the set value during the construction process; 2.2.3) Through continuous fitting, the radius of the hypersphere and the center of the hypersphere are finally calculated. The fitting process can be expressed by the following formula:

4 . The cloud service platform anomaly detection method according to claim 1 , wherein the system metric data in the step 1) includes CPU, memory, disk, and network-related system metric data. 5 . 5. A cloud service platform anomaly detection system, characterized in that, comprising: The acquisition module is used to collect real-time system metric data when the cloud platform host is working normally, and calculate the system operating environment vector according to the system metric data; The model building module is used to use the normal system operating environment vector, combined with the maximum average deviation algorithm MMD and the support vector data description algorithm SVDD training to obtain anomaly detection model; The process of obtaining the anomaly detection model by combining the training of the MMD and SVDD algorithms in the model building module is as follows: 1) Use MMD to cluster the normal operating environment vector of the system host, thereby dividing the cloud platform host into multiple clusters according to the similarity of the system operating environment; 2) According to the normal system operating environment vector, use SVDD to construct a hypersphere for each host cluster, which is used to describe the data distribution of each host during normal operation, so as to detect abnormal data; The detection module is used for classifying the metric data of the host system by using the hypersphere of the cluster where the host is located, so as to detect the abnormality of the host when receiving new host system metric data. 6. The cloud service platform anomaly detection system according to claim 5, wherein the detection module is specifically: 1) When receiving the new host system metric data, calculate the system operating environment vector according to the system metric data; 2) According to the cluster where the host is located, select the hypersphere corresponding to the cluster as the anomaly detection model; 3) When the operating environment vector of the host system falls on the hypersphere or on the hypersphere, it is determined that the host is currently in a normal state; 4) When the operating environment vector of the host system falls outside the hypersphere, it is determined that the host is currently in an abnormal state. 7. cloud service platform anomaly detection system according to claim 5, is characterized in that, the construction process of hypersphere in described model building module is as follows: 1) Considering the operating environment vector set, where N is the number of hosts in the cluster, the construction process of the hypersphere is expressed by the following formula:

At the same time, the above formula satisfies the following constraints: ( _xi -a) ^T ( _xi -a)≤R ² +ξ _i ,ξ _i ≥0 Among them, R is the radius of the hypersphere, a is the center of the hypersphere, the above formula shows that if the data point is inside or on the surface of the hypersphere, ξ _i = 0, otherwise ξ _i > 0, C is a constant; 2) Set the size of the constant C, and ignore the sample points whose distance from the origin exceeds the set value during the construction process; 3) Through continuous fitting, the radius of the hypersphere and the center of the hypersphere are finally calculated, and the fitting process is expressed by the following formula:

8 . The cloud service platform anomaly detection system according to claim 5 , wherein the system metric data in the collection module includes CPU, memory, disk, and network-related system metric data. 9 .

Images