CN110825545A - Cloud service platform anomaly detection method and system - Google Patents
Cloud service platform anomaly detection method and system Download PDFInfo
- Publication number
- CN110825545A CN110825545A CN201910820118.1A CN201910820118A CN110825545A CN 110825545 A CN110825545 A CN 110825545A CN 201910820118 A CN201910820118 A CN 201910820118A CN 110825545 A CN110825545 A CN 110825545A
- Authority
- CN
- China
- Prior art keywords
- host
- hypersphere
- anomaly detection
- measurement data
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 58
- 239000013598 vector Substances 0.000 claims abstract description 58
- 238000000034 method Methods 0.000 claims abstract description 50
- 238000005259 measurement Methods 0.000 claims abstract description 49
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 32
- 230000002159 abnormal effect Effects 0.000 claims abstract description 29
- 238000012549 training Methods 0.000 claims abstract description 20
- 230000005856 abnormality Effects 0.000 claims abstract description 8
- 238000010276 construction Methods 0.000 claims description 20
- 238000004364 calculation method Methods 0.000 description 6
- 238000004891 communication Methods 0.000 description 5
- 238000010801 machine learning Methods 0.000 description 4
- 206010000117 Abnormal behaviour Diseases 0.000 description 2
- 230000009286 beneficial effect Effects 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000002547 anomalous effect Effects 0.000 description 1
- 238000013459 approach Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000006399 behavior Effects 0.000 description 1
- 239000000872 buffer Substances 0.000 description 1
- 238000003066 decision tree Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000010586 diagram Methods 0.000 description 1
- 238000005516 engineering process Methods 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 238000000638 solvent extraction Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0751—Error or fault detection not based on redundancy
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/07—Responding to the occurrence of a fault, e.g. fault tolerance
- G06F11/0703—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
- G06F11/0706—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
- G06F11/0709—Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/21—Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
- G06F18/214—Generating training patterns; Bootstrap methods, e.g. bagging or boosting
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/23—Clustering techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/243—Classification techniques relating to the number of classes
- G06F18/2433—Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/24—Classification techniques
- G06F18/245—Classification techniques relating to the decision surface
- G06F18/2453—Classification techniques relating to the decision surface non-linear, e.g. polynomial classifier
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Quality & Reliability (AREA)
- Computer Hardware Design (AREA)
- Mathematical Physics (AREA)
- Nonlinear Science (AREA)
- Debugging And Monitoring (AREA)
Abstract
The invention discloses a method and a system for detecting the abnormality of a cloud service platform, wherein the method comprises the following steps: 1) collecting system measurement data of a cloud platform host in a normal working state in real time, and calculating a system operation environment vector according to the system measurement data; 2) training by using a normal system operation environment vector in combination with a maximum mean deviation algorithm (MMD) and a support vector data description algorithm (SVDD) to obtain an anomaly detection model; 3) and when new host system measurement data is received, classifying the host system measurement data by using the hypersphere of the cluster where the host is located, thereby detecting the host abnormality. The method combines training of the MMD algorithm and the SVDD algorithm to obtain the anomaly detection model, effectively solves the problem of extreme imbalance of normal and abnormal samples in the cloud service platform, enables the cloud service platform to detect unknown system anomalies in the cloud service platform, and meanwhile, does not need to construct an anomaly detection model for each host, thereby greatly reducing the time for anomaly modeling and the consumption of system resources.
Description
Technical Field
The invention relates to a cloud computing security technology, in particular to a cloud service platform anomaly detection method and system.
Background
The cloud service platform is an open public platform and provides various application services for a large number of users. The reliability of these application services is critical to their consumers. The existence of anomalies in the cloud service platform can make the reliability of the cloud service platform questionable. Due to the size and complexity, the cloud service platform generates a large number of system anomalies, which are mainly caused by cloud platform administrator operation errors, resource over/under configuration, hardware/software failures, network attacks, and the like. Therefore, the method has very important significance in real-time anomaly detection of the system running state of the cloud service platform.
The basic principle of the anomaly detection is that on the basis of system monitoring, the behaviors of a system, a user, a process or a network are taken as corresponding profile models, and when the running state of the system deviates from a normal profile model, the system can be judged to be abnormal. Currently, there are related anomaly detection methods and corresponding anomaly detection systems. The method mainly comprises an anomaly detection algorithm based on statistics and an anomaly detection algorithm based on machine learning.
The anomaly detection method based on statistics firstly adopts a statistical learning method to mine the characteristics of performance data, then calculates the anomaly score of sample data based on the distribution characteristics of the data, and sends out alarm information if the anomaly score exceeds a specified threshold. This approach typically requires knowledge of the time series distribution of system performance data for the cloud service platform hosts, or may not fit well into ever-expanding clusters.
The method based on machine learning first needs to learn modeling from a large number of data samples, and then can detect new data samples to judge whether the data samples are abnormal. Hyunjoo Kim et al propose a network threat detection method based on machine learning, which firstly uses random forests to select significant features, generates clusters by applying K-means and DBSCAN to unmarked data collected from a cloud platform, and marks clustering results by using a new-Kyoto-2006+ data set. Elham beshirat et al use a combination of three different classifiers: the method has high accuracy, but the method needs to construct a neural network and decision tree model for each host, and has high cost.
The existing technical difficulties mainly include: (1) the cloud service platform is in a normal operation state most of the time, and abnormal data samples are far less than normal data samples; (2) the methods based on machine learning generally need to establish an anomaly detection model for each host, and the training process of the anomaly detection model takes a lot of time and system resources, so that the methods are difficult to apply to a large-scale cloud service platform.
Disclosure of Invention
The invention aims to solve the technical problem of providing a cloud service platform anomaly detection method and system aiming at the defects in the prior art.
The technical scheme adopted by the invention for solving the technical problems is as follows: a cloud service platform anomaly detection method comprises the following steps:
1) collecting system measurement data of a cloud platform host in a normal working state in real time, and calculating a system operation environment vector according to the system measurement data;
2) training by utilizing normal system operation environment vectors and combining a maximum mean deviation algorithm MMD and a support vector data description algorithm SVDD (support vector domain description) to obtain an abnormal detection model;
the process of combining the training of the MMD algorithm and the SVDD algorithm in the step 2) to obtain the anomaly detection model is as follows:
step 2.1) clustering the normal operation environment vectors of the system host by using the MMD, thereby dividing the cloud platform host into a plurality of clusters according to the similarity of the system operation environment;
step 2.2) constructing a hypersphere for each host cluster by using the SVDD according to the normal system operation environment vector, and describing the data distribution condition of each host during normal operation so as to detect abnormal data;
3) and when new host system measurement data is received, classifying the host system measurement data by using the hypersphere of the cluster where the host is located, thereby detecting the host abnormality.
According to the scheme, the step 3) is specifically as follows:
step 3.1) when receiving new measurement data of the host system, calculating a system operation environment vector according to the measurement data of the system;
step 3.2) selecting a hypersphere corresponding to the cluster as an anomaly detection model according to the cluster where the host is located;
step 3.3) when the running environment vector of the host system falls into the hypersphere or on the hypersphere, judging that the host is in a normal state currently;
and 3.4) when the running environment vector of the host system falls outside the hypersphere, judging that the host is in an abnormal state currently.
According to the scheme, the construction process of the hypersphere in the step 2.2) is as follows:
2.2.1) consider a set of runtime environment vectors, where N is the number of hosts in the cluster, the process of constructing a hypersphere can be expressed by the following equation:
meanwhile, the above equation satisfies the following constraints:
(xi-a)T(xi-a)≤R2+ξi,ξi≥0
where R is the radius of the hypersphere and a is the center of the hypersphere, the above equation indicates ξ if the data point is inside or on the surface of the hypersphere i0, otherwise ξiIf C is larger than 0, the model is biased to find a larger circle to cover more points as much as possible during model training, and if C is smaller, the model is biased to find a smaller circle;
2.2.2) setting the size of a constant C, and neglecting a sample point which is away from an origin and exceeds a set value in the construction process;
2.2.3) finally calculating to obtain the radius of the hypersphere and the center of the hypersphere through continuous fitting, wherein the fitting process can be expressed by the following formula:
according to the scheme, the system measurement data in the step 1) comprises system measurement data related to a CPU, a memory, a disk and a network.
A cloud service platform anomaly detection system, comprising:
the acquisition module is used for acquiring system measurement data of the cloud platform host during normal work in real time and calculating a system operation environment vector according to the system measurement data;
the model construction module is used for training by utilizing normal system operation environment vectors and combining a maximum mean deviation algorithm MMD and a support vector data description algorithm SVDD (support vector domain description) to obtain an abnormal detection model;
the process of obtaining the abnormal detection model by combining the training of the MMD algorithm and the SVDD algorithm in the model construction module is as follows:
1) clustering the normal operation environment vectors of the system host by using the MMD, thereby dividing the cloud platform host into a plurality of clusters according to the similarity of the system operation environment;
2) constructing a hypersphere for each host cluster by using SVDD according to a normal system operation environment vector, and describing the data distribution condition of each host during normal operation, thereby detecting abnormal data;
and the detection module is used for classifying the measurement data of the host system by using the hypersphere of the cluster where the host is located when receiving the new measurement data of the host system, thereby detecting the abnormality of the host.
According to the scheme, the detection module specifically comprises:
1) when new host system measurement data is received, calculating a system operation environment vector according to the system measurement data;
2) selecting a hypersphere corresponding to a cluster as an anomaly detection model according to the cluster where the host is located;
3) when the operating environment vector of the host system falls in or on the hypersphere, judging that the host is in a normal state currently;
4) and when the running environment vector of the host system falls outside the hypersphere, judging that the host is in an abnormal state currently.
According to the scheme, the construction process of the hypersphere in the model construction module is as follows:
1) considering a running environment vector set, where N is the number of hosts in the cluster, the construction process of the hypersphere can be expressed by the following formula:
meanwhile, the above equation satisfies the following constraints:
(xi-a)T(xi-a)≤R2+ξi,ξi≥0
where R is the radius of the hypersphere and a is the center of the hypersphere, the above equation indicates ξ if the data point is inside or on the surface of the hypersphere i0, otherwise ξiIf C is large, the model training is biased to find a larger circle and cover more points, and if C is small, the model training is biased to find a small circle.
2) Setting the size of a constant C, and neglecting a sample point which is away from the origin and exceeds a set value in the construction process;
3) and finally calculating to obtain the radius of the hypersphere and the center of the hypersphere through continuous fitting, wherein the fitting process can be represented by the following formula:
according to the scheme, the system measurement data in the acquisition module comprises a CPU, a memory, a disk and network-related system measurement data.
The invention has the following beneficial effects:
1. the abnormal detection model is obtained by combining training of the MMD algorithm and the SVDD algorithm, the problem that normal samples and abnormal samples in the cloud service platform are extremely unbalanced is effectively solved, unknown system abnormality in the cloud service platform can be detected, and meanwhile, the abnormal detection model does not need to be built for each host, so that the time for abnormal modeling and the system resource consumption are greatly reduced.
2. Through the constructed model, the measurement data of the host system is classified, online anomaly detection of the cloud service platform host is realized, the abnormal behavior of the system in the cloud service platform is found in time, and the reliability of the cloud service platform is improved.
Drawings
The invention will be further described with reference to the accompanying drawings and examples, in which:
FIG. 1 is a flow chart of a method of an embodiment of the present invention;
FIG. 2 is a pseudo-code diagram of a clustering process of an embodiment of the invention;
FIG. 3 is a flowchart of a process for constructing a hypersphere according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As shown in fig. 1, a cloud service platform anomaly detection method includes:
step 1: the method comprises the following steps of collecting system measurement data related to a CPU/memory/disk/network of a cloud platform host in real time, and calculating a system operation environment vector according to the system measurement data, wherein the calculation process is as follows:
1) calculating the Usage of CPU by CPU system time (SYS), CPU USER time (USER), CPU disk IO latency (IO _ WAIT), CPU hard interrupt event (IRQ), CPU SOFT interrupt event (SOFT _ IRQ) and CPU IDLE time (IDLE) in consideration of CPU measurement datacpuThe calculation formula is as follows:
2) considering the memory measurement data, the memory usage rate is calculated by the TOTAL memory size (TOTAL), the actually used memory size (ACTUAL), and the memory size in the CACHE (CACHE + BUFFERS), and the calculation formula is as follows:
3) considering the disk measurement data, calculating the disk IO frequency through the disk READ times (READ _ COUNT), the disk WRITE times (WRITE _ COUNT) and the maximum disk IO times (MAX _ IO _ COUNT), wherein the calculation formula is as follows:
4) considering the network metric data, calculating the network load through the network inbound traffic SIZE (IN _ SIEZ), the network outbound traffic SIZE (OUT _ SIEZ) and the network bandwidth (MAX _ SIZE), wherein the calculation formula is as follows:
5) obtaining a system operation environment vector according to the calculation result, wherein the system operation environment vector is expressed as follows:
RE＝(Usagecpu,Usagemem,Freqdisk,Loadnet)
step 2: the normal system operating environment vector is utilized, and an MMD algorithm and an SVDD algorithm are combined for training to obtain an anomaly detection model, so that not only can unknown anomalies be detected under the condition that an anomalous sample is missing, but also anomalies of a plurality of hosts with similar operating environments can be detected simultaneously, and thus the modeling time and the resource consumption of the system are greatly reduced;
the training process combining the MMD and SVDD algorithms is as follows:
step 2.1: using MMD to cluster the running environment vectors of the system hosts, partitioning hosts with similar running environments into a cluster, and in hosts of different running environment clusters, the running environments of the hosts are greatly different, and pseudo codes of the clustering process are shown in fig. 2, and the clustering process is as follows:
1) selecting a point closest to an original point from a system operation environment vector set as a first clustering center according to a minimum distance principle, wherein the distance is calculated by adopting a Euclidean distance;
2) selecting a point farthest from the first point from the system operation environment vector set as a second clustering center;
3) dividing the sample points into the nearest clustering centers according to a minimum distance principle, updating the clustering centers, and taking the sample points as new clustering centers if the distances from the sample points to all the clustering centers are greater than a set threshold value;
4) process 3) is repeated until all sample points are divided.
Step 2.2: training the data samples in each cluster by using an SVDD algorithm, and constructing a hypersphere for each cluster, wherein the hypersphere describes the data distribution situation of the running environment of the host in each cluster under the normal running state, the construction process of the hypersphere is shown in FIG. 3, and the construction process is described as follows:
1) considering a running environment vector set, where N is the number of hosts in the cluster, the construction process of the hypersphere can be expressed by the following formula:
meanwhile, the above equation satisfies the following constraints:
(xi-a)T(xi-a)≤R2+ξi,ξi≥0
where R is the radius of the hypersphere and a is the center of the hypersphere, the above equation indicates ξ if the data point is inside or on the surface of the hypersphere i0, otherwise ξiIf C is large, the model will be biased to find a larger circle and try to make the capsuleIncluding more points, if C is smaller, the preference is to find a small circle.
2) Setting the size of a constant C, and neglecting a sample point which is away from the origin and exceeds a set value in the construction process;
3) and finally calculating to obtain the radius of the hypersphere and the center of the hypersphere through continuous fitting, wherein the fitting process can be represented by the following formula:
and step 3: when new host system measurement data is received, classifying the host system measurement data by using a hypersphere of a cluster where a host is located, wherein a classification formula is as follows:
where Ω denotes a hypersphere. When the operating environment vector of the host system falls in or on the hypersphere, judging that the host is in a normal state currently; and when the running environment vector of the host system falls outside the hypersphere, judging that the host is in an abnormal state currently, and giving an alarm through a mail.
The cloud platform anomaly detection method proposed by the implementation has the following beneficial effects:
1) the abnormal detection model is obtained by combining training of the MMD algorithm and the SVDD algorithm, the problem that normal samples and abnormal samples in the cloud service platform are extremely unbalanced is effectively solved, unknown system abnormality in the cloud service platform can be detected, and meanwhile, the abnormal detection model does not need to be built for each host, so that the time for abnormal modeling and the system resource consumption are greatly reduced.
2) Through the constructed model, the measurement data of the host system is classified, online anomaly detection of the cloud service platform host is realized, the abnormal behavior of the system in the cloud service platform is found in time, and the reliability of the cloud service platform is improved.
The invention further provides a cloud service platform anomaly detection system, which comprises an acquisition module, a communication module, a modeling module and a detection module:
the acquisition module runs in the cloud platform host, acquires system resource measurement data related to a CPU/memory/disk/network of the cloud service platform host in real time, and submits the acquired data to the communication module client sub-module.
After the communication module client submodule acquires the data submitted by the acquisition module, the communication module client submodule encapsulates the data, adds a local Mac address in a data packet header to distinguish different host system measurement data, and pushes an encapsulated data packet to a SYS _ METRICS theme of a Kafka message queue.
The communication module service terminal module regularly pulls and analyzes data from the SYS _ METRICS theme of the Kafka message queue, and stores the analyzed data into a database according to a certain format.
The modeling module analyzes and models the system measurement data under normal conditions (namely, the system measurement data which is not attacked to generate an abnormal detection model). The method comprises the following specific steps:
step 1: extracting incremental data from an HBase database, and calculating a system operation environment vector according to system measurement data;
step 2: the method utilizes normal system operation environment vectors, combines training of an MMD algorithm and an SVDD algorithm to obtain an anomaly detection model, enables the anomaly detection model to detect unknown anomalies under the condition of abnormal sample loss, and can detect the anomalies of a plurality of hosts with similar operation environments at the same time, thereby greatly reducing the modeling time and the resource consumption of the system, and the modeling process is as follows:
step 2.1: clustering all the calculated system operation environment vectors by using the MMD, thereby dividing the cloud service platform host into a plurality of clusters according to the similarity of the system operation environments;
step 2.2: and constructing a hypersphere for each host cluster by using the SVDD according to the normal operation environment vector of the system, wherein the hypersphere is used for describing the data distribution condition of the hosts during normal operation.
The detection module loads the hypersphere corresponding to all the host clusters, and when receiving new host system measurement data, calculates the host system operating environment vector according to the host system measurement data; then, according to the cluster where the host computer is located, selecting a hypersphere corresponding to the cluster to classify the system measurement data; when the operating environment vector of the host system falls in or on the hypersphere, judging that the host is in a normal state currently; and when the running environment vector of the host system falls outside the hypersphere, judging that the host is in an abnormal state currently, and giving an alarm through a mail.
It will be understood that modifications and variations can be made by persons skilled in the art in light of the above teachings and all such modifications and variations are intended to be included within the scope of the invention as defined in the appended claims.
Claims (8)
1. A cloud service platform anomaly detection method is characterized by comprising the following steps:
1) collecting system measurement data of a cloud platform host in real time when the cloud platform host works normally, and calculating a system operation environment vector according to the normal and abnormal system measurement data;
2) training by using a normal system operation environment vector in combination with a maximum mean deviation algorithm (MMD) and a support vector data description algorithm (SVDD) to obtain an anomaly detection model;
the process of combining the training of the MMD algorithm and the SVDD algorithm in the step 2) to obtain the anomaly detection model is as follows:
step 2.1) clustering the normal operation environment vectors of the system host by using the MMD, thereby dividing the cloud platform host into a plurality of clusters according to the similarity of the system operation environment;
step 2.2) constructing a hypersphere for each host cluster by using the SVDD according to the normal system operation environment vector, and describing the data distribution condition of each host during normal operation so as to detect abnormal data;
3) and when new host system measurement data is received, classifying the host system measurement data by using the hypersphere of the cluster where the host is located, thereby detecting the host abnormality.
2. The cloud service platform anomaly detection method according to claim 1, wherein the step 3) specifically comprises:
step 3.1) when receiving new measurement data of the host system, calculating a system operation environment vector according to the measurement data of the system;
step 3.2) selecting a hypersphere corresponding to the cluster as an anomaly detection model according to the cluster where the host is located;
step 3.3) when the running environment vector of the host system falls into the hypersphere or on the hypersphere, judging that the host is in a normal state currently;
and 3.4) when the running environment vector of the host system falls outside the hypersphere, judging that the host is in an abnormal state currently.
3. The cloud service platform anomaly detection method according to claim 1, wherein the hypersphere construction process in step 2.2) is as follows:
2.2.1) consider a set of runtime environment vectors, where N is the number of hosts in the cluster, the process of constructing a hypersphere can be expressed by the following equation:
meanwhile, the above equation satisfies the following constraints:
(xi-a)T(xi-a)≤R2+ξi,ξi≥0
where R is the radius of the hypersphere and a is the center of the hypersphere, the above equation indicates ξ if the data point is inside or on the surface of the hyperspherei0, otherwise ξi> 0, C is a constant;
2.2.2) setting the size of a constant C, and neglecting a sample point which is away from an origin and exceeds a set value in the construction process;
2.2.3) finally calculating to obtain the radius of the hypersphere and the center of the hypersphere through continuous fitting, wherein the fitting process can be expressed by the following formula:
4. the cloud service platform anomaly detection method according to claim 1, wherein the system metric data in step 1) includes system metric data related to a CPU, a memory, a disk, and a network.
5. A cloud service platform anomaly detection system, comprising:
the acquisition module is used for acquiring system measurement data of the cloud platform host during normal work in real time and calculating a system operation environment vector according to the system measurement data;
the model construction module is used for training by utilizing a normal system operation environment vector and combining a maximum mean deviation algorithm (MMD) and a support vector data description algorithm (SVDD) to obtain an anomaly detection model;
the process of obtaining the abnormal detection model by combining the training of the MMD algorithm and the SVDD algorithm in the model construction module is as follows:
1) clustering the normal operation environment vectors of the system host by using the MMD, thereby dividing the cloud platform host into a plurality of clusters according to the similarity of the system operation environment;
2) constructing a hypersphere for each host cluster by using SVDD according to a normal system operation environment vector, and describing the data distribution condition of each host during normal operation, thereby detecting abnormal data;
and the detection module is used for classifying the measurement data of the host system by using the hypersphere of the cluster where the host is located when receiving the new measurement data of the host system, thereby detecting the abnormality of the host.
6. The cloud service platform anomaly detection system according to claim 5, wherein the detection module specifically is:
1) when new host system measurement data is received, calculating a system operation environment vector according to the system measurement data;
2) selecting a hypersphere corresponding to a cluster as an anomaly detection model according to the cluster where the host is located;
3) when the operating environment vector of the host system falls in or on the hypersphere, judging that the host is in a normal state currently;
4) and when the running environment vector of the host system falls outside the hypersphere, judging that the host is in an abnormal state currently.
7. The cloud service platform anomaly detection system according to claim 5, wherein a hypersphere construction process in the model construction module is as follows:
1) considering a running environment vector set, where N is the number of hosts in the cluster, the construction process of the hypersphere is expressed by the following formula:
meanwhile, the above equation satisfies the following constraints:
(xi-a)T(xi-a)≤R2+ξi,ξi≥0
where R is the radius of the hypersphere and a is the center of the hypersphere, the above equation indicates ξ if the data point is inside or on the surface of the hyperspherei0, otherwise ξi> 0, C is a constant;
2) setting the size of a constant C, and neglecting a sample point which is away from the origin and exceeds a set value in the construction process;
3) and finally calculating to obtain the radius of the hypersphere and the center of the hypersphere through continuous fitting, wherein the fitting process is represented by the following formula:
8. the cloud service platform anomaly detection system according to claim 5, wherein the system metric data in the collection module includes CPU, memory, disk, and network related system metric data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910820118.1A CN110825545A (en) | 2019-08-31 | 2019-08-31 | Cloud service platform anomaly detection method and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201910820118.1A CN110825545A (en) | 2019-08-31 | 2019-08-31 | Cloud service platform anomaly detection method and system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN110825545A true CN110825545A (en) | 2020-02-21 |
Family
ID=69547898
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201910820118.1A Pending CN110825545A (en) | 2019-08-31 | 2019-08-31 | Cloud service platform anomaly detection method and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN110825545A (en) |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111369339A (en) * | 2020-03-02 | 2020-07-03 | 深圳索信达数据技术有限公司 | Over-sampling improved svdd-based bank client transaction behavior abnormity identification method |
| CN112487363A (en) * | 2020-12-03 | 2021-03-12 | 上海卫星工程研究所 | Method and system for detecting satellite telemetry consistency based on MMD analysis |
| CN112783682A (en) * | 2021-02-01 | 2021-05-11 | 福建多多云科技有限公司 | Abnormal automatic repairing method based on cloud mobile phone service |
| WO2021139249A1 (en) * | 2020-05-28 | 2021-07-15 | 平安科技（深圳）有限公司 | Data anomaly detection method, apparatus and device, and storage medium |
| CN113532866A (en) * | 2020-04-16 | 2021-10-22 | 中国船舶重工集团公司第七一一研究所 | Diesel engine abnormal state detection method and system and computer storage medium |
| CN115563622A (en) * | 2022-09-29 | 2023-01-03 | 国网山西省电力公司 | Method, device and system for detecting operating environment |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101980480A (en) * | 2010-11-04 | 2011-02-23 | 西安电子科技大学 | Semi-supervised anomaly intrusion detection method |
| CN106951776A (en) * | 2017-01-18 | 2017-07-14 | 中国船舶重工集团公司第七0九研究所 | A kind of Host Anomaly Detection method and system |
-
2019
- 2019-08-31 CN CN201910820118.1A patent/CN110825545A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101980480A (en) * | 2010-11-04 | 2011-02-23 | 西安电子科技大学 | Semi-supervised anomaly intrusion detection method |
| CN106951776A (en) * | 2017-01-18 | 2017-07-14 | 中国船舶重工集团公司第七0九研究所 | A kind of Host Anomaly Detection method and system |
Non-Patent Citations (1)
| Title |
|---|
| PING LOU: "《An Anomaly Detection Method for Cloud Service Platform》" * |
Cited By (8)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111369339A (en) * | 2020-03-02 | 2020-07-03 | 深圳索信达数据技术有限公司 | Over-sampling improved svdd-based bank client transaction behavior abnormity identification method |
| CN113532866A (en) * | 2020-04-16 | 2021-10-22 | 中国船舶重工集团公司第七一一研究所 | Diesel engine abnormal state detection method and system and computer storage medium |
| WO2021139249A1 (en) * | 2020-05-28 | 2021-07-15 | 平安科技（深圳）有限公司 | Data anomaly detection method, apparatus and device, and storage medium |
| CN112487363A (en) * | 2020-12-03 | 2021-03-12 | 上海卫星工程研究所 | Method and system for detecting satellite telemetry consistency based on MMD analysis |
| CN112783682A (en) * | 2021-02-01 | 2021-05-11 | 福建多多云科技有限公司 | Abnormal automatic repairing method based on cloud mobile phone service |
| CN112783682B (en) * | 2021-02-01 | 2022-02-22 | 福建多多云科技有限公司 | Abnormal automatic repairing method based on cloud mobile phone service |
| CN115563622A (en) * | 2022-09-29 | 2023-01-03 | 国网山西省电力公司 | Method, device and system for detecting operating environment |
| CN115563622B (en) * | 2022-09-29 | 2024-03-12 | 国网山西省电力公司 | Method, device and system for detecting operation environment |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110825545A (en) | Cloud service platform anomaly detection method and system | |
| CN112769796B (en) | Cloud network side collaborative defense method and system based on end side edge computing | |
| CN110784481B (en) | DDoS detection method and system based on neural network in SDN network | |
| US11570070B2 (en) | Network device classification apparatus and process | |
| US7724784B2 (en) | System and method for classifying data streams using high-order models | |
| CN109067586B (en) | DDoS attack detection method and device | |
| US11516240B2 (en) | Detection of anomalies associated with fraudulent access to a service platform | |
| EP3465515B1 (en) | Classifying transactions at network accessible storage | |
| Zhe et al. | DoS attack detection model of smart grid based on machine learning method | |
| Garg et al. | HyClass: Hybrid classification model for anomaly detection in cloud environment | |
| CN112306820B (en) | Log operation and maintenance root cause analysis method and device, electronic equipment and storage medium | |
| CN117081858B (en) | Intrusion behavior detection method, system, equipment and medium based on multi-decision tree | |
| CN111224984B (en) | Snort improvement method based on data mining algorithm | |
| CN109902754A (en) | A kind of efficiently semi-supervised multi-level intrusion detection method and system | |
| Xiao et al. | A traffic classification method with spectral clustering in SDN | |
| Brandao et al. | Log Files Analysis for Network Intrusion Detection | |
| CN116155581A (en) | Network intrusion detection method and device based on graph neural network | |
| CN108989083B (en) | Fault detection performance optimization method based on hybrid strategy in cloud environment | |
| Rassam et al. | One-class principal component classifier for anomaly detection in wireless sensor network | |
| Nalavade et al. | Evaluation of k-means clustering for effective intrusion detection and prevention in massive network traffic data | |
| CN112888008A (en) | Base station abnormity detection method, device, equipment and storage medium | |
| CN112422546A (en) | Network anomaly detection method based on variable neighborhood algorithm and fuzzy clustering | |
| CN112055007A (en) | Software and hardware combined threat situation perception method based on programmable nodes | |
| Atli et al. | Network intrusion detection using flow statistics | |
| CN111475380B (en) | Log analysis method and device |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20200221 |
|
| RJ01 | Rejection of invention patent application after publication |