CN110825545A - Cloud service platform anomaly detection method and system - Google Patents

Cloud service platform anomaly detection method and system Download PDF

Info

Publication number
CN110825545A
CN110825545A CN201910820118.1A CN201910820118A CN110825545A CN 110825545 A CN110825545 A CN 110825545A CN 201910820118 A CN201910820118 A CN 201910820118A CN 110825545 A CN110825545 A CN 110825545A
Authority
CN
China
Prior art keywords
host
hypersphere
anomaly detection
measurement data
data
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Pending
Application number
CN201910820118.1A
Other languages
Chinese (zh)
Inventor
严俊伟
杨赟
娄平
刘泉
周祖德
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuhan University of Technology WUT
Original Assignee
Wuhan University of Technology WUT
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuhan University of Technology WUT filed Critical Wuhan University of Technology WUT
Priority to CN201910820118.1A priority Critical patent/CN110825545A/en
Publication of CN110825545A publication Critical patent/CN110825545A/en
Pending legal-status Critical Current

Links

Images

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0751Error or fault detection not based on redundancy
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F11/00Error detection; Error correction; Monitoring
    • G06F11/07Responding to the occurrence of a fault, e.g. fault tolerance
    • G06F11/0703Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation
    • G06F11/0706Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment
    • G06F11/0709Error or fault processing not based on redundancy, i.e. by taking additional measures to deal with the error or fault not making use of redundancy in operation, in hardware, or in data representation the processing taking place on a specific hardware platform or in a specific software environment in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/21Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
    • G06F18/214Generating training patterns; Bootstrap methods, e.g. bagging or boosting
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/23Clustering techniques
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/24Classification techniques
    • G06F18/243Classification techniques relating to the number of classes
    • G06F18/2433Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/24Classification techniques
    • G06F18/245Classification techniques relating to the decision surface
    • G06F18/2453Classification techniques relating to the decision surface non-linear, e.g. polynomial classifier

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Artificial Intelligence (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Quality & Reliability (AREA)
  • Computer Hardware Design (AREA)
  • Mathematical Physics (AREA)
  • Nonlinear Science (AREA)
  • Debugging And Monitoring (AREA)

Abstract

The invention discloses a method and a system for detecting the abnormality of a cloud service platform, wherein the method comprises the following steps: 1) collecting system measurement data of a cloud platform host in a normal working state in real time, and calculating a system operation environment vector according to the system measurement data; 2) training by using a normal system operation environment vector in combination with a maximum mean deviation algorithm (MMD) and a support vector data description algorithm (SVDD) to obtain an anomaly detection model; 3) and when new host system measurement data is received, classifying the host system measurement data by using the hypersphere of the cluster where the host is located, thereby detecting the host abnormality. The method combines training of the MMD algorithm and the SVDD algorithm to obtain the anomaly detection model, effectively solves the problem of extreme imbalance of normal and abnormal samples in the cloud service platform, enables the cloud service platform to detect unknown system anomalies in the cloud service platform, and meanwhile, does not need to construct an anomaly detection model for each host, thereby greatly reducing the time for anomaly modeling and the consumption of system resources.

Description

Cloud service platform anomaly detection method and system
Technical Field
The invention relates to a cloud computing security technology, in particular to a cloud service platform anomaly detection method and system.
Background
The cloud service platform is an open public platform and provides various application services for a large number of users. The reliability of these application services is critical to their consumers. The existence of anomalies in the cloud service platform can make the reliability of the cloud service platform questionable. Due to the size and complexity, the cloud service platform generates a large number of system anomalies, which are mainly caused by cloud platform administrator operation errors, resource over/under configuration, hardware/software failures, network attacks, and the like. Therefore, the method has very important significance in real-time anomaly detection of the system running state of the cloud service platform.
The basic principle of the anomaly detection is that on the basis of system monitoring, the behaviors of a system, a user, a process or a network are taken as corresponding profile models, and when the running state of the system deviates from a normal profile model, the system can be judged to be abnormal. Currently, there are related anomaly detection methods and corresponding anomaly detection systems. The method mainly comprises an anomaly detection algorithm based on statistics and an anomaly detection algorithm based on machine learning.
The anomaly detection method based on statistics firstly adopts a statistical learning method to mine the characteristics of performance data, then calculates the anomaly score of sample data based on the distribution characteristics of the data, and sends out alarm information if the anomaly score exceeds a specified threshold. This approach typically requires knowledge of the time series distribution of system performance data for the cloud service platform hosts, or may not fit well into ever-expanding clusters.
The method based on machine learning first needs to learn modeling from a large number of data samples, and then can detect new data samples to judge whether the data samples are abnormal. Hyunjoo Kim et al propose a network threat detection method based on machine learning, which firstly uses random forests to select significant features, generates clusters by applying K-means and DBSCAN to unmarked data collected from a cloud platform, and marks clustering results by using a new-Kyoto-2006+ data set. Elham beshirat et al use a combination of three different classifiers: the method has high accuracy, but the method needs to construct a neural network and decision tree model for each host, and has high cost.
The existing technical difficulties mainly include: (1) the cloud service platform is in a normal operation state most of the time, and abnormal data samples are far less than normal data samples; (2) the methods based on machine learning generally need to establish an anomaly detection model for each host, and the training process of the anomaly detection model takes a lot of time and system resources, so that the methods are difficult to apply to a large-scale cloud service platform.
Disclosure of Invention
The invention aims to solve the technical problem of providing a cloud service platform anomaly detection method and system aiming at the defects in the prior art.
The technical scheme adopted by the invention for solving the technical problems is as follows: a cloud service platform anomaly detection method comprises the following steps:
1) collecting system measurement data of a cloud platform host in a normal working state in real time, and calculating a system operation environment vector according to the system measurement data;
2) training by utilizing normal system operation environment vectors and combining a maximum mean deviation algorithm MMD and a support vector data description algorithm SVDD (support vector domain description) to obtain an abnormal detection model;
the process of combining the training of the MMD algorithm and the SVDD algorithm in the step 2) to obtain the anomaly detection model is as follows:
step 2.1) clustering the normal operation environment vectors of the system host by using the MMD, thereby dividing the cloud platform host into a plurality of clusters according to the similarity of the system operation environment;
step 2.2) constructing a hypersphere for each host cluster by using the SVDD according to the normal system operation environment vector, and describing the data distribution condition of each host during normal operation so as to detect abnormal data;
3) and when new host system measurement data is received, classifying the host system measurement data by using the hypersphere of the cluster where the host is located, thereby detecting the host abnormality.
According to the scheme, the step 3) is specifically as follows:
step 3.1) when receiving new measurement data of the host system, calculating a system operation environment vector according to the measurement data of the system;
step 3.2) selecting a hypersphere corresponding to the cluster as an anomaly detection model according to the cluster where the host is located;
step 3.3) when the running environment vector of the host system falls into the hypersphere or on the hypersphere, judging that the host is in a normal state currently;
and 3.4) when the running environment vector of the host system falls outside the hypersphere, judging that the host is in an abnormal state currently.
According to the scheme, the construction process of the hypersphere in the step 2.2) is as follows:
2.2.1) consider a set of runtime environment vectors, where N is the number of hosts in the cluster, the process of constructing a hypersphere can be expressed by the following equation:
Figure RE-GDA0002354490660000041
meanwhile, the above equation satisfies the following constraints:
(xi-a)T(xi-a)≤R2ii≥0
where R is the radius of the hypersphere and a is the center of the hypersphere, the above equation indicates ξ if the data point is inside or on the surface of the hypersphere i0, otherwise ξiIf C is larger than 0, the model is biased to find a larger circle to cover more points as much as possible during model training, and if C is smaller, the model is biased to find a smaller circle;
2.2.2) setting the size of a constant C, and neglecting a sample point which is away from an origin and exceeds a set value in the construction process;
2.2.3) finally calculating to obtain the radius of the hypersphere and the center of the hypersphere through continuous fitting, wherein the fitting process can be expressed by the following formula:
Figure RE-GDA0002354490660000051
according to the scheme, the system measurement data in the step 1) comprises system measurement data related to a CPU, a memory, a disk and a network.
A cloud service platform anomaly detection system, comprising:
the acquisition module is used for acquiring system measurement data of the cloud platform host during normal work in real time and calculating a system operation environment vector according to the system measurement data;
the model construction module is used for training by utilizing normal system operation environment vectors and combining a maximum mean deviation algorithm MMD and a support vector data description algorithm SVDD (support vector domain description) to obtain an abnormal detection model;
the process of obtaining the abnormal detection model by combining the training of the MMD algorithm and the SVDD algorithm in the model construction module is as follows:
1) clustering the normal operation environment vectors of the system host by using the MMD, thereby dividing the cloud platform host into a plurality of clusters according to the similarity of the system operation environment;
2) constructing a hypersphere for each host cluster by using SVDD according to a normal system operation environment vector, and describing the data distribution condition of each host during normal operation, thereby detecting abnormal data;
and the detection module is used for classifying the measurement data of the host system by using the hypersphere of the cluster where the host is located when receiving the new measurement data of the host system, thereby detecting the abnormality of the host.
According to the scheme, the detection module specifically comprises:
1) when new host system measurement data is received, calculating a system operation environment vector according to the system measurement data;
2) selecting a hypersphere corresponding to a cluster as an anomaly detection model according to the cluster where the host is located;
3) when the operating environment vector of the host system falls in or on the hypersphere, judging that the host is in a normal state currently;
4) and when the running environment vector of the host system falls outside the hypersphere, judging that the host is in an abnormal state currently.
According to the scheme, the construction process of the hypersphere in the model construction module is as follows:
1) considering a running environment vector set, where N is the number of hosts in the cluster, the construction process of the hypersphere can be expressed by the following formula:
Figure RE-GDA0002354490660000071
meanwhile, the above equation satisfies the following constraints:
(xi-a)T(xi-a)≤R2ii≥0
where R is the radius of the hypersphere and a is the center of the hypersphere, the above equation indicates ξ if the data point is inside or on the surface of the hypersphere i0, otherwise ξiIf C is large, the model training is biased to find a larger circle and cover more points, and if C is small, the model training is biased to find a small circle.
2) Setting the size of a constant C, and neglecting a sample point which is away from the origin and exceeds a set value in the construction process;
3) and finally calculating to obtain the radius of the hypersphere and the center of the hypersphere through continuous fitting, wherein the fitting process can be represented by the following formula:
according to the scheme, the system measurement data in the acquisition module comprises a CPU, a memory, a disk and network-related system measurement data.
The invention has the following beneficial effects:
1. the abnormal detection model is obtained by combining training of the MMD algorithm and the SVDD algorithm, the problem that normal samples and abnormal samples in the cloud service platform are extremely unbalanced is effectively solved, unknown system abnormality in the cloud service platform can be detected, and meanwhile, the abnormal detection model does not need to be built for each host, so that the time for abnormal modeling and the system resource consumption are greatly reduced.
2. Through the constructed model, the measurement data of the host system is classified, online anomaly detection of the cloud service platform host is realized, the abnormal behavior of the system in the cloud service platform is found in time, and the reliability of the cloud service platform is improved.
Drawings
The invention will be further described with reference to the accompanying drawings and examples, in which:
FIG. 1 is a flow chart of a method of an embodiment of the present invention;
FIG. 2 is a pseudo-code diagram of a clustering process of an embodiment of the invention;
FIG. 3 is a flowchart of a process for constructing a hypersphere according to an embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail with reference to the following embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.
As shown in fig. 1, a cloud service platform anomaly detection method includes:
step 1: the method comprises the following steps of collecting system measurement data related to a CPU/memory/disk/network of a cloud platform host in real time, and calculating a system operation environment vector according to the system measurement data, wherein the calculation process is as follows:
1) calculating the Usage of CPU by CPU system time (SYS), CPU USER time (USER), CPU disk IO latency (IO _ WAIT), CPU hard interrupt event (IRQ), CPU SOFT interrupt event (SOFT _ IRQ) and CPU IDLE time (IDLE) in consideration of CPU measurement datacpuThe calculation formula is as follows:
Figure RE-GDA0002354490660000091
2) considering the memory measurement data, the memory usage rate is calculated by the TOTAL memory size (TOTAL), the actually used memory size (ACTUAL), and the memory size in the CACHE (CACHE + BUFFERS), and the calculation formula is as follows:
Figure RE-GDA0002354490660000092
3) considering the disk measurement data, calculating the disk IO frequency through the disk READ times (READ _ COUNT), the disk WRITE times (WRITE _ COUNT) and the maximum disk IO times (MAX _ IO _ COUNT), wherein the calculation formula is as follows:
Figure RE-GDA0002354490660000093
4) considering the network metric data, calculating the network load through the network inbound traffic SIZE (IN _ SIEZ), the network outbound traffic SIZE (OUT _ SIEZ) and the network bandwidth (MAX _ SIZE), wherein the calculation formula is as follows:
Figure RE-GDA0002354490660000094
5) obtaining a system operation environment vector according to the calculation result, wherein the system operation environment vector is expressed as follows:
RE=(Usagecpu,Usagemem,Freqdisk,Loadnet)
step 2: the normal system operating environment vector is utilized, and an MMD algorithm and an SVDD algorithm are combined for training to obtain an anomaly detection model, so that not only can unknown anomalies be detected under the condition that an anomalous sample is missing, but also anomalies of a plurality of hosts with similar operating environments can be detected simultaneously, and thus the modeling time and the resource consumption of the system are greatly reduced;
the training process combining the MMD and SVDD algorithms is as follows:
step 2.1: using MMD to cluster the running environment vectors of the system hosts, partitioning hosts with similar running environments into a cluster, and in hosts of different running environment clusters, the running environments of the hosts are greatly different, and pseudo codes of the clustering process are shown in fig. 2, and the clustering process is as follows:
1) selecting a point closest to an original point from a system operation environment vector set as a first clustering center according to a minimum distance principle, wherein the distance is calculated by adopting a Euclidean distance;
2) selecting a point farthest from the first point from the system operation environment vector set as a second clustering center;
3) dividing the sample points into the nearest clustering centers according to a minimum distance principle, updating the clustering centers, and taking the sample points as new clustering centers if the distances from the sample points to all the clustering centers are greater than a set threshold value;
4) process 3) is repeated until all sample points are divided.
Step 2.2: training the data samples in each cluster by using an SVDD algorithm, and constructing a hypersphere for each cluster, wherein the hypersphere describes the data distribution situation of the running environment of the host in each cluster under the normal running state, the construction process of the hypersphere is shown in FIG. 3, and the construction process is described as follows:
1) considering a running environment vector set, where N is the number of hosts in the cluster, the construction process of the hypersphere can be expressed by the following formula:
Figure RE-GDA0002354490660000111
meanwhile, the above equation satisfies the following constraints:
(xi-a)T(xi-a)≤R2ii≥0
where R is the radius of the hypersphere and a is the center of the hypersphere, the above equation indicates ξ if the data point is inside or on the surface of the hypersphere i0, otherwise ξiIf C is large, the model will be biased to find a larger circle and try to make the capsuleIncluding more points, if C is smaller, the preference is to find a small circle.
2) Setting the size of a constant C, and neglecting a sample point which is away from the origin and exceeds a set value in the construction process;
3) and finally calculating to obtain the radius of the hypersphere and the center of the hypersphere through continuous fitting, wherein the fitting process can be represented by the following formula:
Figure RE-GDA0002354490660000112
and step 3: when new host system measurement data is received, classifying the host system measurement data by using a hypersphere of a cluster where a host is located, wherein a classification formula is as follows:
where Ω denotes a hypersphere. When the operating environment vector of the host system falls in or on the hypersphere, judging that the host is in a normal state currently; and when the running environment vector of the host system falls outside the hypersphere, judging that the host is in an abnormal state currently, and giving an alarm through a mail.
The cloud platform anomaly detection method proposed by the implementation has the following beneficial effects:
1) the abnormal detection model is obtained by combining training of the MMD algorithm and the SVDD algorithm, the problem that normal samples and abnormal samples in the cloud service platform are extremely unbalanced is effectively solved, unknown system abnormality in the cloud service platform can be detected, and meanwhile, the abnormal detection model does not need to be built for each host, so that the time for abnormal modeling and the system resource consumption are greatly reduced.
2) Through the constructed model, the measurement data of the host system is classified, online anomaly detection of the cloud service platform host is realized, the abnormal behavior of the system in the cloud service platform is found in time, and the reliability of the cloud service platform is improved.
The invention further provides a cloud service platform anomaly detection system, which comprises an acquisition module, a communication module, a modeling module and a detection module:
the acquisition module runs in the cloud platform host, acquires system resource measurement data related to a CPU/memory/disk/network of the cloud service platform host in real time, and submits the acquired data to the communication module client sub-module.
After the communication module client submodule acquires the data submitted by the acquisition module, the communication module client submodule encapsulates the data, adds a local Mac address in a data packet header to distinguish different host system measurement data, and pushes an encapsulated data packet to a SYS _ METRICS theme of a Kafka message queue.
The communication module service terminal module regularly pulls and analyzes data from the SYS _ METRICS theme of the Kafka message queue, and stores the analyzed data into a database according to a certain format.
The modeling module analyzes and models the system measurement data under normal conditions (namely, the system measurement data which is not attacked to generate an abnormal detection model). The method comprises the following specific steps:
step 1: extracting incremental data from an HBase database, and calculating a system operation environment vector according to system measurement data;
step 2: the method utilizes normal system operation environment vectors, combines training of an MMD algorithm and an SVDD algorithm to obtain an anomaly detection model, enables the anomaly detection model to detect unknown anomalies under the condition of abnormal sample loss, and can detect the anomalies of a plurality of hosts with similar operation environments at the same time, thereby greatly reducing the modeling time and the resource consumption of the system, and the modeling process is as follows:
step 2.1: clustering all the calculated system operation environment vectors by using the MMD, thereby dividing the cloud service platform host into a plurality of clusters according to the similarity of the system operation environments;
step 2.2: and constructing a hypersphere for each host cluster by using the SVDD according to the normal operation environment vector of the system, wherein the hypersphere is used for describing the data distribution condition of the hosts during normal operation.
The detection module loads the hypersphere corresponding to all the host clusters, and when receiving new host system measurement data, calculates the host system operating environment vector according to the host system measurement data; then, according to the cluster where the host computer is located, selecting a hypersphere corresponding to the cluster to classify the system measurement data; when the operating environment vector of the host system falls in or on the hypersphere, judging that the host is in a normal state currently; and when the running environment vector of the host system falls outside the hypersphere, judging that the host is in an abnormal state currently, and giving an alarm through a mail.
It will be understood that modifications and variations can be made by persons skilled in the art in light of the above teachings and all such modifications and variations are intended to be included within the scope of the invention as defined in the appended claims.

Claims (8)

1. A cloud service platform anomaly detection method is characterized by comprising the following steps:
1) collecting system measurement data of a cloud platform host in real time when the cloud platform host works normally, and calculating a system operation environment vector according to the normal and abnormal system measurement data;
2) training by using a normal system operation environment vector in combination with a maximum mean deviation algorithm (MMD) and a support vector data description algorithm (SVDD) to obtain an anomaly detection model;
the process of combining the training of the MMD algorithm and the SVDD algorithm in the step 2) to obtain the anomaly detection model is as follows:
step 2.1) clustering the normal operation environment vectors of the system host by using the MMD, thereby dividing the cloud platform host into a plurality of clusters according to the similarity of the system operation environment;
step 2.2) constructing a hypersphere for each host cluster by using the SVDD according to the normal system operation environment vector, and describing the data distribution condition of each host during normal operation so as to detect abnormal data;
3) and when new host system measurement data is received, classifying the host system measurement data by using the hypersphere of the cluster where the host is located, thereby detecting the host abnormality.
2. The cloud service platform anomaly detection method according to claim 1, wherein the step 3) specifically comprises:
step 3.1) when receiving new measurement data of the host system, calculating a system operation environment vector according to the measurement data of the system;
step 3.2) selecting a hypersphere corresponding to the cluster as an anomaly detection model according to the cluster where the host is located;
step 3.3) when the running environment vector of the host system falls into the hypersphere or on the hypersphere, judging that the host is in a normal state currently;
and 3.4) when the running environment vector of the host system falls outside the hypersphere, judging that the host is in an abnormal state currently.
3. The cloud service platform anomaly detection method according to claim 1, wherein the hypersphere construction process in step 2.2) is as follows:
2.2.1) consider a set of runtime environment vectors, where N is the number of hosts in the cluster, the process of constructing a hypersphere can be expressed by the following equation:
Figure FDA0002187298320000021
meanwhile, the above equation satisfies the following constraints:
(xi-a)T(xi-a)≤R2ii≥0
where R is the radius of the hypersphere and a is the center of the hypersphere, the above equation indicates ξ if the data point is inside or on the surface of the hyperspherei0, otherwise ξi> 0, C is a constant;
2.2.2) setting the size of a constant C, and neglecting a sample point which is away from an origin and exceeds a set value in the construction process;
2.2.3) finally calculating to obtain the radius of the hypersphere and the center of the hypersphere through continuous fitting, wherein the fitting process can be expressed by the following formula:
Figure FDA0002187298320000031
4. the cloud service platform anomaly detection method according to claim 1, wherein the system metric data in step 1) includes system metric data related to a CPU, a memory, a disk, and a network.
5. A cloud service platform anomaly detection system, comprising:
the acquisition module is used for acquiring system measurement data of the cloud platform host during normal work in real time and calculating a system operation environment vector according to the system measurement data;
the model construction module is used for training by utilizing a normal system operation environment vector and combining a maximum mean deviation algorithm (MMD) and a support vector data description algorithm (SVDD) to obtain an anomaly detection model;
the process of obtaining the abnormal detection model by combining the training of the MMD algorithm and the SVDD algorithm in the model construction module is as follows:
1) clustering the normal operation environment vectors of the system host by using the MMD, thereby dividing the cloud platform host into a plurality of clusters according to the similarity of the system operation environment;
2) constructing a hypersphere for each host cluster by using SVDD according to a normal system operation environment vector, and describing the data distribution condition of each host during normal operation, thereby detecting abnormal data;
and the detection module is used for classifying the measurement data of the host system by using the hypersphere of the cluster where the host is located when receiving the new measurement data of the host system, thereby detecting the abnormality of the host.
6. The cloud service platform anomaly detection system according to claim 5, wherein the detection module specifically is:
1) when new host system measurement data is received, calculating a system operation environment vector according to the system measurement data;
2) selecting a hypersphere corresponding to a cluster as an anomaly detection model according to the cluster where the host is located;
3) when the operating environment vector of the host system falls in or on the hypersphere, judging that the host is in a normal state currently;
4) and when the running environment vector of the host system falls outside the hypersphere, judging that the host is in an abnormal state currently.
7. The cloud service platform anomaly detection system according to claim 5, wherein a hypersphere construction process in the model construction module is as follows:
1) considering a running environment vector set, where N is the number of hosts in the cluster, the construction process of the hypersphere is expressed by the following formula:
Figure FDA0002187298320000041
meanwhile, the above equation satisfies the following constraints:
(xi-a)T(xi-a)≤R2ii≥0
where R is the radius of the hypersphere and a is the center of the hypersphere, the above equation indicates ξ if the data point is inside or on the surface of the hyperspherei0, otherwise ξi> 0, C is a constant;
2) setting the size of a constant C, and neglecting a sample point which is away from the origin and exceeds a set value in the construction process;
3) and finally calculating to obtain the radius of the hypersphere and the center of the hypersphere through continuous fitting, wherein the fitting process is represented by the following formula:
Figure FDA0002187298320000051
8. the cloud service platform anomaly detection system according to claim 5, wherein the system metric data in the collection module includes CPU, memory, disk, and network related system metric data.
CN201910820118.1A 2019-08-31 2019-08-31 Cloud service platform anomaly detection method and system Pending CN110825545A (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN201910820118.1A CN110825545A (en) 2019-08-31 2019-08-31 Cloud service platform anomaly detection method and system

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN201910820118.1A CN110825545A (en) 2019-08-31 2019-08-31 Cloud service platform anomaly detection method and system

Publications (1)

Publication Number Publication Date
CN110825545A true CN110825545A (en) 2020-02-21

Family

ID=69547898

Family Applications (1)

Application Number Title Priority Date Filing Date
CN201910820118.1A Pending CN110825545A (en) 2019-08-31 2019-08-31 Cloud service platform anomaly detection method and system

Country Status (1)

Country Link
CN (1) CN110825545A (en)

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111369339A (en) * 2020-03-02 2020-07-03 深圳索信达数据技术有限公司 Over-sampling improved svdd-based bank client transaction behavior abnormity identification method
CN112487363A (en) * 2020-12-03 2021-03-12 上海卫星工程研究所 Method and system for detecting satellite telemetry consistency based on MMD analysis
CN112783682A (en) * 2021-02-01 2021-05-11 福建多多云科技有限公司 Abnormal automatic repairing method based on cloud mobile phone service
WO2021139249A1 (en) * 2020-05-28 2021-07-15 平安科技(深圳)有限公司 Data anomaly detection method, apparatus and device, and storage medium
CN113532866A (en) * 2020-04-16 2021-10-22 中国船舶重工集团公司第七一一研究所 Diesel engine abnormal state detection method and system and computer storage medium
CN115563622A (en) * 2022-09-29 2023-01-03 国网山西省电力公司 Method, device and system for detecting operating environment

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101980480A (en) * 2010-11-04 2011-02-23 西安电子科技大学 Semi-supervised anomaly intrusion detection method
CN106951776A (en) * 2017-01-18 2017-07-14 中国船舶重工集团公司第七0九研究所 A kind of Host Anomaly Detection method and system

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN101980480A (en) * 2010-11-04 2011-02-23 西安电子科技大学 Semi-supervised anomaly intrusion detection method
CN106951776A (en) * 2017-01-18 2017-07-14 中国船舶重工集团公司第七0九研究所 A kind of Host Anomaly Detection method and system

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
PING LOU: "《An Anomaly Detection Method for Cloud Service Platform》" *

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111369339A (en) * 2020-03-02 2020-07-03 深圳索信达数据技术有限公司 Over-sampling improved svdd-based bank client transaction behavior abnormity identification method
CN113532866A (en) * 2020-04-16 2021-10-22 中国船舶重工集团公司第七一一研究所 Diesel engine abnormal state detection method and system and computer storage medium
WO2021139249A1 (en) * 2020-05-28 2021-07-15 平安科技(深圳)有限公司 Data anomaly detection method, apparatus and device, and storage medium
CN112487363A (en) * 2020-12-03 2021-03-12 上海卫星工程研究所 Method and system for detecting satellite telemetry consistency based on MMD analysis
CN112783682A (en) * 2021-02-01 2021-05-11 福建多多云科技有限公司 Abnormal automatic repairing method based on cloud mobile phone service
CN112783682B (en) * 2021-02-01 2022-02-22 福建多多云科技有限公司 Abnormal automatic repairing method based on cloud mobile phone service
CN115563622A (en) * 2022-09-29 2023-01-03 国网山西省电力公司 Method, device and system for detecting operating environment
CN115563622B (en) * 2022-09-29 2024-03-12 国网山西省电力公司 Method, device and system for detecting operation environment

Similar Documents

Publication Publication Date Title
CN110825545A (en) Cloud service platform anomaly detection method and system
CN112769796B (en) Cloud network side collaborative defense method and system based on end side edge computing
CN110784481B (en) DDoS detection method and system based on neural network in SDN network
US11570070B2 (en) Network device classification apparatus and process
US7724784B2 (en) System and method for classifying data streams using high-order models
CN109067586B (en) DDoS attack detection method and device
US11516240B2 (en) Detection of anomalies associated with fraudulent access to a service platform
EP3465515B1 (en) Classifying transactions at network accessible storage
Zhe et al. DoS attack detection model of smart grid based on machine learning method
Garg et al. HyClass: Hybrid classification model for anomaly detection in cloud environment
CN112306820B (en) Log operation and maintenance root cause analysis method and device, electronic equipment and storage medium
CN117081858B (en) Intrusion behavior detection method, system, equipment and medium based on multi-decision tree
CN111224984B (en) Snort improvement method based on data mining algorithm
CN109902754A (en) A kind of efficiently semi-supervised multi-level intrusion detection method and system
Xiao et al. A traffic classification method with spectral clustering in SDN
Brandao et al. Log Files Analysis for Network Intrusion Detection
CN116155581A (en) Network intrusion detection method and device based on graph neural network
CN108989083B (en) Fault detection performance optimization method based on hybrid strategy in cloud environment
Rassam et al. One-class principal component classifier for anomaly detection in wireless sensor network
Nalavade et al. Evaluation of k-means clustering for effective intrusion detection and prevention in massive network traffic data
CN112888008A (en) Base station abnormity detection method, device, equipment and storage medium
CN112422546A (en) Network anomaly detection method based on variable neighborhood algorithm and fuzzy clustering
CN112055007A (en) Software and hardware combined threat situation perception method based on programmable nodes
Atli et al. Network intrusion detection using flow statistics
CN111475380B (en) Log analysis method and device

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
RJ01 Rejection of invention patent application after publication

Application publication date: 20200221

RJ01 Rejection of invention patent application after publication