CN115905845A - Data center anomaly detection method, system, equipment and storage medium - Google Patents
Data center anomaly detection method, system, equipment and storage medium Download PDFInfo
- Publication number
- CN115905845A CN115905845A CN202211280408.XA CN202211280408A CN115905845A CN 115905845 A CN115905845 A CN 115905845A CN 202211280408 A CN202211280408 A CN 202211280408A CN 115905845 A CN115905845 A CN 115905845A
- Authority
- CN
- China
- Prior art keywords
- data
- training
- dimensional
- monitoring data
- encoder
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Debugging And Monitoring (AREA)
Abstract
The application relates to a data center anomaly detection method, system, equipment and storage medium, which comprises the steps of acquiring monitoring data; reconstructing high-dimensional data into a group of low-dimensional data which are linearly independent of each dimension by using the acquired monitoring data based on linear transformation; training the low-dimensional data obtained after dimensionality reduction based on the constructed self-encoder and outputting a predicted value; comparing the predicted value obtained after training with the obtained real value to detect the abnormality; monitoring data obtained through principal component analysis is used, high-dimensional monitoring data are projected to be low-dimensional effective feature space, a self-encoder is trained through normal samples, whether equipment of a data center is in an abnormal state or not can be rapidly predicted at a certain time, after an abnormality occurs, a predicted value is obtained through a support vector machine of a Gaussian kernel function, the abnormality is classified, accordingly, the efficiency of abnormality processing is improved, preventive maintenance of the data center is enhanced, and the effect that operation and maintenance are more intelligent is achieved by reducing the calculation cost.
Description
Technical Field
The present application relates to the field of communications technologies, and in particular, to a method, a system, and a computer device for detecting an anomaly in a data center.
Background
With the advent of the big data era, cloud services and big data are popularized in internet enterprises, the scale of a data center is continuously enlarged, the operation and maintenance management problems of the data center are increasingly remarkable, and the operation and maintenance capacity cannot follow the construction speed of the data center.
In the actual monitoring of the data center, many pieces of information related to the equipment mainly include GPU information, disk information, memory information, and the like. In addition, the physical environment of the data center, such as temperature, humidity, temperature difference, etc., is within the monitoring range of the data center. However, in the traditional operation and maintenance mode, the manual and physical sensors are used for monitoring and risk analysis, and due to the limitation of understanding logic of people to the IT, the traditional operation and maintenance mode can only be limited to completing a calculation task to design an IT system, so that the defect of incomplete logic is caused, and the problems of low monitoring efficiency and high cost exist.
Disclosure of Invention
Based on the above, the application provides a data center anomaly detection method, a data center anomaly detection system and computer equipment, so that the effects of improving the monitoring efficiency of the data center, improving the anomaly processing efficiency, strengthening the preventive maintenance of the data center and reducing the calculation cost to enable the operation and maintenance to be more intelligent based on the technical effect of a machine learning method are achieved.
In a first aspect, the present application provides a method for detecting an anomaly in a data center, where the method includes: acquiring monitoring data; reconstructing high-dimensional data into a group of low-dimensional data which are linearly independent of each dimension by using the acquired monitoring data based on linear transformation; training the low-dimensional data obtained after dimensionality reduction based on the constructed self-encoder and outputting a predicted value; and comparing the predicted value obtained after training with the obtained real value to detect the abnormality.
Optionally, the acquiring the monitoring data includes: obtaining cpu information X CPU And memory information X mem Magnetic disk information X disk Physical environment information X phy And process information X thread ；
Based on the obtained high-dimensional data of the monitoring data for principal component analysis, and described by formula (1),
X＝(X cpu ,X mem ,X disk ,X phy ,X thread ) (1)。
optionally, the reconstructing the acquired monitoring data into a set of low-dimensional data linearly independent of each dimension based on linear transformation includes: performing characteristic matrix on the acquired monitoring data and transposing the acquired monitoring data, wherein the characteristic matrix is described by a formula (2),
in the formula, X train Is the sample characteristic of the pretreated X;
performing feature scaling on the transposed monitoring data, described by equation (3),
in the formula, mu i Is x i The average value of (a) of (b),
and obtaining the ith column and the jth row of data in the monitoring data characteristic matrix.
The covariance matrix is calculated, and the eigenvalue and the eigenvector are calculated, which is described by formula (4),
taking the first k eigenvectors as a dimensionality reduction matrix xi to finally obtainReduced dimension data matrix
Described by the formula (5) below,
and each column of the matrix after dimension reduction is represented by a characteristic after dimension reduction of data to obtain a normal sample.
Optionally, the reconstructing the high-dimensional data into a set of low-dimensional data linearly independent of each dimension based on linear transformation on the acquired monitoring data further includes: and projecting a small amount of abnormal data to the normal sample obtained after dimensionality reduction to obtain a dimensionality reduction table of the abnormal data.
Optionally, the training the dimension-reduced low-dimensional data based on the constructed self-encoder to output a prediction value comprises: calculating a reconstruction error based on the normal sample obtained after dimensionality reduction; calculating Gaussian distribution probability based on half of the normal sample obtained after dimensionality reduction and calculating weighted logarithm probability based on the other half of the normal sample; calculating the reconstruction error calculated by the normal sample, the calculated Gaussian distribution probability and the weighted logarithm probability by the following formula to obtain a loss function to realize the training of the normal sample, wherein the loss function is described by the formula (6),
in the formula, x (i) The original characteristics obtained after the dimensionality reduction are referred to,
refers to the encoder generated reconstruction feature, p (x) (i) |N(μ,σ 2 ) Refers to the probability of obeying a gaussian distribution.
Optionally, the constructing-based self-encoder trains the reduced-dimension data to output a prediction value, further comprising: and training the abnormal data subjected to dimensionality reduction based on the constructed self-encoder, and introducing a support vector machine of a Gaussian kernel function to perform classification analysis on the abnormal data.
Optionally, the constructing-based self-encoder trains the dimensionality-reduced abnormal data and introduces a support vector machine of a gaussian kernel function to perform classification analysis on the abnormal data, and specifically includes: based on the obtained abnormal data which are linearly inseparable, adopting a nonlinear Gaussian kernel function to enable the abnormal data to be linearly separable in a high-dimensional space; calculating the classified abnormal data prediction value through a boundary function based on the abnormal data processed through the Gaussian kernel function, and describing through a formula (7),
where f (x) is the boundary function of the classification, k (x) i X) is a Gaussian kernel function, alpha, y, b are hyperparameters, x i For the anomalous data samples, σ is a manually adjusted parameter.
In a second aspect, the present application provides an industrial control network malicious code detection system, including: the acquisition module is used for acquiring monitoring data; the dimensionality reduction module is used for reconstructing high-dimensional data into a group of low-dimensional data which are linearly independent of each dimensionality based on linear transformation of the acquired monitoring data; the training module is used for training the low-dimensional data obtained after dimensionality reduction based on the constructed self-encoder and outputting a predicted value; and the result output module is used for comparing the predicted value obtained after training with the obtained real value so as to carry out anomaly detection.
In a third aspect, the present application further provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method when executing the computer program.
In a fourth aspect, the present application also provides a computer-readable storage medium having stored thereon a computer program which, when executed by a processor, performs the steps of the method described above.
The application has at least the following advantages:
according to the technical content provided by the embodiment of the application, monitoring data obtained by principal component analysis is used, dimension reduction is carried out on a large amount of monitoring data, high-dimensional monitoring data is projected to be a low-dimensional effective characteristic space, and low-dimensional data obtained after projection is used as the characteristic of data center monitoring data. Because the number of the normal samples and the number of the abnormal samples are unbalanced, only the normal samples are used for training an auto-encoder, meanwhile, the Gaussian distribution of the normal samples is calculated, the trained auto-encoder outputs a predicted value, and the predicted value is compared with a real value, so that whether equipment of a data center is in an abnormal state or not at a certain time can be rapidly predicted. After the abnormity occurs, in order to determine the type of the abnormity and the measures which can be taken, a support vector machine of the Gaussian kernel function is used for obtaining a predicted value to classify the abnormity, so that the abnormity processing efficiency is improved, the preventive maintenance of the data center is enhanced, the calculation cost is reduced, and the operation and maintenance are more intelligent.
Drawings
FIG. 1 is a diagram illustrating an exemplary implementation of a data center anomaly detection method;
FIG. 2 is a schematic flow chart diagram illustrating a method for data center anomaly detection in one embodiment;
FIG. 3 is a block flow diagram that illustrates a method for data center anomaly detection in one embodiment;
FIG. 4 is a schematic flow chart showing step 203 in one embodiment;
FIG. 5 is a schematic diagram of an exemplary embodiment of a self-encoder;
FIG. 6 is a schematic block diagram illustrating a computer device in one embodiment.
Detailed Description
The present application will be described in further detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.
For ease of understanding, the system to which the present application is applied will first be described. The data center anomaly detection method provided by the application can be applied to a system architecture shown in fig. 1. The system comprises: a user space file server 103 and a terminal device 101, wherein the terminal device 101 communicates with the user space file server 103 through a network. The user space file server 103 may be a file server based on NFSv3\ v4 protocol, operating in Linux environment, and NFS (network file system) is a network abstraction over a file system, which may allow a remote client operating on the terminal device 101 to access through a network in a similar manner to a local file system. The terminal device 101 may be, but is not limited to, various personal computers, notebook computers, smart phones, tablet computers, etc., and the user space file server 103 may be implemented by an independent server or a server cluster composed of a plurality of servers.
Fig. 2 is a schematic flowchart of a data center anomaly detection method according to an embodiment of the present application, where the method may be executed by a user space file server in the system shown in fig. 1. As shown in fig. 2 and 3, the method may include the following steps:
step 201: acquiring monitoring data;
in this embodiment, it should be noted that much information related to the equipment in the actual monitoring of the data center mainly includes cpu information X CPU And memory information X mem Disk information X disk And process information X thread And so on. Besides, the system also comprises physical environment information X of the data center phy Information such as temperature, humidity, temperature differential, etc. is also within the scope of the data center monitoring. And a large amount of monitored data is acquired as sample data to provide data support for the subsequent detection training.
Step 203: and reconstructing high-dimensional data into a group of low-dimensional data which are linearly independent of each dimension by using the acquired monitoring data based on linear transformation.
In this embodiment, it should be noted that a principal component analysis method is used to reconstruct the high-dimensional data into the low-dimensional data, and the principal idea is to reconstruct the high-dimensional data into a set of low-dimensional data linearly independent of each dimension through linear transformation, extract the main features of the data, and complete the dimensionality reduction of the data. Through projecting high-dimensional monitoring data into effective characteristic space of low dimension, the low-dimensional data that obtains after the projection is regarded as the characteristic of data center monitoring data to carry out follow-up operation.
Step 205: training the low-dimensional data obtained after dimensionality reduction based on the constructed self-encoder and outputting a predicted value;
in this embodiment, it should be noted that an Auto Encoder (AE) is a kind of Artificial Neural Networks (ans) used in semi-supervised learning and unsupervised learning, and functions to perform representation learning (representation learning) on input information by using the input information as a learning target. The self-encoder has a function of characterizing a learning algorithm in a general sense, and is applied to dimension reduction (dimensional reduction) and outlier detection (anomally detection). Because the number of normal samples and the number of abnormal samples in the calculated sample data are unbalanced, only one self-encoder is trained by using the normal samples, and a predicted value is obtained through training, so that whether equipment of the data center is in an abnormal state at a certain time can be rapidly predicted, and the preventive maintenance of the data center is enhanced.
Step 207: comparing the predicted value obtained after training with the obtained real value to carry out anomaly detection;
in this embodiment, it should be noted that dimension reduction is performed on a large amount of acquired sample data, analysis and operation are performed on the normal data after dimension reduction through a self-encoder, a trained self-encoder obtains a predicted value, whether the normal data is in an abnormal state is judged through comparison between a real value and the predicted value obtained according to the training, if the real value falls within a predicted value range or is equal to the predicted value, the normal data is indicated, otherwise, the abnormal data is indicated, and whether equipment in a data center is in an abnormal state at a certain time can be quickly predicted.
Referring to fig. 2, in some embodiments, in step 201, acquiring the monitoring data specifically includes: obtaining cpu information X CPU And memory information X mem Disk information X disk Physical environment information X phy And process information X thread 。
In this embodiment, it should be further noted that, among them, the cpu information X CPU Including process percent occupied time, kernel percent occupied, percent idle time, percent wait for I/O operations, percent interrupt time, etc., are described by the following equations,
X cpu ＝(cpu_thread,cpu_core,cpu_free,cpu_io,cpu_interrupt.....)；
memory information X mem Including free memory amount, write memory rate, memory read rate, memory access rate, virtual memory size usage, etc., as described by the following equations,
X mem ＝(mem_free,mem_read,mem_write,mem_visit,mem_virtual.....)；
disk information X disk Including disk IO throughput, disk read rate, disk write rate, disk global usage, etc., as described by the following equations,
X disk ＝(disk_io,disk_wirte,disk_read,......disk_visit)；
physical environment information X phy Including temperature, humidity, fan speed, circuit voltage, etc., is described by the following equations,
X phy ＝(hum,tmp,rot_spd,......,cir_voltage)；
process information X thread Including occupying physical memory, processes occupying virtual memory, processes sharing memory, processes occupying cpu run time, etc., as described by the following formulas,
X thread ＝(thread_pyhmem,thread_virmem,......,thread_cpu_time)；
based on the obtained high-dimensional data for principal component analysis of the above-mentioned monitoring data and described by formula (1),
X＝(X cpu ,X mem ,X disk ,X phy ,X thread ) (1)。
through the series of classification processing on the acquired monitoring data, high-dimensional data are obtained, and the high-dimensional data are conveniently processed and reduced into low-dimensional data for computational analysis.
Referring to fig. 2 and 5, in some embodiments, in step 203, the obtained monitoring data is reconstructed into a set of low-dimensional data with linearly independent dimensions based on a linear transformation, including, specifically:
step 2031: performing characteristic matrix on the acquired monitoring data and transposing the acquired monitoring data, wherein the characteristic matrix is described by a formula (2),
in the formula, X train Is the sample characteristic of X after pretreatment, X [1] 、X [2] Etc. with the information X mentioned hereinbefore CPU 、X mem And correspond to each other.
Step 2032: performing feature scaling on the transposed monitoring data, described by equation (3),
in the formula, mu i Is x i The average value of (a) is calculated,
and obtaining the ith column and jth row of data in the monitoring data characteristic matrix, such as the first data in a cpu information column.
Step 2033: the covariance matrix is calculated, and the eigenvalue and the eigenvector are calculated, which is described by formula (4),
step 2034: taking the first k eigenvectors as a dimensionality reduction matrix xi to finally obtain a dimensionality reduced data matrix
Described by the formula (5) below,
each column in the data matrix after dimension reduction is a feature representation of corresponding information after dimension reduction of data, such as cpu feature, gpu feature, disk feature and the like, and the matrix formed by all columns is a feature representation of a normal sample, that is, each column is a feature representation of corresponding information after dimension reduction of data, and each column is a feature representation of a normal sample
Is characteristic of a normal sample.
In this embodiment, it should be noted that the obtained high-dimensional monitoring data is projected to a low-dimensional effective feature space through principal component analysis, and the low-dimensional data obtained after projection is used as the feature of the data center monitoring data, so as to provide data support for obtaining a predicted value through subsequent learning training.
Referring to fig. 2 and 3, in some embodiments, the step 203 reconstructs high-dimensional data into a set of low-dimensional data with linearly independent dimensions based on linear transformation of the acquired monitoring data, and further includes:
and projecting a small amount of abnormal data to the normal sample obtained after dimension reduction to obtain the dimension reduction representation of the abnormal data.
In this embodiment, it should be noted that, by projecting the abnormal data into the normal sample obtained after dimensionality reduction, sample data is enriched conveniently for subsequent analysis and calculation.
Referring to fig. 2 and 4, in some embodiments, the training the reduced-dimension data based on the constructed self-encoder to output a prediction value, step 205, includes: calculating a reconstruction error based on the normal sample obtained after dimensionality reduction; calculating Gaussian distribution probability based on half of the normal samples obtained after dimensionality reduction and calculating weighted logarithmic probability of the other half of the normal samples; calculating the reconstruction error calculated by the normal sample and the calculated Gaussian distribution probability and weighted logarithm probability by the following formulas to obtain a loss function to realize the training of the normal sample, wherein the loss function is described by the formula (6),
in the formula, x (i) The original characteristics obtained after the dimension reduction are pointed out,
refers to the encoder generated reconstruction feature, p (x) (i) |N(μ,σ 2 ) Refers to the probability of obeying a gaussian distribution, where the black areas within the diamond shaped areas shown in fig. 4 represent the gaussian distribution of normal samples and the white areas are a mixture of abnormal samples and irrelevant data distributions.
In this embodiment, it should be noted that the self-encoder includes an encoder and a decoder, the encoder and the decoder are of a four-layer symmetric structure, except for calculating the reconstruction error, half of the normal samples are used to calculate the gaussian distribution, and the other half of the normal samples are used to calculate the weighted logarithmic probability, so as to obtain the loss function. A loss function (loss function) or cost function (cost function) is a function that maps a random event or its associated random variable values to non-negative real numbers to represent the "risk" or "loss" of the random event. In application, the loss function is usually associated with the optimization problem as a learning criterion, i.e. the model is solved and evaluated by minimizing the loss function. For example, in statistics and machine learning, are used for parameter estimation (parametric estimation) of models. The output prediction value is calculated through the loss function, so that the reference standard is subsequently taken for detecting abnormal data of the data center, the abnormal processing efficiency is improved, and the preventive maintenance of the data center is enhanced.
In some embodiments, step 205, training the reduced-dimension data based on the constructed self-encoder to output a prediction value, further comprises training the reduced-dimension abnormal data based on the constructed self-encoder and performing classification analysis on the abnormal data by using a support vector machine introducing a gaussian kernel function.
In this embodiment, it should be noted that the gaussian kernel Function is also called Radial Basis Function (RBF), which is a scalar Function symmetric along a Radial direction. Generally, it is defined as a monotonic function of the euclidean distance between any point x and a center xc in space, which can be written as k (| | x-xc |), and its function is usually to calculate the similarity. The classified abnormal data predicted value is calculated by introducing the Gaussian kernel function so as to be compared with the true value, the problem about what the specific data occurs can be rapidly classified, the abnormal processing efficiency is improved, and the preventive maintenance of the data center is enhanced.
In some embodiments, training the dimensionality-reduced abnormal data based on the constructed self-encoder and performing classification analysis on the abnormal data by using a support vector machine with a gaussian kernel function, specifically including: based on the obtained abnormal data which are linearly inseparable, adopting a nonlinear Gaussian kernel function to enable the abnormal data to be linearly separable in a high-dimensional space; calculating the classified abnormal data prediction value through a boundary function based on the abnormal data processed through the Gaussian kernel function, and describing through a formula (7),
wherein f (x) is the boundary of the classificationFunction, k (x) i X) is a Gaussian kernel function, alpha, y, b are hyper-parameters, x i For the outlier data sample, σ is a manually adjusted parameter.
In this embodiment, it should be noted that the abnormal data includes that the cpu occupancy rate is too high, the disk read-write speed is too slow, the memory access rate is too slow, and the voltage is abnormal. The formula is used for classifying and predicting abnormal data, and the abnormal data is linear inseparable, so that the data is linearly separable in a high-dimensional space by adopting a nonlinear Gaussian kernel function, and the introduction of complex calculated amount is avoided.
The above flows of each step mainly use the monitoring data obtained by principal component analysis, reduce the dimension of a large amount of monitoring data, project the high-dimension monitoring data into effective low-dimension feature space, and use the low-dimension data obtained after projection as the feature of the monitoring data of the data center. Because the number of the normal samples and the number of the abnormal samples are unbalanced, only the normal samples are used for training an auto-encoder, meanwhile, the Gaussian distribution of the normal samples is calculated, the trained auto-encoder outputs a predicted value, and the predicted value is compared with a real value, so that whether equipment of a data center is in an abnormal state or not at a certain time can be rapidly predicted. After the abnormity occurs, in order to determine the type of the abnormity and the measures which can be taken, a support vector machine of a Gaussian kernel function is used for obtaining a predicted value to classify the abnormity, so that the abnormity processing efficiency is improved, and the preventive maintenance of a data center is enhanced.
The present application provides a system for detecting malicious codes in an industrial control network, where the system may include: the device comprises an acquisition module, a dimension reduction module, a training module and a result output module. The main functions of each component module are as follows:
the acquisition module is used for acquiring monitoring data;
the dimensionality reduction module is used for reconstructing high-dimensional data into a group of low-dimensional data which are linearly independent of each dimensionality based on linear transformation of the acquired monitoring data;
the training module is used for training the low-dimensional data obtained after dimensionality reduction based on the constructed self-encoder and outputting a predicted value;
and the result output module is used for comparing the predicted value obtained after training with the obtained true value so as to carry out anomaly detection.
According to an embodiment of the present application, a computer device and a computer-readable storage medium are also provided.
As shown in fig. 6, a block diagram of a computer device according to an embodiment of the present application is shown. Computer apparatus is intended to represent various forms of digital computers or mobile devices. Which may include desktop computers, laptop computers, workstations, personal digital assistants, servers, mainframe computers, and other suitable computers. The mobile device may include a tablet, smartphone, wearable device, and the like.
As shown in fig. 6, the apparatus 600 includes a calculation unit 601, a ROM 602, a RAM 603, a bus 604, and an input/output (I/O) interface 605, the calculation unit 601, the ROM 602, and the RAM 603 being connected to each other via the bus 604. An input/output (I/O) interface 605 is also connected to bus 604.
The calculation unit 601 may perform various processes in the method embodiments of the present application according to computer instructions stored in a Read Only Memory (ROM) 602 or computer instructions loaded from a storage unit 608 into a Random Access Memory (RAM) 603. The computing unit 601 may be a variety of general and/or special purpose processing components having processing and computing capabilities. The computing unit 601 may include, but is not limited to, a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), various dedicated Artificial Intelligence (AI) computing chips, various computing units running machine learning model algorithms, a Digital Signal Processor (DSP), and any suitable processor, controller, microcontroller, or the like. In some embodiments, the methods provided by embodiments of the present application may be implemented as a computer software program tangibly embodied in a computer-readable storage medium, such as storage unit 608.
The RAM 603 can also store various programs and data required for operation of the device 600. Part or all of the computer program may be loaded and/or installed on the device 600 via the ROM 602 and/or the communication unit 609.
An input unit 606, an output unit 607, a storage unit 608 and a communication unit 609 in the device 600 may be connected to the I/O interface 605. The input unit 606 may be, for example, a keyboard, a mouse, a touch screen, a microphone, or the like; the output unit 607 may be, for example, a display, a speaker, an indicator lamp, or the like. The device 600 is capable of exchanging information, data, etc. with other devices via the communication unit 609.
It should be noted that the device may also include other components necessary to achieve proper operation. It may also contain only the components necessary to implement the solution of the present application and not necessarily all of the components shown in the figures.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, field Programmable Gate Arrays (FPGAs), application Specific Integrated Circuits (ASICs), application Specific Standard Products (ASSPs), system on a chip (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and/or combinations thereof.
Computer instructions for implementing the methods of the present application may be written in any combination of one or more programming languages. These computer instructions may be provided to the computing unit 601 such that the computer instructions, when executed by the computing unit 601, such as a processor, cause the steps involved in the method embodiments of the present application to be performed.
The computer-readable storage media provided herein may be tangible media that may contain, or store, computer instructions for performing various steps involved in method embodiments of the present application. The computer readable storage medium may include, but is not limited to, storage media in the form of electronic, magnetic, optical, electromagnetic, and the like.
The above-described embodiments are not intended to limit the scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions may be made in accordance with design requirements and other factors. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present application shall be included in the protection scope of the present application.
Claims (10)
1. A data center anomaly detection method is characterized by comprising the following steps:
acquiring monitoring data;
reconstructing high-dimensional data into a group of low-dimensional data which are linearly independent of each dimension by using the acquired monitoring data based on linear transformation;
training the low-dimensional data obtained after dimensionality reduction based on the constructed self-encoder and outputting a predicted value;
and comparing the predicted value obtained after training with the obtained real value to detect the abnormality.
2. The data center anomaly detection method according to claim 1, wherein said obtaining monitoring data comprises:
obtaining cpu information X CPU And memory information X mem Disk information X disk Physical environment information X phy And process information X thread ；
Based on the obtained high-dimensional data of the monitoring data for principal component analysis, and described by formula (1),
X＝(X cpu ,X mem ,X disk ,X phy ,X thread ) (1)。
3. the data center anomaly detection method according to claim 2, wherein the reconstructing the acquired monitoring data into a set of low-dimensional data with linearly independent dimensions based on linear transformation comprises:
performing characteristic matrix on the acquired monitoring data and transposing the acquired monitoring data, wherein the characteristic matrix is described by a formula (2),
in the formula, X train Is the sample characteristic of the pretreated X;
performing feature scaling on the transposed monitoring data, described by equation (3),
in the formula, mu i Is x i The average value of (a) of (b),
obtaining ith column and jth row of data in the obtained monitoring data characteristic matrix;
the covariance matrix is calculated, and the eigenvalue and the eigenvector are calculated, which is described by formula (4),
taking the first k eigenvectors as a dimensionality reduction matrix xi to finally obtain a dimensionality reduced data matrix
Described by the formula (5),
and each column of the matrix after dimension reduction is represented by a characteristic after dimension reduction of data to obtain a normal sample.
4. The data center anomaly detection method according to claim 3, wherein the reconstructing the acquired monitoring data into a set of low-dimensional data with linearly independent dimensions based on linear transformation further comprises:
and projecting a small amount of abnormal data to the normal sample obtained after dimensionality reduction to obtain dimensionality reduction representation of the abnormal data.
5. The data center anomaly detection method according to claim 3, wherein the training of the reduced-dimension data based on the constructed self-encoder to output predicted values comprises:
calculating a reconstruction error based on the normal sample obtained after dimensionality reduction;
calculating Gaussian distribution probability based on half of the normal samples obtained after dimensionality reduction and calculating weighted logarithmic probability of the other half of the normal samples;
calculating the reconstruction error calculated by the normal sample and the calculated Gaussian distribution probability and weighted logarithm probability by the following formulas to obtain a loss function to realize the training of the normal sample, wherein the loss function is described by a formula (6),
in the formula, x (i) The original characteristics obtained after the dimensionality reduction are referred to,
refers to the encoder-generated reconstruction feature, p (x) (i) |N(μ,σ 2 ) Refers to the probability of obeying a gaussian distribution.
6. The data center anomaly detection method according to claim 4, wherein the building-based self-encoder trains the reduced-dimension data to output a prediction value, further comprising:
and training the abnormal data subjected to dimensionality reduction based on the constructed self-encoder, and introducing a support vector machine of a Gaussian kernel function to perform classification analysis on the abnormal data.
7. The data center abnormality detection method according to claim 6, wherein the training of the dimensionality-reduced abnormal data based on the constructed self-encoder and the classification analysis of the abnormal data by using a support vector machine with a gaussian kernel function are performed, specifically comprising:
based on the obtained abnormal data is linearly inseparable, a nonlinear Gaussian kernel function is adopted to enable the abnormal data to be linearly separable in a high-dimensional space;
calculating the classified abnormal data prediction value through a boundary function based on the abnormal data processed through the Gaussian kernel function, and describing through a formula (7),
wherein f (x) is a boundary function of the classification, k (x) i X) is a Gaussian kernel function, alpha, y, b are hyper-parameters, x i For the outlier data sample, σ is a manually adjusted parameter.
8. A data center anomaly detection system, the system comprising:
the acquisition module is used for acquiring monitoring data;
the dimensionality reduction module is used for reconstructing high-dimensional data into a group of low-dimensional data which are linearly independent of each dimensionality based on linear transformation of the acquired monitoring data;
the training module is used for training the low-dimensional data obtained after dimensionality reduction based on the constructed self-encoder and outputting a predicted value;
and the result output module is used for comparing the predicted value obtained after training with the obtained true value so as to carry out anomaly detection.
9. A computer device comprising a memory, a processor and a computer program stored on the memory and executable on the processor, characterized in that the steps of the method of any of claims 1 to 7 are implemented when the computer program is executed by the processor.
10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211280408.XA CN115905845A (en) | 2022-10-19 | 2022-10-19 | Data center anomaly detection method, system, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202211280408.XA CN115905845A (en) | 2022-10-19 | 2022-10-19 | Data center anomaly detection method, system, equipment and storage medium |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN115905845A true CN115905845A (en) | 2023-04-04 |
Family
ID=86494045
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202211280408.XA Pending CN115905845A (en) | 2022-10-19 | 2022-10-19 | Data center anomaly detection method, system, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115905845A (en) |
Cited By (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117057786A (en) * | 2023-10-11 | 2023-11-14 | 中电科大数据研究院有限公司 | Intelligent operation and maintenance management method, system and storage medium for data center |
-
2022
- 2022-10-19 CN CN202211280408.XA patent/CN115905845A/en active Pending
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117057786A (en) * | 2023-10-11 | 2023-11-14 | 中电科大数据研究院有限公司 | Intelligent operation and maintenance management method, system and storage medium for data center |
| CN117057786B (en) * | 2023-10-11 | 2024-01-02 | 中电科大数据研究院有限公司 | Intelligent operation and maintenance management method, system and storage medium for data center |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Luo et al. | Online learning of interpretable word embeddings | |
| CN111581046A (en) | Data anomaly detection method and device, electronic equipment and storage medium | |
| CN108898181B (en) | Image classification model processing method and device and storage medium | |
| CN113377909B (en) | Paraphrasing analysis model training method and device, terminal equipment and storage medium | |
| US20200082213A1 (en) | Sample processing method and device | |
| CN116822651A (en) | Large model parameter fine adjustment method, device, equipment and medium based on incremental learning | |
| CN111898704B (en) | Method and device for clustering content samples | |
| Yu et al. | Wafer map defect patterns classification based on a lightweight network and data augmentation | |
| CN115905845A (en) | Data center anomaly detection method, system, equipment and storage medium | |
| CN109902731B (en) | Performance fault detection method and device based on support vector machine | |
| WO2024007867A1 (en) | Defect root cause determination method and apparatus, and storage medium | |
| CN112445914A (en) | Text classification method, device, computer equipment and medium | |
| Yoshikawa et al. | Non-linear regression for bag-of-words data via gaussian process latent variable set model | |
| CN109614581B (en) | Non-negative matrix factorization clustering method based on dual local learning | |
| Parker et al. | Nonlinear time series classification using bispectrum‐based deep convolutional neural networks | |
| CN115115920A (en) | Data training method and device | |
| CN113010687B (en) | Exercise label prediction method and device, storage medium and computer equipment | |
| CN115329872A (en) | Sensitive attribute identification method and device based on comparison learning | |
| Walkowiak et al. | Algorithm based on modified angle‐based outlier factor for open‐set classification of text documents | |
| Li et al. | Dataset complexity assessment based on cumulative maximum scaled area under Laplacian spectrum | |
| Otero et al. | EMD-based data augmentation method applied to handwriting data for the diagnosis of Essential Tremor using LSTM networks | |
| CN115700555A (en) | Model training method, prediction method, device and electronic equipment | |
| CN111242195A (en) | Model, insurance wind control model training method and device and electronic equipment | |
| CN112883188A (en) | Emotion classification method and device, electronic equipment and storage medium | |
| CN113673573B (en) | Abnormality detection method based on self-adaptive integrated random fuzzy classification |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |