CN115361307B - Data center anomaly detection method, device and related products - Google Patents
Data center anomaly detection method, device and related products Download PDFInfo
- Publication number
- CN115361307B CN115361307B CN202210982852.XA CN202210982852A CN115361307B CN 115361307 B CN115361307 B CN 115361307B CN 202210982852 A CN202210982852 A CN 202210982852A CN 115361307 B CN115361307 B CN 115361307B
- Authority
- CN
- China
- Prior art keywords
- sequence
- key performance
- data center
- performance index
- network
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000001514 detection method Methods 0.000 title claims abstract description 63
- 238000000034 method Methods 0.000 claims abstract description 42
- 230000002159 abnormal effect Effects 0.000 claims abstract description 16
- 238000013507 mapping Methods 0.000 claims description 12
- 238000000605 extraction Methods 0.000 claims description 7
- 210000002569 neuron Anatomy 0.000 claims description 4
- 230000005856 abnormality Effects 0.000 abstract description 17
- 230000004044 response Effects 0.000 description 9
- 230000006870 function Effects 0.000 description 7
- 238000012549 training Methods 0.000 description 7
- 238000009826 distribution Methods 0.000 description 6
- 239000000284 extract Substances 0.000 description 6
- 238000010586 diagram Methods 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 3
- 238000012545 processing Methods 0.000 description 3
- 230000011218 segmentation Effects 0.000 description 3
- 230000004913 activation Effects 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 2
- 210000004027 cell Anatomy 0.000 description 2
- 230000008859 change Effects 0.000 description 2
- 238000007781 pre-processing Methods 0.000 description 2
- 238000013473 artificial intelligence Methods 0.000 description 1
- 239000000969 carrier Substances 0.000 description 1
- 238000004891 communication Methods 0.000 description 1
- 125000004122 cyclic group Chemical group 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 238000005065 mining Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 230000008569 process Effects 0.000 description 1
- 230000000750 progressive effect Effects 0.000 description 1
- 238000005070 sampling Methods 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/16—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks using machine learning or artificial intelligence
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/50—Network service management, e.g. ensuring proper service fulfilment according to agreements
- H04L41/5003—Managing SLA; Interaction between SLA and QoS
- H04L41/5009—Determining service level performance parameters or violations of service level contracts, e.g. violations of agreed response time or mean time between failures [MTBF]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/16—Threshold monitoring
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Artificial Intelligence (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- Medical Informatics (AREA)
- Software Systems (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
- Debugging And Monitoring (AREA)
Abstract
The application provides a data center abnormality detection method, a data center abnormality detection device and related products, which can be applied to the financial field or other fields. The method comprises the steps of obtaining a key performance index sequence, extracting the characteristics of the key performance index sequence through a preset network capable of automatically extracting the hidden characteristics of data, and generating an index characteristic sequence. And generating a reconstructed sequence by decoding and reconstructing the index feature sequence. Judging whether the key performance index is abnormal or not according to the reconstruction sequence. Therefore, the effective characteristics of the complex and various key performance indexes are extracted through the preset network capable of automatically extracting the hidden characteristics of the data, the problem that the effective characteristics of the complex key performance indexes are difficult to extract by an anomaly detection algorithm based on single statistical characteristics is avoided, the accuracy of anomaly detection of the data center is improved, and the anomaly detection efficiency is improved.
Description
Technical Field
The present application relates to the field of data processing, and in particular, to a method and apparatus for detecting anomalies in a data center, and related products.
Background
With the rapid development of big data technology and cloud computing technology, data centers in the financial industry gradually become main carriers of various types of data. More and more information and services are being processed by data centers in the financial industry, IT operations are increasingly important in the management of data center resources.
In the prior art, in the abnormal detection of the time sequence data of the key performance index, an abnormal detection algorithm based on single statistical characteristics is often adopted to extract effective characteristics in the data of the key performance index. However, in an actual data center operation and maintenance scenario, the key performance indicators are various, and different probability distributions are followed according to different services. This makes it difficult for conventional anomaly detection algorithms based on single statistical features to extract valid features in key performance index data, resulting in lower accuracy in data center anomaly detection. In addition, the abnormality detection algorithm based on the single statistical feature needs to pre-assume and count the key performance index data, and the abnormality detection mode is low in efficiency.
Therefore, how to improve the accuracy of detecting anomalies in a data center and improve the efficiency of detecting anomalies is a technical problem to be solved.
Disclosure of Invention
In view of the above, the present application provides a method, an apparatus and a related product for detecting abnormal data in a data center, which aims to improve the accuracy and the efficiency of detecting abnormal data in the data center by automatically extracting the characteristics of a complex and diverse key performance index sequence.
In a first aspect, an embodiment of the present application provides a method for detecting an anomaly in a data center, where the method includes:
Acquiring a key performance index sequence; the key performance indicator sequence comprises a plurality of key performance indicator data of a data center;
Extracting the characteristics of the key performance index sequence by using a preset network to generate an index characteristic sequence, wherein the preset network is a network for automatically extracting the hidden characteristics of data;
decoding and reconstructing the index feature sequence to generate a reconstructed sequence;
and according to the reconstruction sequence, performing anomaly detection on key performance indexes.
Optionally, the preset network is a variable self-encoder, and the extracting the features of the key performance index sequence by using the preset network to generate an index feature sequence includes:
And inputting the key performance index sequence into a variation self-encoder, and automatically extracting the low-dimensional embedded features of the key performance index sequence to generate an index feature sequence.
Optionally, before the decoding and reconstructing the index feature sequence and generating the reconstructed sequence, the method further includes:
inputting the index feature sequence into a gating circulation unit network for prediction to generate a prediction sequence; the predicted sequence is the predicted sequence of the index feature sequence;
the decoding and reconstructing the index feature sequence to generate a reconstructed sequence, comprising:
The prediction sequence is decoded and reconstructed based on a decoding network of the variable self-encoder to generate a reconstructed sequence.
Optionally, before extracting the features of the key performance indicator sequence by using a preset network, the method further includes:
Normalizing the key performance index sequence to obtain a normalized key performance index sequence;
Dividing the standardized key performance index sequence based on a sliding window to obtain a continuous short window sequence;
The extracting the characteristics of the key performance index sequence by using a preset network comprises the following steps:
and inputting the continuous short window sequence into a preset network, and extracting the characteristics of the key performance index sequence.
Optionally, the performing anomaly detection of the key performance indicator sequence according to the reconstructed sequence includes:
Calculating a reconstruction error according to the reconstruction sequence and the key performance index sequence; the reconstruction error is a model anomaly detection score;
Determining that the key performance index of the data center is abnormal in response to the reconstruction error being greater than a preset error threshold;
And determining that the key performance index of the data center is normal in response to the reconstruction error being not greater than a preset threshold.
Optionally, the calculating a reconstruction error according to the reconstruction sequence and the key performance indicator sequence includes:
Presetting a mapping relation between a reconstruction error and a difference between a reconstruction sequence and a key performance index sequence;
and determining a reconstruction error based on the reconstruction sequence and the key performance index sequence according to the mapping relation.
In a second aspect, an embodiment of the present application provides an apparatus for detecting an anomaly in a data center, where the apparatus includes:
The acquisition module is used for acquiring the key performance index sequence; the key performance indicator sequence comprises a plurality of key performance indicator data of a data center;
The feature extraction module is used for extracting the features of the key performance index sequence by using a preset network to generate an index feature sequence, wherein the preset network is a network for automatically extracting the hidden features of the data;
the reconstruction module is used for decoding and reconstructing the index feature sequence to generate a reconstructed sequence;
and the detection module is used for carrying out abnormality detection on the key performance indexes according to the reconstruction sequence.
Optionally, the preset network is a variable-component self-encoder, and the feature extraction module is further configured to input a key performance index sequence into the variable-component self-encoder, and automatically extract a low-dimensional embedded feature of the key performance index sequence to generate an index feature sequence.
In a third aspect, an embodiment of the present application provides a data center anomaly detection generating apparatus, including: at least one processor and a memory communicatively coupled to the at least one processor;
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of the first aspects.
In a fourth aspect, embodiments of the present application provide a computer readable storage medium having code stored thereon, which when executed by a processor, implements the steps of the method of data center anomaly detection as in any of the first aspects.
The application provides a data center abnormality detection method, a data center abnormality detection device and related products, and the method is implemented as follows: firstly, acquiring a key performance index sequence, and extracting the characteristics of the key performance index sequence through a first preset network capable of automatically extracting the hidden characteristics of data to generate an index characteristic sequence. And then generating a reconstructed sequence by decoding and reconstructing the index feature sequence. And finally, judging whether the key performance index is abnormal or not according to the reconstruction sequence. Therefore, the effective characteristics of the complex and various key performance indexes can be extracted through the first preset network capable of automatically extracting the hidden characteristics of the data, the problem that the effective characteristics of the complex key performance indexes are difficult to extract by an anomaly detection algorithm based on single statistical characteristics is avoided, the accuracy of anomaly detection of the data center is improved, and the anomaly detection efficiency is improved.
Drawings
In order to more clearly illustrate this embodiment or the technical solutions of the prior art, the drawings that are required for the description of the embodiment or the prior art will be briefly described, it being obvious that the drawings in the following description are only some embodiments of the present application, and that other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a method for detecting anomalies in a data center according to an embodiment of the present application;
FIG. 2 is a flowchart of another method for detecting anomalies in a data center according to an embodiment of the present application;
FIG. 3 is a flowchart of a third method for detecting anomalies in a data center according to an embodiment of the present application;
FIG. 4 is a schematic diagram of a network of gated loop cells;
fig. 5 is a schematic structural diagram of an abnormality detection device for a data center according to an embodiment of the present application.
Detailed Description
As described above, currently, an anomaly detection algorithm based on a single statistical feature is often used to extract effective features from key performance index data. However, with the complexity of data center components and service systems, the types of anomalies in key performance indicator data are also becoming increasingly diverse. Such anomaly detection algorithms based on single statistical features have been difficult to satisfy in extracting valid features in complex key performance index data. This results in that a large amount of errors are extremely easily generated at the time of abnormality detection, resulting in low abnormality detection accuracy.
Based on the method, the method and the device for detecting the anomalies in the data center have the advantages that the effective characteristics of complex and diverse key performance indexes are extracted through the first preset network capable of automatically extracting the hidden characteristics of the data, the problem that the effective characteristics of the complex key performance indexes are difficult to extract by an anomaly detection algorithm based on single statistical characteristics is avoided, and therefore the accuracy and the anomaly detection efficiency of the anomaly detection in the data center are improved.
It should be noted that the method for detecting the abnormality of the data center provided by the invention can be used in the fields of artificial intelligence, blockchain, distributed, cloud computing, big data, internet of things, mobile interconnection, digital twin or finance. The foregoing is merely an example, and is not intended to limit the application fields of the method and system for detecting abnormal data in a data center provided by the present invention.
In order to make the present application better understood by those skilled in the art, the following description will clearly and completely describe the technical solutions in the embodiments of the present application with reference to the accompanying drawings, and it is apparent that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Referring to fig. 1, a flowchart of a method for detecting an anomaly of a data center is provided in an embodiment of the present application. The method is applied to a bank data center system. The method at least comprises the following steps:
s101: and acquiring a key performance index sequence.
In the embodiment of the application, a bank data center system firstly acquires a key performance index sequence. Wherein the key performance indicator sequence includes a plurality of key performance indicator data for the data center. In the embodiment of the application, the key performance indexes comprise page access flow, online user number, equipment memory utilization rate, load balancing request number, page response time and the like. In the embodiment of the present application, the key performance index sequence is a sequence formed by the key performance index data according to a preset mode. The preset mode is a mode which is preset and built in a bank data center system server.
Exemplary description: the preset sequence arrangement mode is assumed to be [ page access flow, online user number, equipment memory utilization rate, load balancing request number and page response time ]. The key performance index data acquired by the bank data center system is that the page access flow is a, the online user number is b, the equipment memory utilization rate is c, the load balancing request number is d, the page response time is e, and the key performance index sequence is [ a, b, c, d, e ].
S102: and extracting the characteristics of the key performance index sequence by using a preset network to generate an index characteristic sequence.
In the embodiment of the application, after the key performance index sequence is acquired, the characteristics of the key performance index sequence are extracted by using a preset network, and an index characteristic sequence is generated. In the embodiment of the application, the preset network is a network capable of automatically extracting the hidden characteristics of the data. In an embodiment of the present application, the default network may select the variable self-encoder.
The variational self-encoder is a kind of deep learning and probability statistics technology, and can automatically learn the deeper representation of the sequence data. Structurally, the variant self-encoder comprises an encoding network with a parameter phi and a decoding network with a parameter theta. The variable component self-encoder converts input data x into low-dimensional embedded z, then randomly samples z, and obtains reconstruction data after decodingWherein the approximate distribution q φ (z|x) is constructed based on a variational reasoning technique.
Exemplary description: assuming that q φ (z|x) follows a gaussian distribution, the log-likelihood function of the model is shown as:
The above formula can be rewritten as follows according to the Jensen inequality:
Where L (θ, φ; x) is the under-run of the log-likelihood function log p θ (x), the under-run must be maximized in order to reduce reconstruction errors. The optimization objective of the variation self-encoder is therefore to maximize the function of the variation set-down, as shown in the following equation:
in the embodiment of the application, the variation obtains the mean and the variance corresponding to the probability distribution of the input data from the encoder. In the embodiment of the application, the variation self-encoder can introduce a heavy parameter skill, gaussian noise is added on the encoding network to carry out constraint, so that the generated hidden variable is subjected to Gaussian distribution, and the problem that the direct sampling method is not conductive to the mean value and the variance corresponding to the obtained probability distribution and is difficult to update the weight is solved.
In the embodiment of the present application, the preset network may also select other networks capable of automatically extracting the hidden features of the data, such as GNNS networks.
S103: and decoding and reconstructing the index feature sequence to generate a reconstructed sequence.
In the embodiment of the application, the index feature sequence is a sequence obtained after encoding by an encoder, and after encoding is completed, the index feature sequence needs to be decoded and reconstructed to generate a reconstructed sequence.
In the embodiment of the application, the decoding and the reconstruction can be performed based on the decoding network of the variable self-encoder, and the reconstruction sequence can be generated by adopting other decoding networks for decoding and reconstructing. In the embodiment of the application, the reconstruction sequence comprises the effective characteristics of the complex key performance indexes, so that the reconstruction sequence is utilized to judge the abnormality of the data center, and the accuracy is higher.
S104: and according to the reconstruction sequence, performing anomaly detection on key performance indexes.
In the embodiment of the application, for the generated reconstruction requirement, the abnormality detection of the key performance index is required to be performed in a preset mode. In one embodiment provided by the present application, the reconstruction error may be calculated by using the reconstruction sequence and the key performance indicator sequence; and the reconstruction error is a model anomaly detection score. When the reconstruction error is greater than a preset error threshold, determining that the key performance index of the data center is abnormal; otherwise, determining that the key performance index data of the data center is normal.
In another embodiment of the present application, the reconstruction error may be determined by presetting a mapping relationship between the reconstruction error and a difference between the reconstruction sequence and the key performance indicator sequence, and according to the mapping relationship. Similarly, when the reconstruction error is greater than a preset error threshold, determining that the key performance index of the data center is abnormal; otherwise, determining that the key performance index data of the data center is normal. Those skilled in the art can clearly determine that the mapping relationship can be set according to the needs, and the preset error threshold can also be adjusted according to the needs.
According to the data center anomaly detection method provided by the embodiment of the application, firstly, a key performance index sequence is obtained, and the characteristics of the key performance index sequence are extracted through a first preset network capable of automatically extracting the hidden characteristics of data, so that an index characteristic sequence is generated. And then generating a reconstructed sequence by decoding and reconstructing the index feature sequence. And finally judging whether the key performance index is abnormal or not according to the reconstruction sequence. Therefore, the effective characteristics of the complex and various key performance indexes are extracted through the preset network capable of automatically extracting the hidden characteristics of the data, the problem that the effective characteristics of the complex key performance indexes are difficult to extract by an anomaly detection algorithm based on single statistical characteristics is avoided, the accuracy of anomaly detection of the data center is improved, and the anomaly detection efficiency is improved.
Referring to fig. 2, another flow chart of a data center anomaly detection method according to an embodiment of the present application is provided. The method is applied to a bank data center system. The method comprises the following steps:
s201: and obtaining a key performance index KPI sequence.
S202: and (5) normalizing the KPI sequence to obtain a normalized KPI sequence.
In the embodiment of the application, in order to avoid the influence of obvious differences among different values in KPI index data on an implementation result, the KPI sequence is firstly subjected to standardization processing. The treatment form is as follows:
Where μ is the mean of x, σ is the standard deviation of x, x is the raw data, and x' is the normalized value. And the standardized KPI time sequence data is used for training a preset network model, so that the training time of the model is effectively shortened, and the detection accuracy is improved.
S203: and (5) based on the sliding window segmentation standardized key performance index sequence, acquiring a continuous short window sequence.
In the embodiment of the application, since the KPI data are long-duration data, but the preset network for automatically extracting the hidden characteristics of the data is not a sequence model, training is directly performed on the original data points by adopting the preset network, and inherent important time information in the time sequence data can be ignored. The time correlation may be introduced into the preset network based on a sliding window segmentation standardized key performance index sequence. Specifically, a sliding window with a length of l is applied to the key performance indicators, for example, W t＝[xt-l+1,…,xt represents a window ending at the time t, the whole time sequence is divided into continuous short sequences, and the KPI indicator sequence can be represented as W t＝[wt-(m-1)×l,wt-(m-2)×l,...,wt, and m is a natural number set according to requirements. In this way, a sequence of consecutive windows is obtained as a sequence of KPI time datasets.
S204: and inputting the continuous short window sequence into a preset network, and extracting the characteristics of the key performance index sequence.
And inputting the obtained continuous short window sequence into a preset network, and extracting the characteristics of the key performance index sequence. Exemplary description: the continuous window sequence W t＝[wt-(m-1)×l,wt-(m-2)×l,...,wt is input into a variation self-encoder for learning, a coding network in the variation self-encoder can automatically represent deeper layers in original data, continuously learn the dynamic change rule thereof, obtain the low-dimensional embedded characteristic E t of the original sequence through training,Wherein the method comprises the steps ofRepresenting a low-dimensional embedding of the ith window in KPI sequence W t. The variation self-encoder is provided with two layers, wherein the input dimension of the first layer is 64, the output dimension of the second layer is 32, and the ReLU function with rapid convergence capability is used as an activation function.
S205: and decoding and reconstructing the index feature sequence to generate a reconstructed sequence.
S206: and according to the reconstruction sequence, performing anomaly detection on key performance indexes.
In the embodiment of the present application, steps S201, S205 and S206 are the same as steps S101, S103 and S104, and will not be discussed here.
According to the method provided by the embodiment of the application, the effective characteristics of complex and various key performance indexes can be extracted by the first preset network capable of automatically extracting the hidden characteristics of the data, and the accuracy and the efficiency of abnormality detection of the data center are improved. The time correlation is also considered, and thus, the abnormality detection accuracy can be further improved.
Referring to fig. 3, a flowchart of a third method for detecting an anomaly in a data center according to an embodiment of the present application is provided. The method is applied to a bank data center system, and at least comprises the following steps:
s301: and obtaining a key performance index KPI sequence.
S302: and (5) normalizing the KPI sequence to obtain a normalized KPI sequence.
S303: and (5) based on the sliding window segmentation standardized key performance index sequence, acquiring a continuous short window sequence.
S304: and inputting the continuous short window sequence into a variation self-encoder to generate an index characteristic sequence.
S305: and inputting the index characteristic sequence into a gating circulation unit network for prediction, and generating a prediction sequence.
In the embodiment of the application, the index feature sequence can be input into the gating circulation unit network GRU for prediction. The training speed of the network can be improved in addition to the sequence information with longer time span compared to other networks, such as long and short memory network LSTM. See fig. 4 for a detailed block diagram of the GRU.
S306: the prediction sequence is decoded and reconstructed based on a decoding network of the variable self-encoder to generate a reconstructed sequence.
S307: and according to the reconstruction sequence, performing anomaly detection on the key performance indexes.
Exemplary description: the key performance indicator sequence is W t＝[xt-l+1,…,xt, representing the window ending at time t. The continuous window sequence W t＝[wt-(m-1)×l,wt-(m-2)×l,...,wt is obtained through the processing of steps S302-S303, and m represents the number of non-overlapping windows used for GRU model prediction. The W t＝[wt-(m-1)×l,wt-(m-2)×l,...,wt input variable is learned from the encoder. The coding network can automatically extract deeper representation of the time sequence data, continuously learn the dynamic change rule thereof, obtain the low-dimensional embedded characteristic E t of the original sequence after training,Wherein the method comprises the steps ofRepresenting a low-dimensional embedding of the ith window in KPI sequence W t. And taking the encoded low-dimensional embedded E t as input of the GRU network to predict hidden characteristics.
The first m-1 low-dimensional embeddings generated by the encoded network are used as inputs to the GRU network to train the model. The post m-1 embedments are obtained through the prediction of the GRU model, and the expression form is shown as follows:
The present invention sets the window number m for GRU prediction to 12. By minimizing the prediction error of the last window during training The parameters of the GRU model are continuously optimized, and the prediction result is optimized while the optimal parameters are obtained. The invention adopts a double-layer GRU model, the output of the former layer GRU is used as the input of the latter layer GRU, and the number of neurons of the two layers is the same, and the two layers both contain 256 neurons.
Decoding the m-1 window data obtained by GRU model prediction by using a decoding network of a variation self-encoder to obtain a reconstruction sequenceThe expression form is shown as follows:
Using a mapping relation between a preset reconstruction error and the difference between the reconstruction sequence and the key performance index sequence:
and obtaining a reconstruction error. And when the reconstruction error is larger than a preset error threshold, judging the current time as abnormal.
Referring to fig. 4, a schematic diagram of a structure of a gated loop cell network is shown. As can be seen from fig. 4, the GRU includes a reset gate and an update gate. The outputs of the reset gate and the update gate are related to the input x t at the current time and the hidden state h t-1 at the previous time, and the outputs are calculated using the Sigmoid function as shown in the following equation.
Update door: z t＝Sigmoid(Wz·[ht-1,xt)
Reset gate: r t＝Sigmoid(Wr·[ht-1,xt)
Wherein W is a weight parameter.
Multiplying the output r t of the reset gate at the current moment by the hidden state h t-1 at the previous moment, reserving the state information at the previous moment if the value of an element in the reset gate is close to 1, ignoring the information at the previous moment if the value is close to 0, and then calculating the candidate hidden state by using a Tanh activation function as shown in the following formula.
The hidden state of the time step t is composed of the updated gate z t at the current time, the hidden state h t-1 at the previous time and the candidate hidden state at the current timeThe calculation results are shown in the following formula:
According to the data center anomaly detection method provided by the embodiment of the application, through the capability of automatically mining hidden features among data of the variable self-encoder and the strong time sequence prediction capability of the GRU, the problem that the KPI sequence data volume is huge, the complexity is high, the time correlation is strong, and effective features cannot be accurately extracted is effectively solved, so that the accurate detection of key performance index anomalies of the data center is further improved.
The application also provides a device for detecting the abnormality of the data center. Referring to fig. 5, a schematic structural diagram of a data center anomaly detection device 500 according to an embodiment of the present application is provided. Applied to a banking data center system, the apparatus 500 includes:
an obtaining module 501, configured to obtain a key performance indicator sequence; the key performance indicator sequence comprises a plurality of key performance indicator data of a data center;
the feature extraction module 502 is configured to extract features of the key performance indicator sequence by using a preset network, and generate an indicator feature sequence, where the preset network is a network that automatically extracts hidden features of data;
a reconstruction module 503, configured to decode and reconstruct the index feature sequence, and generate a reconstructed sequence;
and the detection module 504 is configured to perform anomaly detection on the key performance indicators according to the reconstruction sequence.
Optionally, the preset network is a variable-component self-encoder, and the feature extraction module is further configured to input a key performance index sequence into the variable-component self-encoder, and automatically extract a low-dimensional embedded feature of the key performance index sequence to generate an index feature sequence.
Optionally, the apparatus 500 further includes a prediction module, where the prediction module is configured to input the index feature sequence into a gating cyclic unit network to perform prediction, so as to generate a prediction sequence; the predicted sequence is the predicted sequence of the index feature sequence. The reconstruction module 503 is further configured to decode and reconstruct the prediction sequence based on a decoding network of the variable self-encoder, and generate a reconstructed sequence.
Optionally, the apparatus 500 further includes a data preprocessing module, where the data preprocessing module is configured to process the key performance indicator sequence in a standardized manner, so as to obtain a standardized key performance indicator sequence; dividing the standardized key performance index sequence based on a sliding window to obtain a continuous short window sequence; the sequence of consecutive short windows is a sequence of key performance time data sets. The feature extraction module 502 is further configured to input the continuous short window sequence into a preset network, and extract features of the key performance index sequence.
Optionally, the detection module 504 includes:
the calculation unit is used for calculating a reconstruction error according to the reconstruction sequence and the key performance index sequence; the reconstruction error is a model anomaly detection score;
the first response unit is used for determining that the key performance index of the data center is abnormal in response to the reconstruction error being larger than a preset error threshold;
And the second response unit is used for determining that the key performance index of the data center is normal in response to the reconstruction error being not greater than a preset threshold value.
Optionally, the calculating unit is further configured to preset a mapping relationship between the reconstruction error and a difference between the reconstruction sequence and the key performance index sequence; and determining a reconstruction error based on the reconstruction sequence and the key performance index sequence according to the mapping relation.
The embodiment of the application provides a device for detecting the abnormality of a data center. Wherein the acquisition unit 501 acquires a key performance index sequence. The feature extraction unit 502 extracts features of the key performance index sequence through a first preset network capable of automatically extracting hidden features of data, and generates an index feature sequence. The reconstruction unit 503 generates a reconstructed sequence by decoding and reconstructing the index feature sequence. The detection unit 504 determines whether the key performance indicators are abnormal according to the reconstruction sequence. Therefore, the effective characteristics of the complex and various key performance indexes can be extracted through the first preset network capable of automatically extracting the hidden characteristics of the data, the problem that the effective characteristics of the complex key performance indexes are difficult to extract by an anomaly detection algorithm based on single statistical characteristics is avoided, the accuracy of anomaly detection of the data center is improved, and the anomaly detection efficiency is improved.
The embodiment of the application also provides corresponding generating equipment and a computer storage medium, which are used for realizing the scheme provided by the embodiment of the application.
The device comprises a memory for storing instructions or code and a processor for executing the instructions or code to cause the device to perform a method of data center anomaly detection according to any one of the embodiments of the present application.
The computer storage medium has code stored therein that, when executed, causes an apparatus for executing the code to perform the method of any of the embodiments of the present application.
The "first" and "second" in the names of "first", "second" (where present) and the like in the embodiments of the present application are used for name identification only, and do not represent the first and second in sequence.
From the above description of embodiments, it will be apparent to those skilled in the art that all or part of the steps of the above described example methods may be implemented in software plus general hardware platforms. Based on such understanding, the technical solution of the present application may be embodied in the form of a software product, which may be stored in a storage medium, such as a read-only memory (ROM)/RAM, a magnetic disk, an optical disk, etc., and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network communication device such as a router) to perform the method according to the embodiments or some parts of the embodiments of the present application.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the device embodiments, since they are substantially similar to the method embodiments, the description is relatively simple, and reference is made to the description of the method embodiments for relevant points. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. Those of ordinary skill in the art will understand and implement the present invention without undue burden.
The foregoing description of the exemplary embodiments of the application is merely illustrative of the application and is not intended to limit the scope of the application.
Claims (5)
1. A method of data center anomaly detection, the method comprising:
Acquiring a key performance index sequence; the key performance indicator sequence comprises a plurality of key performance indicator data of a data center;
Dividing the key performance index sequence based on a sliding window to obtain a continuous short window sequence; the continuous short window sequence is a key performance time data set sequence;
Inputting the continuous short window sequence into a coding network of a variation self-coder to obtain a first low-dimensional embedded feature; the first low-dimensional embedded feature is obtained by encoding the variation from an encoding network of an encoder, and each element of the first low-dimensional embedded feature represents low-dimensional embedding of a corresponding window in the key performance index sequence; the coding network is a coding network comprising gaussian noise constraints;
Inputting the first low-dimensional embedded features into a GPU network to obtain second low-dimensional embedded features; the GPU network adopts a double-layer GRU model, the output of the former GRU model is the input of the latter GRU model, and the double-layer GPU model is optimized by minimizing the prediction error of the last window; the number of neurons of the double-layer GRU model is the same;
decoding and reconstructing the second low-dimensional embedded feature using a decoding network of the variational self-encoder to generate a reconstructed sequence;
Presetting a mapping relation between a reconstruction error and a sum of squares of differences between a reconstruction sequence and a key performance index sequence; determining a reconstruction error based on the reconstruction sequence and the key performance indicator sequence according to the mapping relation; if the reconstruction error is greater than a preset error threshold, determining that the key performance index of the data center is abnormal; and if the reconstruction error is not greater than a preset threshold, determining that the key performance index of the data center is normal.
2. The method of claim 1, wherein prior to extracting the features of the key performance indicator sequence using a predetermined network, the method further comprises:
and normalizing the key performance index sequence to obtain a normalized key performance index sequence.
3. An apparatus for data center anomaly detection, the apparatus comprising:
The acquisition module is used for acquiring the key performance index sequence; the key performance indicator sequence comprises a plurality of key performance indicator data of a data center;
The feature extraction module is used for inputting the continuous short window sequence into the coding network of the variation self-coder to obtain a first low-dimensional embedded feature; the first low-dimensional embedded feature is obtained by encoding the variation from an encoding network of an encoder, and each element of the first low-dimensional embedded feature represents low-dimensional embedding of a corresponding window in the key performance index sequence; the coding network is a coding network comprising gaussian noise constraints;
Inputting the first low-dimensional embedded features into a GPU network to obtain second low-dimensional embedded features; the GPU network adopts a double-layer GRU model, the output of the former GRU model is the input of the latter GRU model, and the double-layer GPU model is optimized by minimizing the prediction error of the last window; the number of neurons of the double-layer GRU model is the same;
A reconstruction module for decoding and reconstructing the second low-dimensional embedded feature using the decoding network of the variation self-encoder to generate a reconstructed sequence;
the detection module is used for presetting a mapping relation between a reconstruction error and the sum of squares of differences between the reconstruction sequence and the key performance index sequence; determining a reconstruction error based on the reconstruction sequence and the key performance indicator sequence according to the mapping relation; if the reconstruction error is greater than a preset error threshold, determining that the key performance index of the data center is abnormal; and if the reconstruction error is not greater than a preset threshold, determining that the key performance index of the data center is normal.
4. A data center anomaly detection generation apparatus comprising: at least one processor and a memory communicatively coupled to the at least one processor;
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of claim 1 or 2.
5. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a program which, when executed by a processor, implements the steps of the method of data center anomaly detection according to claim 1 or 2.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210982852.XA CN115361307B (en) | 2022-08-16 | 2022-08-16 | Data center anomaly detection method, device and related products |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210982852.XA CN115361307B (en) | 2022-08-16 | 2022-08-16 | Data center anomaly detection method, device and related products |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN115361307A CN115361307A (en) | 2022-11-18 |
| CN115361307B true CN115361307B (en) | 2024-06-28 |
Family
ID=84033803
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210982852.XA Active CN115361307B (en) | 2022-08-16 | 2022-08-16 | Data center anomaly detection method, device and related products |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN115361307B (en) |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112380098A (en) * | 2020-11-19 | 2021-02-19 | 平安科技（深圳）有限公司 | Time sequence abnormity detection method and device, computer equipment and storage medium |
| CN112416643A (en) * | 2020-11-26 | 2021-02-26 | 清华大学 | Unsupervised anomaly detection method and unsupervised anomaly detection device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114640486A (en) * | 2020-12-15 | 2022-06-17 | 华为云计算技术有限公司 | Network flow detection method and related equipment |
-
2022
- 2022-08-16 CN CN202210982852.XA patent/CN115361307B/en active Active
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN112380098A (en) * | 2020-11-19 | 2021-02-19 | 平安科技（深圳）有限公司 | Time sequence abnormity detection method and device, computer equipment and storage medium |
| CN112416643A (en) * | 2020-11-26 | 2021-02-26 | 清华大学 | Unsupervised anomaly detection method and unsupervised anomaly detection device |
Also Published As
| Publication number | Publication date |
|---|---|
| CN115361307A (en) | 2022-11-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| WO2023123941A1 (en) | Data anomaly detection method and apparatus | |
| CN111914873A (en) | Two-stage cloud server unsupervised anomaly prediction method | |
| CN107730040B (en) | RBM-based log information comprehensive feature extraction method and device for power information system | |
| CN113312447B (en) | Semi-supervised log anomaly detection method based on probability label estimation | |
| CN112291098B (en) | Network security risk prediction method and related device thereof | |
| CN114783608B (en) | Construction method of slow patient group disease risk prediction model based on graph self-encoder | |
| US20220207352A1 (en) | Methods and systems for generating recommendations for counterfactual explanations of computer alerts that are automatically detected by a machine learning algorithm | |
| CN111400492B (en) | Hierarchical feature text classification method and system based on SFM-DCNN | |
| CN116402630B (en) | Financial risk prediction method and system based on characterization learning | |
| CN113420868A (en) | Traveling salesman problem solving method and system based on deep reinforcement learning | |
| CN114332500A (en) | Image processing model training method and device, computer equipment and storage medium | |
| CN113255750B (en) | VCC vehicle attack detection method based on deep learning | |
| CN116738354B (en) | Method and system for detecting abnormal behavior of electric power Internet of things terminal | |
| CN115361307B (en) | Data center anomaly detection method, device and related products | |
| CN114418158A (en) | Cell network load index prediction method based on attention mechanism learning network | |
| CN117097541A (en) | API service attack detection method, device, equipment and storage medium | |
| CN116821339A (en) | Misuse language detection method, device and storage medium | |
| CN116910559A (en) | Index anomaly detection method for intelligent operation and maintenance application of power grid supercomputer center | |
| CN114386436B (en) | Text data analysis method, model training method, device and computer equipment | |
| CN116359738A (en) | Method, device, equipment and storage medium for monitoring health state of battery | |
| CN114707829A (en) | Target person rescission risk prediction method based on structured data linear expansion | |
| CN113821401A (en) | WT-GA-GRU model-based cloud server fault diagnosis method | |
| CN114936285B (en) | Crisis information detection method and system based on antagonistic multi-mode automatic encoder | |
| CN114254108B (en) | Method, system and medium for generating Chinese text countermeasure sample | |
| CN117708323A (en) | Power data anomaly detection method and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant |