CN117093879A - Intelligent operation management method and system for data center - Google Patents

Intelligent operation management method and system for data center Download PDF

Info

Publication number
CN117093879A
CN117093879A CN202311352020.0A CN202311352020A CN117093879A CN 117093879 A CN117093879 A CN 117093879A CN 202311352020 A CN202311352020 A CN 202311352020A CN 117093879 A CN117093879 A CN 117093879A
Authority
CN
China
Prior art keywords
data
dimension
data point
neighborhood
range
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
CN202311352020.0A
Other languages
Chinese (zh)
Other versions
CN117093879B (en
Inventor
刘杰
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Wuxi Shanghang Data Co ltd
Original Assignee
Wuxi Shanghang Data Co ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Wuxi Shanghang Data Co ltd filed Critical Wuxi Shanghang Data Co ltd
Priority to CN202311352020.0A priority Critical patent/CN117093879B/en
Publication of CN117093879A publication Critical patent/CN117093879A/en
Application granted granted Critical
Publication of CN117093879B publication Critical patent/CN117093879B/en
Active legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/22Matching criteria, e.g. proximity measures
    • GPHYSICS
    • G06COMPUTING; CALCULATING OR COUNTING
    • G06FELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00Pattern recognition
    • G06F18/20Analysing
    • G06F18/24Classification techniques
    • G06F18/243Classification techniques relating to the number of classes
    • G06F18/2433Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection

Landscapes

  • Engineering & Computer Science (AREA)
  • Data Mining & Analysis (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Biology (AREA)
  • Evolutionary Computation (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Testing And Monitoring For Control Systems (AREA)

Abstract

The invention relates to the technical field of data processing, and provides an intelligent operation management method and system for a data center, wherein the intelligent operation management method comprises the following steps: collecting data of a plurality of dimensions and a plurality of moments of a data center; acquiring an initial correction coefficient of each data point of each dimension according to the distribution of other data points in the neighborhood range of each data point of the same dimension; obtaining the confidence coefficient of each data point of each dimension according to the correlation of the data points of the same moment in different dimensions in a neighborhood range and the change of the data points of the same moment in different days of each dimension; obtaining a correction coefficient and obtaining an adaptive K value of each data point of each dimension; and acquiring local outlier factors of each data point according to the self-adaptive K value of each data point of each dimension, and performing anomaly detection to realize intelligent operation management of the data center. The invention aims to solve the problem that the K value fixed by the LOF algorithm cannot be accurately monitored abnormally due to the correlation of multidimensional data of a data center.

Description

Intelligent operation management method and system for data center
Technical Field
The invention relates to the technical field of data processing, in particular to an intelligent operation management method and system for a data center.
Background
The intelligent operation management of the data center can improve the safety and reliability of the data center, can accurately evaluate the operation condition of the data center through monitoring and analyzing various indexes, equipment states, energy consumption and environmental conditions, timely find and respond to potential problems or risks, and pertinently take measures to adjust and optimize so as to prevent faults and stop, improve the continuous operation capability and business continuity of the data center, and is one of key elements of the data center for realizing modern and efficient operation.
In the running process of the data center, multidimensional data such as temperature, humidity, power consumption, cooling water quantity, bandwidth utilization rate and the like exist; in the existing method, LOF anomaly detection is adopted to analyze data in a single dimension, so as to judge whether equipment failure or energy consumption anomaly occurs in a data center; however, in the traditional LOF anomaly detection, the K adjacent value of each data point is fixed, the data of a single dimension is changed, meanwhile, certain correlation exists between the data of different dimensions to affect each other, the fixed K value can easily cause the missing detection phenomenon of part of the anomaly data points due to overlarge K value, otherwise, the possibility of misjudgment is larger due to the overlarge K value, and too many normal data points are regarded as anomaly data points, so that the anomaly detection result is inaccurate, and the normal operation of the data center is influenced, so that the intelligent operation management of the data center cannot be realized.
Disclosure of Invention
The invention provides an intelligent operation management method and system for a data center, which aim to solve the problem that the existing multidimensional data of the data center are associated and cannot be accurately monitored abnormally through a K value fixed by an LOF algorithm, and the adopted technical scheme is as follows:
in a first aspect, an embodiment of the present invention provides a method for intelligent operation management of a data center, including the steps of:
collecting data of a plurality of dimensions and a plurality of moments of a data center to obtain a plurality of data points of each dimension;
acquiring an initial correction coefficient of each data point of each dimension according to the distribution of other data points in the neighborhood range of each data point of the same dimension;
obtaining the confidence coefficient of each data point of each dimension according to the correlation of the data points of the same moment in different dimensions in a neighborhood range and the change of the data points of the same moment in different days of each dimension; obtaining a correction coefficient through the confidence coefficient and the initial correction coefficient and obtaining a self-adaptive K value of each data point of each dimension;
and acquiring local outlier factors of each data point according to the self-adaptive K value of each data point of each dimension, and performing anomaly detection on the data of multiple dimensions of the data center through the local outlier factors.
Further, the specific acquisition method is as follows:
according to the distribution of adjacent data points in the same dimension, obtaining a neighborhood range, a fluctuation coefficient and a variation range of each data point in each dimension; first, theDimension->Data point initial correction coefficient +.>The calculation method of (1) is as follows:
wherein,indicate->Dimension->Fluctuation coefficient of data point, +.>Indicating the size of the range of variation, +.>Representing ordinal number +.>Weight value of data point of +.>Indicate->Dimension->Data values for data points, +.>Indicate->Dimension->Data values for data points, +.>Representing absolute value; the number of data points in the variation range is 5, < >>Representing ordinal number +.>The sum of the weight coefficients allocated to all the data points in the variation range is 13;
an initial correction coefficient is obtained for each data point for each dimension.
Further, the specific method for obtaining the neighborhood range, the fluctuation coefficient and the variation range of each data point of each dimension includes the following steps:
presetting a time neighborhood sizeFor->No. 5 of the individual dimension>Data point, will be->Data points at->Adjacent front->Data points and adjacent post->A data point is taken as a neighborhood range of the data point; calculating standard deviation of all data points in the data point and neighborhood range, and recording the standard deviation as the neighborhood standard deviation of the data point;
if the neighborhood standard deviation of the data point is larger than the standard deviation threshold, setting the fluctuation coefficient of the data point to be 1; if the neighborhood standard deviation of the data point is smaller than or equal to the standard deviation threshold value, setting the fluctuation coefficient of the data point to be 0;
will be the firstData points and at->Adjacent front->Data points and adjacent back->The data points constitute the range of variation of the data point, +.>The size of the preset variation range;
and acquiring a neighborhood range, a fluctuation coefficient and a variation range of each data point of each dimension.
Further, the specific method for obtaining the confidence coefficient of each data point of each dimension includes the following steps:
obtaining a left neighborhood range and a right neighborhood range of each data point in each dimension and a reference data point, a left reference neighborhood range and a right reference neighborhood range in each other dimension according to the neighborhood ranges of the data points in the same moment in different dimensions;
acquiring fitting data of each data point of each dimension according to the change of the data points of the same time on different days of each dimension; first, theDimension->Confidence of data points->The calculation method of (1) is as follows:
wherein,represents the total number of dimensions, +.>Indicate->Dimension->Data values for data points, +.>Indicate->Dimension->Fitting data of data points, +.>Indicate->No. 5 of the individual dimension>Left neighborhood of data point and data point at +.>Spearman correlation coefficient for left reference neighborhood range of individual dimensions,/->Indicate->No. 5 of the individual dimension>The right neighborhood of a data point is within +.>Spearman correlation coefficient for right reference neighborhood range of individual dimensions,/->Representing absolute value>An exponential function that is based on a natural constant;
confidence is obtained for each data point for each dimension.
Further, the method for obtaining the left neighborhood range and the right neighborhood range of each data point in each dimension, and the reference data point, the left reference neighborhood range and the right reference neighborhood range in each other dimension includes the following specific steps:
for the firstNo. 5 of the individual dimension>Data point, obtain the numberA plurality of data points in other dimensions at the moment corresponding to the data points are used as reference data points of the data points in the corresponding dimensions;
acquiring a neighborhood range of a reference data point as a reference neighborhood range of the data point in a corresponding dimension, forming a left neighborhood range of the data point by the data point before the neighborhood range, and forming a right neighborhood range of the data point by the data point after the neighborhood range; acquiring a left reference neighborhood range and a right reference neighborhood range of the data point in each other dimension according to the reference neighborhood ranges of the other dimensions;
a left neighborhood range and a right neighborhood range of each data point in each dimension are obtained, and a reference data point, a left reference neighborhood range and a right reference neighborhood range in each other dimension are obtained.
Further, the fitting data of each data point in each dimension is obtained by the following specific method:
for the firstNo. 5 of the individual dimension>Data point, the time of obtaining the data point is before +.>Day at the same timeData points of the dimensions, fitting data points as the data points, +.>Is the preset reference days;
for a pair ofFitting the fitting data points by a least square method to obtain a fitting curve, predicting the fitting curve to obtain predicted data of the data points at the corresponding moment on the corresponding days, and marking the predicted data as the +.>Dimension->Fitting data of the data points;
fitting data for each data point for each dimension is obtained.
Further, the method for obtaining the correction coefficient and obtaining the self-adaptive K value of each data point of each dimension through the confidence coefficient and the initial correction coefficient comprises the following specific steps:
carrying out linear normalization on the confidence coefficient of all data points in all dimensions, marking the obtained result as an adjustment parameter of each data point, adding 1 to the adjustment parameter of each data point to obtain a sum, and taking the product of the sum and the initial correction coefficient of the data point as the correction coefficient of each data point;
and acquiring an adaptive K value of each data point of each dimension according to the correction coefficient and a preset K value.
Further, the adaptive K value of each data point in each dimension is specifically obtained by:
presetting a K value asFirst->Dimension->Adaptive K value for data points +.>The calculation method of (1) is as follows:
wherein,representing a preset K value,/->Indicate->Dimension->Correction factors for data points, ">Representing a round up->An exponential function that is based on a natural constant;
an adaptive K value is obtained for each data point for each dimension.
Further, the method for obtaining the local outlier factor of each data point according to the self-adaptive K value of each data point of each dimension includes the following specific steps:
for any data point of any dimension, acquiring the moment corresponding to the data point, and leading the dimension to be before the momentA data point is used as the monitoring range of the data point, wherein +.>For monitoring the range size, +.>Representing the size of the temporal neighborhood; acquiring the +.f. of the data point within the monitoring range based on the adaptive K value of the data point>A distance neighborhood, wherein the distance measurement of the data point adopts Euclidean distance obtained based on time difference and data value difference, and local outlier factors of the data point are obtained through LOF algorithm;
a local outlier factor is obtained for each data point for each dimension.
In a second aspect, another embodiment of the present invention provides an intelligent operation management system for a data center, including:
the data center acquisition data module is used for acquiring data of a plurality of dimensions of the data center at a plurality of moments to obtain a plurality of data points of each dimension;
the multidimensional data analysis processing module: the method comprises the steps of obtaining initial correction coefficients of each data point of each dimension according to the distribution of other data points in the neighborhood range of each data point of the same dimension;
obtaining the confidence coefficient of each data point of each dimension according to the correlation of the data points of the same moment in different dimensions in a neighborhood range and the change of the data points of the same moment in different days of each dimension; obtaining a correction coefficient through the confidence coefficient and the initial correction coefficient and obtaining a self-adaptive K value of each data point of each dimension;
the data anomaly detection management module is used for acquiring local outlier factors of each data point according to the self-adaptive K value of each data point of each dimension, and anomaly detection is carried out on the data of multiple dimensions of the data center through the local outlier factors.
The beneficial effects of the invention are as follows: according to the invention, through obtaining the self-adaptive K values of data points with different dimensions in the data center, obtaining local anomaly factors through an LOF algorithm according to the self-adaptive K values, judging the abnormal data points through the anomaly factors, further carrying out problem risk monitoring on the data center according to the abnormal data, improving the safety and sustainable operation capacity of the data center, and realizing intelligent operation management of the data center; the correction coefficient is obtained through the initial correction coefficient and the confidence coefficient of the data point, the preset K value is adjusted through the correction coefficient to obtain the self-adaptive K value, the fact that the K value of the data point which is possibly abnormal is small and the abnormality is easy to identify is guaranteed, and meanwhile the normal data point is not misjudged due to the fact that the K value is too large; the initial correction coefficient considers the change condition of the data point on the time neighborhood, the confidence coefficient considers the correlation change of the data point under the same moment of different dimensions, the situation that the data mutation of a certain dimension caused by the associated data of different dimensions is misjudged as abnormal data is avoided, the accuracy and precision of the identification of abnormal data of the data center are further improved, and the efficiency of intelligent operation management of the data center is ensured.
Drawings
In order to more clearly illustrate the embodiments of the invention or the technical solutions of the prior art, the drawings which are used in the description of the embodiments or the prior art will be briefly described, it being obvious that the drawings in the description below are only some embodiments of the invention, and that other drawings can be obtained according to these drawings without inventive faculty for a person skilled in the art.
Fig. 1 is a schematic flow chart of an intelligent operation management method for a data center according to an embodiment of the present invention;
fig. 2 is a block diagram of an intelligent operation management system for a data center according to another embodiment of the present invention.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
Referring to fig. 1, a flowchart of a method for intelligent operation management of a data center according to an embodiment of the present invention is shown, and the method includes the following steps:
and S001, collecting data of a plurality of dimensions of the data center at a plurality of moments to obtain a plurality of data points of each dimension.
The purpose of the embodiment is to realize the intelligent operation management of the data center by carrying out instant anomaly detection on the data in each dimension of the data center, so that the data in a plurality of dimensions of the data center are required to be acquired; the method comprises the steps of collecting data in six dimensions of temperature, humidity, voltage, power consumption, cooling water quantity and bandwidth utilization rate of a data center, setting a sampling time interval to be 5 seconds, acquiring temperature and humidity data through a temperature sensor and a humidity sensor, acquiring voltage once every 5 seconds from the data center, directly acquiring the total power consumption and the total cooling water quantity of the past 5 seconds from the data center, directly acquiring the bandwidth utilization rate from the data center, starting running from the data center and collecting the data in each dimension, taking one day as a monitoring period, acquiring data in a plurality of moments in each dimension, and taking each data as a data point; and judging whether abnormal data exist in 10 minutes every 10 minutes.
So far, the data of a plurality of moments of each dimension are obtained.
Step S002, according to the distribution of other data points in the neighborhood range of each data point in the same dimension, obtaining the initial correction coefficient of each data point in each dimension.
After the multidimensional data is acquired, the correction coefficient of each data point is acquired by utilizing the data change condition of the data point in the neighborhood range at each moment; if the data suddenly changes at a certain time, it may be characterized by that the temperature, power consumption and the like at that time are abnormal or the normal data quantity at that time is large, the point with large data change amplitude is given with a high correction coefficient so as to reduce the corresponding timeA value; however, the abnormal detection is performed only by single dimension data, which can lead to the situation of data mutation caused by larger normal data quantity, so that the confidence coefficient of data points needs to be obtained to improve the correction coefficient by combining the relation among the multidimensional data, so as to improve the accuracy of the abnormal detection.
It should be further noted that, for single dimension data, if the change rate of the data in the neighborhood range is higher, the data point is abnormal if the change rate is suddenly increased or suddenly decreased at the corresponding moment, for example, for temperature data, if the temperature data is suddenly increased at a shorter moment, the temperature data may be caused by the failure of a cooling fan, the failure of a cooling system, and the likeThe reason is that at that timeThe value is corrected to a higher degree, namely, a larger correction coefficient is given to improve the detection precision.
Specifically, a time neighborhood size is presetThe present embodiment employs +.>To make a description by->No. 5 of the individual dimension>Data points are taken as examples, the +.>Data points at->The first 10 data points and the last 10 data points adjacent to each other in each dimension are used as a neighborhood range of the data point, standard deviation is calculated for all the data points in the data point and the neighborhood range (standard deviation calculation comprises the data point), the standard deviation is recorded as a neighborhood standard deviation of the data point, a fluctuation coefficient of the data point is obtained according to the neighborhood standard deviation, a standard deviation threshold value is preset, the standard deviation threshold value is described by adopting 8 in the embodiment, and if the neighborhood standard deviation of the data point is larger than the standard deviation threshold value, the fluctuation coefficient of the data point is set to be 1; if the neighborhood standard deviation of the data point is smaller than or equal to the standard deviation threshold value, setting the fluctuation coefficient of the data point to be 0; at the same time add the data point and adjacent front->Data point and adjacent backThe data points constitute the variations of the data pointsSyndrome of transformation of qi>For the preset variation range size, the present embodiment adopts +.>Description is made; then->Dimension->Data point initial correction coefficient +.>The calculation method of (1) is as follows:
wherein,indicate->Dimension->Fluctuation coefficient of data point, +.>Indicating the size of the variation range, the present embodiment uses +.>To make a description of->Representing ordinal number +.>Weight value of data point of +.>Indicate->Dimension->Data values for data points, +.>Indicate->Dimension->Data values for data points, +.>Representing absolute value; the number of data points in the variation range is 5, < >>Representing ordinal number +.>The sum of the weight coefficients assigned to all data points within the variation range is 13, namelyThe method comprises the steps of carrying out a first treatment on the surface of the Obtaining a fluctuation coefficient through the standard deviation of the data in the neighborhood range of the data point to reflect the fluctuation degree, wherein when the fluctuation degree is smaller, the data in the neighborhood range is more normal, the possibility of abnormal data points is smaller, and the obtained initial correction coefficient is 0 after judging by a preset K value; when the fluctuation degree is large, the judgment is needed to be carried out by combining the change range formed by the data points and the left and right adjacent data points, and the change of the data value of each data point and the adjacent previous data point is analyzed in the change rangeThe time distance between each data point and each data point in the variation range is combined to give weight, so that the variation rate of the data point and the adjacent data points is reflected, and the larger the variation rate is, the correction coefficient is adjusted to provide a basis for the subsequent reduction of the K value, and the problem that the normal data point is identified as an abnormal data point due to the overlarge K value is avoided.
Further, according to the method, a neighborhood range, a neighborhood standard deviation, a fluctuation coefficient and a variation range of each data point of each dimension are obtained, and an initial correction coefficient of each data point is obtained; it should be noted that, for a part of data points in each dimension of the first day when the data center starts to run, there are cases where the neighborhood range cannot be completely acquired, that is, the number of data points before the data points is insufficient to form the neighborhood range, and in this embodiment, the neighborhood range is complemented by a quadratic linear interpolation method; meanwhile, as the abnormal data points are judged every 10 minutes, the number of the data points after partial data points exist in each abnormal judgment is insufficient to form a neighborhood range, and the method of secondary linear interpolation is adopted to complement the neighborhood range.
Thus, an initial correction coefficient for each data point in each dimension is obtained.
Step S003, obtaining the confidence coefficient of each data point of each dimension according to the relativity of the data points of the same moment in different dimensions in a neighborhood range and the change of the data points of the same moment in different days of each dimension; and obtaining a correction coefficient through the confidence coefficient and the initial correction coefficient and obtaining an adaptive K value of each data point of each dimension.
It should be noted that, the calculation of the initial correction coefficient considers the situation of data change of single dimension data in the neighborhood range at different moments, but the situation that data mutation is possible to exist at some moments is not caused by equipment abnormality, for example, temperature jump is possible to be caused by the normal phenomenon that the temperature of equipment is increased due to the fact that the data processing capacity of a data center is too large at the moment, however, when the data processing capacity of the data center is too large, the phenomenon that data jump occurs with the situation that the cooling water quantity, the bandwidth utilization rate and the like are all accompanied, namely, certain correlation exists among data in different dimensions, for data points in any one dimension, the confidence coefficient of the data point is quantized through the correlation of the neighborhood range of the data point and the neighborhood range of data points in other dimensions at the same moment, then the initial correction coefficient is adjusted through the confidence coefficient, the correction coefficient of the normal data point is ensured to be sufficiently small due to the characteristic that the correlation change of the normal data point in the neighborhood range is smaller, the correction coefficient of the data point with possible abnormality is larger, and finally the self-adaptive K value of the data point is obtained.
Specifically, for the firstNo. 5 of the individual dimension>The method comprises the steps of obtaining a plurality of data points in other dimensions at the moment corresponding to the data point, taking the data points as reference data points of the data points in the corresponding dimensions, simultaneously obtaining a neighborhood range of the reference data points, taking the neighborhood range of the data points as a reference neighborhood range of the data points in the corresponding dimensions, forming a left neighborhood range of the data points by the data points in front of the neighborhood range, forming a right neighborhood range of the data points by the data points behind the neighborhood range, and similarly obtaining a left reference neighborhood range and a right reference neighborhood range of the data points in each other dimension according to the reference neighborhood ranges of the other dimensions, namely forming the left reference neighborhood range in front of the reference data points, and then forming the right reference neighborhood range; for->No. 5 of the individual dimension>Data point, the time of obtaining the data point is before +.>Day same time->Data points of the dimensions, fitting data points as the data points, +.>For a preset reference number of days, the present embodiment uses +.>Description is made; fitting 10 fitting data points by a least square method to obtain a fitting curve, predicting the fitting curve to obtain predicted data of the data point at the corresponding moment on the corresponding day, and recording the predicted data as the fitting data of the data point, namely predicting the data of the same day by the data of the same moment 10 days before, wherein the least square method and the predicting by the fitting curve are both the prior art, and the embodiment is not repeated; acquiring fitting data of each reference data point of the data point according to the method; then->Dimension->Confidence of data points->The calculation method of (1) is as follows:
wherein,represents the total number of dimensions, +.>Indicate->Dimension->Data values of data points (th->Dimension->The data point is->Dimension->Data points at->Reference data points for each dimension),>indicate->Dimension->Fitting data of data points, +.>Indicate->Dimension->Left neighborhood of data point and data point at +.>Spearman correlation coefficient for left reference neighborhood range of individual dimensions,/->Indicate->Dimension->Right neighborhood range of data points and data points in the sameFirst->The Spearman correlation coefficient of the right reference neighborhood range of each dimension, it should be noted that the Spearman correlation coefficient is a known technology, and is not described in detail in this embodiment, the sequence formed by the data values of all the data points in the left neighborhood range in sequence and the sequence formed by the data values of all the data points in the left reference neighborhood range in sequence are subjected to Spearman correlation coefficient calculation, and the right neighborhood range is the same as the right reference neighborhood range; />The representation is to take the absolute value,representing an exponential function based on natural constants, the present embodiment employs +.>Model to present inverse proportional relationship and normalization process, < ->For inputting the model, an implementer can set an inverse proportion function and a normalization function according to actual conditions;
taking the square value of the difference between the fitting data and the data value of the data points in each dimension as the weight of the correlation change, wherein the larger the difference between the fitting data and the data value is, the smaller the referenceability of the reference data points in the corresponding dimension is, and the smaller the weight is; for the correlation change, the difference of the correlation coefficients of the left and right neighborhood ranges and the reference neighborhood range is quantized, the larger the difference of the two correlation coefficients is, the larger the probability that the corresponding data point is likely to be abnormal is, the larger the correction coefficient is needed to be used for adjusting the K value, and the smaller the K value is, so that the abnormality detection is carried out; by and to all other dimensionsThe feature obtained by the correlation change of each dimension is averaged to finally obtain the confidence coefficient of the data point; acquiring each data point of each dimension according to the methodConfidence level.
Further, the confidence level of all data points in all dimensions is subjected to linear normalization, the obtained result is recorded as an adjustment parameter of each data point, and the product of the sum obtained by adding 1 to the adjustment parameter of each data point and the initial correction coefficient of the data point is used as the correction coefficient of each data point.
Further, a K value is preset asThe present embodiment employs +.>To make a description->Dimension->Adaptive K value for data points +.>The calculation method of (1) is as follows:
wherein,representing a preset K value,/->Indicate->Dimension->Correction factors for data points, ">Representing a round up->Representing an exponential function based on natural constants, the present embodiment employs +.>Model to present inverse proportional relationship and normalization process, < ->For inputting the model, an implementer can set an inverse proportion function and a normalization function according to actual conditions; the larger the correction coefficient is, the smaller the self-adaptive K value is, and the self-adaptive K value of the data point is obtained; and acquiring an adaptive K value of each data point of each dimension according to the method.
Thus, an adaptive K value for each data point for each dimension is obtained.
Step S004, local outlier factors of each data point are obtained according to the self-adaptive K value of each data point of each dimension, and anomaly detection is carried out on the data of a plurality of dimensions of the data center through the local outlier factors, so that intelligent operation management of the data center is achieved.
After the self-adaptive K value of each data point of each dimension is obtained, the moment corresponding to any data point of any dimension is obtained, and the dimension is before the momentA data point is used as the monitoring range of the data point, wherein +.>For monitoring the range size, the present embodiment uses +.>To make a description of->Representing the size of the temporal neighborhood, the present embodiment uses +.>Description is made; root of Chinese characterAcquiring the +.f. of the data point within the monitoring range based on the adaptive K value of the data point>A distance neighborhood, wherein the distance measurement of the data points adopts Euclidean distance obtained based on time difference and data value difference, namely, a coordinate system is constructed by taking time as an abscissa and a data value as an ordinate, the Euclidean distance is calculated according to the coordinates of the coordinate points by converting the data points into coordinate points in the coordinate system; further, local outlier factors of the data point are obtained by LOF algorithm, wherein +.>The distance neighborhood and local outlier factor acquisition are both known techniques of the LOF anomaly detection algorithm, and the embodiment is not described in detail; the local outlier factor of each data point of each dimension is obtained according to the method, and it should be noted that, since the data of each dimension of the data center is collected from the beginning of operation of the data center, and each day is taken as a monitoring period, for the partial data points of each dimension of the first day, it is insufficient to obtain enough data points to form a monitoring range, and LOF anomaly monitoring is performed with data points that can be actually obtained.
Further, an abnormality threshold is preset, in this embodiment, the abnormality threshold is described by 1.1, if the local abnormality factor of a data point is greater than the abnormality threshold, it is indicated that abnormality exists in the data collected by the corresponding dimension of the data point at the moment, whether abnormality exists in 10 minutes is judged every 10 minutes, equipment safety early warning is performed through a sensor of the corresponding dimension, abnormality of the data of a certain dimension caused by improper operation or equipment failure of the data center is found in time, and therefore safety and sustainable operation capability of the data center are improved, and intelligent operation management of the data center is achieved.
Therefore, the intelligent operation management of the data center is realized by carrying out anomaly detection on the data monitored in real time by the plurality of dimensions of the data center.
Referring to fig. 2, a block diagram of an intelligent operation management system for a data center according to an embodiment of the present invention is shown, where the system includes:
the data center collects data at a plurality of dimensions and a plurality of moments in time of the data center, and a plurality of data points in each dimension are obtained.
Multidimensional data analysis processing module 102:
(1) Acquiring an initial correction coefficient of each data point of each dimension according to the distribution of other data points in the neighborhood range of each data point of the same dimension;
(2) Obtaining the confidence coefficient of each data point of each dimension according to the correlation of the data points of the same moment in different dimensions in a neighborhood range and the change of the data points of the same moment in different days of each dimension; and obtaining a correction coefficient through the confidence coefficient and the initial correction coefficient and obtaining an adaptive K value of each data point of each dimension.
The data anomaly detection management module 103 obtains local outlier factors of each data point according to the self-adaptive K value of each data point of each dimension, and performs anomaly detection on the data of multiple dimensions of the data center through the local outlier factors.
The foregoing description of the preferred embodiments of the invention is not intended to be limiting, but rather is intended to cover all modifications, equivalents, alternatives, and improvements that fall within the spirit and scope of the invention.

Claims (5)

1. An intelligent operation management method for a data center is characterized by comprising the following steps:
collecting data of a plurality of dimensions and a plurality of moments of a data center to obtain a plurality of data points of each dimension;
acquiring an initial correction coefficient of each data point of each dimension according to the distribution of other data points in the neighborhood range of each data point of the same dimension;
obtaining the confidence coefficient of each data point of each dimension according to the correlation of the data points of the same moment in different dimensions in a neighborhood range and the change of the data points of the same moment in different days of each dimension; obtaining a correction coefficient through the confidence coefficient and the initial correction coefficient and obtaining a self-adaptive K value of each data point of each dimension;
acquiring local outlier factors of each data point according to the self-adaptive K value of each data point of each dimension, and performing anomaly detection on the data of a plurality of dimensions of the data center through the local outlier factors;
the specific acquisition method comprises the following steps of:
according to the distribution of adjacent data points in the same dimension, obtaining a neighborhood range, a fluctuation coefficient and a variation range of each data point in each dimension; first, theDimension->Data point initial correction coefficient +.>The calculation method of (1) is as follows:
wherein,indicate->Dimension->Fluctuation coefficient of data point, +.>Representation changeSize of the transformation range, < >>Representing ordinal number +.>Weight value of data point of +.>Indicate->Dimension->Data values for data points, +.>Indicate->Dimension->Data values for data points, +.>Representing absolute value; the number of data points in the variation range is 5, < >>Representing ordinal number +.>The sum of the weight coefficients allocated to all the data points in the variation range is 13;
acquiring an initial correction coefficient of each data point of each dimension;
the confidence coefficient of each data point of each dimension is obtained by the following specific methods:
obtaining a left neighborhood range and a right neighborhood range of each data point in each dimension and a reference data point, a left reference neighborhood range and a right reference neighborhood range in each other dimension according to the neighborhood ranges of the data points in the same moment in different dimensions;
acquiring fitting data of each data point of each dimension according to the change of the data points of the same time on different days of each dimension; first, theDimension->Confidence of data points->The calculation method of (1) is as follows:
wherein,represents the total number of dimensions, +.>Indicate->Dimension->Data values for data points, +.>Indicate->Dimension->Fitting data of data points, +.>Indicate->No. 5 of the individual dimension>Left neighborhood of data point and data point at +.>Spearman correlation coefficient for left reference neighborhood range of individual dimensions,/->Indicate->No. 5 of the individual dimension>The right neighborhood of a data point is within +.>Spearman correlation coefficient for right reference neighborhood range of individual dimensions,/->Representing absolute value>An exponential function that is based on a natural constant;
acquiring the confidence coefficient of each data point of each dimension;
the method comprises the following specific steps of: carrying out linear normalization on the confidence coefficient of all data points in all dimensions, marking the obtained result as an adjustment parameter of each data point, adding 1 to the adjustment parameter of each data point to obtain a sum, and taking the product of the sum and the initial correction coefficient of the data point as the correction coefficient of each data point;
acquiring an adaptive K value of each data point of each dimension according to the correction coefficient and a preset K value;
the specific acquisition method is as follows:
presetting a K value asFirst->Dimension->Adaptive K value for data points +.>The calculation method of (1) is as follows:
wherein,representing a preset K value,/->Indicate->Dimension->Correction factors for data points, ">Representing a round up->An exponential function that is based on a natural constant;
acquiring an adaptive K value of each data point of each dimension;
the method for obtaining the local outlier factor of each data point according to the self-adaptive K value of each data point of each dimension comprises the following specific steps:
for any data point of any dimension, acquiring the moment corresponding to the data point, and leading the dimension to be before the momentA data point is used as the monitoring range of the data point, wherein +.>For monitoring the range size, +.>Representing the size of the temporal neighborhood; acquiring the +.f. of the data point within the monitoring range based on the adaptive K value of the data point>A distance neighborhood, wherein the distance measurement of the data point adopts Euclidean distance obtained based on time difference and data value difference, and local outlier factors of the data point are obtained through LOF algorithm;
a local outlier factor is obtained for each data point for each dimension.
2. The method for intelligent operation management of a data center according to claim 1, wherein the obtaining the neighborhood range, the fluctuation coefficient and the variation range of each data point of each dimension comprises the following specific steps:
presetting a time neighborhoodSize and dimensions ofFor->No. 5 of the individual dimension>Data point, will be->Data points at->Adjacent front->Data points and adjacent post->A data point is taken as a neighborhood range of the data point; calculating standard deviation of all data points in the data point and neighborhood range, and recording the standard deviation as the neighborhood standard deviation of the data point;
if the neighborhood standard deviation of the data point is larger than the standard deviation threshold, setting the fluctuation coefficient of the data point to be 1; if the neighborhood standard deviation of the data point is smaller than or equal to the standard deviation threshold value, setting the fluctuation coefficient of the data point to be 0;
will be the firstData points and at->Adjacent front->Data points and adjacent back->The data points constitute the range of variation of the data point, +.>The size of the preset variation range;
and acquiring a neighborhood range, a fluctuation coefficient and a variation range of each data point of each dimension.
3. The method for intelligent operation management of a data center according to claim 1, wherein the obtaining the left neighborhood range and the right neighborhood range of each data point in each dimension, and the reference data point, the left reference neighborhood range and the right reference neighborhood range in each other dimension comprises the following specific steps:
for the firstNo. 5 of the individual dimension>The data point is used for acquiring a plurality of data points in other dimensions at the moment corresponding to the data point and is used as a reference data point of the data point in the corresponding dimension;
acquiring a neighborhood range of a reference data point as a reference neighborhood range of the data point in a corresponding dimension, forming a left neighborhood range of the data point by the data point before the neighborhood range, and forming a right neighborhood range of the data point by the data point after the neighborhood range; acquiring a left reference neighborhood range and a right reference neighborhood range of the data point in each other dimension according to the reference neighborhood ranges of the other dimensions;
a left neighborhood range and a right neighborhood range of each data point in each dimension are obtained, and a reference data point, a left reference neighborhood range and a right reference neighborhood range in each other dimension are obtained.
4. The intelligent operation management method of a data center according to claim 1, wherein the fitting data of each data point in each dimension is obtained by the following specific method:
for the firstNo. 5 of the individual dimension>Data point, the time of obtaining the data point is before +.>Day same time->Data points of the dimensions, fitting data points as the data points, +.>Is the preset reference days;
for a pair ofFitting the fitting data points by a least square method to obtain a fitting curve, predicting the fitting curve to obtain predicted data of the data points at the corresponding moment on the corresponding days, and marking the predicted data as the +.>Dimension->Fitting data of the data points;
fitting data for each data point for each dimension is obtained.
5. An intelligent operation management system for a data center, the system comprising:
the data center acquisition data module is used for acquiring data of a plurality of dimensions of the data center at a plurality of moments to obtain a plurality of data points of each dimension;
the multidimensional data analysis processing module: the method comprises the steps of obtaining initial correction coefficients of each data point of each dimension according to the distribution of other data points in the neighborhood range of each data point of the same dimension;
obtaining the confidence coefficient of each data point of each dimension according to the correlation of the data points of the same moment in different dimensions in a neighborhood range and the change of the data points of the same moment in different days of each dimension; obtaining a correction coefficient through the confidence coefficient and the initial correction coefficient and obtaining a self-adaptive K value of each data point of each dimension;
the data anomaly detection management module is used for acquiring local outlier factors of each data point according to the self-adaptive K value of each data point of each dimension, and anomaly detection is carried out on the data of multiple dimensions of the data center through the local outlier factors.
CN202311352020.0A 2023-10-19 2023-10-19 Intelligent operation management method and system for data center Active CN117093879B (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
CN202311352020.0A CN117093879B (en) 2023-10-19 2023-10-19 Intelligent operation management method and system for data center

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
CN202311352020.0A CN117093879B (en) 2023-10-19 2023-10-19 Intelligent operation management method and system for data center

Publications (2)

Publication Number Publication Date
CN117093879A true CN117093879A (en) 2023-11-21
CN117093879B CN117093879B (en) 2024-01-30

Family

ID=88780663

Family Applications (1)

Application Number Title Priority Date Filing Date
CN202311352020.0A Active CN117093879B (en) 2023-10-19 2023-10-19 Intelligent operation management method and system for data center

Country Status (1)

Country Link
CN (1) CN117093879B (en)

Cited By (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117272192A (en) * 2023-11-22 2023-12-22 青岛洛克环保科技有限公司 Sewage treatment system of magnetic coagulation efficient sedimentation tank based on sewage detection
CN117290799A (en) * 2023-11-24 2023-12-26 山东壹知产数字科技有限公司 Enterprise purchase management method and system based on big data
CN117318310A (en) * 2023-11-28 2023-12-29 浙江隆业电气科技有限公司 Remote unmanned intelligent monitoring system for explosion-proof power distribution cabinet
CN117310348A (en) * 2023-11-23 2023-12-29 东莞市时实电子有限公司 Power adapter fault real-time monitoring method and system
CN117407700A (en) * 2023-12-14 2024-01-16 国网山东省电力公司莱芜供电公司 Method for monitoring working environment in live working process
CN117407728A (en) * 2023-12-15 2024-01-16 湖南南华生物技术有限公司 Storage environment real-time monitoring method and system for stem cells
CN117407828A (en) * 2023-12-15 2024-01-16 深圳市雨博士雨水利用设备有限公司 Data analysis method applied to sponge city rainwater collection system
CN117435874A (en) * 2023-12-21 2024-01-23 河北雄安睿天科技有限公司 Abnormal data detection method and system for water supply and drainage equipment
CN117474539A (en) * 2023-12-25 2024-01-30 青岛天高智慧科技有限公司 Intelligent management method for campus one-card data
CN117474427A (en) * 2023-12-27 2024-01-30 大连金马衡器有限公司 Intelligent pallet cold chain tracing method based on Internet of things technology
CN117574304A (en) * 2024-01-16 2024-02-20 广东世腾智慧科技有限公司 Corrugated paper box production data intelligent analysis method and system based on big data
CN117708748A (en) * 2024-02-05 2024-03-15 苏州众志新环冷却设备有限公司 Operation monitoring system and method for centrifugal fan
CN117807381A (en) * 2024-02-23 2024-04-02 余姚市农业技术推广服务总站 Agricultural technology popularization record data intelligent management system
CN117851974A (en) * 2024-03-07 2024-04-09 艾信智能环境科技(无锡)有限公司 Medical pneumatic type garbage bedding and clothing intelligent collection system
CN117879769A (en) * 2024-03-11 2024-04-12 陕西惠延机械有限公司 Data pushing and transmitting method for tunnel lining trolley cloud platform system
CN117894429A (en) * 2024-03-15 2024-04-16 陕西省中医医院 Intelligent user reference temperature assessment method for moxibustion device control

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220292107A1 (en) * 2021-02-26 2022-09-15 International Business Machines Corporation Quality assessment of extracted features from high-dimensional machine learning datasets
CN115766607A (en) * 2022-11-08 2023-03-07 陕西通信规划设计研究院有限公司 Data compression transmission method based on 5G Internet of things
CN115987295A (en) * 2023-03-20 2023-04-18 河北省农林科学院 Crop monitoring data efficient processing method based on Internet of things
CN116307166A (en) * 2023-03-14 2023-06-23 重庆邮电大学 Method for predicting blueberry disease propagation through spatial mode change
CN116821832A (en) * 2023-07-24 2023-09-29 南京工程学院 Abnormal data identification and correction method for high-voltage industrial and commercial user power load
CN116873156A (en) * 2023-09-05 2023-10-13 山东航宇游艇发展有限公司 Intelligent monitoring method for power abnormality of natural gas ship based on big data

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220292107A1 (en) * 2021-02-26 2022-09-15 International Business Machines Corporation Quality assessment of extracted features from high-dimensional machine learning datasets
CN115766607A (en) * 2022-11-08 2023-03-07 陕西通信规划设计研究院有限公司 Data compression transmission method based on 5G Internet of things
CN116307166A (en) * 2023-03-14 2023-06-23 重庆邮电大学 Method for predicting blueberry disease propagation through spatial mode change
CN115987295A (en) * 2023-03-20 2023-04-18 河北省农林科学院 Crop monitoring data efficient processing method based on Internet of things
CN116821832A (en) * 2023-07-24 2023-09-29 南京工程学院 Abnormal data identification and correction method for high-voltage industrial and commercial user power load
CN116873156A (en) * 2023-09-05 2023-10-13 山东航宇游艇发展有限公司 Intelligent monitoring method for power abnormality of natural gas ship based on big data

Cited By (32)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117272192A (en) * 2023-11-22 2023-12-22 青岛洛克环保科技有限公司 Sewage treatment system of magnetic coagulation efficient sedimentation tank based on sewage detection
CN117272192B (en) * 2023-11-22 2024-03-08 青岛洛克环保科技有限公司 Sewage treatment system of magnetic coagulation efficient sedimentation tank based on sewage detection
CN117310348A (en) * 2023-11-23 2023-12-29 东莞市时实电子有限公司 Power adapter fault real-time monitoring method and system
CN117310348B (en) * 2023-11-23 2024-03-12 东莞市时实电子有限公司 Power adapter fault real-time monitoring method and system
CN117290799A (en) * 2023-11-24 2023-12-26 山东壹知产数字科技有限公司 Enterprise purchase management method and system based on big data
CN117290799B (en) * 2023-11-24 2024-02-02 山东壹知产数字科技有限公司 Enterprise purchase management method and system based on big data
CN117318310A (en) * 2023-11-28 2023-12-29 浙江隆业电气科技有限公司 Remote unmanned intelligent monitoring system for explosion-proof power distribution cabinet
CN117318310B (en) * 2023-11-28 2024-02-09 浙江隆业电气科技有限公司 Remote unmanned intelligent monitoring system for explosion-proof power distribution cabinet
CN117407700A (en) * 2023-12-14 2024-01-16 国网山东省电力公司莱芜供电公司 Method for monitoring working environment in live working process
CN117407700B (en) * 2023-12-14 2024-04-16 国网山东省电力公司莱芜供电公司 Method for monitoring working environment in live working process
CN117407828A (en) * 2023-12-15 2024-01-16 深圳市雨博士雨水利用设备有限公司 Data analysis method applied to sponge city rainwater collection system
CN117407828B (en) * 2023-12-15 2024-03-19 深圳市雨博士雨水利用设备有限公司 Data analysis method applied to sponge city rainwater collection system
CN117407728A (en) * 2023-12-15 2024-01-16 湖南南华生物技术有限公司 Storage environment real-time monitoring method and system for stem cells
CN117407728B (en) * 2023-12-15 2024-03-15 湖南南华生物技术有限公司 Storage environment real-time monitoring method and system for stem cells
CN117435874A (en) * 2023-12-21 2024-01-23 河北雄安睿天科技有限公司 Abnormal data detection method and system for water supply and drainage equipment
CN117435874B (en) * 2023-12-21 2024-03-12 河北雄安睿天科技有限公司 Abnormal data detection method and system for water supply and drainage equipment
CN117474539A (en) * 2023-12-25 2024-01-30 青岛天高智慧科技有限公司 Intelligent management method for campus one-card data
CN117474539B (en) * 2023-12-25 2024-03-29 青岛天高智慧科技有限公司 Intelligent management method for campus one-card data
CN117474427A (en) * 2023-12-27 2024-01-30 大连金马衡器有限公司 Intelligent pallet cold chain tracing method based on Internet of things technology
CN117474427B (en) * 2023-12-27 2024-03-26 大连金马衡器有限公司 Intelligent pallet cold chain tracing method based on Internet of things technology
CN117574304B (en) * 2024-01-16 2024-04-19 广东世腾智慧科技有限公司 Corrugated paper box production data intelligent analysis method and system based on big data
CN117574304A (en) * 2024-01-16 2024-02-20 广东世腾智慧科技有限公司 Corrugated paper box production data intelligent analysis method and system based on big data
CN117708748A (en) * 2024-02-05 2024-03-15 苏州众志新环冷却设备有限公司 Operation monitoring system and method for centrifugal fan
CN117708748B (en) * 2024-02-05 2024-05-03 苏州众志新环冷却设备有限公司 Operation monitoring system and method for centrifugal fan
CN117807381A (en) * 2024-02-23 2024-04-02 余姚市农业技术推广服务总站 Agricultural technology popularization record data intelligent management system
CN117807381B (en) * 2024-02-23 2024-05-24 余姚市农业技术推广服务总站 Agricultural technology popularization record data intelligent management system
CN117851974A (en) * 2024-03-07 2024-04-09 艾信智能环境科技(无锡)有限公司 Medical pneumatic type garbage bedding and clothing intelligent collection system
CN117851974B (en) * 2024-03-07 2024-05-14 艾信智能环境科技(无锡)有限公司 Medical pneumatic type garbage bedding and clothing intelligent collection system
CN117879769A (en) * 2024-03-11 2024-04-12 陕西惠延机械有限公司 Data pushing and transmitting method for tunnel lining trolley cloud platform system
CN117879769B (en) * 2024-03-11 2024-05-28 陕西惠延机械有限公司 Data pushing and transmitting method for tunnel lining trolley cloud platform system
CN117894429A (en) * 2024-03-15 2024-04-16 陕西省中医医院 Intelligent user reference temperature assessment method for moxibustion device control
CN117894429B (en) * 2024-03-15 2024-05-24 陕西省中医医院 Intelligent user reference temperature assessment method for moxibustion device control

Also Published As

Publication number Publication date
CN117093879B (en) 2024-01-30

Similar Documents

Publication Publication Date Title
CN117093879B (en) Intelligent operation management method and system for data center
CN117421687B (en) Method for monitoring running state of digital power ring main unit
CN110222991B (en) Metering device fault diagnosis method based on RF-GBDT
CN117034043A (en) Intelligent building comprehensive energy consumption monitoring method and system based on multi-energy Internet of things
CN116881745B (en) Pressure transmitter abnormality monitoring method based on big data
CN110942137A (en) Power grid information operation and maintenance monitoring method based on deep learning
CN116307944B (en) Distribution box remote monitoring system based on artificial intelligence and Internet of things
CN115512520A (en) Gas use safety early warning method based on intelligent gas and Internet of things system
CN117235557B (en) Electrical equipment fault rapid diagnosis method based on big data analysis
CN110852906B (en) Method and system for identifying electricity stealing suspicion based on high-dimensional random matrix
CN117080624B (en) Battery heating control method and system
CN117150244B (en) Intelligent power distribution cabinet state monitoring method and system based on electrical parameter analysis
CN107808209B (en) Wind power plant abnormal data identification method based on weighted kNN distance
CN116756505B (en) Photovoltaic equipment intelligent management system and method based on big data
CN116151799A (en) BP neural network-based distribution line multi-working-condition fault rate rapid assessment method
CN101923605B (en) Wind pre-warning method for railway disaster prevention
CN113704323A (en) Timing sequence anomaly detection algorithm based on HTM improvement
CN117034177B (en) Intelligent monitoring method for abnormal data of power load
CN117892246B (en) Data processing method for intelligent switch cabinet
CN116150666B (en) Energy storage system fault detection method and device and intelligent terminal
CN117313014B (en) Real-time monitoring and early warning method for abnormal energy consumption data of kiln operation
CN117325699B (en) Safety monitoring system for liquid cooling and super-charging technology
CN115993498A (en) ECOD-based method and device for detecting abnormal rate of area light Fu Xiansun
CN118152829A (en) Health state assessment method and system for oil type iron core reactor
CN117892246A (en) Data processing method for intelligent switch cabinet

Legal Events

Date Code Title Description
PB01 Publication
PB01 Publication
SE01 Entry into force of request for substantive examination
SE01 Entry into force of request for substantive examination
GR01 Patent grant
GR01 Patent grant