CN116860562B - Method and system for monitoring data quality of data center - Google Patents
Method and system for monitoring data quality of data center Download PDFInfo
- Publication number
- CN116860562B CN116860562B CN202311126435.6A CN202311126435A CN116860562B CN 116860562 B CN116860562 B CN 116860562B CN 202311126435 A CN202311126435 A CN 202311126435A CN 116860562 B CN116860562 B CN 116860562B
- Authority
- CN
- China
- Prior art keywords
- data
- sequence
- monitoring
- monitoring information
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012544 monitoring process Methods 0.000 title claims abstract description 164
- 238000000034 method Methods 0.000 title claims abstract description 63
- 230000002159 abnormal effect Effects 0.000 claims abstract description 9
- 238000005259 measurement Methods 0.000 claims description 27
- 238000001514 detection method Methods 0.000 claims description 21
- 238000004364 calculation method Methods 0.000 claims description 11
- 230000005856 abnormality Effects 0.000 claims description 9
- 238000004590 computer program Methods 0.000 claims description 9
- 238000009825 accumulation Methods 0.000 claims description 5
- 230000005540 biological transmission Effects 0.000 claims description 3
- 239000011159 matrix material Substances 0.000 claims description 3
- RGCLLPNLLBQHPF-HJWRWDBZSA-N phosphamidon Chemical compound CCN(CC)C(=O)C(\Cl)=C(/C)OP(=O)(OC)OC RGCLLPNLLBQHPF-HJWRWDBZSA-N 0.000 claims description 3
- 238000011156 evaluation Methods 0.000 abstract description 4
- 230000035945 sensitivity Effects 0.000 abstract description 4
- 230000008030 elimination Effects 0.000 abstract description 3
- 238000003379 elimination reaction Methods 0.000 abstract description 3
- 230000006870 function Effects 0.000 description 8
- 238000013139 quantization Methods 0.000 description 6
- 238000012545 processing Methods 0.000 description 5
- 238000003860 storage Methods 0.000 description 4
- 238000004458 analytical method Methods 0.000 description 3
- 230000009286 beneficial effect Effects 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 238000013467 fragmentation Methods 0.000 description 2
- 238000006062 fragmentation reaction Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 241000894007 species Species 0.000 description 2
- 241000282414 Homo sapiens Species 0.000 description 1
- 230000003321 amplification Effects 0.000 description 1
- 238000003491 array Methods 0.000 description 1
- 238000013523 data management Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 230000007812 deficiency Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 238000013178 mathematical model Methods 0.000 description 1
- 238000003199 nucleic acid amplification method Methods 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000003908 quality control method Methods 0.000 description 1
- 238000012216 screening Methods 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3003—Monitoring arrangements specially adapted to the computing system or computing system component being monitored
- G06F11/3006—Monitoring arrangements specially adapted to the computing system or computing system component being monitored where the computing system is distributed, e.g. networked systems, clusters, multiprocessor systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3058—Monitoring arrangements for monitoring environmental properties or parameters of the computing system or of the computing system component, e.g. monitoring of power, currents, temperature, humidity, position, vibrations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F11/00—Error detection; Error correction; Monitoring
- G06F11/30—Monitoring
- G06F11/3065—Monitoring arrangements determined by the means or processing involved in reporting the monitored data
-
- G—PHYSICS
- G08—SIGNALLING
- G08B—SIGNALLING OR CALLING SYSTEMS; ORDER TELEGRAPHS; ALARM SYSTEMS
- G08B31/00—Predictive alarm systems characterised by extrapolation or other computation using updated historic data
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/08—Configuration management of networks or network elements
- H04L41/0803—Configuration setting
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/10—Protocols in which an application is distributed across nodes in the network
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Computing Systems (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Networks & Wireless Communication (AREA)
- General Engineering & Computer Science (AREA)
- Quality & Reliability (AREA)
- Signal Processing (AREA)
- Mathematical Physics (AREA)
- Health & Medical Sciences (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Business, Economics & Management (AREA)
- Emergency Management (AREA)
- Remote Monitoring And Control Of Power-Distribution Networks (AREA)
Abstract
The application belongs to the technical field of data center station and data quality monitoring, and provides a method and a system for monitoring data quality of a data center station, which specifically comprise the following steps: and configuring a distributed monitoring network of the data center station, continuously acquiring an electric power data image through the distributed monitoring network, acquiring an image turbulence level by utilizing the electric power data image, and finally carrying out early warning on the data center station according to the image turbulence level. The purpose of evaluating the data quality of the monitoring data source end in the electric power data image is achieved through the image turbulence level, the evaluation sensitivity of the data quality of the abnormal data obtained in the data center is improved, the elimination of low-quality data is achieved, the electric power information of each unit managed by one node is continuously obtained, and whether the data acquisition is interfered is analyzed, so that the data center can directly monitor the data quality of the data source end, the reliability of the data obtained by the data center is improved, and the accuracy and the effectiveness of decisions made based on the data center are greatly guaranteed.
Description
Technical Field
The application belongs to the technical field of data center station and data quality monitoring, and particularly relates to a method and a system for monitoring data quality of a data center station.
Background
With the rapid development of social economy and information technology, a great amount of data is generated in daily production and life of human beings, and the processing and analysis of the big data becomes an important decision support and reliable business optimization means for people. However, in the large data processing process, massive data obtained by each system are often stored in different databases, so that data fragmentation and data islanding phenomenon are caused, and data cannot be efficiently integrated and shared. Therefore, the data center is a centralized data management and service platform, and the data of each system is managed in a centralized way, so that people are helped to break the data island and eliminate the data fragmentation.
However, the data in the data center is massive and tedious, has low value density, and often has poor data quality, resulting in low data reliability, thereby affecting the accuracy and effectiveness of decisions made based on the data center. At present, a concern of people monitoring the quality of data is usually whether data at two ends are consistent, namely whether errors occur when a data center station performs data synchronization from a data source end, so that the data is incomplete or inaccurate. In fact, the data quality is low not only because the data at the two ends are inconsistent, but also the data acquired by the data source end may be interfered. In this case, the data is erroneous or inaccurate from the beginning, and even if the data at the two ends are completely consistent in the later stage, only the data center synchronizes the erroneous or inaccurate data, and the data quality cannot be improved naturally. Therefore, a method and a system for monitoring the data quality of a data center station are needed, so that the data center station can judge whether the data acquired by a data source end is interfered or not, and the purpose of directly monitoring the data quality of the data source end is expected to be achieved.
Disclosure of Invention
The application aims to provide a method and a system for monitoring data quality of a data center, which are used for solving one or more technical problems in the prior art and at least providing a beneficial selection or creation condition.
To achieve the above object, according to an aspect of the present application, there is provided a monitoring method for data quality of a data center station, the method comprising the steps of:
s100, configuring a distributed monitoring network of a data center station;
s200, continuously acquiring power data images through a distributed monitoring network;
s300, calculating an image turbulence level by using the electric power data image;
s400, early warning is carried out on the data center according to the image turbulence level.
Further, in step S100, the method for configuring the distributed monitoring network of the station in the data is: the method comprises the steps that a distributed system is adopted for establishing a data center, each server in the distributed system is used as a node, and each node is connected with the data center respectively and performs data exchange or data transmission;
taking a data acquisition point as a unit, wherein the data acquisition point is a position or a place for acquiring data required by a data center station; each unit is provided with an electric power analyzer, monitoring information is collected through the electric power analyzer, the monitoring information collected by one unit comprises two or more than two of voltage, current, power, electric energy, frequency and power factor, and the quantity of the monitoring information collected by the unit is recorded as NSen; the number of units connected to the same node is denoted NUni; the information network built by the individual nodes and units serves as a distributed monitoring network.
Further, in step S200, the method for continuously acquiring the power data image through the distributed monitoring network is as follows: setting a time interval as a measurement interval t1, t1 epsilon [1,60] seconds; each unit acquires power information once every t1, wherein the power information comprises real-time measured values corresponding to all monitoring information; constructing a matrix FMX by taking real-time measured values of different units under the same monitoring information as a column and taking real-time measured values of different monitoring information under the same unit as a row; taking the difference between the maximum value and the minimum value of any row in the FMX as the actual measurement level of the monitoring information corresponding to the row to obtain the actual measurement level corresponding to each monitoring information;
setting a time interval as a measurement interval t2, wherein t2 epsilon [60,120] min; setting a variable tk related to the number of moments, wherein the value range of the variable tk is tk epsilon [1,20], and taking the arithmetic average value of the measured levels of one moment and the previous tk moments as the prediction level of the moment; respectively constructing a sequence according to the actual measurement level and the prediction level of the monitoring information at each moment in the t2 period, and respectively recording the sequence as a monitoring sequence and a prediction sequence; calculating to obtain root mean square error as the mode measurement distance of the monitoring information through the monitoring sequence and the prediction sequence; the mode measuring distance of each monitoring data species constructs a sequence as a mode measuring sequence; and taking the modular measuring sequence as a power data image at the current moment.
Further, in step S300, the method for calculating the image turbulence level using the power data image includes: acquiring average value EFL of each element in the power data image at the current moment, and defining the corresponding monitoring information of one element as first-order monitoring information if the value of the element in the power data image is larger than EFL; comparing the predicted level at each time in any one first-order monitoring information with the measured level, if the predicted level at one time is smaller than the measured level, defining the predicted level at the time as a first turbulence value,
taking root mean square value of each first turbulence value obtained by first-order monitoring information in t2 time as the offset mode distance of the first-order monitoring information; the method for calculating and obtaining the image turbulence deflection ratio DO_Pr comprises the following steps:
；
wherein v1 and v2 are both accumulation variables, MMD v2 Mode detection distance for v2 th first-order monitoring information, DMD v1 For the offset mode distance of the v1 th first-order monitoring information, nq represents the quantity of the first-order monitoring information;
the difference value between the median and the minimum value in each first turbulence value of the first-order monitoring information is recorded as a subscript domain value; the difference between the maximum value and the median in each first turbulence value of the first-order monitoring information is recorded as a superscript threshold value; the image turbulence level DOL is obtained by calculation, and the calculation method comprises the following steps:
；
wherein v3 is an accumulated variable, avg_LFD v3 For the average value of the respective first turbulence values of the v3 th monitoring information LLV v3 And HLV v3 The index threshold and the upper index threshold of the v3 first-order monitoring data are respectively.
The image turbulence level is obtained by calculation after the collected data are combined with a mathematical model, so that the data quality in the electric data image is effectively quantified, however, under the condition of larger value of a measurement interval t2, the phenomenon of insufficient quantization degree of the image turbulence level calculated by the method often occurs, because the method emphasizes the individual measurement mode distances, has equal sensitivity to the data at each moment, cannot effectively amplify and divide the difference between the measurement mode distances in real time, and causes the problem of under-fitting of the turbulence level obtained by processing, and no viable technology exists at present to compensate the phenomenon of insufficient quantization caused by the method, so that the phenomenon of under-fitting of the turbulence level due to the insufficient division of the difference between the measurement mode distances is eliminated, and a more preferable scheme is provided by the application:
preferably, in step S300, the method for calculating the image turbulence level using the power data image is: in the t2 time period, the detection mode distances of the same monitoring information at all moments are obtained to construct a sequence as a detection sequence; searching a sequence number corresponding to the moment when the maximum value appears for the first time from the current moment to the detection sequence corresponding to any monitoring information, and recording the numerical value of the sequence number as an end value interval of the monitoring information; after acquiring the end value interval of each monitoring information, marking the maximum value in the end value interval as MVZ; dividing a sequence into every MVZ elements in each detection sequence to serve as a quasi-sequence RTM;
j is taken as the serial number of the dummy sequence, then the jth dummy sequence is marked as RTM j Wherein j > 1; when the number of the residual elements in the detection sequence cannot meet the requirement of forming the quasi-sequence, the quasi-sequence is not constructed any more; will detect the orderThe first pseudosequence in the column is RTM as the first pseudosequence Fs The remaining pseudosequences as the second pseudosequence RTM j The method comprises the steps of carrying out a first treatment on the surface of the Taking i1 as the sequence number of the element of the pseudo-sequence;
if RTM j (i1)≤RTM Fs (i1) RTM is to j (i1) Marking as low index quantity corresponding to the monitoring information, otherwise marking as high index quantity corresponding to the monitoring information, wherein RTM j (i1) And RTM Fs (i1) I1 st element representing a first pseudo-sequence and a second pseudo-sequence, respectively; after comparing each second pseudo-sequence, respectively forming a low-standard-index sequence LSL and a high-standard-index sequence HSL by each marked index and high-standard-index; the method for calculating the turbulence value DOV of the monitoring information comprises the following steps:
；
where k1 is an accumulated variable, sum<>RTM for sum function Fs For the first pseudo sequence, l_siv (k 1) represents the kth 1 element in the low-index sequence, np is the total number of elements in the low-index sequence;
the image turbulence level DOL is obtained by calculation, and the calculation method comprises the following steps:
；
wherein k2 and k3 are both accumulation variables, exp () is an exponential function with a natural constant e as a base, DOV k2 For the k2 th monitored information sub-turbulence value, TAV k3 LSL as the arithmetic mean of the elements in the first pseudo-sequence under the kth 3 monitored data k3 And HSL k3 Representing the low index sequence and the high index sequence of the kth 3 monitoring information, respectively, NSen is the total amount of the monitoring information.
The beneficial effects are that: the image turbulence level is calculated by comparing and screening the corresponding measuring mode distances among the divided simulated sequences and obtaining the characteristic data, so that the measuring mode distances are regularly compared, the image turbulence level has a relatively stable and reliable overall reference value, the effective amplification of the difference of the measuring mode distances is realized, and the defect of global representativeness deficiency of related derivative calculated values is overcome. The purpose of evaluating the data quality of the monitoring data source end in the electric data image is achieved through the image turbulence level, precise numerical quantization data are formed for numerical feature quantization of the interested area in the image, data support is further conducted on data quality control, evaluation sensitivity of the quality of abnormal data obtained in the data center station is improved, and marking or elimination of low-quality data is achieved.
Further, in step S400, the method for early warning to the data center station according to the image turbulence level is as follows: continuously acquiring the turbulence level of each image obtained by the same server, and judging whether the turbulence level of each image obtained in a time period t2 is abnormal or not through anomaly checking, wherein the anomaly checking method is any one of a Leider criterion method, a Dixon criterion method or a Showler criterion method; if the image turbulence level at the current moment is abnormal, marking all the power information obtained in the previous t1 time period at the current moment as power information abnormality, and sending a data abnormality alarm to a client or a management program, wherein the data abnormality alarm comprises a text, a sound, an image or a table.
Preferably, all undefined variables in the present application, if not explicitly defined, may be thresholds set manually.
The application also provides a monitoring system for the data quality of the data center, which comprises: the method comprises the steps of a method for monitoring data quality of a data center, wherein the monitoring system for the data quality of the data center can be operated in a computing device such as a desktop computer, a notebook computer, a palm computer and a cloud data center, and the operable system can comprise, but is not limited to, a processor, a memory and a server cluster, and the processor executes the computer program to operate in the following units:
the network configuration unit is used for configuring a distributed monitoring network of the data center station;
the data acquisition unit is used for continuously acquiring the electric power data image through the distributed monitoring network;
a quality analysis unit for obtaining an image turbulence level using the electric power data image;
and the monitoring alarm unit is used for carrying out early warning on the data center station according to the image turbulence level.
The beneficial effects of the application are as follows: the application provides a monitoring method and a system for data quality of a data center, which realize the evaluation of the data quality of a monitoring data source end in an electric power data image through the image turbulence level, form numerical quantization data for the numerical feature quantization of an interested area in the image, support the data for further controlling the data quality, improve the evaluation sensitivity of the abnormal data quality obtained in the data center, realize the elimination of low-quality data, and analyze whether the data acquisition of each unit managed by each node is interfered or not by continuously obtaining the electric power information of each unit, thereby enabling the data center to directly monitor the data quality of the data source end, improving the reliability of the data obtained by the data center and greatly guaranteeing the accuracy and the effectiveness of decisions made based on the data center.
Drawings
The above and other features of the present application will become more apparent from the detailed description of the embodiments thereof given in conjunction with the accompanying drawings, in which like reference characters designate like or similar elements, and it is apparent that the drawings in the following description are merely some examples of the present application, and other drawings may be obtained from these drawings without inventive effort to those of ordinary skill in the art, in which:
FIG. 1 is a flow chart of a method for monitoring data quality of a data center station;
fig. 2 is a diagram showing a structure of a monitoring system for data quality of a data center station.
Detailed Description
The conception, specific structure, and technical effects produced by the present application will be clearly and completely described below with reference to the embodiments and the drawings to fully understand the objects, aspects, and effects of the present application. It should be noted that, without conflict, the embodiments of the present application and features of the embodiments may be combined with each other.
A flowchart of a method for monitoring quality of data of a data center station is shown in fig. 1, and a method for monitoring quality of data of a data center station according to an embodiment of the present application is described below with reference to fig. 1, the method comprising the steps of:
s100, configuring a distributed monitoring network of a data center station;
s200, continuously acquiring power data images through a distributed monitoring network;
s300, calculating an image turbulence level by using the electric power data image;
s400, early warning is carried out on the data center according to the image turbulence level.
Further, in step S100, the method for configuring the distributed monitoring network of the station in the data is: the method comprises the steps that a distributed system is adopted for establishing a data center, each server in the distributed system is used as a node, and each node is connected with the data center respectively and performs data exchange or data transmission;
taking a data acquisition point as a unit, wherein the data acquisition point is a position or a place for acquiring data required by a data center station; each unit is provided with an electric power analyzer, monitoring information is collected through the electric power analyzer, the monitoring information collected by one unit comprises two or more than two of voltage, current, power, electric energy, frequency and power factor, and the quantity of the monitoring information collected by the unit is recorded as NSen; the number of units connected to the same node is denoted NUni; the information network built by the individual nodes and units serves as a distributed monitoring network.
Further, in step S200, the method for continuously acquiring the power data image through the distributed monitoring network is as follows: setting a time interval as a measurement interval t1, t1 epsilon [1,60] seconds; each unit acquires power information once every t1, wherein the power information comprises real-time measured values corresponding to all monitoring information; constructing a matrix FMX by taking real-time measured values of different units under the same monitoring information as a column and taking real-time measured values of different monitoring information under the same unit as a row; taking the difference between the maximum value and the minimum value of any row in the FMX as the actual measurement level of the monitoring information corresponding to the row to obtain the actual measurement level corresponding to each monitoring information;
setting a time interval as a measurement interval t2, wherein t2 epsilon [60,120] min; taking the arithmetic average of the measured levels at one time and the previous tk times as the predicted level of the time, tk epsilon [1,20]; respectively constructing a sequence according to the actual measurement level and the prediction level of the monitoring information at each moment in the t2 period, and respectively recording the sequence as a monitoring sequence and a prediction sequence; calculating to obtain root mean square error as the mode measurement distance of the monitoring information through the monitoring sequence and the prediction sequence; the mode measuring distance of each monitoring data species constructs a sequence as a mode measuring sequence; and taking the modular measuring sequence as a power data image at the current moment.
Further, in step S300, the method for calculating the image turbulence level using the power data image includes:
acquiring average value EFL of each element in the power data image at the current moment, and defining the corresponding monitoring information of one element as first-order monitoring information if the value of the element in the power data image is larger than EFL; comparing the predicted level at each time in any one first-order monitoring information with the measured level, if the predicted level at one time is smaller than the measured level, defining the predicted level at the time as a first turbulence value,
taking root mean square value of each first turbulence value obtained by first-order monitoring information in t2 time as the offset mode distance of the first-order monitoring information; the method for calculating and obtaining the image turbulence deflection ratio DO_Pr comprises the following steps:
；
wherein v1 and v2 are both accumulation variables, MMD v2 Mode detection distance for v2 th first-order monitoring information, DMD v1 For the offset mode distance of the v1 th first-order monitoring information, nq represents the quantity of the first-order monitoring information;
the difference value between the median and the minimum value in each first turbulence value of the first-order monitoring information is recorded as a subscript domain value; the difference between the maximum value and the median in each first turbulence value of the first-order monitoring information is recorded as a superscript threshold value; the image turbulence level DOL is obtained by calculation, and the calculation method comprises the following steps:
；
wherein v3 is an accumulated variable, avg_LFD v3 For the average value of the respective first turbulence values of the v3 th monitoring information LLV v3 And HLV v3 The index threshold and the upper index threshold of the v3 first-order monitoring data are respectively.
Further, in step S300, the method for calculating the image turbulence level using the power data image includes: in the t2 time period, the detection mode distances of the same monitoring information at all moments are obtained to construct a sequence as a detection sequence; searching a sequence number corresponding to the moment when the maximum value appears for the first time from the current moment to the detection sequence corresponding to any monitoring information, and recording the numerical value of the sequence number as an end value interval of the monitoring information; after acquiring the end value interval of each monitoring information, marking the maximum value in the end value interval as MVZ; dividing a sequence into every MVZ elements in each detection sequence to serve as a quasi-sequence RTM;
j is taken as the serial number of the dummy sequence, then the jth dummy sequence is marked as RTM j Wherein j > 1; when the number of the residual elements in the detection sequence cannot meet the requirement of forming the quasi-sequence, the quasi-sequence is not constructed any more; RTM using the first dummy sequence in the detection sequence as the first dummy sequence Fs The remaining pseudosequences as the second pseudosequence RTM j The method comprises the steps of carrying out a first treatment on the surface of the Taking i1 as the sequence number of the element of the pseudo-sequence;
if RTM j (i1)≤RTM Fs (i1) RTM is to j (i1) Low marked as corresponding monitoring informationThe index quantity is marked as a high index quantity corresponding to the monitoring information, wherein RTM j (i1) And RTM Fs (i1) I1 st element representing a first pseudo-sequence and a second pseudo-sequence, respectively; after comparing each second pseudo-sequence, respectively forming a low-standard-index sequence LSL and a high-standard-index sequence HSL by each marked index and high-standard-index; the method for calculating the turbulence value DOV of the monitoring information comprises the following steps:
；
where k1 is an accumulated variable, sum<>RTM for sum function Fs For the first pseudo sequence, l_siv (k 1) represents the kth 1 element in the low-index sequence, np is the total number of elements in the low-index sequence;
the image turbulence level DOL is obtained by calculation, and the calculation method comprises the following steps:
；
wherein k2 and k3 are both accumulation variables, exp () is an exponential function with a natural constant e as a base, DOV k2 For the k2 th monitored information sub-turbulence value, TAV k3 LSL as the arithmetic mean of the elements in the first pseudo-sequence under the kth 3 monitored data k3 And HSL k3 Representing the low index sequence and the high index sequence of the kth 3 monitoring information, respectively, NSen is the total amount of the monitoring information.
Further, in step S400, the method for early warning to the data center station according to the image turbulence level is as follows: continuously acquiring the turbulence level of each image obtained by the same server, and judging whether the turbulence level of each image obtained in a time period t2 is abnormal or not through anomaly checking, wherein the anomaly checking method is any one of a Leider criterion method, a Dixon criterion method or a Showler criterion method; if the image turbulence level at the current moment is abnormal, marking all the power information obtained in the previous t1 time period at the current moment as power information abnormality, and sending a data abnormality alarm to a client or a management program, wherein the data abnormality alarm comprises a text, a sound, an image or a table.
Fig. 2 is a block diagram of a monitoring system for data quality of a data center, where the monitoring system for data quality of a data center includes: a processor, a memory and a computer program stored in the memory and executable on the processor, the processor implementing the steps in one of the above embodiments of a monitoring system for data quality of a data center station when the computer program is executed.
The system comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, the processor executing the computer program to run in units of the following system:
the network configuration unit is used for configuring a distributed monitoring network of the data center station;
the data acquisition unit is used for continuously acquiring the electric power data image through the distributed monitoring network;
a quality analysis unit for obtaining an image turbulence level using the electric power data image;
and the monitoring alarm unit is used for carrying out early warning on the data center station according to the image turbulence level.
The monitoring system for the data quality of the data center station can be operated in computing equipment such as a desktop computer, a notebook computer, a palm computer, a cloud server and the like. The system for monitoring the data quality of the data center station can comprise, but is not limited to, a processor and a memory. It will be appreciated by those skilled in the art that the example is merely an example of a monitoring system for data quality of a data center station, and is not limiting of a monitoring system for data quality of a data center station, and may include more or fewer components than examples, or may combine certain components, or different components, e.g., the monitoring system for data quality of a data center station may further include input and output devices, network access devices, buses, etc.
The processor may be a central processing unit (Central Processing Unit, CPU), other general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or the like. The general purpose processor may be a microprocessor or the processor may be any conventional processor or the like, which is a control center of the system for monitoring the quality of data of the data center, and which connects various parts of the entire system for monitoring the quality of data of the data center using various interfaces and lines.
The memory may be used to store the computer program and/or the module, and the processor may implement the various functions of the monitoring system for data quality of the data by running or executing the computer program and/or the module stored in the memory and invoking the data stored in the memory. The memory may mainly include a storage program area and a storage data area, wherein the storage program area may store an operating system, an application program (such as a sound playing function, an image playing function, etc.) required for at least one function, and the like; the storage data area may store data (such as audio data, phonebook, etc.) created according to the use of the handset, etc. In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as a hard disk, memory, plug-in hard disk, smart Media Card (SMC), secure Digital (SD) Card, flash Card (Flash Card), at least one disk storage device, flash memory device, or other volatile solid-state storage device.
Although the present application has been described in considerable detail and with particularity with respect to several described embodiments, it is not intended to be limited to any such detail or embodiment or any particular embodiment so as to effectively cover the intended scope of the application. Furthermore, the foregoing description of the application has been presented in its embodiments contemplated by the inventors for the purpose of providing a useful description, and for the purposes of providing a non-essential modification of the application that may not be presently contemplated, may represent an equivalent modification of the application.
Claims (5)
1. A method for monitoring data quality of a data center, the method comprising the steps of:
s100, configuring a distributed monitoring network of a data center station;
s200, continuously acquiring power data images through a distributed monitoring network;
s300, calculating an image turbulence level by using the electric power data image;
s400, early warning is carried out on the data center according to the image turbulence level;
in step S200, the method for continuously acquiring the power data image through the distributed monitoring network is as follows: setting a time interval as a measurement interval t1, t1 epsilon [1,60] seconds; each unit acquires power information once every t1, wherein the power information comprises real-time measured values corresponding to all monitoring information; constructing a matrix FMX by taking real-time measured values of different units under the same monitoring information as a column and taking real-time measured values of different monitoring information under the same unit as a row; taking the difference between the maximum value and the minimum value of any row in the FMX as the actual measurement level of the monitoring information corresponding to the row to obtain the actual measurement level corresponding to each monitoring information;
setting a time interval as a measurement interval t2, wherein t2 epsilon [60,120] min; setting a variable tk related to the number of moments, wherein the value range of the variable tk is tk epsilon [1,20], and taking the arithmetic average value of the measured levels of one moment and the previous tk moments as the prediction level of the moment; respectively constructing a sequence according to the actual measurement level and the prediction level of the monitoring information at each moment in the t2 period, and respectively recording the sequence as a monitoring sequence and a prediction sequence; calculating to obtain root mean square error as the mode measurement distance of the monitoring information through the monitoring sequence and the prediction sequence; the mode detection distance in each monitoring data is used for constructing a sequence as a mode detection sequence; taking the module measuring sequence as a power data image at the current moment;
in step S300, the method for calculating the image turbulence level using the power data image includes: acquiring average value EFL of each element in the power data image at the current moment, and defining the corresponding monitoring information of one element as first-order monitoring information if the value of the element in the power data image is larger than EFL; comparing the predicted level at each time in any one first-order monitoring information with the measured level, if the predicted level at one time is smaller than the measured level, defining the predicted level at the time as a first turbulence value,
setting a time interval as a measurement interval t2, t2 epsilon [60,120] minutes, and taking root mean square values of first turbulence values obtained by first-order monitoring information in t2 as the offset mode distance of the first-order monitoring information; the method for calculating and obtaining the image turbulence deflection ratio DO_Pr comprises the following steps:
；
wherein v1 and v2 are both accumulation variables, MMD v2 Mode detection distance for v2 th first-order monitoring information, DMD v1 For the offset mode distance of the v1 th first-order monitoring information, nq represents the quantity of the first-order monitoring information;
the difference value between the median and the minimum value in each first turbulence value of the first-order monitoring information is recorded as a subscript domain value; the difference between the maximum value and the median in each first turbulence value of the first-order monitoring information is recorded as a superscript threshold value; the image turbulence level DOL is obtained by calculation, and the calculation method comprises the following steps:
；
wherein v3 is an accumulated variable, avg_LFD v3 For the average value of the respective first turbulence values of the v3 th monitoring information LLV v3 And HLV v3 The index threshold and the upper index threshold of the v3 first-order monitoring data are respectively.
2. A method for monitoring data quality of a data center station according to claim 1, wherein in step S100, the method for configuring a distributed monitoring network of a data center station is: the method comprises the steps that a distributed system is adopted for establishing a data center, each server in the distributed system is used as a node, and each node is connected with the data center respectively and performs data exchange or data transmission;
taking a data acquisition point as a unit, wherein the data acquisition point is a position or a place for acquiring data required by a data center station; each unit is provided with an electric power analyzer, monitoring information is collected through the electric power analyzer, the monitoring information collected by one unit comprises two or more than two of voltage, current, power, electric energy, frequency and power factor, and the quantity of the monitoring information collected by the unit is recorded as NSen; the number of units connected to the same node is denoted NUni; the information network built by the individual nodes and units serves as a distributed monitoring network.
3. The method for monitoring data quality of a data center station according to claim 1, wherein in step S300, the method for calculating the image turbulence level using the electric power data image is: setting a time interval as a measurement interval t2, wherein t2 epsilon [60,120] min, and acquiring the measurement mode distances of the same monitoring information at each moment in the time interval t2 to construct a sequence as a detection sequence; searching a sequence number corresponding to the moment when the maximum value appears for the first time from the current moment to the detection sequence corresponding to any monitoring information, and recording the numerical value of the sequence number as an end value interval of the monitoring information; after acquiring the end value interval of each monitoring information, marking the maximum value in the end value interval as MVZ; dividing a sequence into every MVZ elements in each detection sequence to serve as a quasi-sequence RTM;
j is taken as the serial number of the dummy sequence, then the jth dummy sequence is marked as RTM j Wherein j > 1; when the number of the residual elements in the detection sequence cannot meet the requirement of forming the quasi-sequence, the quasi-sequence is not constructed any more; RTM using the first dummy sequence in the detection sequence as the first dummy sequence Fs Which is provided withResidual pseudosequence as second pseudosequence RTM j The method comprises the steps of carrying out a first treatment on the surface of the Taking i1 as the sequence number of the element of the pseudo-sequence;
if RTM j (i1)≤RTM Fs (i1) RTM is to j (i1) Marking as low index quantity corresponding to the monitoring information, otherwise marking as high index quantity corresponding to the monitoring information, wherein RTM j (i1) And RTM Fs (i1) I1 st element representing a first pseudo-sequence and a second pseudo-sequence, respectively; after comparing each second pseudo-sequence, respectively forming a low-standard-quantity sequence LSL and a high-standard-quantity sequence HSL by each marked low-standard-quantity and high-standard-quantity; and calculating a turbulence value DOV of the monitoring information, and calculating through the turbulence value to obtain an image turbulence level DOL.
4. The method for monitoring data quality of a data center according to claim 1, wherein in step S400, the method for pre-warning the data center according to the image turbulence level is as follows: setting a time interval as a measurement interval t2, t2 epsilon [60,120] min, continuously acquiring the turbulence level of each image acquired by the same server, and judging whether the turbulence level of each image acquired in the time interval t2 is abnormal or not through anomaly checking, wherein the anomaly checking method is any one of a Laida criterion method, a Dixon criterion method or a Showler criterion method; setting a time interval as a measurement interval t1, t1 epsilon [1,60] seconds, if the image turbulence level at the current moment is abnormal, marking all power information obtained in the previous t1 time period at the current moment as power information abnormality, and sending a data abnormality alarm to a client or a management program, wherein the data abnormality alarm comprises a text, a sound, an image or a table.
5. A monitoring system for data quality of a data center station, the monitoring system for data quality of a data center station comprising: a processor, a memory and a computer program stored in the memory and executable on the processor, the processor implementing the steps in a method for monitoring data quality of a data center according to any one of claims 1 to 4 when the computer program is executed, the monitoring system for data quality of a data center being executed in a computing device of a desktop computer, a notebook computer, a palm computer and a cloud data center.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311126435.6A CN116860562B (en) | 2023-09-04 | 2023-09-04 | Method and system for monitoring data quality of data center |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311126435.6A CN116860562B (en) | 2023-09-04 | 2023-09-04 | Method and system for monitoring data quality of data center |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116860562A CN116860562A (en) | 2023-10-10 |
| CN116860562B true CN116860562B (en) | 2023-11-24 |
Family
ID=88234459
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311126435.6A Active CN116860562B (en) | 2023-09-04 | 2023-09-04 | Method and system for monitoring data quality of data center |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116860562B (en) |
Families Citing this family (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN117395699A (en) * | 2023-11-08 | 2024-01-12 | 广东理致技术有限公司 | Monitoring factor energy-saving communication method and system based on Internet of things |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20170078252A (en) * | 2015-12-29 | 2017-07-07 | 삼성에스디에스 주식회사 | Method and apparatus for time series data monitoring |
| CN110457193A (en) * | 2019-07-30 | 2019-11-15 | 深圳供电局有限公司 | Health portrait methods of exhibiting and its system based on power information system operation/maintenance data |
| WO2021164465A1 (en) * | 2020-02-20 | 2021-08-26 | 深圳壹账通智能科技有限公司 | Intelligent early warning method and system |
| JP2021165830A (en) * | 2020-04-02 | 2021-10-14 | キヤノン株式会社 | Image forming apparatus |
| CN114004584A (en) * | 2021-10-22 | 2022-02-01 | 国网重庆市电力公司电力科学研究院 | Power information management method for building user portrait based on data middleboxes |
| CN114004585A (en) * | 2021-10-22 | 2022-02-01 | 国网重庆市电力公司电力科学研究院 | Power consumer data management system |
| CN114640707A (en) * | 2022-03-04 | 2022-06-17 | 微位（深圳）网络科技有限公司 | Message asynchronous processing method and device, electronic equipment and storage medium |
| CN114820047A (en) * | 2022-04-08 | 2022-07-29 | 国网河南省电力公司电力科学研究院 | Multi-user power consumption demand portrait method based on power quality monitoring data |
| CN115114647A (en) * | 2022-08-26 | 2022-09-27 | 湖南华菱电子商务有限公司 | Customer information management method and system for constructing customer portrait based on digital middlebox |
-
2023
- 2023-09-04 CN CN202311126435.6A patent/CN116860562B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| KR20170078252A (en) * | 2015-12-29 | 2017-07-07 | 삼성에스디에스 주식회사 | Method and apparatus for time series data monitoring |
| CN110457193A (en) * | 2019-07-30 | 2019-11-15 | 深圳供电局有限公司 | Health portrait methods of exhibiting and its system based on power information system operation/maintenance data |
| WO2021164465A1 (en) * | 2020-02-20 | 2021-08-26 | 深圳壹账通智能科技有限公司 | Intelligent early warning method and system |
| JP2021165830A (en) * | 2020-04-02 | 2021-10-14 | キヤノン株式会社 | Image forming apparatus |
| CN114004584A (en) * | 2021-10-22 | 2022-02-01 | 国网重庆市电力公司电力科学研究院 | Power information management method for building user portrait based on data middleboxes |
| CN114004585A (en) * | 2021-10-22 | 2022-02-01 | 国网重庆市电力公司电力科学研究院 | Power consumer data management system |
| CN114640707A (en) * | 2022-03-04 | 2022-06-17 | 微位（深圳）网络科技有限公司 | Message asynchronous processing method and device, electronic equipment and storage medium |
| CN114820047A (en) * | 2022-04-08 | 2022-07-29 | 国网河南省电力公司电力科学研究院 | Multi-user power consumption demand portrait method based on power quality monitoring data |
| CN115114647A (en) * | 2022-08-26 | 2022-09-27 | 湖南华菱电子商务有限公司 | Customer information management method and system for constructing customer portrait based on digital middlebox |
Non-Patent Citations (3)
| Title |
|---|
| 基于电力大数据分析模型的防返贫预警监测应用;李丽英;罗志坤;唐敬军;廖丽萍;李爱元;付强;湖南电力;第41卷(第005期);58-62 * |
| 基于电力大数据的标签画像技术与应用研究;吕辉;许道强;仲春林;李昆明;方超;;电力信息与通信技术(第02期);43-48 * |
| 基于聚类的电网低压台区用电画像方法研究;时珊珊;田英杰;金瑞杨;张灏宇;袁志昌;宋洁;苏运;;电工电能新技术(第08期);69-77 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116860562A (en) | 2023-10-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111459700B (en) | Equipment fault diagnosis method, diagnosis device, diagnosis equipment and storage medium | |
| CN109034244B (en) | Line loss abnormity diagnosis method and device based on electric quantity curve characteristic model | |
| CN110703057B (en) | Power equipment partial discharge diagnosis method based on data enhancement and neural network | |
| CN116860562B (en) | Method and system for monitoring data quality of data center | |
| CN110988563B (en) | UPS (uninterrupted Power supply) fault detection method, device, equipment and storage medium | |
| CN111158964B (en) | Disk failure prediction method, system, device and storage medium | |
| CN108491931B (en) | Method for improving nondestructive testing precision based on machine learning | |
| CN115841046B (en) | Accelerated degradation test data processing method and device based on wiener process | |
| CN110634081A (en) | Method and device for processing abnormal data of hydropower station | |
| CN114356900A (en) | Power data anomaly detection method, device, equipment and medium | |
| CN115526641A (en) | Flexible board product production quality tracing method, system, device and storage medium | |
| CN117035563B (en) | Product quality safety risk monitoring method, device, monitoring system and medium | |
| CN117207778B (en) | Nondestructive testing method and system for vehicle parts | |
| CN112801315A (en) | State diagnosis method and device for power secondary equipment and terminal | |
| CN110751201B (en) | SAR equipment task failure cause reasoning method based on textural feature transformation | |
| CN114358121A (en) | Monitoring method and device based on substation equipment and terminal equipment | |
| CN111625525B (en) | Environment data repairing/filling method and system | |
| CN107491576B (en) | Missile component reliability analysis method based on performance degradation data | |
| CN117439899B (en) | Communication machine room inspection method and system based on big data | |
| CN112884167B (en) | Multi-index anomaly detection method based on machine learning and application system thereof | |
| CN117150233B (en) | Power grid abnormal data management method, system, equipment and medium | |
| CN111626586B (en) | Data quality detection method, device, computer equipment and storage medium | |
| WO2022088084A1 (en) | Data processing method, apparatus and system, and electronic device | |
| CN115017127A (en) | Method and device for automatically inspecting acquisition equipment and storage medium | |
| CN116980335A (en) | Method and device for generating risk category identification model and related equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |