CN116866219B - Network data monitoring method and system for IDC data center - Google Patents
Network data monitoring method and system for IDC data center Download PDFInfo
- Publication number
- CN116866219B CN116866219B CN202311107294.3A CN202311107294A CN116866219B CN 116866219 B CN116866219 B CN 116866219B CN 202311107294 A CN202311107294 A CN 202311107294A CN 116866219 B CN116866219 B CN 116866219B
- Authority
- CN
- China
- Prior art keywords
- data packet
- storage
- data
- service node
- node
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000012544 monitoring process Methods 0.000 title claims abstract description 34
- 238000013500 data storage Methods 0.000 claims abstract description 102
- 230000005540 biological transmission Effects 0.000 claims abstract description 95
- 238000004458 analytical method Methods 0.000 claims abstract description 68
- 238000001514 detection method Methods 0.000 claims abstract description 37
- 230000005856 abnormality Effects 0.000 claims abstract description 26
- 230000000737 periodic effect Effects 0.000 claims abstract description 13
- 238000012360 testing method Methods 0.000 claims description 15
- 125000004122 cyclic group Chemical group 0.000 claims description 8
- 238000013112 stability test Methods 0.000 claims description 8
- 238000004364 calculation method Methods 0.000 claims description 6
- 238000009499 grossing Methods 0.000 claims description 5
- 238000001744 unit root test Methods 0.000 claims description 4
- 239000000284 extract Substances 0.000 claims 1
- 230000002159 abnormal effect Effects 0.000 description 4
- 230000007774 longterm Effects 0.000 description 4
- 238000012300 Sequence Analysis Methods 0.000 description 3
- 238000010586 diagram Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000012545 processing Methods 0.000 description 2
- 238000000780 augmented Dickey–Fuller test Methods 0.000 description 1
- 238000012937 correction Methods 0.000 description 1
- 230000003247 decreasing effect Effects 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 238000007689 inspection Methods 0.000 description 1
- 238000012423 maintenance Methods 0.000 description 1
- 230000035772 mutation Effects 0.000 description 1
- 238000012827 research and development Methods 0.000 description 1
- 230000000630 rising effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/14—Network analysis or design
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L43/00—Arrangements for monitoring or testing data switching networks
- H04L43/08—Monitoring or testing based on specific metrics, e.g. QoS, energy consumption or environmental parameters
- H04L43/0876—Network utilisation, e.g. volume of load or congestion level
- H04L43/0894—Packet rate
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Environmental & Geological Engineering (AREA)
- Data Exchanges In Wide-Area Networks (AREA)
Abstract
The application discloses a network data monitoring method and a system for an IDC data center, wherein the method comprises the following steps: drift change analysis, periodic change analysis and time sequence stability analysis are carried out on the storage and transmission data packet to obtain the skewness of the storage and transmission data packet; selecting an abnormality detection algorithm according to the magnitude of the skewness to confirm the abnormality of the network data in the data storage node; and respectively comparing the time stamps before and after data packet transmission between the nodes and the data consistency to obtain the data transmission speed and quality between the nodes. By adopting the application, the network data of the IDC data center is efficiently and real-timely monitored from the network big data storage layer and the network big data transmission layer, and the abnormality of the network data in the data storage node can be detected by timely selecting a proper detection algorithm.
Description
Technical Field
The application relates to the technical field of big data processing, in particular to a network data monitoring method and system for an IDC data center.
Background
An internet data center (Internet Data Center), abbreviated IDC, provides a facility base for operation and maintenance of equipment for centralized collection, storage, processing and transmission of data. IDC typically include a plurality of servers, such as an application service node providing an application service and a data storage node storing data, etc.
In the transmission process of data, the problems that the data quality is affected by data loss, data transmission errors and the like possibly exist, and the quality of the data plays a key role in the accuracy of a data calculation result, and the erroneous data calculation result possibly causes a failure decision according to the data calculation result. If a data transmission problem occurs in the process of transmitting data from the application service node to the data storage node, the data with errors is further transmitted to the data warehouse, which not only wastes network bandwidth resources, but also causes erroneous calculation results.
In addition, a large amount of network data needs to be stored in the data storage node to facilitate the call of the application service node, but the network data monitoring method for the IDC data center is difficult at present to find out the abnormality of the data storage node in time due to the overlarge amount of the network data stored in the data storage node.
Disclosure of Invention
The embodiment of the application provides a network data monitoring method and system for an IDC data center, which are used for intercepting real-time data packets among nodes to calculate skewness and timely selecting a proper detection algorithm to detect the abnormality of data storage nodes.
A first aspect of an embodiment of the present application provides a network data monitoring method for an IDC data center, including:
virtualizing all servers in the IDC data center to obtain a data storage node, an application service node and a business service node;
intercepting a first storage sending data packet sent by the data storage node to the service node direction, receiving a first service receiving data packet sent by the service node from the data storage node direction, sending an application sending data packet sent by the application service node to the service node direction, receiving a second service receiving data packet sent by the service node from the application service node direction, sending a second storage sending data packet sent by the data storage node to the application service node direction, and receiving an application receiving data packet sent by the application service node from the data storage node direction;
carrying out drift change analysis, periodical change analysis and time sequence stability analysis on the second storage sending data packet to obtain the skewness of the second storage sending data packet;
selecting an abnormality detection algorithm according to the magnitude of the skewness to confirm the abnormality of the network data in the data storage node;
and respectively comparing the time stamps and the data consistencies of the first storage sending data packet and the first service receiving data packet, the application sending data packet and the second service receiving data packet, and the second storage sending data packet and the application receiving data packet to obtain the data transmission speed and the data transmission quality between the data storage node and the service node, between the application service node and the service node and between the data storage node and the application service node.
In a possible implementation manner of the first aspect, the performing drift change analysis, periodic change analysis, and time sequence stationarity analysis on the second storage sending data packet obtains a skewness of the second storage sending data packet, which specifically is:
drift change analysis is carried out on the second storage sending data packet;
if the second storage sending data packet has drift, cutting the second storage sending data packet according to the drift point obtained by detection; if the second storage sending data packet does not have drift, keeping the second storage sending data packet unchanged;
performing periodical change analysis and time sequence stability analysis on the second storage sending data packet;
and if the second storage sending data packet does not have periodicity or the second storage sending data packet meets the stability test, calculating the skewness of the second storage sending data packet.
In a possible implementation manner of the first aspect, after the performing a periodic variation analysis and a time sequence stationarity analysis on the second storage sending data packet, the method further includes:
if the second storage transmission data packet has periodicity, cutting the second storage transmission data according to the period span to obtain a plurality of sections of second storage transmission data sections with equal length;
calculating the skewness of the second storage transmission data segment, and taking the skewness of the second storage transmission data segment as the skewness of the second storage transmission data packet.
In a possible implementation manner of the first aspect, the performing drift-change analysis on the second storage sending data packet specifically includes:
extracting the median in a given window according to the size of the given window to acquire a first trend component of a second storage transmission data packet;
smoothing the second storage sending data packet according to the first trend component;
the smoothed second storage sending data packet is an increment data packet or a decrement data packet, and the second storage sending data packet is judged to have no drift;
if the maximum value of the data packet on the left side of the current sample point is smaller than the minimum value of the data packet on the right side of the current sample point, judging that the second storage and transmission data packet has sudden drift; and if the minimum value of the data packet on the left side of the current sample point is larger than the maximum value of the data packet on the right side of the current sample point, judging that the second storage and transmission data packet has sudden drop drift.
In a possible implementation manner of the first aspect, the performing a periodic variation analysis on the second storage transmission data packet specifically includes:
extracting a second trend component in the second storage and transmission data packet;
separating out a residual data packet according to the second trend component;
and calculating a cyclic autocorrelation sequence of the residual data packet, and determining periodicity and corresponding period span according to peak coordinates of the cyclic autocorrelation sequence.
In a possible implementation manner of the first aspect, the performing a time sequence stationarity analysis on the second storage sending data packet specifically includes:
checking the second storage sending data packet through unit root checking;
in the unit root test, if the p value obtained by the test of the second storage transmission data packet in the first preset time period is smaller than the test threshold value, and if the p value obtained by the test of the second storage transmission data packet in the second preset time period is smaller than the test threshold value, the second storage transmission data packet meets the stability test.
In a possible implementation manner of the first aspect, the calculating the skewness of the second storage sending data packet specifically includes:
and converting the expression form of the second storage transmission data packet into a probability distribution form, and calculating the distribution skewness according to the probability distribution form to serve as the skewness of the second storage transmission data packet.
In a possible implementation manner of the first aspect, the determining, by using an anomaly detection algorithm, the anomaly of the network data in the data storage node according to the magnitude of the skewness specifically includes:
if the value of the skewness is larger than 0 and smaller than or equal to a first skewness threshold, detecting the abnormality of the data storage node by adopting an absolute middle bit difference method;
if the value of the skewness is larger than the first skewness threshold and smaller than or equal to the second skewness threshold, detecting the abnormality of the data storage node by adopting a box graph method;
and the value of the skewness is larger than a second skewness threshold and smaller than or equal to a third skewness threshold, and an extremum theory method is selected to detect the abnormality of the data storage node.
In a possible implementation manner of the first aspect, the method further includes:
counting the occurrence times of the same data segment in the first business receiving data packet in a preset counting time period through the business service node, and obtaining a common data segment of the data storage node according to the number of times;
and extracting and analyzing the parameter data segment in the second service receiving data packet within a preset statistical time period by the service node to obtain the monitoring index of the application service node.
A second aspect of an embodiment of the present application provides a network data monitoring system for an IDC data center, comprising:
the virtualization module is used for virtualizing all servers in the IDC data center to obtain a data storage node, an application service node and a business service node;
the intercepting module is used for intercepting a first storage sending data packet sent by the data storage node to the service node direction, receiving a first service receiving data packet sent by the service node from the data storage node direction, sending an application sending data packet sent by the application service node to the service node direction, receiving a second service receiving data packet sent by the service node from the application service node direction, sending a second storage sending data packet sent by the data storage node to the application service node direction and receiving an application receiving data packet sent by the application service node from the data storage node direction;
the skewness calculating module is used for carrying out drift change analysis, periodical change analysis and time sequence stability analysis on the second storage sending data packet to obtain the skewness of the second storage sending data packet;
the detection module is used for selecting an abnormality detection algorithm according to the magnitude of the skewness to confirm the abnormality of the network data in the data storage node;
and the comparison module is used for respectively comparing the time stamps and the data consistencies of the first storage sending data packet and the first service receiving data packet, the application sending data packet and the second service receiving data packet, and the second storage sending data packet and the application receiving data packet to obtain the data transmission speed and the data transmission quality between the data storage node and the service node, between the application service node and the service node and between the data storage node and the application service node.
Compared with the prior art, the embodiment of the application provides a network data monitoring method and a system for an IDC data center, which are used for intercepting data packets among three nodes in real time after the IDC data center is virtualized. The method comprises the steps of carrying out three time sequence analyses on a second storage sending data packet sent by a data storage node to an application service node, namely drift change analysis, periodical change analysis and time sequence stability analysis, then converting a time sequence analysis result from time sequence expression to probability distribution expression form, obtaining skewness according to the probability distribution form as a selection basis of a detection algorithm, and selecting a proper and efficient detection algorithm to detect the data storage node from a network big data storage layer. The second storage sending data packet is acquired in real time, so that the storage condition of the data storage node is reflected in real time, and the detection algorithm is adjusted according to different synchronization of the second storage sending data packet, so that the high efficiency and the accuracy of the detection operation are ensured.
On the other hand, from the aspect of network big data transmission, the data transmission speed and quality between the data storage node and the business service node, between the application service node and the business service node and between the data storage node and the application service node are obtained in real time by analyzing the states before and after data packet transmission among the three nodes and comparing the data packets from the aspects of time and structure.
In summary, in this embodiment, the network data of the IDC data center is efficiently and real-timely monitored from the network big data storage layer and the network big data transmission layer.
Drawings
FIG. 1 is a flow chart of a method for monitoring network data for an IDC data center according to one embodiment of the present application;
fig. 2 is a schematic diagram of a network data monitoring system for an IDC data center according to an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present application will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present application, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
Referring to fig. 1, an embodiment of the present application provides a network data monitoring method for an IDC data center, including:
and S10, virtualizing all servers in the IDC data center to obtain a data storage node, an application service node and a business service node.
S11, intercepting a first storage sending data packet sent by the data storage node to the service node direction, receiving a first service receiving data packet sent by the service node from the data storage node direction, sending an application sending data packet sent by the application service node to the service node direction, receiving a second service receiving data packet sent by the service node from the application service node direction, sending a second storage sending data packet sent by the data storage node to the application service node direction, and receiving an application receiving data packet sent by the application service node from the data storage node direction.
S12, drift change analysis, periodic change analysis and time sequence stability analysis are carried out on the second storage sending data packet, and the skewness of the second storage sending data packet is obtained.
S13, selecting an abnormality detection algorithm according to the degree of the skewness to confirm the abnormality of the network data in the data storage node.
S14, respectively comparing the first storage sending data packet with the first service receiving data packet, the application sending data packet with the second service receiving data packet, and the time stamp and the data consistency of the second storage sending data packet and the application receiving data packet to obtain the data transmission speed and the data transmission quality between the data storage node and the service node, between the application service node and the service node and between the data storage node and the application service node.
In this embodiment, the virtualized IDC data center nodes are divided into three nodes: the system comprises a data storage node, an application service node and a business service node, wherein the data storage node is used for storing a network big data set required by the operation of the IDC data center; the application service node is used for responding to various application applications of the user; the business service node is used for monitoring whether the business targets of the data storage node and the application service node are achieved or not from the data.
S12-S13, efficiently and real-timely monitoring network data of the IDC data center at the network large data storage level, and selecting a proper detection algorithm according to the skewness of the second storage and transmission data packet; and S14, carrying out efficient and real-time monitoring on network data of the IDC data center from a network big data transmission layer according to time stamps and data differences among data packets.
In S14, taking the data transmission speed and quality between the data storage node and the service node as an example, a comparison manner between data packets is illustrated:
and acquiring a sending time stamp from the first storage sending data packet, acquiring a receiving time stamp from the first service receiving data packet, and obtaining the data transmission delay according to the difference value of the sending time stamp and the receiving time stamp.
And comparing the consistency of the first storage sending data packet and the first service receiving data packet, and if the consistency is the same, the transmission efficiency is 100%. If the data blocks are inconsistent, the structure difference and the content difference of the data blocks between the corresponding data block pairs can be compared further according to the data packet being cut into the data blocks.
Based on the abnormal discovery capability of the detection algorithm library, the second storage and transmission data packet can be intercepted based on the data storage node to carry out 7-24 hours inspection on key indexes (drift change analysis, periodical change analysis and time sequence stability), so that risks can be discovered in an abnormal sprouting state, the abnormal exposure can be carried out earlier, and research and development personnel are assisted to position and stop damage before the problem is worsened.
The situation that the mean value of the second storage sending data packet changes significantly or a global mutation point exists over time is called as a drifting scene. In order to accurately capture the latest trend of the second storage transmission packet, it is necessary to determine whether or not there is a drift phenomenon in combination with the history data.
Illustratively, S12 is specifically:
and carrying out drift change analysis on the second storage sending data packet.
If the second storage sending data packet has drift, cutting the second storage sending data packet according to the drift point obtained by detection; and if the second storage sending data packet does not have drift, keeping the second storage sending data packet unchanged.
Performing periodical change analysis and time sequence stability analysis on the second storage sending data packet;
and if the second storage sending data packet does not have periodicity or the second storage sending data packet meets the stability test, calculating the skewness of the second storage sending data packet.
Illustratively, after the periodically-varying analysis and the time-sequence stationarity analysis are performed on the second storage transmission data packet, the method further includes:
if the second storage transmission data packet has periodicity, cutting the second storage transmission data according to the period span to obtain a plurality of sections of second storage transmission data sections with equal length;
calculating the skewness of the second storage transmission data segment, and taking the skewness of the second storage transmission data segment as the skewness of the second storage transmission data packet.
Illustratively, the drift change analysis for the second storage sending data packet specifically includes:
extracting the median in a given window according to the size of the given window to acquire a first trend component of a second storage transmission data packet;
smoothing the second storage sending data packet according to the first trend component;
the smoothed second storage sending data packet is an increment data packet or a decrement data packet, and the second storage sending data packet is judged to have no drift;
if the maximum value of the data packet on the left side of the current sample point is smaller than the minimum value of the data packet on the right side of the current sample point, judging that the second storage and transmission data packet has sudden drift; and if the minimum value of the data packet on the left side of the current sample point is larger than the maximum value of the data packet on the right side of the current sample point, judging that the second storage and transmission data packet has sudden drop drift.
The embodiment uses a drift detection method based on median filtering, and the main process comprises the following links:
median smoothing link: and extracting the median in the window according to the size of the given window to acquire the trend component of the second storage transmission data packet. The window of the link needs to be large enough to avoid the influence of periodic factors and filter delay correction is performed. The reason why median, rather than mean, smoothing is used is to circumvent the effects of abnormal samples.
Judging whether the smoothed second storage sending data packet has a long-term trend or not: if each point is greater (less) than the previous point, the sequence is an increment (decrement) packet. If the second stored transmitted packet has a strictly increasing or strictly decreasing nature, the indicator has a significant long-term trend, which can be terminated prematurely.
Traversing the smoothed second storage sending data packet, and judging whether drift phenomenon exists or not by using the following two rules:
a. if the maximum value of the data packet on the left of the current sample point is smaller than the minimum value of the data packet on the right of the current sample point, then the sudden drift (rising trend) exists.
b. If the minimum value of the data packet on the left of the current sample point is larger than the maximum value of the data packet on the right of the current sample point, then there is a dip drift (falling trend).
Illustratively, in S12, the periodic variation analysis is performed on the second storage transmission data packet, which specifically includes:
extracting a second trend component in the second storage and transmission data packet;
separating out a residual data packet according to the second trend component;
and calculating a cyclic autocorrelation sequence of the residual data packet, and determining periodicity and corresponding period span according to peak coordinates of the cyclic autocorrelation sequence.
As an example, a moving average method may be used to extract long-term trend terms from the second stored transmit data packet, and to make a difference to the second stored transmit data packet to obtain a residual data packet.
The autocorrelation sequence is calculated by performing a vector point multiplication operation with the residual sequence after cyclic shifting of the residual sequence (cyclic autocorrelation can avoid delay decay).
Finally, a series of local highest peaks of the autocorrelation sequence are extracted, and the interval of the abscissa is taken as a period. If the autocorrelation value corresponding to the periodic point is less than a given threshold, no significant periodicity is considered.
Illustratively, in S12, performing a time sequence stationarity analysis on the second storage sending data packet specifically includes:
checking the second storage sending data packet through unit root checking;
in the unit root test, if the p value obtained by the test of the second storage transmission data packet in the first preset time period is smaller than the test threshold value, and if the p value obtained by the test of the second storage transmission data packet in the second preset time period is smaller than the test threshold value, the second storage transmission data packet meets the stability test.
For a packet, if its properties do not change with changes in the observed time at any time, the packet may be considered to be stationary. Thus, for second storage transmitted packets having long-term trend components or periodic components, they are all not smooth.
The present embodiment determines whether the packet is stationary by a root-by-root Test (Augmented Dickey-Fuller Test). Specifically, for the history data of one given time range index, it is considered that the timing is smooth in the case where the following conditions are simultaneously satisfied.
1. The p value obtained by the adfuller test of the last 7 days of time series data is less than 0.08.
2. The p value obtained by the adfuller test of the last 24 days of time series data is less than 0.08.
Illustratively, the calculating the skewness of the second storage transmission data packet in S12 specifically includes:
and converting the expression form of the second storage transmission data packet into a probability distribution form, and calculating the distribution skewness according to the probability distribution form to serve as the skewness of the second storage transmission data packet.
In this embodiment, the time sequence index is substantially converted into a probability distribution diagram, and then the skewness of the distribution is calculated, and if the absolute value of the skewness exceeds the threshold, the threshold is output by the extremum theory. If the absolute value of the skewness is smaller than the threshold value, the threshold value is output by a box graph or an absolute median.
Illustratively, S13 specifically includes:
if the value of the skewness is larger than 0 and smaller than or equal to a first skewness threshold, detecting the abnormality of the data storage node by adopting an absolute middle bit difference method;
if the value of the skewness is larger than the first skewness threshold and smaller than or equal to the second skewness threshold, detecting the abnormality of the data storage node by adopting a box graph method;
and the value of the skewness is larger than a second skewness threshold and smaller than or equal to a third skewness threshold, and an extremum theory method is selected to detect the abnormality of the data storage node.
The present embodiment focuses on determining which one of the three intervals the skewness belongs to—the low skewness interval: (0, a first bias threshold), a middle bias interval (a first bias threshold, a second bias threshold), a high bias interval (a second bias threshold, a third bias threshold), an absolute medium bit difference (MAD) detection algorithm if the bias interval is low, a box plot (box plot) detection algorithm if the bias interval is medium, and an extremum theory (EVT) detection algorithm if the bias interval is Gao Piandu.
The network data monitoring method for the IDC data center further comprises the following steps:
counting the occurrence times of the same data segment in the first business receiving data packet in a preset counting time period through the business service node, and obtaining a common data segment of the data storage node according to the number of times;
and extracting and analyzing the parameter data segment in the second service receiving data packet within a preset statistical time period by the service node to obtain the monitoring index of the application service node.
And the frequency of the same data segment in the first service receiving data packet is larger than a preset common threshold value, and the data segment is used as the common data segment of the data storage node. All the commonly used data segments can be sequenced according to the number of times of occurrence, so that the data segments which belong to the data needing important protection can be displayed more clearly.
The parameter data segments of the second service receiving data packet comprise a JVM heap memory parameter segment, a GC parameter segment, a CPU utilization parameter segment, a thread number parameter segment and a throughput parameter segment.
Compared with the prior art, the embodiment of the application provides a network data monitoring method and a system for an IDC data center, which are used for intercepting data packets among three nodes in real time after the IDC data center is virtualized. The method comprises the steps of carrying out three time sequence analyses on a second storage sending data packet sent by a data storage node to an application service node, namely drift change analysis, periodical change analysis and time sequence stability analysis, then converting a time sequence analysis result from time sequence expression to probability distribution expression form, obtaining skewness according to the probability distribution form as a selection basis of a detection algorithm, and selecting a proper and efficient detection algorithm to detect the data storage node from a network big data storage layer. The second storage sending data packet is acquired in real time, so that the storage condition of the data storage node is reflected in real time, and the detection algorithm is adjusted according to different synchronization of the second storage sending data packet, so that the high efficiency and the accuracy of the detection operation are ensured.
On the other hand, from the aspect of network big data transmission, the data transmission speed and quality between the data storage node and the business service node, between the application service node and the business service node and between the data storage node and the application service node are obtained in real time by analyzing the states before and after data packet transmission among the three nodes and comparing the data packets from the aspects of time and structure.
In summary, in this embodiment, the network data of the IDC data center is efficiently and real-timely monitored from the network big data storage layer and the network big data transmission layer.
An embodiment of the present application provides a network data monitoring system for an IDC data center, comprising: the system comprises a virtualization module 20, an interception module 21, a skewness calculation module 22, a detection module 23 and a comparison module 24.
And the virtualization module 20 is configured to virtualize all servers in the IDC data center to obtain a data storage node, an application service node and a service node.
The interception module 21, the method comprises the steps that a first storage sending data packet sent by a data storage node to a service node direction is intercepted, the service node receives a first service receiving data packet sent by the data storage node direction, an application sending data packet sent by an application service node to the service node direction, a second service receiving data packet sent by the service node to the application service node direction, a second storage sending data packet sent by the data storage node to the application service node direction and an application receiving data packet sent by the application service node to the data storage node direction are intercepted.
And the skewness calculating module 22 is configured to perform drift change analysis, periodic change analysis, and time sequence stability analysis on the second storage sending data packet to obtain the skewness of the second storage sending data packet.
And the detection module 23 is used for selecting an abnormality detection algorithm according to the magnitude of the skewness to confirm the abnormality of the network data in the data storage node.
And the comparison module 24 is configured to compare the time stamps and the data consistencies of the first storage sending data packet and the first service receiving data packet, the application sending data packet and the second service receiving data packet, and the second storage sending data packet and the application receiving data packet, respectively, so as to obtain data transmission speeds and qualities between the data storage node and the service node, between the application service node and the service node, and between the data storage node and the application service node.
It will be clear to those skilled in the art that, for convenience and brevity of description, the specific working process of the identification system described above may refer to the corresponding process in the foregoing method embodiment, which is not described herein again.
Compared with the prior art, the embodiment of the application provides a network data monitoring system for an IDC data center, which is used for intercepting data packets among three nodes in real time after the IDC data center is virtualized. The method comprises the steps of carrying out three time sequence analyses on a second storage sending data packet sent by a data storage node to an application service node, namely drift change analysis, periodical change analysis and time sequence stability analysis, then converting a time sequence analysis result from time sequence expression to probability distribution expression form, obtaining skewness according to the probability distribution form as a selection basis of a detection algorithm, and selecting a proper and efficient detection algorithm to detect the data storage node from a network big data storage layer. The second storage sending data packet is acquired in real time, so that the storage condition of the data storage node is reflected in real time, and the detection algorithm is adjusted according to different synchronization of the second storage sending data packet, so that the high efficiency and the accuracy of the detection operation are ensured.
On the other hand, from the aspect of network big data transmission, the data transmission speed and quality between the data storage node and the business service node, between the application service node and the business service node and between the data storage node and the application service node are obtained in real time by analyzing the states before and after data packet transmission among the three nodes and comparing the data packets from the aspects of time and structure.
In summary, in this embodiment, the network data of the IDC data center is efficiently and real-timely monitored from the network big data storage layer and the network big data transmission layer.
While the foregoing is directed to the preferred embodiments of the present application, it will be appreciated by those skilled in the art that changes and modifications may be made without departing from the principles of the application, such changes and modifications are also intended to be within the scope of the application.
Claims (7)
1. A method for monitoring network data for an IDC data center, comprising:
virtualizing all servers in the IDC data center to obtain a data storage node, an application service node and a business service node; the business service node is used for monitoring whether the business targets of the data storage node and the application service node are achieved or not from the data;
intercepting a first storage sending data packet sent by the data storage node to the service node direction, receiving a first service receiving data packet sent by the service node from the data storage node direction, sending an application sending data packet sent by the application service node to the service node direction, receiving a second service receiving data packet sent by the service node from the application service node direction, sending a second storage sending data packet sent by the data storage node to the application service node direction, and receiving an application receiving data packet sent by the application service node from the data storage node direction;
drift change analysis, periodic change analysis and time sequence stability analysis are carried out on the second storage sending data packet to obtain the skewness of the second storage sending data packet, wherein the skewness is specifically as follows: drift change analysis is carried out on the second storage sending data packet; if the second storage sending data packet has drift, cutting the second storage sending data packet according to the drift point obtained by detection; if the second storage sending data packet does not have drift, keeping the second storage sending data packet unchanged; performing periodical change analysis and time sequence stability analysis on the second storage sending data packet; if the second storage sending data packet does not have periodicity or the second storage sending data packet meets the stability test, calculating the skewness of the second storage sending data packet; and carrying out time sequence stability analysis on the second storage sending data packet, wherein the time sequence stability analysis specifically comprises the following steps: checking the second storage sending data packet through unit root checking; in the unit root test, if the p value obtained by the second storage and transmission data packet test in the first preset time period is smaller than 0.08, and the p value obtained by the second storage and transmission data packet test in the second preset time period is smaller than 0.08, the second storage and transmission data packet meets the stability test;
selecting an abnormality detection algorithm according to the magnitude of the skewness to confirm the abnormality of the network data in the data storage node;
respectively comparing the time stamps and the data consistencies of the first storage sending data packet and the first service receiving data packet, the application sending data packet and the second service receiving data packet, and the second storage sending data packet and the application receiving data packet to obtain the data transmission speed and the data transmission quality between the data storage node and the service node, between the application service node and the service node and between the data storage node and the application service node;
counting the occurrence times of the same data segment in the first business receiving data packet in a preset counting time period through the business service node, and obtaining a common data segment of the data storage node according to the number of times; and extracting and analyzing the parameter data segment in the second service receiving data packet within a preset statistical time period by the service node to obtain the monitoring index of the application service node.
2. The network data monitoring method for IDC data center as in claim 1, further comprising, after the periodic variation analysis and the time-series stationarity analysis of the second stored transmit data packet:
if the second storage transmission data packet has periodicity, cutting the second storage transmission data according to the period span to obtain a plurality of sections of second storage transmission data sections with equal length;
calculating the skewness of the second storage transmission data segment, and taking the skewness of the second storage transmission data segment as the skewness of the second storage transmission data packet.
3. The method for monitoring network data in IDC data center according to claim 1, wherein the drift-change analysis of the second stored transmit data packet specifically comprises:
extracting the median in a given window according to the size of the given window to acquire a first trend component of a second storage transmission data packet;
smoothing the second storage sending data packet according to the first trend component;
the smoothed second storage sending data packet is an increment data packet or a decrement data packet, and the second storage sending data packet is judged to have no drift;
if the maximum value of the data packet on the left side of the current sample point is smaller than the minimum value of the data packet on the right side of the current sample point, judging that the second storage and transmission data packet has sudden drift; and if the minimum value of the data packet on the left side of the current sample point is larger than the maximum value of the data packet on the right side of the current sample point, judging that the second storage and transmission data packet has sudden drop drift.
4. The method for monitoring network data in IDC data center according to claim 1, wherein the periodically changing analysis of the second stored transmission data packet specifically comprises:
extracting a second trend component in the second storage and transmission data packet;
separating out a residual data packet according to the second trend component;
and calculating a cyclic autocorrelation sequence of the residual data packet, and determining periodicity and corresponding period span according to peak coordinates of the cyclic autocorrelation sequence.
5. The network data monitoring method for IDC data center as in claim 1, wherein the calculating the skewness of the second stored transmit data packet comprises:
and converting the expression form of the second storage transmission data packet into a probability distribution form, and calculating the distribution skewness according to the probability distribution form to serve as the skewness of the second storage transmission data packet.
6. The network data monitoring method for IDC data center according to claim 1, wherein the selecting an anomaly detection algorithm according to the magnitude of the skewness confirms the anomalies of the network data in the data storage node, specifically comprising:
if the value of the skewness is larger than 0 and smaller than or equal to a first skewness threshold, detecting the abnormality of the data storage node by adopting an absolute middle bit difference method;
if the value of the skewness is larger than the first skewness threshold and smaller than or equal to the second skewness threshold, detecting the abnormality of the data storage node by adopting a box graph method;
and the value of the skewness is larger than a second skewness threshold and smaller than or equal to a third skewness threshold, and an extremum theory method is selected to detect the abnormality of the data storage node.
7. A network data monitoring system for an IDC data center, comprising:
the virtualization module is used for virtualizing all servers in the IDC data center to obtain a data storage node, an application service node and a business service node; the business service node is used for monitoring whether the business targets of the data storage node and the application service node are achieved or not from the data;
the intercepting module is used for intercepting a first storage sending data packet sent by the data storage node to the service node direction, receiving a first service receiving data packet sent by the service node from the data storage node direction, sending an application sending data packet sent by the application service node to the service node direction, receiving a second service receiving data packet sent by the service node from the application service node direction, sending a second storage sending data packet sent by the data storage node to the application service node direction and receiving an application receiving data packet sent by the application service node from the data storage node direction;
the bias calculation module is configured to perform drift change analysis, periodic change analysis, and time sequence stability analysis on the second storage transmission data packet to obtain a bias of the second storage transmission data packet, where the bias specifically is: drift change analysis is carried out on the second storage sending data packet; if the second storage sending data packet has drift, cutting the second storage sending data packet according to the drift point obtained by detection; if the second storage sending data packet does not have drift, keeping the second storage sending data packet unchanged; performing periodical change analysis and time sequence stability analysis on the second storage sending data packet; if the second storage sending data packet does not have periodicity or the second storage sending data packet meets the stability test, calculating the skewness of the second storage sending data packet; and carrying out time sequence stability analysis on the second storage sending data packet, wherein the time sequence stability analysis specifically comprises the following steps: checking the second storage sending data packet through unit root checking; in the unit root test, if the p value obtained by the second storage and transmission data packet test in the first preset time period is smaller than 0.08, and the p value obtained by the second storage and transmission data packet test in the second preset time period is smaller than 0.08, the second storage and transmission data packet meets the stability test;
the detection module is used for selecting an abnormality detection algorithm according to the magnitude of the skewness to confirm the abnormality of the network data in the data storage node;
the comparison module is used for respectively comparing the first storage sending data packet with the first service receiving data packet, the application sending data packet with the second service receiving data packet, and the time stamp and the data consistency of the second storage sending data packet with the application receiving data packet to obtain the data transmission speed and the data transmission quality between the data storage node and the service node, between the application service node and the service node and between the data storage node and the application service node;
the network data monitoring system counts the times of the occurrence of the same data segment in the first service receiving data packet in a preset counting time period through the service node, and obtains the common data segment of the data storage node according to the times; and the network data monitoring system also extracts and analyzes the parameter data segment in the second service receiving data packet within a preset statistical time period through the service node to obtain the monitoring index of the application service node.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311107294.3A CN116866219B (en) | 2023-08-31 | 2023-08-31 | Network data monitoring method and system for IDC data center |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311107294.3A CN116866219B (en) | 2023-08-31 | 2023-08-31 | Network data monitoring method and system for IDC data center |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN116866219A CN116866219A (en) | 2023-10-10 |
| CN116866219B true CN116866219B (en) | 2023-11-24 |
Family
ID=88228893
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311107294.3A Active CN116866219B (en) | 2023-08-31 | 2023-08-31 | Network data monitoring method and system for IDC data center |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116866219B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104135395A (en) * | 2014-03-10 | 2014-11-05 | 腾讯科技（深圳）有限公司 | Method and system of monitoring data transmission quality in IDC (Internet Data Center) network |
| CN110213125A (en) * | 2019-05-23 | 2019-09-06 | 南京维拓科技股份有限公司 | Abnormality detection system based on time series data under a kind of cloud environment |
| CN113328906A (en) * | 2021-04-22 | 2021-08-31 | 成都欧珀通信科技有限公司 | Flow real-time monitoring method and device, storage medium and electronic equipment |
| CN115473830A (en) * | 2022-09-06 | 2022-12-13 | 中国银行股份有限公司 | Data center performance index abnormity detection method, device, system and medium |
| CN115794544A (en) * | 2022-11-16 | 2023-03-14 | 苏州盈虚有数信息科技有限公司 | Middleware abnormity monitoring device and method |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10742673B2 (en) * | 2017-12-08 | 2020-08-11 | Nicira, Inc. | Tracking the dynamics of application-centric clusters in a virtualized datacenter |
-
2023
- 2023-08-31 CN CN202311107294.3A patent/CN116866219B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104135395A (en) * | 2014-03-10 | 2014-11-05 | 腾讯科技（深圳）有限公司 | Method and system of monitoring data transmission quality in IDC (Internet Data Center) network |
| CN110213125A (en) * | 2019-05-23 | 2019-09-06 | 南京维拓科技股份有限公司 | Abnormality detection system based on time series data under a kind of cloud environment |
| CN113328906A (en) * | 2021-04-22 | 2021-08-31 | 成都欧珀通信科技有限公司 | Flow real-time monitoring method and device, storage medium and electronic equipment |
| CN115473830A (en) * | 2022-09-06 | 2022-12-13 | 中国银行股份有限公司 | Data center performance index abnormity detection method, device, system and medium |
| CN115794544A (en) * | 2022-11-16 | 2023-03-14 | 苏州盈虚有数信息科技有限公司 | Middleware abnormity monitoring device and method |
Non-Patent Citations (1)
| Title |
|---|
| 基于 AI 算法的数据库异常监测系统的设计与实现;知乎;《https://zhuanlan.zhihu.com/p/561208803》;1-11 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN116866219A (en) | 2023-10-10 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110955575A (en) | Business system fault positioning method based on correlation analysis model | |
| CN115118581B (en) | Internet of things data all-link monitoring and intelligent guaranteeing system based on 5G | |
| JP6564799B2 (en) | Threshold determination device, threshold determination method and program | |
| US20100046375A1 (en) | Congestion Control Using Application Slowdown | |
| CN109193650A (en) | A kind of power grid weak spot appraisal procedure based on higher-dimension Random Matrices Theory | |
| CN115936428B (en) | Incremental power distribution network external damage prevention constant value optimizing system | |
| CN102955719A (en) | Determination method and device of suspected memory leak | |
| CN115776449B (en) | Train Ethernet communication state monitoring method and system | |
| US9235463B2 (en) | Device and method for fault management of smart device | |
| CN117439827B (en) | Network flow big data analysis method | |
| CN112734977B (en) | Equipment risk early warning system and algorithm based on Internet of things | |
| CN115529595A (en) | Method, device, equipment and medium for detecting abnormity of log data | |
| JP5963974B2 (en) | Information processing apparatus, information processing method, and program | |
| CN118051863B (en) | Health data acquisition system and method based on digital metering technology | |
| CN113934536B (en) | Data acquisition method facing edge calculation | |
| CN109844779A (en) | For analyzing measurement-yield correlation method and system | |
| CN115685013A (en) | Transformer fault diagnosis method and system based on multi-monitoring parameter statistical characteristics | |
| CN116866219B (en) | Network data monitoring method and system for IDC data center | |
| EP4198803A1 (en) | Fault processing method and apparatus, network device and storage medium | |
| CN117456707A (en) | Intelligent bus duct temperature and humidity abnormality early warning method and device | |
| CN113285847A (en) | Communication network anomaly detection method and system of intelligent converter station monitoring system | |
| CN117076886A (en) | Petrochemical pipe gallery data analysis and decision system and operation method thereof | |
| CN116866222A (en) | Intelligent network cable fault detection method, system, equipment and storage medium | |
| CN113205215A (en) | Knowledge-based battlefield situation prediction method | |
| CN117056171B (en) | Kafka abnormity monitoring method and device based on AI algorithm |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |